Thoughts on using Metaflow w/ CI/CD systems: right now, the lack of a programmatic API is really painful when it comes to triggering stepfunctions runs in CI/CD context.  I think aside from pattern matching on the stdout of the metaflow CLI, there's no way for a CI/CD system to get the execution id of the flow it just started if it wants to track its progress (for example, if you want to run an actual flow as part of an integration test). Per @straight-shampoo-11124, there is a

run --run-id-file

option for local runs; could we add something similar for step-functions?

Yep, that's a straight-forward addition.

let me look into this, should be pretty easy to add

Technically you can do the same thing with

step-functions trigger

as well - with the caveat that

step-functions trigger

is async and you may have to wait a bit before the actual state-machine is launched which will then register the metadata with the metadata service.

it also means the CI/CD system needs to have Metaflow API access, but for our particular usecase we just want it to trigger the sfn execution and fail a build if the execution fails.

created a PR here https://github.com/Netflix/metaflow/pull/590

Tested, approved and merged!

Thanks guys!!! This is a big help for us using stepfunctions in integration tests 😄

📣 in case you haven't followed this thread, we just released a small new feature (

step-functions trigger --run-id-file

) that helps automating SFN deployments and testing

This is great! We had the same problem and the team was chatting about this last week. Thanks for this 🙏

a good reason to give feedback early and frequently 😄