Hi all, I’ve searched the conversations, the Metaflow docs and the fantastic Outerbound guides, but couldn’t find an answer: Is it possible to run metaflow workflows on AWS Batch asynchronously in a fire and forget way, or does that absolutely require using step functions for coordination? I essentially want to be able to close the terminal without the job dying on me.

hi Vincent! you could do it by running the flow on a cloud instance using e.g.

screen

or

tmux

that keep the process running even if the connection dies

We do exactly that. The flows are run from a small AWS instance in a tmux window.

ok, does it work for you or would you prefer some other approach?

Works well! 🙂 Sorry for not being clear.

cool. Indeed a production scheduler like Step Functions, Argo Workflows, or Airflow is another option to keep them running without human attention or laptops

I see. Yeah, in that case we could use github codespaces, although this would incur additional running costs. I guess step functions is the way to go. Thank you for your reply

Indeed step functions are the go-to for production workcases. I created an API Gateway Cloudformation that triggers a lambda function that parses the parameters and then triggers the step function. It’s been great, since I can go from local testing, to online test, to pushing the changes to the step function in minutes. Robust solution, expected results.