Hello team! Apologies for a naive question in adva...
# ask-metaflowq
quick-lighter-52296
03/06/2024, 5:59 AMHello team! Apologies for a naive question in advance, I think I am missing something obvious but not sure what. I am trying to leverage the recently publicized support for distributed training with metaflow. But getting the error that:
Copy code
Deploying flows with @parallel decorator(s) to AWS Step Functions is not supported currently.đź‘€ 1
âś… 1
c
crooked-jordan-29960
03/06/2024, 6:01 AMHi! Unfortunately the multinode batch jobs do not work with step functions.
crooked-jordan-29960
03/06/2024, 6:06 AMInstead of step functions, you'll need to use the run command. One suggestion is to do this inside a tmux session in an ec2 instance or similar setup that can run the process without you needing to keep it alive by interacting with a laptop/workstation.
q
quick-lighter-52296
03/06/2024, 7:13 AMHi, thanks for the help here. Is this a step function limitatation that can never be overcome, or do you foresee this limitation being removed from future releases? It’s actually a bummer that we can not “deploy” such flows as our use case does involve being able to programatically run them.
c
crooked-jordan-29960
03/06/2024, 2:42 PMYa, this is an AWS-limitation I believe. Based on this documentation, step functions cannot submit batch jobs using the parameters needed, namely the `nodeOverrides` param in the multinode job definition.
đź’Ż 1
crooked-jordan-29960
03/06/2024, 2:51 PMIf you need this, and are open to using Kubernetes for compute instead of batch, Outerbounds will soon open-source a set of modifications to @kubernetes that enable gang-scheduled/@parallel jobs like a more flexible version of batch multinode. This will have an integration with the Argo functionality that is analogous to Step Functions in the pure-AWS approach.
❤️ 1
12 Views