Hi all, I am looking at

metaflow-ray

for distributed GPU XGBoost on AWS Batch, replacing the current Coiled + Dask setup, but I’d ideally like to keep Step Functions for scheduling/orchestration. My understanding is that

@parallel

/

num_parallel

isn’t supported by Metaflow’s Step Functions integration (

step-functions create

rejects it), and that Argo Workflows requires

@kubernetes

, so it can’t drive

@batch

multi-node jobs either. Has anyone done something similar? Curious what pattern people use in production e.g. Step Functions kicking off a separate Ray training flow, Argo/EKS, EventBridge, plain Batch multi-node, etc.