hey all. I’m looking into potentially bringing Metaflow into my team. We are heavy on Sagemaker/AWS and common complain is the time it takes for Sagemaker to spin up the container for a training job, specially for short experiments. Has anyone compared spin up times of Sagemaker training job against using Step Functions with Batch?

Hey Matias - The job startup times are quite quick with AWS Batch if you have available capacity (in the order of couple of seconds)

interesting, the times we see are in 1-2min range, thanks for the info

For fast/short running tasks, can you “batch” them together into a single meta task and then AWS batch loops over them all to run? Node spin up time and also container size transfer time are a thing.