Hey everyone, I am using metaflow with aws batch and I wanted to know what would be the best way to refactor a large step into smaller functions/steps. I have a step which runs on aws batch and uses a custom image. But the problem is it has a lot of lines and those can be split up but depend on the dependencies in the custom image. What would be the best way to refactoring this? Can I use the same aws batch instance in multiple steps or how can I break the step into functions? Thanks for the help.

@clean-spring-46187 you can use the same custom image across multiple steps. That should make it a bit easier to split up your giant step.

But each step would be on a different aws batch instance right? The main issue is AWS batch takes some time to get GPU instances, so it would unnecessary wait.

each step will be a different batch job but depending on how you have configured your compute environment, we could trade away utilization for availability and ensure that the next job executes immediately. Also, AWS Batch doesn't spin down the instance immediately, so there is a good chance that the next job will be scheduled rather quickly.

Is there some resource for how this can be achieved? Thanks for the help.

the user guide for AWS Batch does a good job at explaining all the knobs and levers available to you. Happy to chat further about exactly what outcome you are optimizing for and the path to that.

Thanks. I will go through this.