Hey folks! Been a lurker a long time. I’ve been using Metaflow for a project with short-lived workflows; typically in the 1-5 minute range. My code is simple enough to be all on the start function. I am using AWS Batch with ECS on EC2. It took me a ridicilously long time to figure out that the end step also runs on a container, but in my project it isn’t needed to. Is there any way I can skip the end step from running on ECS, and let’s say, spoof end’s step success?

You can run the steps locally without using @batch Edit: if you are just running using "--with batch", then remove that and add only @batch to the start step where you actually want to use batch.

Thanks for the suggestion! I failed to mention that I export my workflow on a step function. So the @end step automatically gets exported to a step, that is an AWS Batch Execution step. So essentially, I love the export to step function feature as it gives me an interface to the workflow that's super easy to integrate, but I can't figure out how to skip the end step from launching a container since it's a step in the Step Function. I tried simply removing it, but obviously metaflow ultimately fails the execution since @end step is missing. Although the end step finishes almost instantly since it's empty, it does take a toll on my compute environment VPU capacity, since it constantly requests 1 VCPU for essentially no work

right. As of today

start

and

end

steps are required. With Step Functions, every step (task) runs as a separate container, so inevitably you have to do container execution at least for

start

and

end

even if the functions are empty

Exactly right. Usually 4-8 seconds. I do have them set to 128mb or Ram though. What I haven't tried is, declaring another docker image, an empty one. I may shave-off a couple of seconds from my default one of ~400mb EDIT: Thanks for the prompt response too! What a magnificent framework this is!