Slow parallel execution in devstack/sandbox Hi everyone, I'm new here and have what feels like a pretty basic issue We are trying out metaflow and love the premise. the ability to go from python code that can execute locally to a deployed production flow is very appealing. We are however running into issues with some simple examples, but this is almost certainly a skill issue. we have the following code:

from metaflow import FlowSpec, step
from time import sleep


class SleepFlow(FlowSpec):

    @step
    def start(self):
        self.my_var = "hello world"
        self.tasks = [1]*50
        self.next(self.a, foreach='tasks')

    @step
    def a(self):
        sleep(self.input)
        self.next(self.join)
    
    @step
    def join(self, inputs):
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    SleepFlow()

When running without the devstack with say

python3 sleepFlow.py run

it executes this in a little under 6 seconds when doing the same thing from the dev shell it instead takes 26 minutes and there is some strange behavior (images showing timing with and without kubernetes option from the dev shell) finally when running this from the outerbounds sandbox it doesn't even complete at all and individual steps fail randomly while having this same long running behavior we did do a run where we used parallel_map, multiprocessing, or dask

from time import sleep, time
from metaflow import FlowSpec, step, parallel_map, resources, pypi, conda
from dask.distributed import Client, wait
from multiprocessing import Pool

def a(input):
    sleep(input)

class ParSleepFlow(FlowSpec):
    
    #@conda(python="3.11", packages={"dask":"", "distributed":""})
    @resources(cpu=8)
    @step
    def start(self):
        self.my_var = "hello world"
        self.tasks = [1]*50

        start = time()        
        parallel_map(a, self.tasks)
        end = time()
        print("parallel map " + str(end - start))

        c= Client()
        start = time()
        wait(c.map(a, self.tasks))
        end = time()
        print("dask map " + str(end - start))

        start = time()
        with Pool(50) as p:
            p.map(a, self.tasks)
        end = time()
        print("standard map " + str(end - start))
        #self.next(self.a, foreach='tasks')
        self.next(self.end)    
    
    
    # @step
    # def join(self, inputs):
    #     self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    ParSleepFlow()

which produced much better results

(intro-to-mf) metaflow-sandbox$ python sleep_flow.py run

Metaflow 2.11.15.3+ob(v1) executing ParSleepFlow for user:sandbox

Validating your flow...

The graph looks good!

Running pylint...

Pylint is happy!

2025-07-16 21:07:25.076 Workflow starting (run-id 17), see it in the UI at <https://ui-pw-1643727799.outerbounds.dev/ParSleepFlow/17>

2025-07-16 21:07:26.043 [17/start/177 (pid 9017)] Task is starting.

2025-07-16 21:07:34.476 [17/start/177 (pid 9017)] parallel map 7.200414419174194

2025-07-16 21:07:36.310 [17/start/177 (pid 9017)] dask map 1.017256259918213

2025-07-16 21:07:37.568 [17/start/177 (pid 9017)] standard map 1.2571454048156738

2025-07-16 21:07:39.053 [17/start/177 (pid 9017)] Task finished successfully.

2025-07-16 21:07:39.323 [17/end/178 (pid 9267)] Task is starting.

2025-07-16 21:07:41.288 [17/end/178 (pid 9267)] Task finished successfully.

2025-07-16 21:07:41.383 Done! See the run in the UI at <https://ui-pw-1643727799.outerbounds.dev/ParSleepFlow/17>

is this mainly just a wrong tool for the job issue in that the work is heavily dominated by the overhead? or are we doing something incorrectly here? what would a more fair test look like? the reason we ran into this at all is that we were trying to create a pipeline to process some sound files and ran into similar slow downs leading us to setup these simple flows to try and understand the system itself better and rule out issues in our code

hi stephan! devstack is significantly bit under-powered by default, so it's not a good setup for benchmarking perf. it's intended use case is for developing metaflow and running ci/cd. the perf numbers that you are seeing are wholly expected with devstack. you can tweak the resources allocated to each component in this file if you would still like to use devstack

Ah ok thanks for the quick reply. that makes sense. we will fiddle with that a bit more.

sounds good! the nature of workload you are running is a great fit. happy to share the outerbounds sandbox. if you were using metaflow.org/sandbox - that's oss metaflow sandbox and not the outerbounds one.

metaflow.org/sandbox currently redirects to https://docs.outerbounds.com/sandbox/

yes - that's all open source metaflow and not the outerbounds product

yeah we saw that in the demo that will did for us. really slick looking. we would likely want that its more that we are specifically interested in the ability to debug the development of a flow iteratively from entirely local to the dev stack and ultimately into a deployed cluster such as one provided by outerbounds. we were mainly just trying to understand if we were doing it wrong or if the dev stack is merely a bit buggy. we played around with the makefile and the tilt file a bit and scaled up the cluster a bit but that didnt seem to make much of a difference. due to how repeatable it was we then figured it must be the code itself.

python simple_flow.py run --with kubernetes --max-workers 2

this immediately improved the situation significantly with runs being executed much more evenly. this managed to get the total execution under 10 minutes on the oss sandbox with similar performance on the dev stack really seems as if at 50 parallel sleeps we very much were just saturating the single node minikube cluster. it seems that the dev cluster is actually probably going to be a good debugging tool for us then its just a matter of being more realistic about its capabilities on a single machine.