Hi, everyone! We are evaluating Metaflow as a main power horse in our company, and, of course, as with many people before us, dependencies installation is the major blocker. We use Google Cloud to execute workflows, so our main option is to go with Argo Workflows then, and we also have plenty of internal Python packages, which makes the Conda approach a bit cumbersome, as we don't have it now and hesitant about introducing it into our stack. I tried to experiment with speeding up the dependencies installation via pip to bootstrap the execution environment as quick as possible, but it still takes up to a couple minutes; obviously, it is too much when we are talking about hundreds of parallel tasks. I skimmed through the Metaflow sources to understand more about how it works and I got this crazy thought, which I'm trying to evaluate right now and I would appreciate any help with it. The idea is to build container images on the fly with all dependencies baked into them, so they can be reused between workflow executions. It aligns nicely with our flow, as we rarely upgrade the dependencies, but execute many small tasks. In the abstract, it can be done similarly to these

@conda

/

@conda_base

decorators Metaflow has, but it leaves two questions open so far: 1. How to determine that workflow is being executed locally. So far I think that for local execution we would prefer just to build a virtual environment locally for each workflow step and use them (but this idea is also being evaluated right now) 2. How can this container build process be injected into the Metaflow "scheduling" process? The image itself can be provided by the

@kubernetes

decorator, which can be applied on the fly to the workflow step, but I'm not sure yet, at what point we should build this image first. I would appreciate any thoughts on this idea, thanks in advance 🙂

Great idea! We have been thinking and working along the same lines. You can already supply a custom docker image using

@kubernetes(image='foo')

@square-wire-39606, thanks for the reply!

You can already supply a custom docker image using

@kubernetes(image='foo')

I saw it, but I hadn't checked yet if it would be possible to run this workflow locally if the

@kubernetes

decorator applied on it.

one straightforward mechanism for running locally in the same environment would be to create a venv and run the flow within that venv

Ideally, I would love to have venv created on the fly, because it makes it a better developer experience. So my idea basically narrows to this pseudo-code decorator:

def dependencies(func, dependencies: Dict[str, str]):
    qualified_name = f"{current.flow_name}-{current.step_name}-{hash(dependencies)}"
    if current.runs_locally:
        def wrapper():
            venv.create(name=qualified_name)
            venv.install_dependencies(dependencies)
            func()
        return wrapper
    else:
        def wrapper():
            docker.build_image(name=qualified_name, from="python3.11:slim")
            docker.push_image(name=qualified_name)
            func = kubernetes(func, image=qualified_name)
            return func

I hope it is clear enough

you can also check out https://github.com/Netflix/metaflow-nflx-extensions. It’s like @conda but includes support for pip as well (you have @pip for example). It takes care of the “local venv on the fly type of approach — it still uses conda under the hood but you are no longer tied to just the conda packages and can use internal .whl files for example). It does not take care of the “build docker image and push out” but acts like @conda for remote execution (it will hydrate the environment remotely when it executes there. The pre-baking stuff would be cool to do though although ideally you would be able to do it in the local case too to make the local experience as close to the remote one (though not sure if the cost is worth it there ie if the downside of making the local experimenting experience slower is worth it).