hey Metaflow folks, I have a conda-related question/comment for everyone here: conda_base has a disabled parameter that I'm not sure is working as intended. In the flow below I would expect that steps that aren't decorated with @conda, the base environment (custom image or local conda) would kick in:

@conda_base(disabled=True)
class CondaQFlow(FlowSpec):
    
    @conda(libraries={'pandas':'1.5.3'})
    @step
    def start(self):
        import pandas as pd
        print(pd.__version__)
        self.next(self.end)
    
    @step
    def end(self):
        import pandas as pd
        print(pd.__version__)


if __name__ == "__main__":
    CondaQFlow()

For me this outputs

2.0.2

for both steps. If I were to specify a different pandas version in conda_base, then that version would correctly get picked up in the

end

step and

start

would output

1.5.3

. I know I can get the desired behavior by putting

@conda(disabled=True)

at the end step but for long flows with a lot of steps my OCD gets triggered a bit by having to put

@conda

everywhere. I support a few teams using metaflow and this pops up pretty frequently when onboarding new metaflowers. They expect the default behavior to be

@conda(disabled=True)

for steps where they don't use the decorator.

good question! Indeed doing

--environment=conda

enables

@conda

on all steps, so you have to disable it explicitly to expose outside packages to steps. This is intentional - the idea with

--environment=conda

is to control all dependencies tightly. If it wasn't like this, every user might get a different result when they execute the flow in their environment that contains a bespoke set of packages, which could be surprising too

Thanks Ville! The default behavior makes total sense now. Tested the above example w

METAFLOW_DECOSPECS

set and worked like a charm.

make sense. One way to handle such migrations is to call the legacy code as a subprocess, so it can work fine even with

@conda

ahh didn’t think about that - seems like it would be pretty tricky though to share objects from within the run. Guess I could just serialize things to disk but that would be kind of user-unfriendly. Think I’ll just add the environment variable to the lifecycle config and see how that goes until I can package everything up.

your journey would make a super interesting blog article 🤗

hah yeah think theres one or two in there, think the migration story from sagemaker pipelines in particular would be a good one

you can set all of them through environment variables at least - most in the config file too