Hi, does anyone know if there’s a way to prevent one’s local environmental variables ending up in flow steps when starting Flows from a local terminal? I’m using the

@environment

decorator to set values, but I found that my personal credentials (in my local terminal environment) were available in Metaflow.

you can empty

os.environ

in your own code (e.g in a custom decorator) prior to execution of any other custom code. Would that work?

Yeah, that would work. I’m curious if that was a deliberate design choice- is it a more common use case for users to want all variables available within Metaflow?

yes, it’s deliberate. It’s common to have all kinds of credentials and other settings as we vars. Hiding them by default would cause friction especially during prototyping you are right that reproducibility would benefit from a stricter approach. The philosophy is to make it easy to get started and allow projects to be hardened gradually through features like

@secrets

Thanks for the explanation. Are virtual environments also packaged and shipped off as part of the flow? I thought it was only subdirectories of the Flow file. Should I expect that packages installed in my local venv (from which I trigger the flow) will be available in the dockerized Step?

re: virtual env - everything under the directory hierarchy where the flow file is stored is packaged automatically. You need to include packages you want to include by default in the directory hierarchy by hosting the package in the same repo, by symlinking, or by using

git subtree

etc. In addition or alternatively, you can use

@conda

to include packages

In order to support an internal company package, I’ve cloned this library into a

dependencies

folder in the same repo (but in a parent directory) of my flow. Based on the documentation, this would not be present in the K8s containers for each step.

do you want to always use the Github version and never use the local version?

I at least want to be able to choose

re: choosing, one approach could be:

if self.development_mode:
   from experimental.my_package import my_module
else:
   from subprocess import check_call
   check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])
   from my_package import my_module

now if you had a directory structure like this

flow.py
experimental/
   my_package/
     my_module.py

you could use a parameter

development_mode

to choose between the experimental local version or installing a clean upstream version on the fly

Sure

yes, by default it uses all packages available in the environment like any python script or notebook. If you want to control the environment, you need to use

@conda

. Even if you don't install any Conda packages, you can just add an empty

@conda

and then

pip

install packages on the fly to get a clean environment.

@straight-shampoo-11124 how can I install an in house package like this: pip install git+https://github.com/bla/amazingstuff.git using @conda ? ideally flow wide to be able to run flow in cloud? I have this at the moment:

@conda_base(
    libraries={
        "numpy": "1.26.4",
        "pandas": "2.2"
    },
    python="3.11.9"
)

replying here too for completeness 🙂 https://outerboundsco.slack.com/archives/C02116BBNTU/p1725034633913169?thread_ts=1725032915.173759&cid=C02116BBNTU