Hi Guys, do you know if we have a way to install the python packages on runtime on each step? currently we use conda decorators to manage the external libraries, for example @conda(libraries={“pandas”: “0.22.0"}). Could we read the libraries name from config file and add it to the decorator?

One way to do this is use pip install inside each step. You can use a requirements.txt file for that.

here's an example of a simple custom decorator,

@with_requirements

which reads requirements from

requirements.txt

that has lines like this

numpy==1.20.3
matplotlib==3.4.2

this works with

@batch

and

@kubernetes

and production schedulers too, if you set an environment variable

METAFLOW_PACKAGE_SUFFIXES=.txt

Is there any reason for the generator function?

no, you can read the lines however you want

Cool thanks Ville 🙏

Thanks for this solution @straight-shampoo-11124 We used something similar - a python module as config with

@conda(libraries=CONFIG["requirements"])

While this works, the one common thing between both these solutions is that the location of requirements/config is hard coded in the flow. I was wondering if there is a way to specify this configuration file location. This is related to https://github.com/Netflix/metaflow/issues/431

you could load the configuration file name e.g. from an environment variable, replacing

open('requirements.txt')

with something like

open(os.environ['LIBRARIES_SPEC'])

this seems to only support hard-pinned versions? (splitting on

==

)

correct

Interesting! It should be easy to extend this to parse the dependencies from a

pyproject.toml

using Poetry 👍

If we deploy the code as step function, would the step function able to load the external library each time it executes? I ran some testing for the step function deployed. The testing process is follows: I used the similar code to load an external config file from s3, then I deploy the step function with “”python foo.py --environment=conda run”. The first time the python version in the config file is right, so the step function works fine. Then I changed the config file with the versions that should fail the flow and I am expecting the step function will fail. To my surprise, the step function runs fine(that means that it didn’t read the config file gain but use the old config file when it is deployed).

@straight-shampoo-11124 Sorry I missed your message earlier, but your env variable solution works great, thank you! Similar to @limited-agent-85591’s question, I have a question about passing env variable to a deployed step function at run time. I'm aware of the

@environment

decorator, but it seems this can only be used to pass env variables to steps of a flow, rather than to the flow itself. Basically I have a flow like this

config_file_loc = os.getenv("METAFLOW_CONFIG_FILE", str(pathlib.Path(__file__).parent / "configs" / "pricing_config.yaml"))
parsed = urlparse(config_file_loc, allow_fragments=False)
if parsed.scheme == "s3":
    import boto3
    s3 = boto3.client("s3")
    CONFIG = yaml.safe_load(s3.get_object(Bucket=parsed.netloc, Key=parsed.path.lstrip("/"))["Body"].read())
else:
    CONFIG = yaml.safe_load(open(config_file_loc, "r"))

class ExampleFlow(FlowSpec):
    config_file = IncludeFile(
        "config_file",
        is_text=True,
        default=os.getenv(
            "METAFLOW_CONFIG_FILE",
            str(pathlib.Path(__file__).parent / "configs" / "example_config.yaml"),
        ),
    )

    @card
    @conda(pip=CONFIG["environment"])
    @step
    def start(self):
        print("reading config")
        self.config = yaml.safe_load(self.config_file)

I can trigger the flow like this

METAFLOW_CONFIG_FILE=/home/ubuntu/example_config.yaml python example_flow/run.py --environment=conda run

Both the param

config_file

which contains details other than the environment and the variable

CONFIG

that's used to set the conda decorator are set correctly based on the yaml file location defined in the env variable

METAFLOW_CONFIG_FILE

. When

step-functions create

is run, do the

@conda

decorators get actual values based on the defaults or is the flow deployed as is? If it is the latter, how can I pass env variables to the step function at run-time (using the

trigger

command)?

yeah, environment variables are applied at the deploy time. Also take a look at deploy-time functions which might come in handy in your case