hello I m trying to use the ` pypi` decorator on a step whil Outerbounds #ask-metaflow

hello, I'm trying to use the `@pypi` decorator on ...

helpful-honey-48997

03/20/2024, 2:30 PM

hello, I'm trying to use the

@pypi

decorator on a step while also having the

netflix_ext

installed... I'm getting this error when running:

Copy code

(metaflow) ruben@ruben-A6:~/work/test-metaflow$ python test_flows/sfgpflow_local.py run
Metaflow 2.11.5.post2+gitfbe4aa1 executing SFGPFlow for user:ruben
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
    Incompatible environment:
    The @pypi decorator requires --environment=conda

I ended up getting super confused because I was trying to add log statements to the

pypi_decorator.py

in the core metaflow repo, but then I found out that the error is actually originating from here: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py#L67 Any thoughts about how I can go about debugging this?

✅ 1

helpful-honey-48997

03/20/2024, 2:39 PM

note: I can't just use

--environment=conda

because of this error:

Copy code

(metaflow) ruben@ruben-A6:~/work/test-metaflow$ python test_flows/sfgpflow_local.py --environment=conda run
Metaflow 2.11.5.post2+gitfbe4aa1 executing SFGPFlow for user:ruben
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 3 environments ...Environment at '/home/ruben/miniforge3/envs/metaflow_builder_3533666f3385b36f0424ee6844b766191c3233c9_1226250b7b43a9ad83820c579b8412e406ff3b86' already created and valid
    Conda ran into an error while setting up environment.:
    Cannot create a relocatable environment as it depends on local files or non wheels and no storage backend is defined: <https://files.pythonhosted.org/packages/30/dc/d9748ec2b61b6a1fcbaff3891961297c83cfa34c9f9984761fb6f6ae65d0/rpy2-3.5.14.tar.gz>

dry-beach-38304

03/20/2024, 4:03 PM

This is telling you that it can’t create an environment that it can save properly because it has no place to save it to.

dry-beach-38304

03/20/2024, 4:04 PM

You do need the —environment argument. To solve that second error you need to add a storage backend to metaflow. S3 or gcs or azure

dry-beach-38304

03/20/2024, 4:05 PM

Or let me know what the dependencies you are using are. There may be another set that would work and just have wheels.

helpful-honey-48997

03/20/2024, 4:19 PM

ah! okay I was hoping I would be able to just run this locally without setting up the backend, I guess I'm still learning 🙂 these are the dependencies:

Copy code

@pypi(
        python="3.9.18",
        packages={
            "numpy": "1.26.2",
            "pandas": "1.5.3",
            "rpy2": "3.5.14",
            "cmdstanpy": "1.2.0",
            "rpy2-arrow": "0.0.8",
            "requests": "2.31.0",
            "dataclasses-json": "0.5.9",
            "nbdev": "2.3.13",
        },
    )

Just to confirm about the storage backend, I need to follow the steps here for this for the case of using AWS? https://outerbounds.com/engineering/operations/configure-metaflow/ Just want to make sure I'm following the right instructions!

helpful-honey-48997

03/20/2024, 5:45 PM

ooh crap, I think I was fundamentally misunderstanding the development workflow for metaflows... I was thinking it would bypass / remove the dependency on and server infrastructure if I'm just trying to run something locally, but I guess I actually have to remove all of the `pypi`/`conda` decorators in order to just purely run a given workflow locally

dry-beach-38304

03/20/2024, 7:31 PM

your misunderstanding is understandable and it doesn’t always need a storage backend. In this case though, it does because one of the packages you need doesn’t have a wheel file so it needs to build it for you to be then able to reproducibly install it. That’s the error. I could relax that condition if needed and the regular pypi (not the netflix ext) may not have this constraint. Typically though, if you are running just locally, you probably don’t have as much a need for pypi/conda unless your steps have wildly different requirements. pypi/conda are really meant to give you a reproducible environment across executions/machines.

💯 1

helpful-honey-48997

03/21/2024, 8:15 AM

thanks for the explanation! yeah I think there is no work needed here, everything makes sense after I slept on it 🙂 not super necessary, but it might be useful to have a guide around the "development workflow"... I guess it's kind of baked into the tutorials a little bit (starting with something local then going into remote), but IMHO it's always a cycle of working locally on something unproven, deploy and scale for experiments, then come back to local testing/prototyping. It'd be awesome to understand how people manage/navigate this back-and-forth without messing around with commenting and uncommenting decorators everywhere!

dry-beach-38304

03/21/2024, 8:27 AM

At least here, most people run “locally” even with a datastore. It’s not 100% local but good enough for local dev

💯 1

helpful-honey-48997

03/21/2024, 12:44 PM

ah, that makes sense! thank you 🙂

9 Views

Open in Slack

Previous Next