https://outerbounds.com/ logo
Join Slack
Powered by
Hi everyone, How can we define a separate and inde...
# ask-metaflow
b
Hi everyone, How can we define a separate and independent environment for metaflow steps using pip? I am using this pip decorator (https://github.com/Netflix/metaflow/issues/24#issuecomment-571976372 ) to define environments for steps, but instead of generating an independent environment for each step it just overwrites the existing environment. 
For ex, in this sample code in the start step the ruamel.yaml version is 0.17.19, in step process_message I install ruamel.yaml 0.17.15 and in step end it stays 0.17.15 instead of going back to 0.17.19.
Copy code
class LinearFlow(FlowSpec):
    @step
    def start(self):
        self.message = 'Thanks for reading.'
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)

        self.next(self.process_message)

    @pip(libraries={'ruamel.yaml':'0.17.15'})
    @step
    def process_message(self):
        print('the message is: %s' % self.message)
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)
        self.next(self.end)

    @step
    def end(self):
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)
        print('the message is still: %s' % self.message)
output:
Copy code
2023-03-17 20:43:24.495 Workflow starting (run-id 1679085804401214):
2023-03-17 20:43:24.540 [1679085804401214/start/1 (pid 19437)] Task is starting.
2023-03-17 20:43:26.363 [1679085804401214/start/1 (pid 19437)] ruamel.yaml==0.17.19
2023-03-17 20:43:26.516 [1679085804401214/start/1 (pid 19437)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:26.526 [1679085804401214/start/1 (pid 19437)] Task finished successfully.
2023-03-17 20:43:26.581 [1679085804401214/process_message/2 (pid 19455)] Task is starting.
2023-03-17 20:43:28.051 [1679085804401214/process_message/2 (pid 19455)] 03/17/2023 20:43:28 - INFO - decorators -   pip-install ruamel.yaml:  with version 0.17.15
2023-03-17 20:43:30.225 [1679085804401214/process_message/2 (pid 19455)] Collecting ruamel.yaml==0.17.15
2023-03-17 20:43:30.299 [1679085804401214/process_message/2 (pid 19455)] Downloading ruamel.yaml-0.17.15-py3-none-any.whl (108 kB)
2023-03-17 20:43:31.076 [1679085804401214/process_message/2 (pid 19455)] Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in /fsx/jay/workspace/metis_eval/removetmux/metis/env/lib/python3.6/site-packages (from ruamel.yaml==0.17.15) (0.2.7)
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Installing collected packages: ruamel.yaml
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Attempting uninstall: ruamel.yaml
2023-03-17 20:43:31.542 [1679085804401214/process_message/2 (pid 19455)] Found existing installation: ruamel.yaml 0.17.19
2023-03-17 20:43:31.672 [1679085804401214/process_message/2 (pid 19455)] Uninstalling ruamel.yaml-0.17.19:
2023-03-17 20:43:36.769 [1679085804401214/process_message/2 (pid 19455)] Successfully uninstalled ruamel.yaml-0.17.19
2023-03-17 20:43:38.613 [1679085804401214/process_message/2 (pid 19455)] Successfully installed ruamel.yaml-0.17.15
2023-03-17 20:43:38.682 [1679085804401214/process_message/2 (pid 19455)] the message is: Thanks for reading.
2023-03-17 20:43:39.024 [1679085804401214/process_message/2 (pid 19455)] ruamel.yaml==0.17.15
2023-03-17 20:43:39.164 [1679085804401214/process_message/2 (pid 19455)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:39.173 [1679085804401214/process_message/2 (pid 19455)] Task finished successfully.
2023-03-17 20:43:39.221 [1679085804401214/end/3 (pid 19518)] Task is starting.
2023-03-17 20:43:40.963 [1679085804401214/end/3 (pid 19518)] ruamel.yaml==0.17.15
2023-03-17 20:43:40.965 [1679085804401214/end/3 (pid 19518)] ruamel.yaml.clib==0.2.7
2023-03-17 20:43:40.965 [1679085804401214/end/3 (pid 19518)] the message is still: Thanks for reading.
2023-03-17 20:43:41.118 [1679085804401214/end/3 (pid 19518)] Task finished successfully.
2023-03-17 20:43:41.126 Done!
✅ 1
d
There is no official support for this yet. You can check out an extension to metaflow that allows you to install pip packages within a conda environment (so you can do all pip there). It is here https://github.com/Netflix/metaflow-nflx-extensions and on pypi. If it doesn’t work for you, let me know. As I said, it’s not officially supported but happy to help if you run into issues. Please read the read me for details.
h
would the @enviroment decorator be applicable here?
b
thank you for the reply. I'll check out the extension.
getting an import error with the new environment
Copy code
Traceback (most recent call last):
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow/extension_support/plugins.py", line 137, in resolve_plugins
    plugin_module = importlib.import_module(path)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py", line 52, in <module>
    from .conda import Conda
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 20, in <module>
    from typing import (
ImportError: cannot import name 'OrderedDict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lflow.py", line 2, in <module>
    from metaflow import FlowSpec, step
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow/__init__.py", line 115, in <module>
    from .plugins.datatools import S3
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow/plugins/__init__.py", line 125, in <module>
    STEP_DECORATORS = resolve_plugins("step_decorator")
  File "/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.6/site-packages/metaflow/extension_support/plugins.py", line 140, in resolve_plugins
    "Cannot locate %s plugin '%s' at '%s'" % (category, name, path)
ValueError: Cannot locate step_decorator plugin 'conda' at 'metaflow_extensions.netflix_ext.plugins.conda.conda_step_decorator'
pip freeze -
Copy code
metaflow==2.8.1
metaflow-netflixext==0.0.9
d
ah — it seems OrderedDict is 3.7.2+ (https://docs.python.org/3/library/typing.html?highlight=ordereddict#typing.OrderedDict) and you seem to be on 3.6. Let me see if I can replace it with something else; I’ll get back to you shortly. If you can upgrade to 3.7+, it should also work. I haven’t tested on older than 3.7 but let me see if I can get it to work on something lower.
b
I should be able to get it to 3.7+.
👍 1
d
cool — let me know if that works for you — if it does, I’ll update the doc to specify that it’s 3.7+. I suspect other things may break pre 3.7.
👍 1
but if you need it, I can try to get it to work pre 3.7 — it should definitely be do-able 🙂
b
ok. code runs with python 3.7. but, it doesnt resolves the issue. it still uninstalls 0.17.19 ruamel.yaml from the pip environment and then step 2 and 3 have 0.17.15. I am trying to have an independent environment for step 2. without affecting step 3.
d
You have to use the @conda with pip_libraries. It doesn’t include a new @pip just allows you to have pip libraries specified in a conda environment. I am not at my desk right now but this evening I can send you a code snippet that will do what you want.
b
thank you. that would be really helpful. btw, I tried it with the conda decroator and got a mamba related error.
Copy code
python lflow.py --environment=conda run
/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/lib/python3.7/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Metaflow 2.8.1+netflix-ext(0.0.9) executing LinearFlow for user:jay
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 2 environments in flow ...Falling back to 'mamba' to resolve as conda-lock not installed
No STDOUT
Pretty-printed STDERR:
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage]
             [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
             [--tags TAGS]
             [specs [specs ...]]
mamba: error: unrecognized arguments: --json

No STDOUT
Pretty-printed STDERR:
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage]
             [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
             [--tags TAGS]
             [specs [specs ...]]
mamba: error: unrecognized arguments: --json

    Conda ran into an error while setting up environment.:
    Conda command '['/fsx/jay/workspace/metis_eval/removetmux/extension/metis/env/bin/mamba', 'info', '--json']' returned error (2); see pretty-printed error above
edited step 2 to look like this
Copy code
@conda(libraries={'ruamel.yaml':'0.17.15'})
    @step
    def process_message(self):
        print('the message is: %s' % self.message)
        pkgs = freeze.freeze()
        for pkg in pkgs:
            if 'yaml' in str(pkg):
                print(pkg)
        self.next(self.end)
d
ok, here is a sample flow I used:
Copy code
from metaflow import FlowSpec, step, conda

class LinearFlow(FlowSpec):
    @conda(pip_packages={"ruamel.yaml": "0.17.19"})
    @step
    def start(self):
        self.message = 'Thanks for reading.'
        import ruamel.yaml
        print("Version is %s" % ruamel.yaml.__version__)

        self.next(self.process_message)

    @conda(pip_packages={"ruamel.yaml":"0.17.15"})
    @step
    def process_message(self):
        print('the message is: %s' % self.message)
        import ruamel.yaml
        print("Version is %s" % ruamel.yaml.__version__)

        self.next(self.end)

    @conda(pip_packages={"ruamel.yaml":"0.17.19"})
    @step
    def end(self):
        import ruamel.yaml
        print("Version is %s" % ruamel.yaml.__version__)
        print('the message is still: %s' % self.message)

if __name__ == "__main__":
    LinearFlow()
This produces the following output:
Copy code
Metaflow 2.7.22+netflix-ext(0.0.9) executing LinearFlow for user:romain
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 2 environments in flow ... done in 32 seconds.
2023-03-21 08:48:32.078 Workflow starting (run-id 1679388512011593):
2023-03-21 08:48:33.260 Using existing Conda environment 39cbd7faeda48facc6cb5dda03471b854d1e0f2a (a5763f00bf2e15a9a7f9242dbf3ec0ee32fbdfa9)
2023-03-21 08:48:33.264 [1679388512011593/start/1 (pid 31718)] Task is starting.
2023-03-21 08:48:33.678 [1679388512011593/start/1 (pid 31718)] Version is 0.17.19
2023-03-21 08:48:33.721 [1679388512011593/start/1 (pid 31718)] Task finished successfully.
2023-03-21 08:48:33.747 Using existing Conda environment 7ad0716657309427345d0f5842d57c75bf2b468b (98222207a6bb6c9f2bb98889701a11614f0c1170)
2023-03-21 08:48:33.750 [1679388512011593/process_message/2 (pid 31727)] Task is starting.
2023-03-21 08:48:34.090 [1679388512011593/process_message/2 (pid 31727)] the message is: Thanks for reading.
2023-03-21 08:48:34.101 [1679388512011593/process_message/2 (pid 31727)] Version is 0.17.15
2023-03-21 08:48:34.147 [1679388512011593/process_message/2 (pid 31727)] Task finished successfully.
2023-03-21 08:48:34.186 Using existing Conda environment 39cbd7faeda48facc6cb5dda03471b854d1e0f2a (a5763f00bf2e15a9a7f9242dbf3ec0ee32fbdfa9)
2023-03-21 08:48:34.189 [1679388512011593/end/3 (pid 31730)] Task is starting.
2023-03-21 08:48:34.543 [1679388512011593/end/3 (pid 31730)] Version is 0.17.19
2023-03-21 08:48:34.543 [1679388512011593/end/3 (pid 31730)] the message is still: Thanks for reading.
2023-03-21 08:48:34.639 [1679388512011593/end/3 (pid 31730)] Task finished successfully.
2023-03-21 08:48:34.641 Done!
(or something like that)
Please be sure to update the extension btw — I did make a small change and push out a bunch of other pending updates I had.
make sure you also have an updated version of mamba (that error message you had — I couldn’t reproduce)
Finally, note that ruamel.yaml is available on conda-forge so you can also, in your case, use just the regular conda decorator. version .15 does not seem there though (it goes from .14 to .16 for some reason) and some versions are not compatible with never versions of python so there may be a bit of wrangling to do there. At any rate, the above code does work for me and hopefully is what you are looking for.
you do need conda-lock as well for solving pip packages (that’s in the readme). Let me know if that helps and/or if you have additional questions.
b
ok. so, it looks like we have to reinstall the package. For some reason I assumed that these decorators create a standalone environment independent from other steps.
Thanks for your help.
d
the 
@conda
one does, the 
@pip
you are using doesn’t
3 Views
Open in Slack
PreviousNext
Powered by