https://outerbounds.com/ logo
Join Slack
Powered by
I am trying to use @pypi to decorate a step: ```@p...
# ask-metaflow
g
I am trying to use @pypi to decorate a step:
Copy code
@pypi(python='3.10.12', 
      packages={'numpy': '1.22.2',
                'pandas': '1.5.2'})
But am running into the following error:
Copy code
Bootstrapping virtual environment(s) ...
    Pip ran into an error while setting up environment:
    Unable to locate a Micromamba managed virtual environment
    for id 181b3f65db3b0f7
Has anyone else experienced this issue?
a
it is likely that the local conda envs are corrupted. do you see this id in the output of 
micromamba env list
?
g
micromamba env list
gives me the the following:
Copy code
Name  Active  Path
────────────────────────────────────────────────
  base          /Users/kerrymcgowan/micromamba
                /usr/local/anaconda3
a
and what is the output of 
micromamba info
- specifically the envs directories field?
g
Copy code
libmamba version : 1.5.10
     micromamba version : 1.5.10
           curl version : libcurl/8.6.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.61.0
     libarchive version : libarchive 3.7.6 zlib/1.2.12 liblzma/5.6.2 bz2lib/1.0.8 liblz4/1.10.0 libzstd/1.5.6
       envs directories : /Users/kerrymcgowan/micromamba/envs
          package cache : /Users/kerrymcgowan/micromamba/pkgs
                          /Users/kerrymcgowan/.mamba/pkgs
            environment : /Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow (active)
           env location : /Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow
      user config files : /Users/kerrymcgowan/.mambarc
 populated config files :
       virtual packages : __unix=0=0
                          __osx=14.5=0
                          __archspec=1=x86_64_v3
               channels :
       base environment : /Users/kerrymcgowan/micromamba
               platform : osx-64
a
great! can you list the directories in 
/Users/kerrymcgowan/micromamba/envs
- if you find any that has the id 
181b3f65db3b0f7
- can you try nuking it?
g
That fixed one issue, thank you! I'm now getting a different error tho: 
TypeError: expected str, bytes or os.PathLike object, not NoneType
. Here's the full stack trace:
Copy code
Internal error
Traceback (most recent call last):
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/cli.py", line 1134, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/cli.py", line 585, in wrapper
    return func(args, kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/cli.py", line 786, in run
    runtime.execute()
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/runtime.py", line 436, in execute
    self._launch_workers()
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/runtime.py", line 877, in _launch_workers
    task = self._new_task(step, task_kwargs)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/runtime.py", line 186, in _new_task
    return Task(
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/runtime.py", line 1186, in __init__
    deco.runtime_task_created(
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/plugins/pypi/conda_decorator.py", line 222, in runtime_task_created
    self.environment.interpreter(self.step)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/plugins/pypi/conda_environment.py", line 205, in interpreter
    return self.solvers["conda"].interpreter(id_)
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/site-packages/metaflow/plugins/pypi/micromamba.py", line 215, in interpreter
    return os.path.join(self.path_to_environment(id_), "bin/python")
  File "/Users/kerrymcgowan/opt/anaconda3/envs/iteration1flow/lib/python3.10/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
a
interesting - what happens if you also nuke your local .metaflow folder and then try?
g
I still get the same error
a
could you try changing the version of any of the packages?
it is likely that the micromamba installation is somehow corrupted. are you using this on a mac or a linux instance?
i
@gentle-motherboard-71754 were you able to eventually resolve this? I am also seeing the same error
Copy code
File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/plugins/pypi/micromamba.py", line 215, in interpreter
    return os.path.join(self.path_to_environment(id_), "bin/python")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType
tried completely removing 
micromamba/envs/metaflow
directory, but that didn't seem to work either
a
What version of metaflow are you on?
i
2.12.38
a
Can you try with a slightly older version (and nuke the micromamba directory)? We are also debugging internally.
i
sure, which version should i try?
a
2.12.30 should be safe to rule out issues with metaflow
i
still seeing the same issue
Copy code
Metaflow 2.12.30 executing PlayListFlow for user:ferras.hamad
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2024-12-10 18:13:08.092 Bootstrapping virtual environment(s) ...
2024-12-10 18:13:22.104 Virtual environment(s) bootstrapped!
2024-12-10 18:13:22.452 Workflow starting (run-id 309):
2024-12-10 18:13:25.810 Workflow failed.
2024-12-10 18:13:25.810 Terminating 0 active tasks...
2024-12-10 18:13:25.810 Flushing logs...
    Internal error
Traceback (most recent call last):
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/cli.py", line 1166, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/cli.py", line 586, in wrapper
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/cli.py", line 818, in run
    runtime.execute()
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/runtime.py", line 436, in execute
    self._launch_workers()
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/runtime.py", line 877, in _launch_workers
    task = self._new_task(step, task_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/runtime.py", line 186, in _new_task
    return Task(
           ^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/runtime.py", line 1186, in __init__
    deco.runtime_task_created(
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_decorator.py", line 222, in runtime_task_created
    self.environment.interpreter(self.step)
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_environment.py", line 205, in interpreter
    return self.solvers["conda"].interpreter(id_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ferras.hamad/miniconda3/lib/python3.12/site-packages/metaflow/plugins/pypi/micromamba.py", line 215, in interpreter
    return os.path.join(self.path_to_environment(id_), "bin/python")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType
i am running it within an active conda environment on osx (apple silicon)
g
@icy-kite-33230 unfortunately, I was not. If anything changes, I will post an update.
i
thanks @gentle-motherboard-71754 I might try switching to the nflx implementation: https://github.com/Netflix/metaflow-nflx-extensions
@gentle-motherboard-71754 ok coming back to this. My problem was that i had both micromamba and anaconda installed try running 
micromamba env list
and then uninstalling one in my case
Copy code
(venv) (base)  00-helloworld % /opt/homebrew/bin/micromamba env list
  Name  Active  Path                          
────────────────────────────────────────────────
  base          /Users/ferras.hamad/micromamba
                /opt/homebrew/anaconda3
Copy code
brew uninstall anaconda
Copy code
sudo chmod 777 /Users/ferras.hamad/.conda/environments.txt
^ cc @square-wire-39606 for future reference
👍🏼 1
👍 2
a
@thankful-ambulance-42457 fyi
14 Views
Open in Slack
PreviousNext
Powered by