https://outerbounds.com/ logo
Join Slack
Powered by
Also we are running into an issue with `@pypi` on ...
# ask-metaflow
s
Also we are running into an issue with 
@pypi
on a linux machine. We are using the linux machine to launch flows programmatically to a batch cluster cc @chilly-quill-46782
✅ 1
👍 1
Here are the details of the issue @chilly-quill-46782 is running into: here’s the full output (running on Linux):
Copy code
$ python metaflow/hello_full_example_flow.py --environment=pypi run
Metaflow 2.12.15+netflix-ext(1.2.2) executing FullExampleFlow for user:bibek
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
    Incompatible environment:
    The @pypi decorator requires --environment=conda
After I use 
--environment=conda
, I get this:
Copy code
$ python metaflow/hello_full_example_flow.py --environment=conda run
Metaflow 2.12.15+netflix-ext(1.2.2) executing FullExampleFlow for user:bibek
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Incompatible environment:
    No mamba binary found
After I installed 
mamba
, I got this:
Copy code
$ python metaflow/hello_full_example_flow.py --environment=conda run
Metaflow 2.12.15+netflix-ext(1.2.2) executing FullExampleFlow for user:bibek
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
No STDOUT
Pretty-printed STDERR:
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage]
             [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
             [--tags TAGS]
             [specs ...]
mamba: error: unrecognized arguments: --json

    Conda ran into an error while setting up environment.:
    Conda command '['/home/bibek/envs/torch/bin/mamba', 'info', '--json']' returned error (2); see pretty-printed error above
also a bit confused on why the 
@pypi
decorator need 
--environment=conda
👍 1
s
the error seems to be originating from the bleeding edge decorators. let me pull in @dry-beach-38304 for the resolution. re: 
--environment=conda
- behind the scenes metaflow installs the packages in an isolated virtual environment which is generated through mamba/conda.
d
I’ll take a look.
I suspect you installed this : https://pypi.org/project/mamba/
which is not mamba
(or micromamba or conda but one of those 🙂 )
s
Thanks! The package resolution happens via conda? Is there a way to use pip?
d
the package resolution happens via pip
if you use just 
@pypi
mamba/conda/micromamba is used to provide the base python environment but all packages and resolution will come from pip
👍 1
as a side note, you can definitely just use micromamba as well
which is just a single binary.
you just need to change the configuration to use it.
if you don’t want a full mamba install
s
@chilly-quill-46782 FYI ^^ looks like the mamba package on pypi isn’t what it claims to be.
Got it, we don’t mind doing the full mamba installation.
c
@dry-beach-38304 Thanks for the tip! Please let me know if I got this right: I have to activate the mamba/conda environment and not use my virtualenv for this to work, right? When I installed mamba and kept my virtualenv environment, I got the same error immediately 
The @conda decorator requires --environment=conda
. But when i activated the conda environment, did 
pip install metaflow
inside that environment, then the flow was able to run.
d
You can mix and match venvs and conda if you want. It’s just that Metaflow needs to be able to find your mamba installation. Basically, if 
which mamba
works, it should be all good.
c
which mamba
works, but I when I run the exact same command, I get the error: 
The @pypi decorator requires --environment=conda
d
what command are you running?
c
python metaflow/example_flow.py --environment=pypi run
d
you need 
--environment=conda
instead of 
--environment=pypi
. But rest assured, they are literal synonyms in the mainline work: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pypi_environment.py#L4
😂 3
18 Views
Open in Slack
PreviousNext
Powered by