Greetings Metaflow Team 👋 , Quick question on the support for Pure PyPi Images for metaflow-nflx-extensions - As per the known issues, “plugin relies on conda, mamba, and micromamba”. Enabling this plugin is not allowing pure PyPI environments to be created. Is there a configuration of the env variables/parameters with which we can create a pure PYPI environment that does not go to conda for resolving the environment? Or is it not possible at all to do so? Background • We want to only rely on PYPI to get our dependencies and do not want to hit conda repos to get the packages, due to VPN & Security boundaries. • Tried with only metaflow package, without the extensions and it works with PyPI repositories for pypi_base decorator - but as soon as we install metaflow-nflx-extensions, it goes to the conda repos for resolving the environment. • Leveraging AWS Batch to run metaflow & currently only using

@pypi_base

as decorator for managing the environments. Setting following env variables

CONDA_PYPI_DEPENDENCY_RESOLVER=pip METAFLOW_CONDA_DEPENDENCY_RESOLVER=micromamba CONDA_DEPENDENCY_RESOLVER=micromamba CONDA_CHANNEL_PRIORITY=flexible METAFLOW_DEBUG_CONDA=1 CONDA_MIXED_DEPENDENCY_RESOLVER=conda-lock CONDA_CHANNELS='conda-forge'  --environment conda

Thanks a lot for your support and response in advance.

Hello. The non Netflix extension version of pypi_base also uses Conda for the base environment (it basically installs Python that way). If you want to get a similar behavior you need to set METAFLOW_CONDA_DEPENDENCY_RESOLVER to micromamba as you currently do. You may need to setup your channels too depending on your restrictions but you should get the same behavior as the non Netflix extension one. If I may ask though, what functionality are you looking to get from the extension (since you want to use it a bit more “basically”.

Thanks for your response @dry-beach-38304 - what should be the channel if we want to leverage only pypi-repositories ? - As currently its ‘conda-forge’, not sure what it should be for PyPI

apologies for the confusion: • you need at least one conda channel for both the basic Metaflow pypi decorator AND the one for the extension. Typically this is conda-forge so if it works for basic Metaflow pypi decorator, it should work for the extension. • are you getting a specific error message that may help me figure out where hte problem is coming from. There shouldn’t be much difference but I am maybe missing something.

Currently the issue is that as soon as we add this package it tries to connect to conda - while just metaflow, it does not do so. Here is the log for it

debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:518]: Builder environments: {EnvID(req_id='9ecf751db3a818702ebe5023daaacacd10f24bbe', full_id='_default', arch='linux-64'): {'id': EnvID(req_id='9ecf751db3a818702ebe5023daaacacd10f24bbe', full_id='_default', arch='linux-64'), 'steps': ['query_data', 'deploy_model'], 'user_deps': {'conda': ['python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools'], 'npconda': [], 'sys': ['__glibc==2.35=0']}, 'deps': {'conda': ['python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools'], 'npconda': [], 'sys': ['__glibc==2.35=0']}, 'sources': {'conda': ['conda-forge']}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': False, 'resolved': None, 'already_resolved': False, 'env_type': <EnvType.CONDA_ONLY: 'conda-only'>}}
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:278]: Conda call: ['/usr/bin/micromamba', 'create', '--yes', '--prefix', '/tmp/tmpefsoe5pa/prefix', '--dry-run', 'python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools', '-r', '/opt/conda', '--json']

debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/resolvers/pip_resolver.py:103]: Creating builder conda environment
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:278]: Conda call: ['/usr/bin/micromamba', 'info', '--json']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: ld_impl_linux-64-2.43-h712a8e2_1 -> preferred .conda @ web
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: _libgcc_mutex-0.1-conda_forge -> preferred .tar.bz2 @ web

We want to stop this from happening - but unable to figure out what should be the parameters to use only PYPI repo

and this does not happen with basic @pypi (ie: no extension)

If we only install metaflow on the container and run the program with

@pypi_base

, then we dont see this behaviour. If we install the extensions package and then run the same code, it tries to go to conda and throws these errors

right above that, it shoudl print out the command line it is executing, could you let me know what that is so I can see how it should match with the one from non extension? You can see here that the main code will use micromamba to resolve an environment: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/micromamba.py#L53 (you could add a print here https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/micromamba.py#L220 to be able to compare to the command line that is being executed in the extension. I am a bit stumped as to why it is working in one case and not the other but it’s probably some subtle difference in arguments. Both definitely use micromamba to resolve a base python environment.

These are the commands that it seemed to be executing with extension

2024-09-30 16:35:08.755 Creating local datastore in current directory (/metaflow/.metaflow)
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at <s3://cnn-mls-metaflow-science/metaflow/conda_env>
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:459]: For step start, got flow req: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]; step req: StepReq[disabled=True]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:466]: For step start, merged requirement: StepReq[disabled=True]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:459]: For step query_data, got flow req: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]; step req: StepReq[]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:466]: For step query_data, merged requirement: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:335]: Binary call: ['/opt/conda/bin/cph', '--version']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:335]: Binary call: ['/opt/conda/bin/conda-lock', '--version']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:335]: Binary call: ['/opt/conda/bin/pip', '--version']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:278]: Conda call: ['/usr/bin/micromamba', 'info', '--json']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:335]: Binary call: ['/opt/conda/bin/pip', 'config', 'list']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:619]: For step query_data, final req: StepReq[disabled=False; python=3.8; packages={'pypi': {'numpy': '1.19.4', 'boto3': '>=1.14.0', 'requests': '>=2.21.0', 'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2'}, 'conda': {'python': '3.8'}, 'sys': {'__glibc': '2.35=0'}}; sources={'conda': ['conda-forge'], 'pypi': ['<https://pypi.org/simple']}]>
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:590]: EnvID(req_id='0669c556ab12d33fd1720672a5337f35af4edcba', full_id='_default', arch='linux-64') not found locally
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:206]: Added environment to resolve {'id': EnvID(req_id='0669c556ab12d33fd1720672a5337f35af4edcba', full_id='_default', arch='linux-64'), 'steps': ['query_data'], 'user_deps': {'pypi': ['numpy==1.19.4', 'boto3==>=1.14.0', 'requests==>=2.21.0', 'snowflake-connector-python[pandas]==2.9', 'scipy==1.5.3', 'joblib==1.2'], 'conda': ['python==3.8'], 'sys': ['__glibc==2.35=0']}, 'deps': {'pypi': ['numpy==1.19.4', 'boto3==>=1.14.0', 'requests==>=2.21.0', 'snowflake-connector-python[pandas]==2.9', 'scipy==1.5.3', 'joblib==1.2'], 'conda': ['python==3.8'], 'sys': ['__glibc==2.35=0']}, 'sources': {'conda': ['conda-forge'], 'pypi': ['<https://pypi.org/simple']>}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': None, 'already_resolved': False, 'env_type': <EnvType.PYPI_ONLY: 'pypi-only'>, 'force': False}
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:459]: For step deploy_model, got flow req: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]; step req: StepReq[]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:466]: For step deploy_model, merged requirement: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:335]: Binary call: ['/opt/conda/bin/pip', 'config', 'list']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:619]: For step deploy_model, final req: StepReq[disabled=False; python=3.8; packages={'pypi': {'numpy': '1.19.4', 'boto3': '>=1.14.0', 'requests': '>=2.21.0', 'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2'}, 'conda': {'python': '3.8'}, 'sys': {'__glibc': '2.35=0'}}; sources={'conda': ['conda-forge'], 'pypi': ['<https://pypi.org/simple']}]>
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:590]: EnvID(req_id='0669c556ab12d33fd1720672a5337f35af4edcba', full_id='_default', arch='linux-64') not found locally
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:211]: Environment 'EnvID(req_id='0669c556ab12d33fd1720672a5337f35af4edcba', full_id='_default', arch='linux-64')' is also needed by 'deploy_model'
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:459]: For step end, got flow req: StepReq[disabled=False; python=3.8; packages={'pypi': {'snowflake-connector-python[pandas]': '2.9', 'scipy': '1.5.3', 'joblib': '1.2', 'numpy': '1.19.4'}}; sources={'pypi': []}]; step req: StepReq[disabled=True]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:466]: For step end, merged requirement: StepReq[disabled=True]
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:231]: Resolving environments:
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:234]: 0669c556ab12d33fd1720672a5337f35af4edcba (_default): {'id': EnvID(req_id='0669c556ab12d33fd1720672a5337f35af4edcba', full_id='_default', arch='linux-64'), 'steps': ['query_data', 'deploy_model'], 'user_deps': {'pypi': ['numpy==1.19.4', 'boto3==>=1.14.0', 'requests==>=2.21.0', 'snowflake-connector-python[pandas]==2.9', 'scipy==1.5.3', 'joblib==1.2'], 'conda': ['python==3.8'], 'sys': ['__glibc==2.35=0']}, 'deps': {'pypi': ['numpy==1.19.4', 'boto3==>=1.14.0', 'requests==>=2.21.0', 'snowflake-connector-python[pandas]==2.9', 'scipy==1.5.3', 'joblib==1.2'], 'conda': ['python==3.8'], 'sys': ['__glibc==2.35=0']}, 'sources': {'conda': ['conda-forge'], 'pypi': ['<https://pypi.org/simple']>}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': None, 'already_resolved': False, 'env_type': <EnvType.PYPI_ONLY: 'pypi-only'>, 'force': False}
    Resolving 1 environment ...debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:590]: EnvID(req_id='9ecf751db3a818702ebe5023daaacacd10f24bbe', full_id='_default', arch='linux-64') not found locally
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:518]: Builder environments: {EnvID(req_id='9ecf751db3a818702ebe5023daaacacd10f24bbe', full_id='_default', arch='linux-64'): {'id': EnvID(req_id='9ecf751db3a818702ebe5023daaacacd10f24bbe', full_id='_default', arch='linux-64'), 'steps': ['query_data', 'deploy_model'], 'user_deps': {'conda': ['python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools'], 'npconda': [], 'sys': ['__glibc==2.35=0']}, 'deps': {'conda': ['python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools'], 'npconda': [], 'sys': ['__glibc==2.35=0']}, 'sources': {'conda': ['conda-forge']}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': False, 'resolved': None, 'already_resolved': False, 'env_type': <EnvType.CONDA_ONLY: 'conda-only'>}}
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:278]: Conda call: ['/usr/bin/micromamba', 'create', '--yes', '--prefix', '/tmp/tmpefsoe5pa/prefix', '--dry-run', 'python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools', '-r', '/opt/conda', '--json']

debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/resolvers/pip_resolver.py:103]: Creating builder conda environment
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:278]: Conda call: ['/usr/bin/micromamba', 'info', '--json']
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: ld_impl_linux-64-2.43-h712a8e2_1 -> preferred .conda @ web
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: _libgcc_mutex-0.1-conda_forge -> preferred .tar.bz2 @ web
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: ca-certificates-2024.8.30-hbcca054_0 -> preferred .conda @ web
debug[conda /opt/conda/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1521]: libgomp-14.1.0-h77fa898_1 -> preferred .conda @ web

Apologies for not being very clear: • in both cases, something is happening with regards to conda. In the second case, it doesn’t print much but between the

Bootstrapping virtual environment(s)

and

Virtual environment(s) bootstrapped!

you can see that over a minute goes by and it’s doing something. • clearly one work and one doesn’t so something the extension is doing is slightly different and your firewall/proxy/whatever-security-thing-your-company-puts is blocking it. • since I don’t have access to that “thing” that is blocking access, I can only speculate and I am trying to narrow down what the issue may be. • You can see for example that in the extension case, this command seems to work:

['/usr/bin/micromamba', 'create', '--yes', '--prefix', '/tmp/tmpefsoe5pa/prefix', '--dry-run', 'python==3.8', 'pip', 'wheel==0.42.0', 'tomli', 'setuptools', '-r', '/opt/conda', '--json']

but it then has issues downloading the packages themselves (at least that is what I am guessing). • In the non-extension case, packages are downloaded here: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/micromamba.py#L120 which is why I was suggesting instrumenting the _call function a bit. I’m really sorry about trying to gather more information this way but this is not something I can readily reproduce since it heavily depends on your corporate environment. I can only speculate and, to be best of my knowledge, both versions WILL download stuff. I am wondering what the exact different is and why it happens to help you better. There are a bunch of other variables (things can get cached, etc) which can also impact the result but that’s hard for me to speculate on without a bit more information and unfortunately the non-extension pypi implementation gives NO details.

Thanks for your response @dry-beach-38304 - this is very helpful information to dig deeper. Would not it be using this one - https://github.com/Netflix/metaflow/blob/255cdbeba3ddbcde2444f1483de2a3f935807a26/metaflow/plugins/pypi/pip.py#L74 when we are doing only PyPI base ?

after yes — not for the base environment. It needs a base python environment to go on.

Got it - thanks for your guidance @dry-beach-38304. Will see if I could debug with more granular info and will get back to you