https://outerbounds.com/ logo
Join Slack
Powered by
`Pip ran into an error while setting up environmen...
# ask-metaflow
f
Pip ran into an error while setting up environment:
command '/Users/ed/.metaflowconfig/micromamba/bin/micromamba run --prefix /Users/ed/micromamba/envs/metaflow/osx-arm64/166809f7a3e0372 pip3 --disable-pip-version-check --no-color --no-input --isolated install --dry-run --only-binary=:all: --upgrade-strategy=only-if-needed --target=/var/folders/y6/vkmmb9lj41q4_0dq4q_h0xyh0000gn/T/tmpg8_fi5i9 --report=/var/folders/y6/vkmmb9lj41q4_0dq4q_h0xyh0000gn/T/tmpg8_fi5i9/report.json --progress-bar=off --quiet --abi none --abi abi3 --abi cp312 --platform macosx_10_8_universal2 --platform macosx_14_0_arm64 --platform macosx_10_6_universal2 --platform macosx_13_0_universal2 --platform macosx_13_0_arm64 --platform macosx_10_12_universal2 --platform macosx_10_7_universal2 --platform macosx_12_0_arm64 --platform macosx_10_11_universal2 --platform macosx_11_0_arm64 --platform macosx_10_15_universal2 --platform macosx_11_0_universal2 --platform macosx_10_13_universal2 --platform macosx_10_14_universal2 --platform macosx_12_0_universal2 --platform macosx_10_9_universal2 --platform macosx_14_0_universal2 --platform macosx_10_4_universal2 --platform any --platform macosx_10_16_universal2 --platform macosx_10_5_universal2 --platform macosx_10_10_universal2 requests>=2.21.0 boto3>=1.14.0 keras-cv==0.9.0' returned error (1)
error    libmamba Could not set lock (Resource temporarily unavailable)
warning  libmamba Cannot lock '/Users/ed/.cache/mamba/proc'
Waiting for other mamba process to finish
ERROR: Cannot install keras-cv because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
✅ 1
d
NOt sure why it is failing. It looks like it is doing some cross platform thing when it doesn’t really need to. I suspect thgouh that you have some issue with your micromamba install. You can try wiping out the 
.metaflowconfig/micromamba
directory or also try the bleeding edge ones that work differently and I verified can resolve this easily: https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators. They can use any installation of mamba/conda/micromamba that you have instead of downloading one. I’d try wiping the dir first and if that doesn’t work giving the extension a try.
f
Great suggestions, I’ll give these a shot. Didn’t know about the bleeding edge decorators, these look cool
d
Yes, very cool. But am biased (it’s what we use here at Netflix).
f
Quick question. I tried installing the metaflow-extension library, and then running a simple metaflow step, but I keep getting 
Permission denied: '/opt/mf_env-creation.lock'
. What is going on? Why can’t I run any metaflow step without getting this error? I tried using a clean conda env, but this didn’t fix anything. For what it’s worth, the 
mf_env-creation.lock
doesn’t exist in /opt/. I tried creating one, but it didn’t help.
I noticed that when setting “”METAFLOW_CONDA_DEPENDENCY_RESOLVER”: “micromamba”" in config.json, it works. But this is strange, shouldn’t ‘mamba’ also work?
I’m also noticing that when adding an @batch decorator, that the behavior of @pypi changes. For example the following code with both decorators:
@pypi(packages={'tensorflow': '2.16.1', 'keras-cv': '0.9.0'})
@batch(gpu=1, image="<http://docker.io/tensorflow/tensorflow:latest-gpu|docker.io/tensorflow/tensorflow:latest-gpu>", queue="job-queue-gpu-metaflow",)
This causes an error: 
The conflict is caused by:
tensorflow-datasets 4.9.6 depends on promise
tensorflow-datasets 4.9.5 depends on promise
tensorflow-datasets 4.9.4 depends on promise
tensorflow-datasets 4.9.3 depends on promise
tensorflow-datasets 4.9.2 depends on promise
tensorflow-datasets 4.9.1 depends on promise
tensorflow-datasets 4.9.0 depends on promise
tensorflow-datasets 4.8.3 depends on promise
tensorflow-datasets 4.8.2 depends on promise
tensorflow-datasets 4.8.1 depends on promise
tensorflow-datasets 4.8.0 depends on promise
tensorflow-datasets 4.7.0 depends on promise
tensorflow-datasets 4.6.0 depends on promise
tensorflow-datasets 4.5.2 depends on promise
tensorflow-datasets 4.5.1 depends on promise
tensorflow-datasets 4.5.0 depends on promise
tensorflow-datasets 4.4.0 depends on promise
tensorflow-datasets 4.3.0 depends on promise
tensorflow-datasets 4.2.0 depends on promise
tensorflow-datasets 4.1.0 depends on promise
tensorflow-datasets 4.0.1 depends on promise
tensorflow-datasets 4.0.0 depends on promise
tensorflow-datasets 3.2.1 depends on promise
tensorflow-datasets 3.2.0 depends on promise
tensorflow-datasets 3.1.0 depends on promise
tensorflow-datasets 3.0.0 depends on promise
tensorflow-datasets 2.1.0 depends on promise
tensorflow-datasets 2.0.0 depends on promise
tensorflow-datasets 1.3.2 depends on promise
tensorflow-datasets 1.3.1 depends on promise
tensorflow-datasets 1.3.0 depends on promise
tensorflow-datasets 1.2.0 depends on promise
tensorflow-datasets 1.1.0 depends on promise
tensorflow-datasets 1.0.2 depends on promise
tensorflow-datasets 1.0.1 depends on promise
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
Pretty-printed STDERR:
ERROR: Cannot install keras-cv because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
Whereas just running the @pypi decorator works. Thanks in advance for all of the help. I really want to love Metaflow, but I keep getting stopped by these weird conflicts 🙃
d
Going backwards: • adding 
batch
does change the behavior in the sense that it now resolves for a linux-64 enviornment (instead of the mac if you are running on a mac for example) • for mamba and that weird error, I suspect that 
mamba info
returns /opt as its envs directories (or at least as the first one) ◦ this is what we use to determine where to go. You can probably fix it by configuring mamba to put your environments elsewhere or changing the permission on /opt To further elucidate the issue, it’s because mamba/conda and the likes, absolutely do not like it whne there are multiple things doing concurrent modifications so this is an attempt to add a lock to make sure concurrent processes behave.
you can run with 
METAFLOW_DEBUG_CONDA=1
and it can give more detail too. Note for both those packages too, I think they are on conda so it is usually preferable to use that (resolution is usually better)
f
Here is my 
mamba info
, I don’t see /opt in here:
mamba version : 1.5.8
active environment : base
active env location : /Users/ed/miniconda3
shell level : 2
user config file : /Users/ed/.condarc
populated config files : /Users/ed/.condarc
conda version : 24.5.0
conda-build version : not installed
python version : 3.11.8.final.0
solver : libmamba (default)
virtual packages : __archspec=1=m1
__conda=24.5.0=0
__osx=14.5=0
__unix=0=0
base environment : /Users/ed/miniconda3  (writable)
conda av data dir : /Users/ed/miniconda3/etc/conda
conda av metadata url : None
channel URLs : <https://conda.anaconda.org/conda-forge/osx-arm64>
<https://conda.anaconda.org/conda-forge/noarch>
<https://conda.anaconda.org/Microsoft/osx-arm64>
<https://conda.anaconda.org/Microsoft/noarch>
<https://repo.anaconda.com/pkgs/main/osx-arm64>
<https://repo.anaconda.com/pkgs/main/noarch>
<https://repo.anaconda.com/pkgs/r/osx-arm64>
<https://repo.anaconda.com/pkgs/r/noarch>
package cache : /Users/ed/miniconda3/pkgs
/Users/ed/.conda/pkgs
envs directories : /Users/ed/miniconda3/envs
/Users/ed/.conda/envs
platform : osx-arm64
user-agent : conda/24.5.0 requests/2.31.0 CPython/3.11.8 Darwin/23.5.0 OSX/14.5 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.8
UID:GID : 501:20
netrc file : /Users/ed/.netrc
offline mode : False
d
oh tath’s interesting. could you run with the METAFLOW_DEBUG_CONDA=1.
f
Debug soup incoming:
✅ 1
Metaflow 2.12.0+netflix-ext(1.2.0) executing main_flow for user:ed
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py:164]: Conda root is at <s3://metaflow-metaflows3bucket-d52svurmzuek/metaflow/conda_env>
Validating your flow...
The graph looks good!
Running pylint...
Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step start, got flow req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}}; sources={'pypi': []}]; step req: StepReq[]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step start, merged requirement: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}}; sources={'pypi': []}]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/ed/miniconda3/envs/plant-object-detection/bin/cph', '--version']
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/ed/miniconda3/envs/plant-object-detection/bin/conda-lock', '--version']
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/ed/miniconda3/envs/plant-object-detection/bin/pip', '--version']
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/Users/ed/miniconda3/envs/plant-object-detection/bin/mamba', 'info', '--json']
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step start, final req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}, 'conda': {'boto3': '>=1.14.0', 'python': '3.12.3', 'requests': '>=2.21.0'}, 'sys': {}}; sources={'conda': ['conda-forge', 'Microsoft', '<https://repo.anaconda.com/pkgs/main/osx-arm64>', '<https://repo.anaconda.com/pkgs/main/noarch>', '<https://repo.anaconda.com/pkgs/r/osx-arm64>', '<https://repo.anaconda.com/pkgs/r/noarch']}]>
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:204]: Added environment to resolve {'id': EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='_default', arch='osx-arm64'), 'steps': ['start'], 'user_deps': {'pypi': [], 'conda': ['boto3==>=1.14.0', 'python==3.12.3', 'requests==>=2.21.0'], 'sys': []}, 'deps': {'pypi': [], 'conda': ['boto3==>=1.14.0', 'python==3.12.3', 'requests==>=2.21.0'], 'sys': []}, 'sources': {'conda': ['conda-forge', 'Microsoft', '<https://repo.anaconda.com/pkgs/main/osx-arm64>', '<https://repo.anaconda.com/pkgs/main/noarch>', '<https://repo.anaconda.com/pkgs/r/osx-arm64>', '<https://repo.anaconda.com/pkgs/r/noarch']>}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': <metaflow_extensions.netflix_ext.plugins.conda.env_descr.ResolvedEnvironment object at 0x106008440>, 'already_resolved': True, 'env_type': <EnvType.CONDA_ONLY: 'conda-only'>, 'force': False}
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step parse_and_transform_records, got flow req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}}; sources={'pypi': []}]; step req: StepReq[disabled=False; packages={'pypi': {'tensorflow': '2.16.1', 'keras-cv': '0.9'}}; sources={'pypi': []}]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step parse_and_transform_records, merged requirement: StepReq[disabled=False; python=3.12.3; packages={'pypi': {'tensorflow': '2.16.1', 'keras-cv': '0.9'}}; sources={'pypi': []}]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/Users/ed/miniconda3/envs/plant-object-detection/bin/pip', 'config', 'list']
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step parse_and_transform_records, final req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {'tensorflow': '2.16.1', 'boto3': '>=1.14.0', 'keras-cv': '0.9', 'requests': '>=2.21.0'}, 'conda': {'python': '3.12.3'}, 'sys': {}}; sources={'conda': ['conda-forge', 'Microsoft', '<https://repo.anaconda.com/pkgs/main/osx-arm64>', '<https://repo.anaconda.com/pkgs/main/noarch>', '<https://repo.anaconda.com/pkgs/r/osx-arm64>', '<https://repo.anaconda.com/pkgs/r/noarch>'], 'pypi': ['<https://pypi.org/simple']}]>
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='adc33cc7b10c7cf22bee06205ee3a2ce52ed949a', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:204]: Added environment to resolve {'id': EnvID(req_id='adc33cc7b10c7cf22bee06205ee3a2ce52ed949a', full_id='_default', arch='osx-arm64'), 'steps': ['parse_and_transform_records'], 'user_deps': {'pypi': ['tensorflow==2.16.1', 'boto3==>=1.14.0', 'keras-cv==0.9', 'requests==>=2.21.0'], 'conda': ['python==3.12.3'], 'sys': []}, 'deps': {'pypi': ['tensorflow==2.16.1', 'boto3==>=1.14.0', 'keras-cv==0.9', 'requests==>=2.21.0'], 'conda': ['python==3.12.3'], 'sys': []}, 'sources': {'conda': ['conda-forge', 'Microsoft', '<https://repo.anaconda.com/pkgs/main/osx-arm64>', '<https://repo.anaconda.com/pkgs/main/noarch>', '<https://repo.anaconda.com/pkgs/r/osx-arm64>', '<https://repo.anaconda.com/pkgs/r/noarch>'], 'pypi': ['<https://pypi.org/simple']>}, 'extras': {}, 'conda_format': ['_any'], 'base': None, 'base_accurate': None, 'resolved': <metaflow_extensions.netflix_ext.plugins.conda.env_descr.ResolvedEnvironment object at 0x106031880>, 'already_resolved': True, 'env_type': <EnvType.PYPI_ONLY: 'pypi-only'>, 'force': False}
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:461]: For step end, got flow req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}}; sources={'pypi': []}]; step req: StepReq[]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:468]: For step end, merged requirement: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}}; sources={'pypi': []}]
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py:642]: For step end, final req: StepReq[disabled=False; python=3.12.3; packages={'pypi': {}, 'conda': {'boto3': '>=1.14.0', 'python': '3.12.3', 'requests': '>=2.21.0'}, 'sys': {}}; sources={'conda': ['conda-forge', 'Microsoft', '<https://repo.anaconda.com/pkgs/main/osx-arm64>', '<https://repo.anaconda.com/pkgs/main/noarch>', '<https://repo.anaconda.com/pkgs/r/osx-arm64>', '<https://repo.anaconda.com/pkgs/r/noarch']}]>
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:209]: Environment 'EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='_default', arch='osx-arm64')' is also needed by 'end'
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py:229]: Resolving environments:
All packages already cached in s3.
All environments already cached in s3.
2024-06-11 16:25:06.079 Workflow starting (run-id 34):
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:579]: EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='_default', arch='osx-arm64') found locally
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2031]: Locating MF environment metaflow_1e7721c8402615bc78f3a75fea7d0a317f8a336a_b56b2591df5f48c534f4a2a99d57e99fe41bcd65
debug[conda /Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='1e7721c8402615bc78f3a75fea7d0a317f8a336a', full_id='b56b2591df5f48c534f4a2a99d57e99fe41bcd65', arch='osx-arm64') at dir_name /Users/ed/micromamba/envs/metaflow_1e7721c8402615bc78f3a75fea7d0a317f8a336a_b56b2591df5f48c534f4a2a99d57e99fe41bcd65
2024-06-11 16:25:09.231 Workflow failed.
2024-06-11 16:25:09.232 Terminating 0 active tasks...
2024-06-11 16:25:09.232 Flushing logs...
Internal error
Traceback (most recent call last):
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/cli.py", line 1069, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
return self.main(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
✅ 1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
return callback(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/cli.py", line 563, in wrapper
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/cli.py", line 746, in run
runtime.execute()
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/runtime.py", line 325, in execute
self._launch_workers()
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/runtime.py", line 732, in _launch_workers
self._launch_worker(task)
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/runtime.py", line 758, in _launch_worker
worker = Worker(task, self._max_log_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/runtime.py", line 1383, in __init__
self._proc = self._launch()
^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow/runtime.py", line 1428, in _launch
deco.runtime_step_cli(
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_step_decorator.py", line 532, in runtime_step_cli
existing_env_info = conda.created_environment(self._env_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 507, in created_environment
envs = self._created_envs(prefix, full_match=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2064, in _created_envs
with CondaLock(self.echo, self._env_lock_file(env)):
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2680, in __enter__
self._acquire()
File "/Users/ed/miniconda3/envs/plant-object-detection/lib/python3.12/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2645, in _acquire
self.fd = os.open(self.lock, os.O_CREAT | os.O_EXCL | os.O_RDWR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/opt/mf_env-creation.lock'
d
could you do: 
mamba info --json
and let me know if there is opt in “envs”
f
There is, at the very bottom
“envs”: [
“/Users/ed/micromamba/envs/metaflow/osx-arm64/096fd957f5cb98f”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/0b6f7471a194edb”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/104c8e9cf769d5d”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/1170879434cf520”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/147c53213985583",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/147d7bb838d3137”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/15bb12525fdb62e”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/165a00e0f124577”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/166809f7a3e0372",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/19afa77fa748979”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/1a63b255b97314b”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/2316f16e16c1247”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/2910d02ea49ac25",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/2a9406d48ef1b20”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/308d7adc18025a9",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/3a7b08298c8d50e”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/3d5268f6f9d834f”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/43abcfdf19b3c86”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/47d6a873a6084b3",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/488fd6f831505fb”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/4aa686e34d47cd0",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/573a8b5800c7faf”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/5807c0d141bbfa5",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/595458e4caadfbd”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/5aa4afb667a9d9b”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/5bc90e4ede8da36”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/699eeb61ea70215",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/6eaa003b3922939”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/6ee3f6a8e82466b”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/6ff04e29180f9a5”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/70647977ee6f4b5",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/8635294ec051f2a”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/8bc7ec47fde13dc”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/9550889d33a96c9”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/9856723fa6ae1f5",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ac1e0584a19e20d”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ad79c2602ca462d”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ad947e625a7680d”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/b5905d75e574c2d”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ba9f730514e5dfc”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/bebefe5f284810b”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/c2996a5dda3d10c”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/cc88d9543911cd3",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/cd32c77cf3ae2b3”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/d16bca205c510ad”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/dfefbdafb5fc274”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/e81ccf5164820a4",
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ec82b865d36ea24”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/ed6879bd141f68a”,
“/Users/ed/micromamba/envs/metaflow/osx-arm64/fdff87d8fc7a5d4”,
“/Users/ed/micromamba/envs/metaflow_0253a4419b63768a4cfadabd8fae2f02c2fffb4a_57de1c553dc50f7a8c4cf4dbf66d91592d97c0b3",
“/Users/ed/micromamba/envs/metaflow_1e7721c8402615bc78f3a75fea7d0a317f8a336a_b56b2591df5f48c534f4a2a99d57e99fe41bcd65”,
“/Users/ed/micromamba/envs/metaflow_7bcdfeaad051e9cdf26c53fbb84ab5e9bfa572b3_998178228bc2e9460d01673df4343dd5625df632",
“/Users/ed/micromamba/envs/metaflow_850d1cc64f1bf1c28f1022aa2c3619da56d862b8_f41587451ec930beec4d8c891d9a848fba95e445”,
“/Users/ed/micromamba/envs/metaflow_adc33cc7b10c7cf22bee06205ee3a2ce52ed949a_92a5c54a2a36184f93b30cc254903e1c0189dbec”,
“/Users/ed/micromamba/envs/metaflow_builder_99e7f13a79f71ac6869a725ddb8138fd38cd8495_7542f3473141911703ac6273e1e396989f00a960”,
“/Users/ed/miniconda3",
“/Users/ed/miniconda3/envs/Azure”,
“/Users/ed/miniconda3/envs/CICDTesting”,
“/Users/ed/miniconda3/envs/FilmBot”,
“/Users/ed/miniconda3/envs/LLMDetect”,
“/Users/ed/miniconda3/envs/NoOps”,
“/Users/ed/miniconda3/envs/TerraformTutorial”,
“/Users/ed/miniconda3/envs/bigger-boat”,
“/Users/ed/miniconda3/envs/cv-test-env”,
“/Users/ed/miniconda3/envs/dummy_test_pypi”,
“/Users/ed/miniconda3/envs/keras-cv”,
“/Users/ed/miniconda3/envs/langchaintutorial”,
“/Users/ed/miniconda3/envs/new-tfmetal”,
“/Users/ed/miniconda3/envs/plant-object-detection”,
“/Users/ed/miniconda3/envs/plant-object-detection/envs/metaflow_builder_99e7f13a79f71ac6869a725ddb8138fd38cd8495_731b7ad7647f70062282d1d08cca666517474087",
“/Users/ed/miniconda3/envs/plant-object-detection/envs/metaflow_builder_d11123cefdef3b6980078906f1a978f8805f3d8e_79c85c7d8b7d7b752daf2c5c1e98ae64e1b4b52b”,
“/Users/ed/miniconda3/envs/post-modern-stack”,
“/Users/ed/miniconda3/envs/soda_tutorial”,
“/Users/ed/miniconda3/envs/test-gpu”,
“/Users/ed/miniconda3/envs/torch”,
“/opt/anaconda3"
],
Also here is 
env_vars
: 
"env_vars": {
"CIO_TEST": "<not set>",
"CONDA_DEFAULT_ENV": "base",
"CONDA_EXE": "/Users/ed/miniconda3/bin/conda",
"CONDA_PREFIX": "/Users/ed/miniconda3",
"CONDA_PREFIX_1": "/opt/anaconda3",
"CONDA_PROMPT_MODIFIER": "(base) ",
"CONDA_PYTHON_EXE": "/Users/ed/miniconda3/bin/python",
"CONDA_ROOT": "/Users/ed/miniconda3",
"CONDA_SHLVL": "2",
"CURL_CA_BUNDLE": "<not set>",
"INFOPATH": "/opt/homebrew/share/info:",
"LD_PRELOAD": "<not set>",
"MANPATH": "/opt/homebrew/share/man::",
"PATH": "/Users/ed/.rbenv/shims:/Users/ed/miniconda3/bin:/opt/anaconda3/condabin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/ed/Applications/iTerm.app/Contents/Resources/utilities:/Users/ed/Library/Application Support/JetBrains/Toolbox/scripts",
"REQUESTS_CA_BUNDLE": "<not set>",
"SSL_CERT_FILE": "<not set>"
},
d
Ok. So it’s that last one causing issues. I can work around it most likely by ifniring the error but the fix for now would be to remove that from mamba. Try unsetting CONDA_PREFIX_1. Is that from an older installation?
f
Got it. I was able to remove the opt/anaconda reference (this was an old installation of Anaconda. Found the reference in my bash_profile). Now mamba, conda, and micromamba all work to resolve the libraries when NOT using batch. Unfortunately, the issue when adding the @batch decorator still doesn’t work. I still get the 
tensorflow-datasets x.x.x depends on promise
issue. Installing just tensorflow is OK, but installing TF and Keras-CV leads to this issue. I found a work around by basically running pip install keras-cv after the AWS batch instance is created, but I know this isn’t ideal. Have you run into an issue like this before?
Thanks again for all of your help! You’re doing the community a big favor ❤️
Could be because I am using a docker image: 
@batch(gpu=1, image="<http://docker.io/tensorflow/tensorflow:latest-gpu|docker.io/tensorflow/tensorflow:latest-gpu>", queue="job-queue-gpu-metaflow",)
. Perhaps there is some conflict with a pre-built library I am not aware of?
Nope removing the docker container didn’t fix it.
d
I’ll take a look into that. I am not 100% sure what that message means but does it work if you specify promise explicitly?
could you also do 
metaflow environment show <hash>:<hash>
or 
metaflow environment show --pathspec myflow/run/step
?
f
Unfortunately I can’t do 
metaflow environment show --pathspec myflow/run/step
, because the flow never gets to an actual run. It breaks before the run occurs.
Trying to specify promise with the latest version yields an error, I believe because there is no wheel for it (it is a.tar.gz file).
Using @conda instead of @pypi resolves, but the conda package of keras_cv is old, necessitating using an older python, and older tensorflow, which becomes its own nightmare
d
Ah, sorry, I was misreading the issue.
ok, that makes sense now, yes, when cross-compiling, we can’t use .tar.gz (or other source distributions — this is a pypi issue not something metaflow specific)
ok, hold on.
I think you can use just conda though here right, you are looking for tensorflow 2.16.1 and keras-cv 0.9.0 both of which are on conda-forge:
Copy code
### Environment for architecture linux-64
Environment of type conda-only full hash bd4c3e523c5acc969a41f64ca105f9034ae85018:d069ca52caf9055d2640f68235e48bffeb23de1b
Arch linux-64
Available on linux-64

Resolved on 2024-06-12 22:03:02.930316
Resolved by rcledat

User-requested packages sys::__glibc==2.27, conda::boto3==>=1.14.0, conda::cffi==>=1.13.0,!=1.15.0, conda::fastavro==>=1.6.0, conda::keras-cv==0.9.0, conda::pandas==>=0.24.0, conda::pyarrow==>=0.17.1, conda::python==3.10.11, conda::requests==>=2.21.0, conda::tensorflow==2.16.1
User sources conda::conda-forge

Conda Packages installed _libgcc_mutex==0.1-conda_forge, _openmp_mutex==4.5-2_gnu, absl-py==2.1.0-pyhd8ed1ab_0, astunparse==1.6.3-pyhd8ed1ab_0, aws-c-auth==0.7.22-h9137712_5, aws-c-cal==0.6.15-h88a6e22_0, aws-c-common==0.9.19-h4ab18f5_0, aws-c-compression==0.2.18-h83b837d_6, aws-c-event-stream==0.4.2-h0cbf018_13, aws-c-http==0.8.2-h360477d_2, aws-c-io==0.14.9-h2d549f9_2, aws-c-mqtt==0.10.4-hf85b563_6, aws-c-s3==0.5.10-h679ed35_3, aws-c-sdkutils==0.1.16-h83b837d_2, aws-checksums==0.1.18-h83b837d_6, aws-crt-cpp==0.26.10-h8bc9c4d_4, aws-sdk-cpp==1.11.329-h3eaa560_4, boto3==1.34.124-pyhd8ed1ab_0, botocore==1.34.124-pyge310_1234567_0, brotli-python==1.1.0-py310hc6cd4ac_1, bzip2==1.0.8-hd590300_5, c-ares==1.28.1-hd590300_0, ca-certificates==2024.6.2-hbcca054_0, cached-property==1.5.2-hd8ed1ab_1, cached_property==1.5.2-pyha770c72_1, certifi==2024.6.2-pyhd8ed1ab_0, cffi==1.16.0-py310h2fee648_0, charset-normalizer==3.3.2-pyhd8ed1ab_0, click==8.1.7-unix_pyh707e725_0, colorama==0.4.6-pyhd8ed1ab_0, dill==0.3.8-pyhd8ed1ab_0, dm-tree==0.1.8-py310ha8c1f0e_4, etils==1.7.0-pyhd8ed1ab_0, fastavro==1.9.4-py310h2372a71_0, flatbuffers==24.3.25-h59595ed_0, gast==0.5.4-pyhd8ed1ab_0, gflags==2.2.2-he1b5a44_1004, giflib==5.2.2-hd590300_0, glog==0.7.0-hed5481d_0, google-pasta==0.2.0-pyh8c360ce_0, googleapis-common-protos==1.63.1-pyhd8ed1ab_0, grpcio==1.62.2-py310h1b8f574_0, h5py==3.11.0-nompi_py310hf054cd7_102, hdf5==1.14.3-nompi_hdf9ad27_105, icu==73.2-h59595ed_0, idna==3.7-pyhd8ed1ab_0, importlib-metadata==7.1.0-pyha770c72_0, importlib_resources==6.4.0-pyhd8ed1ab_0, jmespath==1.0.1-pyhd8ed1ab_0, kagglehub==0.2.5-pyhd8ed1ab_0, keras==3.3.3-pyhd8ed1ab_0, keras-core==0.1.7-pyhd8ed1ab_0, keras-cv==0.9.0-pyhd8ed1ab_0, keyutils==1.6.1-h166bdaf_0, krb5==1.21.2-h659d440_0, ld_impl_linux-64==2.40-hf3520f5_3, libabseil==20240116.2-cxx17_h59595ed_0, libaec==1.1.3-h59595ed_0, libarrow==16.1.0-h9696e77_8_cpu, libarrow-acero==16.1.0-hac33072_8_cpu, libarrow-dataset==16.1.0-hac33072_8_cpu, libarrow-substrait==16.1.0-h7e0c224_8_cpu, libblas==3.9.0-22_linux64_openblas, libbrotlicommon==1.1.0-hd590300_1, libbrotlidec==1.1.0-hd590300_1, libbrotlienc==1.1.0-hd590300_1, libcblas==3.9.0-22_linux64_openblas, libcrc32c==1.1.2-h9c3ff4c_0, libcurl==8.8.0-hca28451_0, libedit==3.1.20191231-he28a2e2_2, libev==4.33-hd590300_2, libevent==2.1.12-hf998b51_1, libffi==3.4.2-h7f98852_5, libgcc-ng==13.2.0-h77fa898_8, libgfortran-ng==13.2.0-h69a702a_8, libgfortran5==13.2.0-h3d2ce59_8, libgomp==13.2.0-h77fa898_8, libgoogle-cloud==2.25.0-h2736e30_0, libgoogle-cloud-storage==2.25.0-h3d9a0c8_0, libgrpc==1.62.2-h15f2491_0, libjpeg-turbo==3.0.0-hd590300_1, liblapack==3.9.0-22_linux64_openblas, libnghttp2==1.58.0-h47da74e_1, libnsl==2.0.1-hd590300_0, libopenblas==0.3.27-pthreads_h413a1c8_0, libparquet==16.1.0-h6a7eafb_8_cpu, libpng==1.6.43-h2797004_0, libprotobuf==4.25.3-h08a7969_0, libre2-11==2023.09.01-h5a48ba9_2, libsqlite==3.46.0-hde9e2c9_0, libssh2==1.11.0-h0841786_0, libstdcxx-ng==13.2.0-hc0a3c3a_8, libthrift==0.19.0-hb90f79a_1, libutf8proc==2.8.0-h166bdaf_0, libuuid==2.38.1-h0b41bf4_0, libzlib==1.3.1-h4ab18f5_1, lz4-c==1.9.4-hcb278e6_0, markdown==3.6-pyhd8ed1ab_0, markdown-it-py==3.0.0-pyhd8ed1ab_0, markupsafe==2.1.5-py310h2372a71_0, mdurl==0.1.2-pyhd8ed1ab_0, ml_dtypes==0.3.2-py310hcc13569_0, namex==0.0.8-pyhd8ed1ab_0, ncurses==6.5-h59595ed_0, numpy==1.26.4-py310hb13e2d6_0, openssl==3.3.1-h4ab18f5_0, opt_einsum==3.3.0-pyhc1e730c_2, optree==0.11.0-py310hd41b1e2_0, orc==2.0.1-h17fec99_1, packaging==24.1-pyhd8ed1ab_0, pandas==2.2.2-py310hf9f9076_1, pip==24.0-pyhd8ed1ab_0, promise==2.3-py310hff52083_8, protobuf==4.25.3-py310ha8c1f0e_0, psutil==5.9.8-py310h2372a71_0, pyarrow==16.1.0-py310hb7f781d_3, pyarrow-core==16.1.0-py310h46b3431_3_cpu, pycparser==2.22-pyhd8ed1ab_0, pygments==2.18.0-pyhd8ed1ab_0, pysocks==1.7.1-pyha2e5f31_6, python==3.10.11-he550d4f_0_cpython, python-dateutil==2.9.0-pyhd8ed1ab_0, python-flatbuffers==24.3.25-pyh59ac667_0, python-tzdata==2024.1-pyhd8ed1ab_0, python_abi==3.10-4_cp310, pytz==2024.1-pyhd8ed1ab_0, re2==2023.09.01-h7f4b329_2, readline==8.2-h8228510_1, regex==2024.5.15-py310hc51659f_0, requests==2.32.3-pyhd8ed1ab_0, rich==13.7.1-pyhd8ed1ab_0, s2n==1.4.16-he19d79f_0, s3transfer==0.10.1-pyhd8ed1ab_0, setuptools==70.0.0-pyhd8ed1ab_0, six==1.16.0-pyh6c4a22f_0, snappy==1.2.0-hdb0a2a9_1, tensorboard==2.16.2-pyhd8ed1ab_0, tensorboard-data-server==0.7.0-py310h75e40e8_1, tensorflow==2.16.1-cpu_py310h49b650b_0, tensorflow-base==2.16.1-cpu_py310h224022f_0, tensorflow-datasets==4.8.3-pyhd8ed1ab_0, tensorflow-estimator==2.16.1-cpu_py310hc6dcfef_0, tensorflow-metadata==1.15.0-pyhd8ed1ab_0, termcolor==2.4.0-pyhd8ed1ab_0, tk==8.6.13-noxft_h4845f30_101, toml==0.10.2-pyhd8ed1ab_0, tqdm==4.66.4-pyhd8ed1ab_0, typing-extensions==4.12.2-hd8ed1ab_0, typing_extensions==4.12.2-pyha770c72_0, tzdata==2024a-h0c530f3_0, urllib3==2.2.1-pyhd8ed1ab_0, werkzeug==3.0.3-pyhd8ed1ab_0, wheel==0.43.0-pyhd8ed1ab_1, wrapt==1.16.0-py310h2372a71_0, xz==5.2.6-h166bdaf_0, zipp==3.19.2-pyhd8ed1ab_0, zstd==1.5.6-ha6fb4c9_0
if that doesn’t work for you, the workaround for these types of issues is to build the environment on the target machine first and then use it. You can do this in a flow fairly easily. See this for example: https://netflix.slack.com/archives/C02116BBNTU/p1712816625810499?thread_ts=1712008712.280789&amp;cid=C02116BBNTU
f
Using @conda worked! I had an issue where the conda package was not equal to the pip package (not uncommon.) I had to use a lesser version of Tensorflow and Python to make it work, but finally I’m able to use the decorators as intended. Thanks again for all of your help!
That being said, I am curious about your solution regarding building the environment on the target machine. Unfortunately I was not able to click on the link (looks like I need a Netflix account). Is there somewhere on the github repo where I can see this?
d
I’ll paste it here. I think it’s slack connect acting up 🙂
2 Views
Open in Slack
PreviousNext
Powered by