https://outerbounds.com/ logo
Join Slack
Powered by
Hello everyone. I'm trying to use the `@pypi_base...
# ask-metaflow
r
Hello everyone. I'm trying to use the 
@pypi_base
with a package that is in a private repository but I haven't been able to get it to work. I've defined the pip.conf file as follows:
Copy code
[global]
extra-index-url = https://<USER>:<TOKEN>@gitlab.com/api/v4/projects/<PROJECT_ID>/packages/pypi/simple

[install]
trusted-host = <http://gitlab.com|gitlab.com>
When installing the package with pip outside Metaflow it works as expected. But when Metaflow tries to bootstrap the virtual environment containing the package, I receive an error stating that a package that is a dependency of the private library could not be found (lightgbm in this case):
Copy code
ERROR: Could not find a version that satisfies the requirement lightgbm<5.0.0,>=4.3.0 (from <MY_PRIVATE_LIB>) (from versions: none)
ERROR: No matching distribution found for lightgbm<5.0.0,>=4.3.0
Does anyone have an idea of what the problem might be?
✅ 1
d
it should work. It seems to be looking for lightgbm<5 and more than 4.3. Do you ahve a version of that in your private repo for the version of python you are resolving for (may be different than your outside python version)
r
I don't have the dependencies of the library in the private repository. I expected it would look for the private library in the private respository but it would use pypi.org for its dependencies, because it works like this when I run 
pip install <MY_LIB>
outside the context of Metaflow. Does Metaflow work differently?
Actually the problem is related to lightgbm, When I define it as the only required package in the decorator the error still persists. So I guess the problem is related to the fact that lighgbm doesn't provide a wheel for arm mac (I'm using a M2 Mac) and Metaflow environment resolution doesn't allow to build it from source. Is that right?
d
If you are resolving cross platform no. If you are not, I thought it would work but if not the bleeding edge one will (not cross platform again but on same platform it will work).
r
I've changed to the bleeding edge, but now my private library is not found. Does it ignore the pip.conf file?
d
it does not. You can also try to pass it direction in 
@pypi_base
using 
extra_indices
. If that works but not the other way, I’d be curious as to why. You can run with 
METAFLOW_DEBUG_CONDA=1
and it tells you what it is doing so I can help debug.
r
Using 
extra_indices
the private library is found. If I remove it the command run by Metaflow does not include extra-index-url:
Copy code
Binary call: ['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow_builder_4d09c70acb8c9c8e3165254752b96c84dd309431_299531e902601bbf419c2aa2ec1226b76bf94fec/bin/python', '-m', 'pip', '-v', '--isolated', 'install', '--ignore-installed', '--dry-run', '--target', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpzz3p1gp_', '--report', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpzz3p1gp_/out.json', '-i', '<https://pypi.org/simple>', '<MY_PRIVATE_LIB>==0.3', 'requests>=2.21.0', 'boto3>=1.14.0']
d
awesome. Thanks for that info. I’ll check why that is and push a fix. In the meantime though, are you unblocked? (ie: is the original problem solved?)
r
yes, thanks. Now I just have to figure out how to deal with lightgbm, because since it does not have a wheel for macosx_arm I can't create the environment locally and run the flow, unless I don't use 
@pypi or @conda
decorators. But I need them to run the flow on kubernetes, so it would represent a problem in terms of reproducibility
d
I am confused, does the bleeding edge decorator not build the wheel for you (not cross platform but if there is a wheel for linux, it should work ok on kube too — using that wheel)
r
Do you mean by using something like 
metaflow environment resolve --python "3.10.*" --alias my_named_env -r requirements.txt --arch linux-64
? But even if this work for kubernertes, how could I use the same environment locally in a Mac M1 laptop If I can't resolve it for macosx_arm? Also, I tried running the command above and it failed to find lightgbm 4.3.0. Looking at the command:
Copy code
Binary command for Conda '['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow_builder_4d09c70acb8c9c8e3165254752b96c84dd309431_299531e902601bbf419c2aa2ec1226b76bf94fec/bin/python', '-m', 'pip', '-v', '--isolated', 'install', '--ignore-installed', '--dry-run', '--target', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmptakiezp7', '--report', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmptakiezp7/out.json', '-i', '<https://pypi.org/simple>', '--extra-index-url', '<MY_PRIVATE_REPO>', '--only-binary=:all:', '--abi', 'none', '--abi', 'cp310', '--abi', 'abi3', '--platform', 'manylinux_2_23_x86_64', '--platform', 'manylinux2010_x86_64', '--platform', 'manylinux_2_21_x86_64', '--platform', 'any', '--platform', 'manylinux_2_18_x86_64', '--platform', 'manylinux_2_19_x86_64', '--platform', 'manylinux_2_20_x86_64', '--platform', 'manylinux_2_24_x86_64', '--platform', 'manylinux_2_25_x86_64', '--platform', 'manylinux_2_17_x86_64', '--platform', 'manylinux_2_26_x86_64', '--platform', 'manylinux_2_27_x86_64', '--platform', 'manylinux1_x86_64', '--platform', 'manylinux2014_x86_64', '--platform', 'linux_x86_64', 'requests>=2.21.0', '<MY_PRIVATE_LIB>==0.3', 'boto3>=1.14.0']' returned error (1); see pretty-printed error above
It does not include the platform 'manylinux_2_28_x86_64', which is the one lightgbm uses in the 4.3.0 wheel: https://pypi.org/project/lightgbm/#files
d
Ooh, lots of things here: • first, the 2_28: that’s because by default the bleeding edge decorator assumes you have GLIBC 2.27. This is something you can easily change by setting `CONDA_OVERRIDE_GLIBC`(set it to whatever your linux box supports. Something like “2.35”; I think I support up to 2.38 for now which should be more than what you need) • second, you can actually resolve the enviornment for both architectures if you want. Of course, it’s not the same environment (different architectures) but it should work pretty much how you expect it to. If you do this on the mac arm, it will: ◦ resolve it on linux and pull the pre-built wheel (since it is available) ◦ resolve it on the mac and there it should build the wheel for you and save it ◦ at that point, the environment will be available on both architectures ◦ NOTE: This is not something ppl use very often so it may have a few kinks but it should work ◦ To do it, just pass --arch multiple times. • However, what I was saying is that if you set 
@kubernetes
on any of the steps, it should resolve the environment for Linux (the GLIBC version thing still applies though). You therefore shouldn’t need to use named environments unless you care deeply about an environment being simultaneously resolved on both mac and linux for your set of requirements.
r
Oh, I didn't know about CONDA_OVERRIDE_GLIBC, that solved the lightgbm problem, thx. But I still don't get how I can resolve the environment for macosx_arm, because the wheel is not available for it, so when I try to resolve the env it can't find lightgbm. Actually I was not able to resolve the env for linux either, because there is a dependency that don't provide a wheel, only source dist: https://pypi.org/project/thrift/0.16.0/#files
Copy code
Pretty-printed STDERR:
ERROR: Could not find a version that satisfies the requirement thrift<0.17.0,>=0.16.0 (from databricks-sql-connector) (from versions: none)
ERROR: No matching distribution found for thrift<0.17.0,>=0.16.0
So I think the bottom line of my problems is that I can't resolve dependencies that don't provide a wheel.
looking closer at the command run by Metaflow, it's adding the option '--only-binary=all', so it's preventing the build of packages from source. Is there a way to change this option?
d
I’m out for a bit. I’ll send you a way to do this. It’s not that without wheels it won’t work — it’s no wheels and cross platform.
If you just resolve on Mac for Mac does it not work?
And no, can’t remove that option. Pip requires it if you specify architectures.
r
It's not working for mac, I get the error:
Copy code
Some built pypi packages are not compatible with the target architecture. This can happen when some dependencies do not have an available wheel for the target architecture and you are resolving an environment for a different architecture than the one you are on. Requirements would have us build: packages/pypi/files.pythonhosted.org/packages/74/d1/2e4b02e4611ab36647639c4eea8c4520bb90f948563e00a3bec583a9f9f5/lightgbm-4.3.0.tar.gz
d
You are using the latest extension. Could you send me the full output of the debug run of this. For non cross platform, it should build it for you.
r
Yes, it is the latest version of the extension. There are some sensitive data in the logs, so I will paste only the portion I think is relevant. Let me know if it's not enough:
Copy code
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:165]: Going to build packages lightgbm-4.3.0, thrift-0.16.0
 (building PYPI packages from repositories)debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:172]: Creating builder environment to build PYPI packages
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/opt/homebrew/bin/mamba', 'info', '--json']
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2022]: Found EnvID(req_id='4d09c70acb8c9c8e3165254752b96c84dd309431', full_id='e7415af007f806dd174c49c417ef844272454ab4', arch='osx-arm64') at dir_name /opt/homebrew/Caskroom/miniforge/base/envs/metaflow_builder_4d09c70acb8c9c8e3165254752b96c84dd309431_e7415af007f806dd174c49c417ef844272454ab4
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:275]: Conda call: ['/opt/homebrew/bin/mamba', 'info', '--json']
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:218]: Going to fetch sources for lightgbm-4.3.0, thrift-0.16.0
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1508]: lightgbm-4.3.0 -> preferred .tar.gz @ local
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1508]: thrift-0.16.0 -> preferred .tar.gz @ local
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:332]: packages/pypi/files.pythonhosted.org/packages/74/d1/2e4b02e4611ab36647639c4eea8c4520bb90f948563e00a3bec583a9f9f5/lightgbm-4.3.0.tar.gz: building from '/opt/homebrew/Caskroom/miniforge/base/pkgs/pypi/pypi/lightgbm-4.3.0.tar.gz' in '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpoyb_lukl/build_0'
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:332]: packages/pypi/files.pythonhosted.org/packages/e4/23/dd951c9883cb49a73b750bdfe91e39d78e8a3f1f7175608634f381a197d5/thrift-0.16.0.tar.gz: building from '/opt/homebrew/Caskroom/miniforge/base/pkgs/pypi/pypi/thrift-0.16.0.tar.gz' in '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpoyb_lukl/build_1'
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow_builder_4d09c70acb8c9c8e3165254752b96c84dd309431_e7415af007f806dd174c49c417ef844272454ab4/bin/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--progress-bar', 'off', '-w', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpoyb_lukl/build_1', '/opt/homebrew/Caskroom/miniforge/base/pkgs/pypi/pypi/thrift-0.16.0.tar.gz']
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:328]: Binary call: ['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow_builder_4d09c70acb8c9c8e3165254752b96c84dd309431_e7415af007f806dd174c49c417ef844272454ab4/bin/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--progress-bar', 'off', '-w', '/var/folders/0d/0pmk6g2908x9q2x30n2sxqs80000gn/T/tmpoyb_lukl/build_0', '/opt/homebrew/Caskroom/miniforge/base/pkgs/pypi/pypi/lightgbm-4.3.0.tar.gz']
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:303]: Package for 'packages/pypi/files.pythonhosted.org/packages/e4/23/dd951c9883cb49a73b750bdfe91e39d78e8a3f1f7175608634f381a197d5/thrift-0.16.0.tar.gz' built in '/opt/homebrew/Caskroom/miniforge/base/pkgs/pypi/thrift-0.16.0-cp310-cp310-macosx_11_0_arm64.whl'
Copy code
debug[conda /Users/altbank647/Library/Caches/pypoetry/virtualenvs/model-training-d200h8ks-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/pypi_package_builder.py:266]: Package 'packages/pypi/files.pythonhosted.org/packages/74/d1/2e4b02e4611ab36647639c4eea8c4520bb90f948563e00a3bec583a9f9f5/lightgbm-4.3.0.tar.gz' was built with frozenset({<py3-none-macosx_14_0_arm64 @ 4390018944>}) but target architecture supports [<cp310-cp310-macosx_11_0_arm64 @ 4377262976>, <cp310-cp310-macosx_11_0_universal2 @ 4377295360>, <cp310-cp310-macosx_10_16_universal2 @ 4377295488>, <cp310-cp310-macosx_10_15_universal2 @ 4377295680>, <cp310-cp310-macosx_10_14_universal2 @ 4377295872>, <cp310-cp310-macosx_10_13_universal2 @ 4377296064>, <cp310-cp310-macosx_10_12_universal2 @ 4377296256>, <cp310-cp310-macosx_10_11_universal2 @ 4377296448>, <cp310-cp310-macosx_10_10_universal2 @ 4377296640>, <cp310-cp310-macosx_10_9_universal2 @ 4375430720>, <cp310-cp310-macosx_10_8_universal2 @ 4377362560>, <cp310-cp310-macosx_10_7_universal2 @ 4377362752>, <cp310-cp310-macosx_10_6_universal2 @ 4377362944>, <cp310-cp310-macosx_10_5_universal2 @ 4377363136>, <cp310-cp310-macosx_10_4_universal2 @ 4377363328>, <cp310-abi3-macosx_11_0_arm64 @ 4377363520>, <cp310-abi3-macosx_11_0_universal2 @ 4377363712>, <cp310-abi3-macosx_10_16_universal2 @ 4377363904>, <cp310-abi3-macosx_10_15_universal2 @ 4377364096>, <cp310-abi3-macosx_10_14_universal2 @ 4377364288>, <cp310-abi3-macosx_10_13_universal2 @ 4377364480>, <cp310-abi3-macosx_10_12_universal2 @ 4377364672>, <cp310-abi3-macosx_10_11_universal2 @ 4377364864>, <cp310-abi3-macosx_10_10_universal2 @ 4377365056>, <cp310-abi3-macosx_10_9_universal2 @ 4377365248>, <cp310-abi3-macosx_10_8_universal2 @ 4377365440>, <cp310-abi3-macosx_10_7_universal2 @ 4377365632>, <cp310-abi3-macosx_10_6_universal2 @ 4377365824>, <cp310-abi3-macosx_10_5_universal2 @ 4377366016>, <cp310-abi3-macosx_10_4_universal2 @ 4377366208>, <cp310-none-macosx_11_0_arm64 @ 4377366400>, <cp310-none-macosx_11_0_universal2 @ 4377366592>, <cp310-none-macosx_10_16_universal2 @ 4377366784>, <cp310-none-macosx_10_15_universal2 @ 4377366976>, <cp310-none-macosx_10_14_universal2 @ 4377367168>, <cp310-none-macosx_10_13_universal2 @ 4377367360>, <cp310-none-macosx_10_12_universal2 @ 4377367552>, <cp310-none-macosx_10_11_universal2 @ 4377367744>, <cp310-none-macosx_10_10_universal2 @ 4377367936>, <cp310-none-macosx_10_9_universal2 @ 4377368128>, <cp310-none-macosx_10_8_universal2 @ 4377368320>, <cp310-none-macosx_10_7_universal2 @ 4377368512>, <cp310-none-macosx_10_6_universal2 @ 4377368704>, <cp310-none-macosx_10_5_universal2 @ 4377368896>, <cp310-none-macosx_10_4_universal2 @ 4377369088>, <cp39-abi3-macosx_11_0_arm64 @ 4377262144>, <cp39-abi3-macosx_11_0_universal2 @ 4377369280>, <cp39-abi3-macosx_10_16_universal2 @ 4377369408>, <cp39-abi3-macosx_10_15_universal2 @ 4377369600>, <cp39-abi3-macosx_10_14_universal2 @ 4377369792>, <cp39-abi3-macosx_10_13_universal2 @ 4377369984>, <cp39-abi3-macosx_10_12_universal2 @ 4377370176>, <cp39-abi3-macosx_10_11_universal2 @ 4377370368>, <cp39-abi3-macosx_10_10_universal2 @ 4377370560>, <cp39-abi3-macosx_10_9_universal2 @ 4377370752>, <cp39-abi3-macosx_10_8_universal2 @ 4377370944>, <cp39-abi3-macosx_10_7_universal2 @ 4377371136>, <cp39-abi3-macosx_10_6_universal2 @ 4377371328>, <cp39-abi3-macosx_10_5_universal2 @ 4377371520>, <cp39-abi3-macosx_10_4_universal2 @ 4377371712>, <cp38-abi3-macosx_11_0_arm64 @ 4377371904>, <cp38-abi3-macosx_11_0_universal2 @ 4377372096>, <cp38-abi3-macosx_10_16_universal2 @ 4377372288>, <cp38-abi3-macosx_10_15_universal2 @ 4377372480>, <cp38-abi3-macosx_10_14_universal2 @ 4377372672>, <cp38-abi3-macosx_10_13_universal2 @ 4377372864>, <cp38-abi3-macosx_10_12_universal2 @ 4377373056>, <cp38-abi3-macosx_10_11_universal2 @ 4377373248>, <cp38-abi3-macosx_10_10_universal2 @ 4377373440>, <cp38-abi3-macosx_10_9_universal2 @ 4377373632>, <cp38-abi3-macosx_10_8_universal2 @ 4377373824>, <cp38-abi3-macosx_10_7_universal2 @ 4377374016>, <cp38-abi3-macosx_10_6_universal2 @ 4377374208>, <cp38-abi3-macosx_10_5_universal2 @ 4377374400>, <cp38-abi3-macosx_10_4_universal2 @ 4377374592>, <cp37-abi3-macosx_11_0_arm64 @ 4377374784>, <cp37-abi3-macosx_11_0_universal2 @ 4377374976>, <cp37-abi3-macosx_10_16_universal2 @ 4377375168>, <cp37-abi3-macosx_10_15_universal2 @ 4377375360>, <cp37-abi3-macosx_10_14_universal2 @ 4377375552>, <cp37-abi3-macosx_10_13_universal2 @ 4377375744>, <cp37-abi3-macosx_10_12_universal2 @ 4377375936>, <cp37-abi3-macosx_10_11_universal2 @ 4377376128>, <cp37-abi3-macosx_10_10_universal2 @ 4377376320>, <cp37-abi3-macosx_10_9_universal2 @ 4377376512>, <cp37-abi3-macosx_10_8_universal2 @ 4377376704>, <cp37-abi3-macosx_10_7_universal2 @ 4377376896>, <cp37-abi3-macosx_10_6_universal2 @ 4377377088>, <cp37-abi3-macosx_10_5_universal2 @ 4377377280>, <cp37-abi3-macosx_10_4_universal2 @ 4377377472>, <cp36-abi3-macosx_11_0_arm64 @ 4377377664>, <cp36-abi3-macosx_11_0_universal2 @ 4377377856>, <cp36-abi3-macosx_10_16_universal2 @ 4377378048>, <cp36-abi3-macosx_10_15_universal2 @ 4377378240>, <cp36-abi3-macosx_10_14_universal2 @ 4377378432>, <cp36-abi3-macosx_10_13_universal2 @ 4377378624>, <cp36-abi3-macosx_10_12_universal2 @ 4377378880>, <cp36-abi3-macosx_10_11_universal2 @ 4377379072>, <cp36-abi3-macosx_10_10_universal2 @ 4377379264>, <cp36-abi3-macosx_10_9_universal2 @ 4377379456>, <cp36-abi3-macosx_10_8_universal2 @ 4377379648>, <cp36-abi3-macosx_10_7_universal2 @ 4377379840>, <cp36-abi3-macosx_10_6_universal2 @ 4377380032>, <cp36-abi3-macosx_10_5_universal2 @ 4377380224>, <cp36-abi3-macosx_10_4_universal2 @ 4377380416>, <cp35-abi3-macosx_11_0_arm64 @ 4377380608>, <cp35-abi3-macosx_11_0_universal2 @ 4377380800>, <cp35-abi3-macosx_10_16_universal2 @ 4377380992>, <cp35-abi3-macosx_10_15_universal2 @ 4377381184>, <cp35-abi3-macosx_10_14_universal2 @ 4377381376>, <cp35-abi3-macosx_10_13_universal2 @ 4377381568>, <cp35-abi3-macosx_10_12_universal2 @ 4377381760>, <cp35-abi3-macosx_10_11_universal2 @ 4377381952>, <cp35-abi3-macosx_10_10_universal2 @ 4377382144>, <cp35-abi3-macosx_10_9_universal2 @ 4377382336>, <cp35-abi3-macosx_10_8_universal2 @ 4377382528>, <cp35-abi3-macosx_10_7_universal2 @ 4377382720>, <cp35-abi3-macosx_10_6_universal2 @ 4377382912>, <cp35-abi3-macosx_10_5_universal2 @ 4377383104>, <cp35-abi3-macosx_10_4_universal2 @ 4377383296>, <cp34-abi3-macosx_11_0_arm64 @ 4377383488>, <cp34-abi3-macosx_11_0_universal2 @ 4377383680>, <cp34-abi3-macosx_10_16_universal2 @ 4377383872>, <cp34-abi3-macosx_10_15_universal2 @ 4377384064>, <cp34-abi3-macosx_10_14_universal2 @ 4377384256>, <cp34-abi3-macosx_10_13_universal2 @ 4377384448>, <cp34-abi3-macosx_10_12_universal2 @ 4377384640>, <cp34-abi3-macosx_10_11_universal2 @ 4377384832>, <cp34-abi3-macosx_10_10_universal2 @ 4377385024>, <cp34-abi3-macosx_10_9_universal2 @ 4377385216>, <cp34-abi3-macosx_10_8_universal2 @ 4377385408>, <cp34-abi3-macosx_10_7_universal2 @ 4377385600>, <cp34-abi3-macosx_10_6_universal2 @ 4377385792>, <cp34-abi3-macosx_10_5_universal2 @ 4377385984>, <cp34-abi3-macosx_10_4_universal2 @ 4377386176>, <cp33-abi3-macosx_11_0_arm64 @ 4377386368>, <cp33-abi3-macosx_11_0_universal2 @ 4377386560>, <cp33-abi3-macosx_10_16_universal2 @ 4377386752>, <cp33-abi3-macosx_10_15_universal2 @ 4377386944>, <cp33-abi3-macosx_10_14_universal2 @ 4377387136>, <cp33-abi3-macosx_10_13_universal2 @ 4377387328>, <cp33-abi3-macosx_10_12_universal2 @ 4377387520>, <cp33-abi3-macosx_10_11_universal2 @ 4377387712>, <cp33-abi3-macosx_10_10_universal2 @ 4377387904>, <cp33-abi3-macosx_10_9_universal2 @ 4377388096>, <cp33-abi3-macosx_10_8_universal2 @ 4377388288>, <cp33-abi3-macosx_10_7_universal2 @ 4377388480>, <cp33-abi3-macosx_10_6_universal2 @ 4377388672>, <cp33-abi3-macosx_10_5_universal2 @ 4377388864>, <cp33-abi3-macosx_10_4_universal2 @ 4377389056>, <cp32-abi3-macosx_11_0_arm64 @ 4377389248>, <cp32-abi3-macosx_11_0_universal2 @ 4377389440>, <cp32-abi3-macosx_10_16_universal2 @ 4377389632>, <cp32-abi3-macosx_10_15_universal2 @ 4377389824>, <cp32-abi3-macosx_10_14_universal2 @ 4377390016>, <cp32-abi3-macosx_10_13_universal2 @ 4377390208>, <cp32-abi3-macosx_10_12_universal2 @ 4377390400>, <cp32-abi3-macosx_10_11_universal2 @ 4377390592>, <cp32-abi3-macosx_10_10_universal2 @ 4377390784>, <cp32-abi3-macosx_10_9_universal2 @ 4377390976>, <cp32-abi3-macosx_10_8_universal2 @ 4377391168>, <cp32-abi3-macosx_10_7_universal2 @ 4377391360>, <cp32-abi3-macosx_10_6_universal2 @ 4377391552>, <cp32-abi3-macosx_10_5_universal2 @ 4377391744>, <cp32-abi3-macosx_10_4_universal2 @ 4377391936>, <cp310-none-any @ 4377392320>, <py310-none-any @ 4377392448>, <py3-none-any @ 4377392704>, <py39-none-any @ 4377393024>, <py38-none-any @ 4377393216>, <py37-none-any @ 4377393472>, <py36-none-any @ 4377393728>, <py35-none-any @ 4377393984>, <py34-none-any @ 4377394240>, <py33-none-any @ 4377394496>, <py32-none-any @ 4377394752>, <py31-none-any @ 4377395008>, <py30-none-any @ 4377428096>]
    Conda ran into an error while setting up environment.:
    Some built pypi packages are not compatible with the target architecture. This can happen when some dependencies do not have an available wheel for the target architecture and you are resolving an environment for a different architecture than the one you are on. Requirements would have us build: packages/pypi/files.pythonhosted.org/packages/74/d1/2e4b02e4611ab36647639c4eea8c4520bb90f948563e00a3bec583a9f9f5/lightgbm-4.3.0.tar.gz
d
ok perfect. yep this helps and makes sense. Let me see if I can repro and fix. The TL;DR is that when the wheel is built, it says it is built for 
py3-none-macosx_14_0_arm64
and I don’t have that in the list of supported architecture. This ia probably a bug on my side since that is supported. I probably have to extend the supported tag list a bit for this use case.
Quick update, this is the code that generates the list of supported architectures: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/utils.py#L228. It’s very weird that it has the cp* ones with the macosx stuff but not the py3 thing since the “platforms” portion should be the same for both calls. I’ll dig a little more but that’s where the issue is coming from.
r
Ok, thank you very much for the help. I was able to resolve the environment by using a linux-64 devcontainer, it's not the ideal solution but it woks fine. I've given up on macos-arm for now, too much trouble.
d
Cool — I am going to look at it though. It should work for sure. In terms of using a devcontainer, yes, that was going to be my proposal (like here: https://netflix.slack.com/archives/C02116BBNTU/p1712816625810499?thread_ts=1712008712.280789&amp;cid=C02116BBNTU).
👍 1
basically you can run a flow that has a step that resolves the environment.
17 Views
Open in Slack
PreviousNext
Powered by