https://outerbounds.com/ logo
Join Slack
Powered by
Has anyone else run into this issue with `@pypi` `...
# ask-metaflow
c
Has anyone else run into this issue with 
@pypi
Copy code
"torch": "2.2.0",
EXTRA_INDICES = <https://download.pytorch.org/whl/cu121>

[pod t-fe04d0d8-ql8xq-dps2v] OSError: libcudart.so.12: cannot open shared object file: No such file or directory
I've seen it once before but the fix I had last time doesn't seem to work. https://outerbounds-community.slack.com/archives/C02116BBNTU/p1713984982732359
✅ 1
d
could you maybe print out the LD_LIBRARY_PATH that is loaded to see. It should be set properly but not sure what may be going on. It’s also possible that it would work better if you moved some of the deps to the conda decorator. What are your current deps?
c
Copy code
"git+<https://github.com/mbr/asciitree.git@0.3.3#egg=asciitree>": "",
    "git+<https://github.com/Bayer-Group/tiffslide.git@v2.3.1#egg=tiffslide>": "",
    "git+<https://github.com/valarlabs/xarray.git@main#egg=xarray>": "",
    "zarr": "2.16.1",
    "attrs": "23.2.0",
    "tqdm": "4.66.1",
    "numpy": "1.26.3",
    "pandas": "2.2.0",
    "geopandas": "0.14.3",
    "s3fs": "2023.12.2",
    "typing_extensions": "4.9.0",
    "rich": "13.7.0",
    "matplotlib": "3.8.2",
    "plotly": "5.18.0",
    "dask": "2023.3.2",
    "opencv-python-headless": "4.8.0.74",
    "torch": "2.2.0",
    "torchvision": "",
    "torchmetrics": "",
    "rasterio": "1.3.9",
    "git+<https://github.com/nalepae/pandarallel.git@v1.6.5#egg=pandarallel>": "",
    "omegaconf": "",
    "python-dotenv": "1.0.1",
    "timm": "0.9.12",
    "albumentations": "1.3.1",
    "einops": "0.7.0",
    "numcodecs[msgpack]": "0.12.1",
    "pydantic": "2.6.1",
    "transformers": "4.37.2",
    "numba": "0.59.0",
    "ujson": "5.9.0",
    "pypeln": "",
    "pyjson5": "1.6.6",
d
ok, so let’s try to move some of the stuff out of the pypi decorator into the conda one, specifically all the torch stuff: list in the conda decorator
Copy code
pytorch == 2.2.0
torchvision
torchmetrics
pytorch-cuda == 12.1
and add pytorch and nvidia channels
c
I'll give it try. I'm using 
pypi_base
does that matter? Does the order of decorators matter?
d
order doesn’t matter. and 
pypi_base
is fine, you can use that and 
conda_base
among us party 1
c
Is there a way around this. (I think I saw in the README that this isn't supported in a combined conda pypi)
Copy code
metaflow._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    git+<https://github.com/valarlabs/xarray.git@main#egg=xarray>
d
gimme a few mins and I’ll try it out. I think there is a way. I should update the doc.
🙏 1
could you send me your two decorators whenever you get a chance.
c
Copy code
PYTHON_VERSION = "3.11.7"
PACKAGES = {
    "git+<https://github.com/mbr/asciitree.git@0.3.3#egg=asciitree>": "",
    "git+<https://github.com/Bayer-Group/tiffslide.git@v2.3.1#egg=tiffslide>": "",
    "git+<https://github.com/valarlabs/xarray.git@main#egg=xarray>": "",
    "zarr": "2.16.1",
    "attrs": "23.2.0",
    "tqdm": "4.66.1",
    "numpy": "1.26.3",
    "pandas": "2.2.0",
    "geopandas": "0.14.3",
    "s3fs": "2023.12.2",
    "typing_extensions": "4.9.0",
    "rich": "13.7.0",
    "matplotlib": "3.8.2",
    "plotly": "5.18.0",
    "dask": "2023.3.2",
    "opencv-python-headless": "4.8.0.74",
    # "torch": "2.2.0",
    # "torchvision": "",
    # "torchmetrics": "",
    # "pytorch_cuda": "12.1",
    "rasterio": "1.3.9",
    "git+<https://github.com/nalepae/pandarallel.git@v1.6.5#egg=pandarallel>": "",
    "omegaconf": "",
    "python-dotenv": "1.0.1",
    "timm": "0.9.12",
    "albumentations": "1.3.1",
    "einops": "0.7.0",
    "numcodecs[msgpack]": "0.12.1",
    "pydantic": "2.6.1",
    "transformers": "4.37.2",
    "numba": "0.59.0",
    "ujson": "5.9.0",
    "pypeln": "",
    "pyjson5": "1.6.6",
}

CONDA_LIBRARIES = {
    "pytorch::pytorch": "",
    "pytorch::torchvision": "",
    # "pytorch::torchmetrics": "",
    # "pytorch::pytorch-cuda": "12.1",
}

@conda_base(libraries=CONDA_LIBRARIES, python=PYTHON_VERSION)
@pypi_base(python=PYTHON_VERSION, packages=PACKAGES)
....
I'm on latest Metaflow 2.12.13+netflix-ext(1.2.2)
d
awesome. I am in a few meetings but will take a look shortly.
🙏 1
ok. I’ve tried a few things and it doesn’t work straight out. Let me try a few more things and see if I can get it to work. I am hopeful it can.
I’ll get back to you shortly.
c
Thank you 🙏
d
Alright, so here is the modified packages:
Copy code
PYTHON_VERSION = "3.11.7"
PACKAGES = {
    "asciitree @ git+<https://github.com/mbr/asciitree.git@0.3.3>": "",
    "tiffslide @ git+<https://github.com/Bayer-Group/tiffslide.git@v2.3.1>": "",
    "xarray @ git+<https://github.com/valarlabs/xarray.git@main>": "",
    "zarr": "2.16.1",
    "attrs": "23.2.0",
    "tqdm": "4.66.1",
    "numpy": "1.26.3",
    "pandas": "2.2.0",
    "geopandas": "0.14.3",
    "s3fs": "2023.12.2",
    "typing_extensions": "4.9.0",
    "rich": "13.7.0",
    "matplotlib": "3.8.2",
    "plotly": "5.18.0",
    "dask": "2023.3.2",
    "opencv-python-headless": "4.8.0.74",
    # "torch": "2.2.0",
    # "torchvision": "",
    # "torchmetrics": "",
    # "pytorch_cuda": "12.1",
    "rasterio": "1.3.9",
    "pandarallel @ git+<https://github.com/nalepae/pandarallel.git@v1.6.5>": "",
    "omegaconf": "",
    "python-dotenv": "1.0.1",
    "timm": "0.9.12",
    "albumentations": "1.3.1",
    "einops": "0.7.0",
    "numcodecs[msgpack]": "0.12.1",
    "pydantic": "2.6.1",
    "transformers": "4.37.2",
    "numba": "0.59.0",
    "ujson": "5.9.0",
    "pypeln": "",
    "pyjson5": "1.6.6",
}

CONDA_LIBRARIES = {
    "pytorch::pytorch": "",
    "pytorch::torchvision": "",
    "torchmetrics": "",
    "pytorch::pytorch-cuda": "12.1",
}

@conda_base(libraries=CONDA_LIBRARIES, channels=["nvidia", "pytorch"])
@pypi_base(python=PYTHON_VERSION, packages=PACKAGES)
you also need a version in a branch — let me know if it works for you.
I’ll add a test to incorporate that as well but it should at least work
not sure if it will solve that other problem though but it was a feature that was missing anyways 🙂
c
Good morning, I tried installing the update 
pip install git+<https://github.com/Netflix/metaflow-nflx-extensions.git@ef5229d0afb66348f2614780de331f09dfb165d7>
but I see this error.
Copy code
metaflow._vendor.packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    git+<https://github.com/Bayer-Group/tiffslide.git@v2.3.1#egg=tiffslide>
Did I install it incorrectly?
d
right — you need to edit those as I pasted above. 
"tiffslide @ git+<https://github.com/Bayer-Group/tiffslide.git@v2.3.1>": "",
(see my modified file with the packages and what not a few messages above)
c
Ah my apologies, missed that message 🤦
That works thank you so much 🙂
d
so fully solved this? As in, it ran?
c
Yes
Thank you
d
cool!!
I’ll merge this and release then.
9 Views
Open in Slack
PreviousNext
Powered by