Hi all, I'm using the new and improved

@conda

decorator and metaflow

2.10.2

, but having some problems installing the external dependencies. Attempting this:

@conda_base(
    libraries={
        "pytorch::pytorch": "1.12.1",
        "pytorch::torchvision": "0.13.0",
        "conda-forge::matplotlib": "3.5.3",
        "conda-forge::sentencepiece": "0.1.97",
        "conda-forge::pandas": "1.5.3",
        "conda-forge::pytorch-lightning": "1.9.4",
        "conda-forge::scikit-learn": "1.2.2",
        "conda-forge::deepspeed": "0.9.2"
    },
    python="3.10.1",
)

What is the error that you are running into?

nothing provides __cuda needed by pytorch-1.12.0-cuda102py310hdf4a2db_200

I have mamba installed. It's using Sagemaker Studio, which is using a custom docker image. The docker image is Amazon Linux 2.

Thanks! And what is the CUDA version that you are on?

Oh! CUDA isn't installed in the docker image.

Can you add another package as the dependency here - __cuda with say 11.8 as the version?

you should e able to use CONDA_OVERRIDE_CUDA maybe?

That would work as well

@ancient-application-36103, Getting this error now:

nothing provides requested __cuda 11.8
    nothing provides __cuda needed by pytorch-1.12.0-cuda102py310hdf4a2db_200

try with CONDA_CHANNEL_PRIORITY=flexible as well?

Thanks Romain, Unfortunately still get an error:

nothing provides cuda-cudart >=11.7,<11.8 needed by pytorch-cuda-11.7-h778d358_3

Have you set the default conda channel to conda-forge? cudart is available in https://prefix.dev/channels/conda-forge

I will try that. I ended up switching to

@pypi

which seemed to work.

Sorry, was busy with something else. Anyways, to answer your questions: • Bleeding edge decorators: https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators • The issue seems to be that torchvision 0.13.0 requires pytorch 1.12.0 and not 1.12.1. You can use torchvision 0.13.1 and it will work (at least with the bleeding edge decorators.

CONDA_OVERRIDE_CUDA=11.8 metaflow environment resolve --dry-run -f riley.yml --python 3.10.1

gives you:

Environment of type conda-only full hash 2a810119796e7d4f87820710fcac9433ba2c9bd3:2366432560451c0f61fbc061a8d2a3f4daab4f68
Arch linux-64
Available on linux-64

Resolved on 2023-10-13 21:43:12.847752
Resolved by rcledat

User-requested packages sys::__cuda==11.8=0, conda::boto3==>=1.14.0, conda::cffi==>=1.13.0,!=1.15.0, conda::conda-forge::deepspeed==0.9.2, conda::conda-forge::matplotlib==3.5.3, conda::conda-forge::pandas==1.5.3, conda::conda-forge::pytorch-lightning==1.9.4, conda::conda-forge::scikit-learn==1.2.2, conda::conda-forge::sentencepiece==0.1.97, conda::fastavro==>=1.6.0, conda::pandas==>=0.24.0, conda::pyarrow==>=0.17.1, conda::python==3.10.1, conda::pytorch::pytorch==1.12.1, conda::pytorch::torchvision==0.13.1, conda::requests==>=2.21.0
User sources conda::conda-forge

Conda Packages installed _libgcc_mutex==0.1-conda_forge, _openmp_mutex==4.5-2_kmp_llvm, alsa-lib==1.2.8-h166bdaf_0, annotated-types==0.6.0-pyhd8ed1ab_0, aom==3.5.0-h27087fc_0, attr==2.5.1-h166bdaf_1, aws-c-auth==0.7.0-hf8751d9_2, aws-c-cal==0.6.0-h93469e0_0, aws-c-common==0.8.23-hd590300_0, aws-c-compression==0.2.17-h862ab75_1, aws-c-event-stream==0.3.1-h9599702_1, aws-c-http==0.7.11-hbe98c3e_0, aws-c-io==0.13.28-h3870b5a_0, aws-c-mqtt==0.8.14-h2e270ba_2, aws-c-s3==0.3.13-heb0bb06_2, aws-c-sdkutils==0.1.11-h862ab75_1, aws-checksums==0.1.16-h862ab75_1, aws-crt-cpp==0.20.3-he9c0e7f_4, aws-sdk-cpp==1.10.57-hbc2ea52_17, blas==2.116-mkl, blas-devel==3.9.0-16_linux64_mkl, boto3==1.28.63-pyhd8ed1ab_0, botocore==1.31.63-pyhd8ed1ab_0, brotli==1.0.9-h166bdaf_9, brotli-bin==1.0.9-h166bdaf_9, brotli-python==1.0.9-py310hd8f1fbe_9, bzip2==1.0.8-h7f98852_4, c-ares==1.20.1-hd590300_0, ca-certificates==2023.7.22-hbcca054_0, cairo==1.16.0-ha61ee94_1014, certifi==2023.7.22-pyhd8ed1ab_0, cffi==1.16.0-py310h2fee648_0, charset-normalizer==3.3.0-pyhd8ed1ab_0, colorama==0.4.6-pyhd8ed1ab_0, cudatoolkit==11.6.2-hfc3e2af_12, cycler==0.12.1-pyhd8ed1ab_0, dbus==1.13.6-h5008d03_3, deepspeed==0.9.2-cpu_py310h11dbdba_0, expat==2.5.0-hcb278e6_1, fastavro==1.8.4-py310h2372a71_0, ffmpeg==5.1.2-gpl_h8dda1f0_106, fftw==3.3.10-nompi_hc118613_108, font-ttf-dejavu-sans-mono==2.37-hab24e00_0, font-ttf-inconsolata==3.000-h77eed37_0, font-ttf-source-code-pro==2.038-h77eed37_0, font-ttf-ubuntu==0.83-hab24e00_0, fontconfig==2.14.2-h14ed4e7_0, fonts-conda-ecosystem==1-0, fonts-conda-forge==1-0, fonttools==4.43.1-py310h2372a71_0, freetype==2.12.1-h267a509_2, fsspec==2023.9.2-pyh1a96a4e_0, gettext==0.21.1-h27087fc_0, gflags==2.2.2-he1b5a44_1004, glib==2.78.0-hfc55251_0, glib-tools==2.78.0-hfc55251_0, glog==0.6.0-h6f12383_0, gmp==6.2.1-h58526e2_0, gnutls==3.7.8-hf3e180e_0, graphite2==1.3.13-h58526e2_1001, gst-plugins-base==1.22.0-h4243ec0_2, gstreamer==1.22.0-h25f0c4b_2, gstreamer-orc==0.4.34-hd590300_0, harfbuzz==6.0.0-h8e241bc_0, hjson-py==3.1.0-pyhd8ed1ab_0, icu==70.1-h27087fc_0, idna==3.4-pyhd8ed1ab_0, jack==1.9.22-h11f4161_0, jmespath==1.0.1-pyhd8ed1ab_0, joblib==1.3.2-pyhd8ed1ab_0, jpeg==9e-h0b41bf4_3, keyutils==1.6.1-h166bdaf_0, kiwisolver==1.4.5-py310hd41b1e2_1, krb5==1.20.1-h81ceb04_0, lame==3.100-h166bdaf_1003, lcms2==2.15-hfd0df8a_0, ld_impl_linux-64==2.40-h41732ed_0, lerc==4.0.0-h27087fc_0, libabseil==20230125.3-cxx17_h59595ed_0, libaio==0.3.113-h166bdaf_0, libarrow==12.0.1-hd2d78f0_7_cpu, libblas==3.9.0-16_linux64_mkl, libbrotlicommon==1.0.9-h166bdaf_9, libbrotlidec==1.0.9-h166bdaf_9, libbrotlienc==1.0.9-h166bdaf_9, libcap==2.67-he9d0100_0, libcblas==3.9.0-16_linux64_mkl, libclang==15.0.7-default_h7634d5b_3, libclang13==15.0.7-default_h9986a30_3, libcrc32c==1.1.2-h9c3ff4c_0, libcups==2.3.3-h36d4200_3, libcurl==8.1.2-h409715c_0, libdb==6.2.32-h9c3ff4c_0, libdeflate==1.17-h0b41bf4_0, libdrm==2.4.114-h166bdaf_0, libedit==3.1.20191231-he28a2e2_2, libev==4.33-h516909a_1, libevent==2.1.10-h28343ad_4, libexpat==2.5.0-hcb278e6_1, libffi==3.4.2-h7f98852_5, libflac==1.4.3-h59595ed_0, libgcc-ng==13.2.0-h807b86a_2, libgcrypt==1.10.1-h166bdaf_0, libgfortran-ng==13.2.0-h69a702a_2, libgfortran5==13.2.0-ha4646dd_2, libglib==2.78.0-hebfc3b9_0, libgoogle-cloud==2.12.0-hac9eb74_1, libgpg-error==1.47-h71f35ed_0, libgrpc==1.54.3-hb20ce57_0, libhwloc==2.9.1-hd6dc26d_0, libiconv==1.17-h166bdaf_0, libidn2==2.3.4-h166bdaf_0, liblapack==3.9.0-16_linux64_mkl, liblapacke==3.9.0-16_linux64_mkl, libllvm15==15.0.7-hadd5161_1, libnghttp2==1.52.0-h61bc06f_0, libnsl==2.0.0-hd590300_1, libnuma==2.0.16-h0b41bf4_1, libogg==1.3.4-h7f98852_1, libopus==1.3.1-h7f98852_1, libpciaccess==0.17-h166bdaf_0, libpng==1.6.39-h753d276_0, libpq==15.3-hbcd7760_1, libprotobuf==3.21.12-hfc55251_2, libsentencepiece==0.1.97-h180e1df_2, libsndfile==1.2.2-hc60ed4a_1, libsqlite==3.43.2-h2797004_0, libssh2==1.11.0-h0841786_0, libstdcxx-ng==13.2.0-h7e041cc_2, libsystemd0==253-h8c4010b_1, libtasn1==4.19.0-h166bdaf_0, libthrift==0.18.1-h5e4af38_0, libtiff==4.5.0-h6adf6a1_2, libtool==2.4.7-h27087fc_0, libudev1==253-h0b41bf4_1, libunistring==0.9.10-h7f98852_0, libutf8proc==2.8.0-h166bdaf_0, libuuid==2.38.1-h0b41bf4_0, libva==2.18.0-h0b41bf4_0, libvorbis==1.3.7-h9c3ff4c_0, libvpx==1.11.0-h9c3ff4c_3, libwebp-base==1.3.2-hd590300_0, libxcb==1.13-h7f98852_1004, libxkbcommon==1.5.0-h79f4944_1, libxml2==2.10.3-hca2bb57_4, libzlib==1.2.13-hd590300_5, lightning-utilities==0.9.0-pyhd8ed1ab_0, llvm-openmp==17.0.2-h4dfa4b3_0, lz4-c==1.9.4-hcb278e6_0, matplotlib==3.5.3-py310hff52083_2, matplotlib-base==3.5.3-py310h8d5ebf3_2, mkl==2022.1.0-h84fe81f_915, mkl-devel==2022.1.0-ha770c72_916, mkl-include==2022.1.0-h84fe81f_915, mpg123==1.32.3-h59595ed_0, munkres==1.1.4-pyh9f0ad1d_0, mysql-common==8.0.33-hf1915f5_5, mysql-libs==8.0.33-hca2cd23_5, ncurses==6.4-hcb278e6_0, nettle==3.8.1-hc379101_1, nspr==4.35-h27087fc_0, nss==3.94-h1d7d5a4_0, numpy==1.26.0-py310hb13e2d6_0, openh264==2.3.1-hcb278e6_2, openjpeg==2.5.0-hfec8fc6_2, openssl==3.1.3-hd590300_0, orc==1.9.0-h2f23424_1, p11-kit==0.24.1-hc5aa10d_0, packaging==23.2-pyhd8ed1ab_0, pandas==1.5.3-py310h9b08913_1, pcre2==10.40-hc3806b6_0, pillow==9.4.0-py310h023d228_1, pip==23.2.1-pyhd8ed1ab_0, pixman==0.42.2-h59595ed_0, ply==3.11-py_1, psutil==5.9.5-py310h2372a71_1, pthread-stubs==0.4-h36c2ea0_1001, pulseaudio==16.1-hcb278e6_3, pulseaudio-client==16.1-h5195f5e_3, pulseaudio-daemon==16.1-ha8d29e2_3, py-cpuinfo==9.0.0-pyhd8ed1ab_0, pyarrow==12.0.1-py310h0576679_7_cpu, pycparser==2.21-pyhd8ed1ab_0, pydantic==2.4.2-pyhd8ed1ab_0, pydantic-core==2.10.1-py310hcb5633a_0, pyparsing==3.1.1-pyhd8ed1ab_0, pyqt==5.15.9-py310h04931ad_5, pyqt5-sip==12.12.2-py310hc6cd4ac_5, pysocks==1.7.1-pyha2e5f31_6, python==3.10.1-h543edf9_2_cpython, python-dateutil==2.8.2-pyhd8ed1ab_0, python_abi==3.10-4_cp310, pytorch==1.12.1-py3.10_cuda11.6_cudnn8.3.2_0, pytorch-lightning==1.9.4-pyhd8ed1ab_1, pytorch-mutex==1.0-cuda, pytz==2023.3.post1-pyhd8ed1ab_0, pyyaml==6.0.1-py310h2372a71_1, qt-main==5.15.8-h5d23da1_6, rdma-core==28.9-h59595ed_1, re2==2023.03.02-h8c504da_0, readline==8.2-h8228510_1, requests==2.31.0-pyhd8ed1ab_0, s2n==1.3.46-h06160fa_0, s3transfer==0.7.0-pyhd8ed1ab_0, scikit-learn==1.2.2-py310hf7d194e_2, scipy==1.11.3-py310hb13e2d6_1, sentencepiece==0.1.97-hff52083_2, sentencepiece-python==0.1.97-py310hebdb9f0_2, sentencepiece-spm==0.1.97-h180e1df_2, setuptools==68.2.2-pyhd8ed1ab_0, sip==6.7.11-py310hc6cd4ac_1, six==1.16.0-pyh6c4a22f_0, snappy==1.1.10-h9fff704_0, sqlite==3.43.2-h2c6b66d_0, svt-av1==1.4.1-hcb278e6_0, tbb==2021.9.0-hf52228f_0, threadpoolctl==3.2.0-pyha21a80b_0, tk==8.6.13-h2797004_0, toml==0.10.2-pyhd8ed1ab_0, tomli==2.0.1-pyhd8ed1ab_0, torchmetrics==1.2.0-pyhd8ed1ab_0, torchvision==0.13.1-py310_cu116, tornado==6.3.3-py310h2372a71_1, tqdm==4.66.1-pyhd8ed1ab_0, typing-extensions==4.8.0-hd8ed1ab_0, typing_extensions==4.8.0-pyha770c72_0, tzdata==2023c-h71feb2d_0, ucx==1.14.1-h64cca9d_5, unicodedata2==15.1.0-py310h2372a71_0, urllib3==1.26.17-pyhd8ed1ab_0, wheel==0.41.2-pyhd8ed1ab_0, x264==1!164.3095-h166bdaf_2, x265==3.5-h924138e_3, xcb-util==0.4.0-h516909a_0, xcb-util-image==0.4.0-h166bdaf_0, xcb-util-keysyms==0.4.0-h516909a_0, xcb-util-renderutil==0.3.9-h166bdaf_0, xcb-util-wm==0.4.1-h516909a_0, xkeyboard-config==2.38-h0b41bf4_0, xorg-fixesproto==5.0-h7f98852_1002, xorg-kbproto==1.0.7-h7f98852_1002, xorg-libice==1.1.1-hd590300_0, xorg-libsm==1.2.4-h7391055_0, xorg-libx11==1.8.4-h0b41bf4_0, xorg-libxau==1.0.11-hd590300_0, xorg-libxdmcp==1.1.3-h7f98852_0, xorg-libxext==1.3.4-h0b41bf4_2, xorg-libxfixes==5.0.3-h7f98852_1004, xorg-libxrender==0.9.10-h7f98852_1003, xorg-renderproto==0.11.1-h7f98852_1002, xorg-xextproto==7.3.0-h0b41bf4_1003, xorg-xproto==7.0.31-h7f98852_1007, xz==5.2.6-h166bdaf_0, yaml==0.2.5-h7f98852_2, zlib==1.2.13-hd590300_5, zstd==1.5.5-hfc55251_0

Dry-run -- not caching or aliasing

Very helpful! Thanks Romain. Will definitely give this a try!

(not sure what the error message was about for your resolution, what led me to this was the more helpful:

The following packages are incompatible
├─ pytorch 1.12.1  is installable with the potential options
│  ├─ pytorch 1.12.1 would require
│  │  └─ pytorch-mutex 1.0 cpu, which can be installed;
│  └─ pytorch 1.12.1 would require
│     └─ pytorch-mutex 1.0 cuda, which conflicts with any installable versions previously reported;
└─ torchvision 0.13.0  is not installable because it requires
   ├─ pytorch 1.12.0  but there are no viable options
   │  ├─ pytorch 1.12.0 conflicts with any installable versions previously reported;
   │  ├─ pytorch 1.12.0 would require
   │  │  └─ pytorch-mutex 1.0 cpu, which can be installed;
   │  └─ pytorch 1.12.0 would require
   │     └─ pytorch-mutex 1.0 cuda, which conflicts with any installable versions previously reported;
   └─ pytorch-mutex [1.0 cpu|1.0 cuda] with the potential options
      ├─ pytorch-mutex 1.0 conflicts with any installable versions previously reported;
      └─ pytorch-mutex 1.0, which can be installed.

Hi Romain, Thanks for checking in and thanks so much for your help! Yes I did. Seems to be working. I didn't try the bleeding edge decorator though. Just the regular

@conda

decorator and the CONDA_OVERRIDE_CUDA was sufficient.