https://outerbounds.com/ logo
Join Slack
Powered by
I'm trying to create an environment with `@pypi` `...
# ask-metaflow
c
I'm trying to create an environment with 
@pypi
Copy code
@pypi(
        python="3.11.7",
        packages={
            "tiffslide": "",
        }
)
This specific package 
tiffslide
no matter what gives me
Copy code
Pip ran into an error while setting up environment:
    command '/Users/mikebentleymills/.metaflowconfig/micromamba/bin/micromamba run --prefix /Users/mikebentleymills/micromamba/envs/metaflow/osx-arm64/bd951bc278e2ed8 pip3 --disable-pip-version-check --no-input --no-color --isolated install --dry-run --only-binary=:all: --upgrade-strategy=only-if-needed --target=/var/folders/kw/3xtlvj6120vg50rfhvylbzp00000gn/T/tmpaz0830ur --report=/var/folders/kw/3xtlvj6120vg50rfhvylbzp00000gn/T/tmpaz0830ur/report.json --progress-bar=off --quiet --abi cp311 --abi none --abi abi3 --platform macosx_10_8_universal2 --platform macosx_11_0_arm64 --platform macosx_14_0_universal2 --platform macosx_14_0_arm64 --platform macosx_10_10_universal2 --platform macosx_10_15_universal2 --platform macosx_12_0_universal2 --platform macosx_13_0_universal2 --platform macosx_10_9_universal2 --platform macosx_10_6_universal2 --platform macosx_10_12_universal2 --platform macosx_11_0_universal2 --platform macosx_10_11_universal2 --platform macosx_13_0_arm64 --platform macosx_10_4_universal2 --platform macosx_10_14_universal2 --platform macosx_10_13_universal2 --platform macosx_10_5_universal2 --platform macosx_12_0_arm64 --platform macosx_10_7_universal2 --platform macosx_10_16_universal2 --platform any requests>=2.21.0 boto3>=1.14.0 tiffslide' returned error (1)
    ERROR: Cannot install tiffslide because these package versions have conflicting dependencies.
    ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
It doesn't appear to matter what version of python It does install on my mac locally without issue using 
pip
. Any ideas?
✅ 1
I narrowed it down 
pypi
not being able to find the dependency 
imagecodecs
which is available.
d
not sure what the issue is with that decorator but you can try the bleeding edge one . It seems to resolve for me:
Copy code
metaflow environment resolve --dry-run -r test.txt
### Environment for architecture osx-arm64
Environment of type pypi-only full hash 5b2e9d46f16fae9b07b64da44f3b6a09e42db532:06991aecff93b36bcfb36e395839fba18b0a8ea8
Arch osx-arm64
Available on osx-arm64

Resolved on 2024-03-20 12:30:32.248956
Resolved by rcledat

User-requested packages pypi::boto3==>=1.14.0, pypi::cffi==>=1.13.0,!=1.15.0, pypi::fastavro==>=1.6.0, pypi::pandas==>=0.24.0, pypi::pyarrow==>=0.17.1, conda::python==3.11.7, pypi::requests==>=2.21.0, pypi::tiffslide
User sources conda::conda-forge, pypi::<https://pypi.netflix.net/simple>

Conda Packages installed bzip2==1.0.8-h93a5062_5, ca-certificates==2024.2.2-hf0a4a13_0, libexpat==2.6.2-hebf3989_0, libffi==3.4.2-h3422bc3_5, libsqlite==3.45.2-h091b4b1_0, libzlib==1.2.13-h53f4e23_5, ncurses==6.4.20240210-h078ce10_0, openssl==3.2.1-h0d3ecfb_1, pip==24.0-pyhd8ed1ab_0, python==3.11.7-hdf0ec26_1_cpython, readline==8.2-h92ec313_1, setuptools==69.2.0-pyhd8ed1ab_0, tk==8.6.13-h5083fa2_1, tomli==2.0.1-pyhd8ed1ab_0, tzdata==2024a-h0c530f3_0, wheel==0.42.0-pyhd8ed1ab_0, xz==5.2.6-h57fd34a_0
Pypi Packages installed asciitree==0.3.3, boto3==1.34.66, botocore==1.34.66, certifi==2024.2.2, cffi==1.16.0, charset-normalizer==3.3.2, fastavro==1.9.4, fasteners==0.19, fsspec==2024.3.1, idna==3.6, imagecodecs==2024.1.1, jmespath==1.0.1, numcodecs==0.12.1, numpy==1.26.4, pandas==2.2.1, pillow==10.2.0, pyarrow==15.0.2, pycparser==2.21, python-dateutil==2.9.0.post0, pytz==2024.1, requests==2.31.0, s3transfer==0.10.1, six==1.16.0, tifffile==2024.2.12, tiffslide==2.4.0, typing-extensions==4.10.0, tzdata==2024.1, urllib3==2.2.1, zarr==2.17.1

Dry-run -- not caching or aliasing
(see here: https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators)
c
Thanks @dry-beach-38304
d
(might be sligtly different for you as I have a few other deps but should be similar)
c
Seems to have worked. Thanks. I did need to say 
--environment=conda
I assume thats cool?
d
yes. That is normal.
👍 1
(it’s all conda whether or not you use the bleeding edge — the --environment=pypi is just an alias in the non bleeding edge)
c
With this new decorator does it support install from github?
Copy code
"xarray": "@git+<https://github.com/valarlabs/xarray.git@7ef77892da25fc6bd3c7fa6dcc06193c3c8cb735#egg=xarray>"
?
@dry-beach-38304 I'm trying to build the above and I've figured out how I think with this
Copy code
@pypi(
        python="3.11.7",
        packages={
            "git+<https://github.com/valarlabs/xarray.git@valar#egg=xarray>": "",
 }
But I get this error
Copy code
metaflow._vendor.packaging.version.InvalidVersion: Invalid version: '2023.5.24.dev84+g7ef77892+mfbuild'
Any way to get around this? Also how does it determine the name afaik the 
2023.5.24.dev84
doesn't appear in our forked repo at all. Thanks.
d
definitely supports builds from github. Let me check that.
I may need to rename the version. It gets it from the setup.py or whatever. I don’t generate it.
I add the +mfbuild which I should probably not add if there is already something there.
that should be an easy change.
c
do you know if pip reads that name itself?
d
it probably reads it while building. Trying now to see the debug logs
ok. I see what the problem is. Let me try fixing this and a few other things and pushing an update tonight. I’ve been a bit busy but this is a tiny fix so hoping to get to it today or so.
c
Thanks really appreciate it
d
fixed — will release tomorrow hopefully:
Copy code
Environment of type pypi-only full hash 8cc85270ef481a5d654c24c036f3c118fec9abab:412f59d24366b2f3e555aae6835a3c8323fdd847
Arch linux-64
Available on linux-64

Resolved on 2024-03-22 07:25:47.047918
Resolved by rcledat

User-requested packages sys::__glibc==2.27=0, pypi::boto3==>=1.14.0, pypi::cffi==>=1.13.0,!=1.15.0, pypi::fastavro==>=1.6.0, pypi::pandas==>=0.24.0, pypi::pyarrow==>=0.17.1, conda::python==3.11.7, pypi::requests==>=2.21.0, pypi::xarray@git+<https://github.com/valarlabs/xarray.git@valar>
User sources conda::conda-forge, pypi::<https://pypi.netflix.net/simple>

Conda Packages installed _libgcc_mutex==0.1-conda_forge, _openmp_mutex==4.5-2_gnu, bzip2==1.0.8-hd590300_5, ca-certificates==2024.2.2-hbcca054_0, ld_impl_linux-64==2.40-h41732ed_0, libexpat==2.6.2-h59595ed_0, libffi==3.4.2-h7f98852_5, libgcc-ng==13.2.0-h807b86a_5, libgomp==13.2.0-h807b86a_5, libnsl==2.0.1-hd590300_0, libsqlite==3.45.2-h2797004_0, libuuid==2.38.1-h0b41bf4_0, libxcrypt==4.4.36-hd590300_1, libzlib==1.2.13-hd590300_5, ncurses==6.4.20240210-h59595ed_0, openssl==3.2.1-hd590300_1, pip==24.0-pyhd8ed1ab_0, python==3.11.7-hab00c5b_1_cpython, readline==8.2-h8228510_1, setuptools==69.2.0-pyhd8ed1ab_0, tk==8.6.13-noxft_h4845f30_101, tomli==2.0.1-pyhd8ed1ab_0, tzdata==2024a-h0c530f3_0, wheel==0.43.0-pyhd8ed1ab_0, xz==5.2.6-h166bdaf_0
Pypi Packages installed boto3==1.34.68, botocore==1.34.68, certifi==2024.2.2, cffi==1.16.0, charset-normalizer==3.3.2, fastavro==1.9.4, idna==3.6, jmespath==1.0.1, numpy==1.26.4, packaging==24.0, pandas==2.2.1, pyarrow==15.0.2, pycparser==2.21, python-dateutil==2.9.0.post0, pytz==2024.1, requests==2.31.0, s3transfer==0.10.1, six==1.16.0, tzdata==2024.1, urllib3==2.2.1, xarray==2023.5.24.dev84+g7ef77892.mfbuild

Dry-run -- not caching or aliasing
c
Thanks.
d
running a tad late.
Haven’t released yet but try installing from here: https://github.com/Netflix/metaflow-nflx-extensions/pull/35
5 Views
Open in Slack
PreviousNext
Powered by