https://outerbounds.com/ logo
Join Slack
Powered by
Hi I'm encountering an issue while trying to use ...
# ask-metaflow
n
Hi I'm encountering an issue while trying to use the 
@pypi_base
decorator for installing 
kedro 0.19.7
in my flow. The error I'm facing is as follows:
Copy code
Pip ran into an error while setting up environment:
    command '/Users/srikanth.bethi/.local/bin/micromamba run --prefix /Users/srikanth.bethi/micromamba/envs/metaflow/osx-arm64/baa1f76cc5191d6 pip3 --disable-pip-version-check --no-color --no-input --isolated install --dry-run --only-binary=:all: --upgrade-strategy=only-if-needed --target=/var/folders/b4/377scmjj57ldfs7c7fdcfwhm0000gn/T/tmp_pyvxzqm --report=/var/folders/b4/377scmjj57ldfs7c7fdcfwhm0000gn/T/tmp_pyvxzqm/report.json --progress-bar=off --quiet --abi cp310 --abi abi3 --abi none --platform any --platform macosx_10_10_universal2 --platform macosx_14_0_arm64 --platform macosx_13_0_universal2 --platform macosx_10_4_universal2 --platform macosx_11_0_universal2 --platform macosx_12_0_universal2 --platform macosx_14_0_universal2 --platform macosx_10_15_universal2 --platform macosx_10_16_universal2 --platform macosx_13_0_arm64 --platform macosx_10_12_universal2 --platform macosx_10_5_universal2 --platform macosx_10_11_universal2 --platform macosx_12_0_arm64 --platform macosx_10_13_universal2 --platform macosx_10_6_universal2 --platform macosx_10_7_universal2 --platform macosx_11_0_arm64 --platform macosx_10_8_universal2 --platform macosx_10_9_universal2 --platform macosx_10_14_universal2 requests>=2.21.0 kedro==0.19.7' returned error (1) 
    ERROR: Cannot install kedro because these package versions have conflicting dependencies.
    ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>
Interestingly, using the 
@conda_base
decorator resolves this issue, but I need to rely on private repositories, which require sticking with 
@pypi
. I've also tried the 
metaflow-nflx-extensions
with a conda environment and 
@pypi_base
, yet the issue persists with Kedro. Could anyone advise on how to successfully include Kedro as a dependency under these circumstances? Thank you!
here is test flow I have used to simulate this issue
Copy code
from metaflow import FlowSpec, step, pypi_base


@pypi_base(python='3.10',
        packages={'kedro': '0.19.7'})

class TestFlow(FlowSpec):
    @step
    def start(self):
        self.my_var = 'hello world'
        self.next(self.end)

    @step
    def end(self):
        print('the data artifact is still: %s' % self.my_var)

if __name__ == '__main__':
    TestFlow()
I have tested with poetry to see if version dependency is an issue with 
poetry
with the following 
pyproject.toml
file and confirmed it is not an issue.
Copy code
[tool.poetry]
name = "test-metaflow-packages"
version = "0.1.0"
description = ""
authors = ["srikanth bethi <srikanthreddy.bethi@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
requests = ">=2.21.0"
kedro = "0.19.7"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
I believe 
requests>=2.21.0
dependency is added by metaflow
this is the error I get with 
netflix-ext
Copy code
The conflict is caused by:
    omegaconf 2.3.0 depends on antlr4-python3-runtime==4.9.*
    omegaconf 2.2.3 depends on antlr4-python3-runtime==4.9.*
    omegaconf 2.2.2 depends on antlr4-python3-runtime==4.9.*
    omegaconf 2.2.1 depends on antlr4-python3-runtime==4.9.*
    omegaconf 2.1.2 depends on antlr4-python3-runtime==4.8
    omegaconf 2.1.1 depends on antlr4-python3-runtime==4.8

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict


Pretty-printed STDERR:
ERROR: Cannot install kedro because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts>

    Conda ran into an error while setting up environment.:
    Binary command for Conda '['/home/jovyan/micromamba/envs/metaflow_builder_53264742bca41d06203c41c9b64f3db6cb643d5c_62b8d5a8be4a6706f608efb10a61450d226552ce/bin/python', '-m', 'pip', '-v', '--isolated', 'install', '--ignore-installed', '--dry-run', '--target', '/tmp/tmpzf2ic0b6', '--report', '/tmp/tmpzf2ic0b6/out.json', '-i', '<https://pypi.org/simple>', '--extra-index-url', '<https://srikanth.bethi:cmVmdGtuOjAxOjE3NTIyNDY0NDE6R2p1VGVsaExzcmwzRjRUUVBzUTRKbUxqdnBo@physicsx.jfrog.io/artifactory/api/pypi/px-pypi-release/simple>', '--only-binary=:all:', '--abi', 'cp310', '--abi', 'none', '--abi', 'abi3', '--platform', 'manylinux_2_25_x86_64', '--platform', 'manylinux_2_24_x86_64', '--platform', 'manylinux_2_18_x86_64', '--platform', 'manylinux_2_23_x86_64', '--platform', 'manylinux_2_17_x86_64', '--platform', 'manylinux1_x86_64', '--platform', 'manylinux2014_x86_64', '--platform', 'manylinux_2_21_x86_64', '--platform', 'linux_x86_64', '--platform', 'manylinux_2_19_x86_64', '--platform', 'manylinux_2_27_x86_64', '--platform', 'manylinux_2_20_x86_64', '--platform', 'any', '--platform', 'manylinux2010_x86_64', '--platform', 'manylinux_2_26_x86_64', 'requests>=2.21.0', 'kedro==0.19.7']' returned error (1); see pretty-printed error above
@dry-beach-38304, Could you please help me?
d
I’ll take a look today.
🙏 1
I can’t seem to reproduce (yet) this running 2.12.25 and 1.2.2 of the netflix extensions. I can’t repro on my prod enviornment either. Based on the command line, I assumed you were running this on a mac but resolving for a linux-64 environment. It seems to resolve ok for me. Could you run again with 
METAFLOW_DEBUG_CONDA=1
and let me know the output and I can take another look.
Here is what I get when I do resolve it:
Copy code
### Environment for architecture linux-64
Environment of type pypi-only full hash 934276d25a234e06cdea98018984833df4607b50:ff1ca533037b4c133e19d6a9501935c6d36cd4f7
Arch linux-64
Available on linux-64

Resolved on 2024-10-21 08:59:28.983451
Resolved by rcledat

User-requested packages sys::__glibc==2.27, pypi::kedro==0.19.7, conda::python==3.10.x, pypi::requests==>=2.21.0
User sources conda::conda-forge, pypi::<https://pypi.netflix.net/simple>

Conda Packages installed _libgcc_mutex==0.1-conda_forge, _openmp_mutex==4.5-2_gnu, bzip2==1.0.8-h4bc722e_7, ca-certificates==2024.8.30-hbcca054_0, ld_impl_linux-64==2.43-h712a8e2_1, libffi==3.4.2-h7f98852_5, libgcc==14.2.0-h77fa898_1, libgcc-ng==14.2.0-h69a702a_1, libgomp==14.2.0-h77fa898_1, libnsl==2.0.1-hd590300_0, libsqlite==3.46.1-hadc24fc_0, libuuid==2.38.1-h0b41bf4_0, libxcrypt==4.4.36-hd590300_1, libzlib==1.3.1-hb9d3cd8_2, ncurses==6.5-he02047a_1, openssl==3.3.2-hb9d3cd8_0, pip==24.2-pyh8b19718_1, python==3.10.15-h4a871b0_2_cpython, readline==8.2-h8228510_1, setuptools==75.1.0-pyhd8ed1ab_0, tk==8.6.13-noxft_h4845f30_101, tomli==2.0.2-pyhd8ed1ab_0, tzdata==2024b-hc8b5060_0, wheel==0.42.0-pyhd8ed1ab_0, xz==5.2.6-h166bdaf_0
Pypi Packages installed antlr4-python3-runtime==4.9.3, appdirs==1.4.4, arrow==1.3.0, attrs==24.2.0, binaryornot==0.4.4, build==1.2.2.post1, cachetools==5.5.0, certifi==2024.8.30, chardet==5.2.0, charset-normalizer==3.4.0, click==8.1.7, cookiecutter==2.6.0, dynaconf==3.2.6, fsspec==2024.10.0, gitdb==4.0.11, gitpython==3.1.43, idna==3.10, importlib-metadata==8.5.0, importlib-resources==6.4.5, jinja2==3.1.4, kedro==0.19.7, kedro-telemetry==0.6.1, markdown-it-py==3.0.0, markupsafe==3.0.2, mdurl==0.1.2, more-itertools==10.5.0, omegaconf==2.3.0, packaging==24.1, parse==1.20.2, platformdirs==4.3.6, pluggy==1.5.0, pre-commit-hooks==5.0.0, pygments==2.18.0, pyproject-hooks==1.2.0, python-dateutil==2.9.0.post0, python-slugify==8.0.4, pytoolconfig==1.3.1, pyyaml==6.0.2, requests==2.32.3, rich==13.9.2, rope==1.13.0, ruamel-yaml==0.18.6, ruamel-yaml-clib==0.2.12, six==1.16.0, smmap==5.0.1, text-unidecode==1.3, toml==0.10.2, tomli==2.0.2, types-python-dateutil==2.9.0.20241003, typing-extensions==4.12.2, urllib3==2.2.3, zipp==3.20.2

Dry-run -- not caching or aliasing
I did request python “3.10.*” (and not 3.10)
I don’t think that should make a huge difference but pointing it out.
h
I was able to reproduce using the following. Seems to be related to the internal pypi repo, because I also can't reproduce without setting 
--extra-index-url.
This repo seems to be missing the required versions for 
antlr4-python3-runtime
somehow?
Copy code
pip -v --isolated install --ignore-installed --dry-run --target /tmp/tmpzf2ic0b6 --report /tmp/tmpzf2ic0b6/out.json -i <https://pypi.org/simple> --extra-index-url <https://srikanth.bethi:cmVmdGtuOjAxOjE3NTIyNDY0NDE6R2p1VGVsaExzcmwzRjRUUVBzUTRKbUxqdnBo@physicsx.jfrog.io/artifactory/api/pypi/px-pypi-release/simple> --only-binary=:all: --abi cp310 --abi none --abi abi3 --platform manylinux_2_25_x86_64 --platform manylinux_2_24_x86_64 --platform manylinux_2_18_x86_64 --platform manylinux_2_23_x86_64 --platform manylinux_2_17_x86_64 --platform manylinux1_x86_64 --platform manylinux2014_x86_64 --platform manylinux_2_21_x86_64 --platform linux_x86_64 --platform manylinux_2_19_x86_64 --platform manylinux_2_27_x86_64 --platform manylinux_2_20_x86_64 --platform any --platform manylinux2010_x86_64 --platform manylinux_2_26_x86_64 'requests>=2.21.0' 'kedro==0.19.7'
Copy code
pip -v --isolated install --ignore-installed --dry-run --target /tmp/tmpzf2ic0b6 --report /tmp/tmpzf2ic0b6/out.json -i <https://pypi.org/simple> --extra-index-url <https://srikanth.bethi:cmVmdGtuOjAxOjE3NTIyNDY0NDE6R2p1VGVsaExzcmwzRjRUUVBzUTRKbUxqdnBo@physicsx.jfrog.io/artifactory/api/pypi/px-pypi-release/simple> --only-binary=:all: --abi cp310 --abi none --abi abi3 --platform manylinux_2_25_x86_64 --platform manylinux_2_24_x86_64 --platform manylinux_2_18_x86_64 --platform manylinux_2_23_x86_64 --platform manylinux_2_17_x86_64 --platform manylinux1_x86_64 --platform manylinux2014_x86_64 --platform manylinux_2_21_x86_64 --platform linux_x86_64 --platform manylinux_2_19_x86_64 --platform manylinux_2_27_x86_64 --platform manylinux_2_20_x86_64 --platform any --platform manylinux2010_x86_64 --platform manylinux_2_26_x86_64 'antlr4-python3-runtime==4.9.*'
Using pip 24.2 from /Users/npow/anaconda3/envs/py310/lib/python3.10/site-packages/pip (python 3.10)
Looking in indexes: <https://pypi.org/simple>, <https://srikanth.bethi>:****@physicsx.jfrog.io/artifactory/api/pypi/px-pypi-release/simple

ERROR: Could not find a version that satisfies the requirement antlr4-python3-runtime==4.9.* (from versions: 4.11.0, 4.11.1, 4.12.0, 4.13.0, 4.13.1, 4.13.2)
ERROR: No matching distribution found for antlr4-python3-runtime==4.9.*
d
thanks @hundreds-rainbow-67050
62 Views
Open in Slack
PreviousNext
Powered by