https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, I can use: ```@conda_base( libraries...
# ask-metaflow
h
Hi all, I can use:
Copy code
@conda_base(
    libraries={
        "numpy": "1.26.4",
        "pandas": "2.2.2",
        "scikit-learn": "1.5",
        "lightgbm": "4.3.0",
        # "pyarrow": ">=10.0.1,<10.1.0",
        "pyarrow": "15.0.2",
        "s3fs": "2024.3.1",
        "xgboost": "2.1.1",
        "memory_profiler": "0.61.0"
    },
    python="3.11.9",
)
to define dependencies if metaflow runs in the cloud, which is great. How can I add “internal packages” (not sure if correct term?) I/we maintain, which are installed like so:
Copy code
pip install git+<https://github.com/bla/amazingstuff.git>
Our AWS env. where our metaflow runs and the repo above should play ball/be accessible security wise … Thanks. Chris (edited)
✅ 1
v
you can try
Copy code
@pypi(packages={"git+<https://github.com/bla/amazingstuff.git>" : ""})
obviously you need the right credentials to access the repo in the environment where you run the flow
h
Thanks. My understanding from reading some doc, @conda_base and @pypi are mutually exclusive? So I have to adapt/get rid off (?): @conda_base( libraries={ “numpy”: “1.26.4", “pandas”: “2.2.2", “scikit-learn”: “1.5", “lightgbm”: “4.3.0", # “pyarrow”: “>=10.0.1,<10.1.0", “pyarrow”: “15.0.2", “s3fs”: “2024.3.1", “xgboost”: “2.1.1", “memory_profiler”: “0.61.0" }, python=“3.11.9”, ) and cannot use? python Test.py --with batch:cpu=2,memory=3840,queue=metaflow-1234-55ff636 --environment=conda run
v
right, Conda and Pypi are two different universes of packages that don't mix (easily). If you want to use private 
pypi
packages, you need to specify all dependencies from Pypi, or have two separate flows if that's feasible
d
you can also use the netflix extensions if you want
that allows mixing
(I implemented mixing of git repos and conda deps there — it’s not merged yet so you have to use the branch in that thread)
working on adding tests and hopefully merging soon. Seeing if I can add even more support like local repos and what not — hopefully should work 🙂
h
Thanks for your replies. I am bit in a pickle this works fine: @pypi(packages={ “pyodbc”:“5.1.0" , “git+https://github.com/bla/amazingstuff.git” : “” }) but the package uses pyodbc to access sql server and I get this error: 024-08-30 172351.207 [252/start/1168 (pid 99643)] [f48570de-a8da-42ca-ab84-fec529c02952] File “/metaflow/linux-64/379285aa0b6bee6/lib/python3.11/site-packages/sqlserver_utils/helpers.py”, line 1, in <module> 2024-08-30 172351.207 [252/start/1168 (pid 99643)] [f48570de-a8da-42ca-ab84-fec529c02952] import pyodbc 2024-08-30 172351.207 [252/start/1168 (pid 99643)] [f48570de-a8da-42ca-ab84-fec529c02952] ImportError: libodbc.so.2: cannot open shared object file: No such file or directory I think this gets all to complicated and I have to copy and paste stuff rather trying to reuse code in this data access package ….
d
ya, the pyodbc pip package does not seem to contain the pyodbc lib whereas conda does
the extension should definitely work for this case (branch one)
h
ah yes good point thanks. use conda for pyodbc. did not even think about it - genius thanks!
@dry-beach-38304 - is “branch one” = update-black
d
h
@dry-beach-38304 I may try this tonight. Can you please tell me how to install using pip? This: pip install git+https://github.com/Netflix/metaflow-nflx-extensions/tree/fix/git-pkg-conda-lock does not work
d
Something like metaflow-nflx-extensions@git-pkg-conda-lock. I’ll check when I get to a computer.
👍 1
h
thanks! i may try it when i have time - thanks for other response
d
exact syntax: 
pip install "git+<https://github.com/Netflix/metaflow-nflx-extensions.git@fix/git-pkg-conda-lock>"
h
thanks - sorry must have forgotten ” in my above attempt
@dry-beach-38304 got around to install the above. I just tried: 
python GreatFlow.py --with batch:cpu=36,memory=60000,queue=metaflow-684414486554-55ff636,image=3.11.9-slim-bookworm --environment=conda
and decorated my class using:
Copy code
@conda_base(
    libraries={
        "pandas": "2.2.2",
        "pyarrow": "10.0.1",
        "numpy": "1.26.4",
        "pandas": "2.2",
        "scikit-learn": "1.5.1",
        "lightgbm": "4.5.0",
        "s3fs": "2024.3.1",
        "xgboost": "2.1.1",
        "snowflake-connector-python": "3.11.0"
    },
    python="3.11.9",
)
and my steps:
Copy code
@pypi(packages={
    "git+<https://github.com/my_company/Repo1.git>" : "",
    "git+<https://github.com/my_company/Repo2.git>": ""
},
python="3.11.9",
)
Alas I get:
Copy code
Metaflow 2.12.18+netflix-ext(1.2.2) executing MetaFlow_CDCNSLScorePriceModel for user:user1
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
No STDOUT
Pretty-printed STDERR:
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage]
             [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
             [--tags TAGS]
             [specs ...]
mamba: error: unrecognized arguments: --json

    Conda ran into an error while setting up environment.:
    Conda command '['/Users/user1/miniconda3/envs/meta_flow/bin/mamba', 'info', '--json']' returned error (2); see pretty-printed error above
Any ideas? Thanks in advance.
d
This is not mamba. There is, annoyingly, a Python package named mamba which has nothing to do with the mamba installer.
Am out right now but you need to install either Conda, mamba or micromamba
Also for your packages it should be something f like “mypackage @ git+https://github.com/…/mypackage.git”: “”
h
Thanks. I have conda installed locally. Bit confusing though, as this should also be w.r.t. this “cloud” image: 
image=3.11.9-slim-bookworm
when metaflow tries to bootstrap the docker image to be used in AWS batch? Is this where conda has to be installed, which may not be part of this image?
d
The first stension can use mamba, Conda or micromamba. By default it uses mamba. In your case, mamba actually points to something else. You can set it to use Conda. When I get back to a computer I’ll point you to a previous message here that had the exact same problem.
👍 1
Here is one thread that had the same problem: https://netflix.slack.com/archives/C02116BBNTU/p1724526386871959. If you have conda already (as in 
which conda
points to a proper conda setup), you can also set 
METAFLOW_CONDA_DEPENDENCY_RESOLVER
to 
conda
and it will use that instead.
given this has happened twice, I’ll add a better warning for this.
17 Views
Open in Slack
PreviousNext
Powered by