https://outerbounds.com/ logo
Join Slack
Powered by
I would like to use the ranx package. It does not...
# ask-metaflow
f
I would like to use the ranx package. It does not have a condas package. When I attempt to run a flow using Pypi, I'm getting that there's a dependency mismatch, despite the fact that poetry has blessed the configuration of packages I'm using. Is there any sane way to debug the pypi decorator? ie, can I just see the actual package mismatch?
✅ 1
the actual error is
Copy code
command '/home/mark/.metaflowconfig/micromamba/bin/micromamba run --prefix /home/mark/micromamba/envs/metaflow/linux-64/6551c37ac2ebcb6 pip3 --disable-pip-version-check --no-input --no-color --isolated install --dry-run --only-binary=:all: --upgrade-strategy=only-if-needed --target=/tmp/tmp626fgjw3 --report=/tmp/tmp626fgjw3/report.json --progress-bar=off --quiet --abi cp310 --abi none --abi abi3 --platform manylinux1_x86_64 --platform manylinux_2_17_x86_64 --platform linux_x86_64 --platform manylinux2014_x86_64 --platform manylinux_2_18_x86_64 --platform manylinux_2_21_x86_64 --platform manylinux_2_25_x86_64 --platform manylinux_2_20_x86_64 --platform manylinux2010_x86_64 --platform manylinux_2_19_x86_64 --platform manylinux_2_23_x86_64 --platform manylinux_2_26_x86_64 --platform any --platform manylinux_2_24_x86_64 --platform manylinux_2_27_x86_64 requests>=2.21.0 boto3>=1.14.0 pandas==2.2.0 pyarrow==15.0.0 openai==1.14.2 tiktoken==0.5.2 smart_open==6.4.0 ujson==5.9.0 ranx==0.3.19' returned error (1)
Note that boto3 and requests are ">=" constraints, which contradicts the documentation that constraints are hardcoded to specific versions for pypi: https://docs.metaflow.org/scaling/dependencies/libraries
d
you can try the bleeding edge decorators which has more debugging option (it should print out the pip error for one and if you set METAFLOW_DEBUG_CONDA=1, it prints out a LOT more information). You can alternatively run the command that you print out to see what error it gives.
f
ordinarily, I would expect an error like what appears in this documentation, but I'm not getting it https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
running that line gives no error
apart from "resolution impossible"
yeah, something's bustedhere
poetry has no problem with the libraries as specified
Copy code
[tool.poetry]
name = "garbage"
version = "0.1.0"
description = ""
authors = ["Mark Roden <mroden@crexi.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
pandas = "2.2.0"
pyarrow = "15.0.0"
openai= "1.14.2"
tiktoken = "0.5.2"
smart_open = "6.4.0"
ujson = "5.9.0"
scipy = "1.12.0"
ranx = "0.3.19"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
for the poetry file, with a simple ``````
Copy code
import pandas as pd
import ujson as json
import tiktoken
from smart_open import open
import scipy
from ranx import Qrels, Run


if __name__ == '__main__':
    print("what up")
returns "what up"
maybe it's because that boto3 requirement is super super old? But I'm looking for something more recent.... is this an issue?
yeah this is ridiculous, I've been beating my head against the wall for some time now
poetry says all those versions are compatible, I have no idea what micromamba's problem is
OK, I figured it out by going to metaflow 2.11.9 and then setting all of my pypi dependencies to ""
I would dearly love to know what versions were selected so I can use this in the future with pinned versions
d
hey — sorry. Busy end of the day. A few ideas: • there may be one or more package that is not available in .whl format and since you seem to be cross-compiling (ie: building a linux env from a mac for example), it may be having trouble resolving it. • for debugging what it found, there should be a conda.cnd file (or something like that) in 
.metaflow
which you can try to parse to see what it is doing. • alternatively, as I suggested earlier, I would try with the Netflix extensions which have a more verbose (among other things) implementation of the decorator which may give more clue (see here: https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators). ◦ with that version, you may get more debugging messages on failure ◦ you can also use 
METAFLOW_DEBUG_CONDA=1
which gives a lot of information ◦ and finally, once it does resolve (since you seem to have managed to get it to resolve), you can do 
metaflow environment show …
and that will show you all the packages that were installed which would answer your question a bit more easily than parsing the 
conda.cnd
file (but of course, the information is available in both). Finally, another thing to try is the line that gives no error, you can modify it a bit to remove some of the flags like 
--quiet
to see if you get more detail (that’s one of the changes in the bleeding edge one — I found that --quiet made it impossible to debug so I actually went one step further and run the command with ‘-v’ which is typically much more helpful. You can manually do that as well here to see if you get more information.
f
Thanks, I'll give those a shot!
👍 1
d
let me know what works and what doesn’t and I can try to help further.
d
I had the same issue with 
mteb[beir]==1.1.2
Just this package is enough to trigger the issue (works on poetry). I tracked it down to 
--only-binary=:all:
flag that is being used when building the environment bundle. Not sure what the workaround would be for this. In my case I think it would be acceptable to allow non-binaries. Any ideas on how I can proceed on this?
d
only binary all is required unfortunately with cross platform builds (pip requirement).
There are ways around this though (with the bleeding edge decorators). Essentially, you can run a flow to build your environment for you. Some hints at the bottom of this thread: https://netflix.slack.com/archives/C02116BBNTU/p1697198947028709 and let me see, I thought I had pasted a bigger example
❤️ 1
ok, can’t find it, but basically looks like this isntead of what is there at the end of that thread:
Copy code
subprocess.check_call(
                [
                    sys.executable,
                    "-m",
                    "metaflow.cmd.main_cli",
                    "environment",
                    "resolve",
                    "-r",
                    req_file.name,
                    "--alias",
                    "named_env_alias_you_pick"
                ],
Once that flow runs on batch/kube/whatever but basically on the target platform, you can then simply use 
@named_env(name="named_env_alias_you_pick")
in your flow and you should be all set. Let me know if you have questions getting it to work but it should be pretty straightforward. Basically, you have a flow that builds the environment in a non-cross platform way and then you can use it later (Metaflow will remember it).
6 Views
Open in Slack
PreviousNext
Powered by