Hi! How would someone go about including a private...
# ask-metaflow
q
Hi! How would someone go about including a private dependency which contains a
pyproject.toml
file which itself contains other dependencies? Is there a way to make
@pypi
include such dependencies and also specify a set of optional dependencies as if you were to execute
pip install ../my-lib[foo,bar,baz]
?
1
g
do you mean a package pushed to a private python registry?
Copy code
@pypi_base(
    python="3.10",
    extra_indices=[
        f"<https://_token>:{os.getenv('GOOGLE_AUTH_TOKEN')}@us-python.pkg.dev/project-name/repo-name/simple"
    ],
    packages={
        "my-package[foo,bar,baz]": "1.2.3",
    },
)
q
We don't have a proper Python registry, but just a private git repo 🙂
Thanks for the example! I'll try to let it reference a local or remote git repo.
Hello! I ran into an issue where I am unable to bootstrap the python environment:
Copy code
2024-07-23 22:50:23.459 Bootstrapping virtual environment(s) ...
    Pip ran into an error while setting up environment:
    Could not find a binary distribution for untokenize<0.2.0,>=0.1.1
    for the platform linux-64

    Note that @pypi does not currently support source distributions
I am using the following decorator:
Copy code
@pypi_base(
    python="3.11",
    packages={
        "../../my-lib[x,y,z]": "",
    }
)
which has
untokenize
as one of its transitive dependencies. Is anyone aware of a possible workaround for this issue?
a
currently there is a hard requirement for packages to be made available as wheels for a proper env set. you could try the extensions version of
@pypi
to see if that addresses your use case - https://github.com/Netflix/metaflow-nflx-extensions
q
Thanks for your reply! I tried to install the extensions but it seems to require mamba instead of automatically downloading and running micromamba. I guess it needs more features that aren't present in micromamba? Now that you said so, I'll try to give that a proper shot.
d
it can use mamba, micromamba or conda but relies on one being installed on the system (any of them will do). You need to configure the one you want to use though. By default it’s mamba.
the extension won’t do cross-platform in all cases but will do source and build for you if needed.
q
Okay great. I fixed a small issue in
metaflow-nflx-extensions
but I encountered a different issue now:
Copy code
Metaflow 2.12.8+netflix-ext(1.2.1) executing UFlow for user:ili
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ...No STDOUT
Pretty-printed STDERR:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'poetry'

    Conda ran into an error while setting up environment.:
    Binary command for Conda '['/home/ili/miniforge3/envs/metaflow_builder_6c040771535a4ea9ee50654d4c796ac87e38c144_ec6837a0d5a15074db30835d62f58f934e8a12f6/bin/python', '-c', "import tomli; f = open('/home/ili/work/src/my-lib/pyproject.toml', mode='rb'); d = tomli.load(f); print(d.get('poetry', d['tool']['poetry'])['name']); print(d.get('poetry', d['tool']['poetry'])['version'])"]' returned error (1); see pretty-printed error above
Does it assume poetry is also installed or is metaflow supposed to install it?
d
you don’t need to install it. What does your pyproject.toml look like for the package you are trying to build?
It’s trying to get the name of the package and clearly makes the wrong assumptions — easy to extend. And thanks for the PR, I’ll merge it
👍 1
q
The pyproject toml looks something like:
Copy code
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my-lib"
version = "0.1"
description = "Foo"
readme = "README.md"
dependencies = [
    "numpy~=1.23.5",
    "scipy~=1.10.1",
    "pandas~=2.0.3",
    "boto3>=1.20",
    "typing_extensions~=4.7.1",
    "haversine>=2.0.0",
    "aiohttp~=3.9.5",
    "iso8601~=1.0.2",
    "scikit-learn~=1.3.2",
    "torch~=2.1.2",
    "tqdm~=4.66.1",
    "aim~=3.23.0",
]

[project.optional-dependencies]
private = ["some-other-private-lib @ <git+ssh://git@github.com/ilian/some-private-lib.git>"]
linters = [
    "mypy==1.4.1",
    "black[jupyter]==23.7.0",
    "autoflake==2.2.0",
    "isort==5.12.0",
    "docformatter==1.7.5",
]
research = [
    "python-dotenv~=1.0.0",
    "jupyterlab~=4.0.5",
    "ipywidgets~=8.1.0",
    "widgetsnbextension~=4.0.8",
    "seaborn~=0.13.0",
    "matplotlib~=3.7.2",
    "tensorboard~=2.13.0",
    "pymongo~=4.6.1",
    "SQLAlchemy[pymysql]~=2.0.31",
    "dill~=0.3.8",
]
tests = [
    "pytest~=7.4.3"
]

[tool.setuptools.packages.find]
include = ["mylib", "mylib.*", "research", "research.*"]
d
ok, I’ll fi that. As a workaround could you add:
Copy code
[tool.poetry]
name = "my-lib"
version = "0.1"
and let me know how that goes.
q
@dry-beach-38304 That worked! Thank you very much 🙌 Let me know as soon as this workaround isn't necessary anymore.
a
Hi, I use the similar configuration(
@pypi_base
and local package) but I got this error:
Copy code
2024-07-24 17:07:37.270 Bootstrapping virtual environment(s) ...
    Internal error
Traceback (most recent call last):
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 1126, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 585, in wrapper
    return func(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 746, in run
    before_run(obj, tags, decospecs)
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 825, in before_run
    obj.package = MetaflowPackage(
                  ^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_environment.py", line 176, in init_environment
    _ = list(map(lambda x: self.solvers[solver].download(x), results))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_environment.py", line 176, in <lambda>
    _ = list(map(lambda x: self.solvers[solver].download(x), results))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/pip.py", line 196, in download
    hash=package["hash"],
         ~~~~~~~^^^^^^^^
KeyError: 'hash'
a
@adorable-dusk-49894 what are your contents of
@pypi
?
a
Copy code
@pypi_base(
    python="3.10",
    packages={
        "../some-local-package": ""
    }
)
d
@adorable-dusk-49894 — the previous user tried with the netflix extension (see here: https://pypi.org/project/metaflow-netflixext/). I am not 100% sure local packages work without the extension. You can try with that package and it should hopefully work.
@quaint-exabyte-4933 — sure thing. I have some pending work on this whole thing (other bugs and what not). I am running late but am going to try to carve out time some time soon-ish. In other related news, I am finally making some progress on updating conda-lock (which this uses) to use a newer version of poetry which should also improve the extension 🙂
🙌 1
@quaint-exabyte-4933 — fixed, will push it out shortly.
109 Views