quaint-exabyte-4933
07/23/2024, 7:04 PMpyproject.toml
file which itself contains other dependencies?
Is there a way to make @pypi
include such dependencies and also specify a set of optional dependencies as if you were to execute pip install ../my-lib[foo,bar,baz]
?great-father-37686
07/23/2024, 7:09 PMgreat-father-37686
07/23/2024, 7:22 PM@pypi_base(
python="3.10",
extra_indices=[
f"<https://_token>:{os.getenv('GOOGLE_AUTH_TOKEN')}@us-python.pkg.dev/project-name/repo-name/simple"
],
packages={
"my-package[foo,bar,baz]": "1.2.3",
},
)
quaint-exabyte-4933
07/23/2024, 7:23 PMquaint-exabyte-4933
07/23/2024, 7:25 PMquaint-exabyte-4933
07/23/2024, 8:59 PM2024-07-23 22:50:23.459 Bootstrapping virtual environment(s) ...
Pip ran into an error while setting up environment:
Could not find a binary distribution for untokenize<0.2.0,>=0.1.1
for the platform linux-64
Note that @pypi does not currently support source distributions
I am using the following decorator:
@pypi_base(
python="3.11",
packages={
"../../my-lib[x,y,z]": "",
}
)
which has untokenize
as one of its transitive dependencies.
Is anyone aware of a possible workaround for this issue?ancient-application-36103
07/23/2024, 9:59 PM@pypi
to see if that addresses your use case - https://github.com/Netflix/metaflow-nflx-extensionsquaint-exabyte-4933
07/23/2024, 10:02 PMdry-beach-38304
07/24/2024, 6:41 AMdry-beach-38304
07/24/2024, 6:42 AMquaint-exabyte-4933
07/24/2024, 6:56 AMmetaflow-nflx-extensions
but I encountered a different issue now:
Metaflow 2.12.8+netflix-ext(1.2.1) executing UFlow for user:ili
Validating your flow...
The graph looks good!
Running pylint...
Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
Resolving 1 environment ...No STDOUT
Pretty-printed STDERR:
Traceback (most recent call last):
File "<string>", line 1, in <module>
KeyError: 'poetry'
Conda ran into an error while setting up environment.:
Binary command for Conda '['/home/ili/miniforge3/envs/metaflow_builder_6c040771535a4ea9ee50654d4c796ac87e38c144_ec6837a0d5a15074db30835d62f58f934e8a12f6/bin/python', '-c', "import tomli; f = open('/home/ili/work/src/my-lib/pyproject.toml', mode='rb'); d = tomli.load(f); print(d.get('poetry', d['tool']['poetry'])['name']); print(d.get('poetry', d['tool']['poetry'])['version'])"]' returned error (1); see pretty-printed error above
Does it assume poetry is also installed or is metaflow supposed to install it?dry-beach-38304
07/24/2024, 6:58 AMdry-beach-38304
07/24/2024, 6:58 AMquaint-exabyte-4933
07/24/2024, 7:02 AM[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "my-lib"
version = "0.1"
description = "Foo"
readme = "README.md"
dependencies = [
"numpy~=1.23.5",
"scipy~=1.10.1",
"pandas~=2.0.3",
"boto3>=1.20",
"typing_extensions~=4.7.1",
"haversine>=2.0.0",
"aiohttp~=3.9.5",
"iso8601~=1.0.2",
"scikit-learn~=1.3.2",
"torch~=2.1.2",
"tqdm~=4.66.1",
"aim~=3.23.0",
]
[project.optional-dependencies]
private = ["some-other-private-lib @ <git+ssh://git@github.com/ilian/some-private-lib.git>"]
linters = [
"mypy==1.4.1",
"black[jupyter]==23.7.0",
"autoflake==2.2.0",
"isort==5.12.0",
"docformatter==1.7.5",
]
research = [
"python-dotenv~=1.0.0",
"jupyterlab~=4.0.5",
"ipywidgets~=8.1.0",
"widgetsnbextension~=4.0.8",
"seaborn~=0.13.0",
"matplotlib~=3.7.2",
"tensorboard~=2.13.0",
"pymongo~=4.6.1",
"SQLAlchemy[pymysql]~=2.0.31",
"dill~=0.3.8",
]
tests = [
"pytest~=7.4.3"
]
[tool.setuptools.packages.find]
include = ["mylib", "mylib.*", "research", "research.*"]
dry-beach-38304
07/24/2024, 7:29 AM[tool.poetry]
name = "my-lib"
version = "0.1"
and let me know how that goes.quaint-exabyte-4933
07/24/2024, 9:16 AMadorable-dusk-49894
07/24/2024, 3:10 PM@pypi_base
and local package) but I got this error:
2024-07-24 17:07:37.270 Bootstrapping virtual environment(s) ...
Internal error
Traceback (most recent call last):
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 1126, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
return self.main(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
return callback(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 585, in wrapper
return func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 746, in run
before_run(obj, tags, decospecs)
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/cli.py", line 825, in before_run
obj.package = MetaflowPackage(
^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/package.py", line 70, in __init__
environment.init_environment(echo)
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_environment.py", line 176, in init_environment
_ = list(map(lambda x: self.solvers[solver].download(x), results))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/conda_environment.py", line 176, in <lambda>
_ = list(map(lambda x: self.solvers[solver].download(x), results))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/easongao/opt/anaconda3/envs/metaflow/lib/python3.12/site-packages/metaflow/plugins/pypi/pip.py", line 196, in download
hash=package["hash"],
~~~~~~~^^^^^^^^
KeyError: 'hash'
ancient-application-36103
07/24/2024, 3:12 PM@pypi
?adorable-dusk-49894
07/24/2024, 3:12 PM@pypi_base(
python="3.10",
packages={
"../some-local-package": ""
}
)
dry-beach-38304
07/25/2024, 8:03 AMdry-beach-38304
07/25/2024, 8:04 AMdry-beach-38304
08/05/2024, 1:45 AM