Hi Folks, has anyone run into this issue with the

metaflow extensions

and

conda-lock v1.3.0 or v1.4.0

? The poetry Config class doesn’t seem to have a

create()

attribute.

@conda_base(
    libraries={
        "pandas": "1.5.3",
        "snowflake-connector-python": "3.0.2",
        "cryptography": "38.0.4",
    },
    pip_packages={"some_internal_package": "1.0"},
    pip_sources=[
        "<https://custom.source/packages-pypi-virtual/simple>"
    ],
)

Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 2 environments in flow ...    Internal error
Traceback (most recent call last):
  File ".../.venv/lib/python3.10/site-packages/metaflow/cli.py", line 1175, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File ".../.venv/lib/python3.10/site-packages/metaflow/tracing_noop.py", line 19, in wrapper_func
    return func(args, kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow/cli.py", line 692, in wrapper
    return func(args, kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow/cli.py", line 830, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File ".../.venv/lib/python3.10/site-packages/metaflow/cli.py", line 888, in before_run
    obj.package = MetaflowPackage(
  File ".../.venv/lib/python3.10/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File ".../.venv/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 120, in init_environment
    self._resolve_environments(echo, need_resolution)
  File ".../.venv/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 242, in _resolve_environments
    env_id, resolved_env = f.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(self.args, self.kwargs)
  File ".../.venv/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 203, in _resolve
    self._conda.resolve(
  File ".../.venv/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 188, in resolve
    packages = self._resolve_env_with_conda_lock(
  File ".../.venv/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1459, in _resolve_env_with_conda_lock
    poetry_config = Config.create()
AttributeError: type object 'Config' has no attribute 'create'

hey @clean-school-46881: let me take a look at that. It works for me internally but ya, that is a very hacky bit of trying to pass sources to poetry. I am actually working on adding more functionality to this (including a pure pip through pip implementation) but I’ll take a look at this error. Can I get back to you by EOD?

Sure, thanks. I wonder if I needed to set a special configuration in order to use a custom pip source

ah just looked at the code.

Thanks!

and then below, change https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/conda.py#L1408 to

python_exec = sys.executable

Cool, so it was able to move past that (thanks) and is giving me another resolving error:

Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.10.4', 'requests >=2.21.0', 'boto3 >=1.14.0', 'pandas 1.5.3', 'snowflake-connector-python 3.0.2', 'cryptography 38.0.4', 'pip *']
Traceback (most recent call last):
  File ".../.venv/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
    result = resolve_version(
  File ".../.venv/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File ".../.venv/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 83, in solve
    self._propagate(next)
  File ".../.venv/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 123, in _propagate
    root_cause = self._resolve_conflict(incompatibility)
  File ".../.venv/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 321, in _resolve_conflict
    raise SolveFailure(incompatibility)
conda_lock._vendor.poetry.mixology.failure.SolveFailure: Because freezegun (0.1.11) depends on both python-dateutil (<empty>) and python-dateutil (<empty>), freezegun is forbidden.
So, because -dummy-package- depends on tala_feature_store (=1.0.7rc2) which depends on freezegun (0.1.11), version solving failed.

There’s more to it, but is this the typical “relax your constraints” issue, or is it related?

this is a poetry solving issue — I am not 100% sure what is happening but strongly suspect that ya, it is something to do with constraints. The way conda-lock works is that it first solves the conda environment and then passes all those locked dependencies to poetry and adds the one you specify for pip. I can’t really know for sure why this environment is not resolving but yes, you may have to play with it a bit. You can try to move some things from conda to pip to see if that gives poetry a bit more lattitude. In all cases btw, you can specify the package versions using something like “>=…” or whatever. It should work in a comma separated form as usual.

Oh, that makes sense then. In that case I’ll definitely choose one or the other instead of letting conda lock its reqs

yes, the new version will have an explicit @pip decorator which will use conda just to lock the python base environment and then use pip for everything else.