Hey all, I'm getting an environment resolution errors when running a metaflow pipeline on my local machine. I'm running the flow with

--environment=conda

and getting an error like this (partial error log):

File "/local/home/usr/.cache/pypoetry/virtualenvs/env-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2114, in _resolve_env_with_pip
    self._call_binary(args, binary=builder_python)
  File "local/home/usr/.cache/pypoetry/virtualenvs/env-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 3983, in _call_binary
    elif not os.path.isfile(binary) or not os.access(binary, os.X_OK):
  File "/home/usr/anaconda3/lib/python3.10/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

Any insights?

This is using the conda extension (at least based on logs). Make sure you are using the latest and rerun with METAFLOW_DEBUG_CONDA=1 and I can take a look at the full log (responses may be delayed as I am out of the country). Based on the trace, it is trying to create a Python environment to build a source package (or something) and somehow that is not working.

Tried with

METAFLOW_DEBUG_CONDA=1

but the log doesn't really change:

Traceback (most recent call last):
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/cli.py", line 828, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/cli.py", line 886, in before_run
    obj.package = MetaflowPackage(
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 79, in init_environment
    resolver.resolve_environments(echo)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py", line 305, in resolve_environments
    self._resolve_environments(echo, need_resolution)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py", line 712, in _resolve_environments
    env_id, resolved_env, addl_builder_env = f.result()
  File "/home/usr/anaconda3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/usr/anaconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/usr/anaconda3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(self.args, self.kwargs)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py", line 507, in _resolve
    resolved_env, builder_env = self._conda.resolve(
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 376, in resolve
    packages, returned_builder_env = self._resolve_env_with_pip(
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2114, in _resolve_env_with_pip
    self._call_binary(args, binary=builder_python)
  File "/local/home/usr/.cache/pypoetry/virtualenvs/env-7zk57AIM-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 3983, in _call_binary
    elif not os.path.isfile(binary) or not os.access(binary, os.X_OK):
  File "/home/usr/anaconda3/lib/python3.10/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

it should print a lot more stuff (like for that other log you sent me) — is this both out and err?