Hey I am having issues using pre-resolved environments with

named_env

. I am developing from my

osx-arm64

and running steps via batch / step functions at some point. Basic env file

name: bpy
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip
  - opencv
  - pip:
    - bpy==4.0.0

Issue #1 - Pre resolving environments for linux-64 on MacOS As far as I understand this should work, according to the docs with

metaflow environment resolve -f env.yml --arch linux-64 --alias bpy_test

I get the following error from

conda-lock

when using

conda

for

METAFLOW_CONDA_DEPENDENCY_RESOLVER

Error: Got unexpected extra argument (virtual_yml.spec)

    raise CondaException(
metaflow_extensions.netflix_ext.plugins.conda.utils.CondaException: Binary command for Conda '['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/bin/conda-lock', 'lock', '-f', 'pyproject.toml', '-p', 'linux-64', '--filename-template', '/tmp/conda-lock-gen-tmpqo43j8dk', '-k', 'explicit', '--conda', '/opt/homebrew/bin/conda', '--conda', '--virtual-package-spec', 'virtual_yml.spec']' returned error (2); see pretty-printed error above

When using the

mamba

deefault for

METAFLOW_CONDA_DEPENDENCY_RESOLVER

I get:

Resolving 1 environment ...Pretty-printed STDOUT:
/private/var/folders/xm/zy4s2pl903jbbr1hqzkhgrbm0000gp/T/tmp52xemgjw/virtual_yml.spec
Virtual package spec: %s {'subdirs': {'linux-64': {'packages': {'__archspec': '1=m2', '__conda': '24.3.0=0', '__osx': '14.5=0', '__unix': '0=0', '__glibc': 2.27}}}}

Pretty-printed STDERR:
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1404, in lock
    lock_func(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1112, in run_lock
    make_lock_files(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 318, in make_lock_files
    virtual_package_repo = virtual_package_repo_from_specification(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/virtual_package.py", line 260, in virtual_package_repo_from_specification
    spec = VirtualPackageSpec.parse_obj(data)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/pydantic/main.py", line 1135, in parse_obj
    return cls.model_validate(obj)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/pydantic/main.py", line 568, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for VirtualPackageSpec
subdirs.linux-64.packages.__glibc
  Input should be a valid string [type=string_type, input_value=2.27, input_type=float]
    For further information visit <https://errors.pydantic.dev/2.8/v/string_type>

Which seems to be a trivial pydantic parsing error (__glibc being a number not a string and it not wanting to parse it) Issue #2 - Starting a flow from MacOS that uses a pre resolved environment for linux-64 To circumvent Issue #1 I logged into a linux ec2 instance and resolved the env from there -> works check Then I hoped I could at least launch Flows that use that environment

@named_env(name="bpy_test")

in a

@batch

step. However I get the same errors like in Issue #1 even when running via step functions or

--with batch

I would love to be able to at least schedule my flows from my Macbook, as all the environments are presolved and cached anyways. Am I doing something wrong or is this a known issue? Thanks so much!

@dry-beach-38304 Since this is about

@named_env

, you are the best person to help 🙂

I am out this week with (very) limited connectivity. For Issue #1, it has been reported before but have not had time to dig into it. For the second one, if the env resolves, it should work. I would make sure that all your steps have the same named env and all are on batch. My guess is that at least one step has a different environment.

To issue #2: I am using

named_env_base(name="bpy")

to give the whole Flow the same environment On MacOS

python dependency_check.py --environment=conda run

Result:

Base environment 'bpy' exists but is not available for architecture 'osx-arm64'

Correct ✅ Run it on batch then:

python dependency_check.py --environment=conda run --with batch

Result

Metaflow 2.12.8+netflix-ext(1.2.1) executing DependencyFlow for user:kokottc
Validating your flow...
    The graph looks good!
Running pylint...
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ...Pretty-printed STDOUT:
subdirs.linux-64.packages.__glibc
... ERROR
  Input should be a valid string [type=string_type, input_value=2.27, input_type=float]
    For further information visit <https://errors.pydantic.dev/2.8/v/string_type>

Same when trying to create the step function

python dependency_check.py --environment=conda step-functions create

Same error What I do not understand yet is why is it trying to bootstrap the conda environment on my machine in the first place if I am just trying to run the whole thing remote with batch / step functions. Flow is very simple

@named_env_base(name="bpy")
class DependencyFlow(FlowSpec):
    @step
    def start(self):
        print("Starting..")

        self.next(self.end)

    @step
    def end(self):
        print("Ending..")

ok — so fixed the first two problems (conda and mamba resolution from macos). For the other problem, could you rerun with

METAFLOW_DEBUG_CONDA=1

and send me the output so I can see what is happening exactly.

Thank you @dry-beach-38304, please excuse my late reply, I was on vacation! I upgraded to the latest version (1.2.2) and re-ran through my issues. • I can now start a flow from MacOS that uses a pre resolved environment for linux-64 in batch ✅ (this was the "other issue" that seems to be resolved now after the update) • Conda and mamba resolution from MacOS seems to have improved (do not get the previous error I received) but I am having an issue with a specific package: I am trying to pre-resolve the following environment for

--arch linux-64

from

osx-arm64

using

metaflow environment resolve -f env.yml --arch linux-64 --alias bpy_test

name: bpy
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip
  - pip:
      - bpy==4.0.0

I get the following error (

Unable to find installation candidates for bpy (4.0.0)

)

Virtual package spec: %s {'subdirs': {'linux-64': {'packages': {'__archspec': '1=m2', '__conda': '24.3.0=0', '__osx': '14.6.1=0', '__unix': '0=0', '__glibc': '2.27'}}}}
...
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/pypi_solver.py", line 402, in get_requirements
    link = chooser.choose_for(op.package)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/_vendor/poetry/installation/chooser.py", line 72, in choose_for
    raise RuntimeError(
RuntimeError: Unable to find installation candidates for bpy (4.0.0)

If I run the environment resolution from an actual

linux-64

machine it resolves without issues. (Using

metaflow environment resolve -f env.yml --alias bpy_test

) Any idea what the issue could be here?

yes — by default, we inject GLIBC 2.27 but it seems like bpy uses GLIBC 2.28 or more. You have a few choices here: • Modify this line: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/config/mfextinit_netflixext.py#L132 • use

METAFLOW_CONDA_DEFAULT_PACKAGES = {"linux-64": {"__glibc": "2.35"}}

• use

CONDA_OVERRIDE_GLIBC = "2.35"

(or whatever version you have on your machine). This is only injected when cross building which is why you don’t get the issue on linux directly since we just use the version from the system there.

Thank you @dry-beach-38304! This worked perfectly Just in case someone reads in this thread

• use

METAFLOW_CONDA_DEFAULT_PACKAGES = {"linux-64": {"__glibc": "2.35"}}

Should be

METAFLOW_CONDA_SYS_DEFAULT_PACKAGES

thanks. and sorry for getting the name wrong. Typed too fast 😞