Hey I am having issues using pre-resolved environments with

named_env

. I am developing from my

osx-arm64

and running steps via batch / step functions at some point. Basic env file

name: bpy
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip
  - opencv
  - pip:
    - bpy==4.0.0

Issue #1 - Pre resolving environments for linux-64 on MacOS As far as I understand this should work, according to the docs with

metaflow environment resolve -f env.yml --arch linux-64 --alias bpy_test

I get the following error from

conda-lock

when using

conda

for

METAFLOW_CONDA_DEPENDENCY_RESOLVER

Error: Got unexpected extra argument (virtual_yml.spec)

    raise CondaException(
metaflow_extensions.netflix_ext.plugins.conda.utils.CondaException: Binary command for Conda '['/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/bin/conda-lock', 'lock', '-f', 'pyproject.toml', '-p', 'linux-64', '--filename-template', '/tmp/conda-lock-gen-tmpqo43j8dk', '-k', 'explicit', '--conda', '/opt/homebrew/bin/conda', '--conda', '--virtual-package-spec', 'virtual_yml.spec']' returned error (2); see pretty-printed error above

When using the

mamba

deefault for

METAFLOW_CONDA_DEPENDENCY_RESOLVER

I get:

Resolving 1 environment ...Pretty-printed STDOUT:
/private/var/folders/xm/zy4s2pl903jbbr1hqzkhgrbm0000gp/T/tmp52xemgjw/virtual_yml.spec
Virtual package spec: %s {'subdirs': {'linux-64': {'packages': {'__archspec': '1=m2', '__conda': '24.3.0=0', '__osx': '14.5=0', '__unix': '0=0', '__glibc': 2.27}}}}

Pretty-printed STDERR:
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1404, in lock
    lock_func(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1112, in run_lock
    make_lock_files(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 318, in make_lock_files
    virtual_package_repo = virtual_package_repo_from_specification(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/conda_lock/virtual_package.py", line 260, in virtual_package_repo_from_specification
    spec = VirtualPackageSpec.parse_obj(data)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/pydantic/main.py", line 1135, in parse_obj
    return cls.model_validate(obj)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/metaflow/lib/python3.10/site-packages/pydantic/main.py", line 568, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for VirtualPackageSpec
subdirs.linux-64.packages.__glibc
  Input should be a valid string [type=string_type, input_value=2.27, input_type=float]
    For further information visit <https://errors.pydantic.dev/2.8/v/string_type>

Which seems to be a trivial pydantic parsing error (__glibc being a number not a string and it not wanting to parse it) Issue #2 - Starting a flow from MacOS that uses a pre resolved environment for linux-64 To circumvent Issue #1 I logged into a linux ec2 instance and resolved the env from there -> works check Then I hoped I could at least launch Flows that use that environment

@named_env(name="bpy_test")

in a

@batch

step. However I get the same errors like in Issue #1 even when running via step functions or

--with batch

I would love to be able to at least schedule my flows from my Macbook, as all the environments are presolved and cached anyways. Am I doing something wrong or is this a known issue? Thanks so much!

@dry-beach-38304 Since this is about

@named_env

, you are the best person to help 🙂

I am out this week with (very) limited connectivity. For Issue #1, it has been reported before but have not had time to dig into it. For the second one, if the env resolves, it should work. I would make sure that all your steps have the same named env and all are on batch. My guess is that at least one step has a different environment.

To issue #2: I am using

named_env_base(name="bpy")

to give the whole Flow the same environment On MacOS

python dependency_check.py --environment=conda run

Result:

Base environment 'bpy' exists but is not available for architecture 'osx-arm64'

Correct ✅ Run it on batch then:

python dependency_check.py --environment=conda run --with batch

Result

Metaflow 2.12.8+netflix-ext(1.2.1) executing DependencyFlow for user:kokottc
Validating your flow...
    The graph looks good!
Running pylint...
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ...Pretty-printed STDOUT:
subdirs.linux-64.packages.__glibc
... ERROR
  Input should be a valid string [type=string_type, input_value=2.27, input_type=float]
    For further information visit <https://errors.pydantic.dev/2.8/v/string_type>

Same when trying to create the step function

python dependency_check.py --environment=conda step-functions create

Same error What I do not understand yet is why is it trying to bootstrap the conda environment on my machine in the first place if I am just trying to run the whole thing remote with batch / step functions. Flow is very simple

@named_env_base(name="bpy")
class DependencyFlow(FlowSpec):
    @step
    def start(self):
        print("Starting..")

        self.next(self.end)

    @step
    def end(self):
        print("Ending..")

ok — so fixed the first two problems (conda and mamba resolution from macos). For the other problem, could you rerun with

METAFLOW_DEBUG_CONDA=1

and send me the output so I can see what is happening exactly.