Any one had success in getting the

ray

library to work with metaflow on apple silicon? I get the error message

nothing provides requested ray-tune 2.3.0

when attempting to include

"ray-tune": "2.3.0",

in the

@conda_base

decorator with PYTHON_VERSION = "3.10.13". I am able to successfully construct a conda python

3.10.13

environment with ray-tune 2.3.0

what is the full environment you are trying to work with?

This just ran on a mac with M1 chip

from metaflow import FlowSpec, step, conda_base

@conda_base(
    python='3.10.13', 
    libraries={
        'ray-tune': '2.3.0'
    }
)
class RayAppleSiliconSmokeTest(FlowSpec):

    @step
    def start(self):
        import ray
        ray.init()
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    RayAppleSiliconSmokeTest()

Thanks @crooked-jordan-29960 and @glamorous-night-24898! Here is the set of dependencies

TRAINING_PACKAGES = {
    "pandas": "2.1.1",
    "pyarrow": "11.0.0",
    "numpy": "1.26.0",
    "s3fs": "2023.4.0",
    "transformers": "4.32.1",
    "datasets": "2.14.6",
    "pytorch::pytorch": "2.1.1",
    "accelerate": "0.24.1",
    "conda-forge::hyperopt": "0.2.7",  
    "ray-tune": "2.3.0",
}

PYTHON_VERSION = "3.10.13"

@project(name="my_project")
@conda_base(packages=TRAINING_PACKAGES, python=PYTHON_VERSION)

How did you install micromamba? This runs on my macbook 🤔

micromamba create --yes --quiet --dry-run --no-extra-safety-checks --repodata-ttl=86400 --retry-clean-cache --prefix=/var/folders/0r/h_wgmt5j18z8t37d0kg2tr5c0000gn/T/tmpck71h5oq/prefix --channel=conda-forge ray-tune==2.3.0 python==3.10.13

Interesting. I have no recollection of the micromamba install. Will remove and give another go.

possibly, ray has been spotty with conda in my experience, but they distribute through pypi as first class. if it is ok for your workflow I'd check that route

micromamba can be weird too — so it’s not always on the package maintainers. You can try with mamba/conda which sometimes have better compatibility (I know, it’s weird since it shares some code with mamba at least) with the bleeding edge decorators. If you hae mamba and can resolve that enviornment, it may also work that way.

Well pypi had the same result

from metaflow import FlowSpec, step, pypi


class RayAppleSiliconSmokeTest(FlowSpec):
    @pypi(python="3.10.13", packages={"ray-tune": "2.3.0"})
    @step
    def start(self):
        import ray

        ray.init()
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    RayAppleSiliconSmokeTest()

Metaflow 2.10.8 executing RayAppleSiliconSmokeTest for user:cearl
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping virtual environment(s) ...
    Pip ran into an error while setting up environment:

    ERROR: Could not find a version that satisfies the requirement ray-tune==2.3.0 (from versions: none)
    ERROR: No matching distribution found for ray-tune==2.3.0

The package may be called

ray[tune]

. I don’t know if it makes a difference.

Virtual environment(s) bootstrapped!

Woot!

awesome!

Thanks so much 👏

Whr do you mean?

For example a local library packaged up by setup.py, etc. my-local-lib-0.1.0-py3-none-any.whl

The bleeding edge decorators support this (or should). Not sure about stock ones though.

Thanks! I was unsure of the precise syntax for getting load of requirements.txt files accomplished

You can use metaflow environment resolve to resolve a regular requirements.txt and give the environment a name or you can just use the decorators. Both should work. Let me know what your requirements are and I can give you both syntaxes.

Yes, so suppose I had a

my-local-lib-0.1.0-py3-none-any.whl

file that I wanted to include via decorators & that I've loaded bleeding edge. What syntax would you suggest?

something like:

@pypi(packages={"/root/flows/conda_tests/my-local-lib-0.1.0-py3-none-any.whl": ""})

should work (just tried it on a trivial example)

Beautiful