Let me know if that doesn’t work for you.

conda not even installed

Under the hood it builds the wheel for you and caches it. It will refuse to do it if you are running on a Mac and want to target a Linux box for example.

I am planning on relaxing thst and allowing that for no arch builds at least.

You can have a flow that resolves your environmenr first and then uses it.

And once it is done once you are typically good to go.

Even if you want to then resolve in a mac. It just needs to run once on a Linux box.

you just need to run that build flow once and you should be set.

so

When you run this, here is what happens: • Metaflow will resolve the environment and figure out all the packages it needs • For the CLIP package, it will realize it’s not a .whl package and it wants .whl so it can reliably install them the next time it needs the environment • It then will build the wheel on the machine that is running the resolve (so in your case, the machine that is running the metaflow step so a batch instance somewhere) • Metaflow will then cache all the packages it needed for this environment to S3 so that it can access them more quickly next time (and doesn’t have to download them from the web). So that means it downloads any packages it needs and then uploads them to S3. In the case of the CLIP package, it will upload the wheel directly to S3 as well • The next time you run (from your mac), Metaflow does the same thing: ◦ It tries to resolve the packages ◦ It sees that it needs a CLIP package and realizes it is not a wheel ◦ Metaflow is lazy here and instead of building it, it will first check to see if it has already built it ◦ Here it will see that it has indeed built it already so it doesn’t need to rebuild it ◦ It will upload to S3 any additional packages needed (it only uploads packages that are not already in S3 so here this means it won’t upload any more CLIP related stuff) • When it runs (on batch again, this time the real flow): ◦ Packages are downloaded from S3 (including the CLIP package built by that unrelated build flow) So that’s why I am saying that if all goes well, running that build flow once is all you need (until CLIP gets updated in which case Metaflow won’t “find” the prebuilt wheel because it won’t be looking for the same one since CLIP would have moved to a different commit). There are some subtleties to that point but it’s generally correct.

Here's what I'm trying

import subprocess
import sys
import tempfile

from metaflow import FlowSpec, conda, step

class BuildClipWheel(FlowSpec):

    @conda(disabled=True)
    @step
    def start(self):
        with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8") as req_file:
            req_file.write("clip @ git+<https://github.com/openai/CLIP.git@a1d071733d7111c9c014f024669f959182114e33>")
            req_file.flush()
            subprocess.check_call(
                [
                    sys.executable,
                    "-m",
                    "metaflow.cmd.main_cli",
                    "environment",
                    "resolve",
                    "-r",
                    req_file.name
                ]
            )
        self.next(self.end)
    @step
    def end(self):
        import clip

if __name__ == "__main__":
    BuildClipWheel()

And the error I'm currently getting:

Metaflow 2.10.2+netflix-ext(1.0.4) executing BuildClipWheel for user:tgallagher
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Internal error
Traceback (most recent call last):
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/cli.py", line 691, in wrapper
    return func(args, kwargs)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/cli.py", line 828, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/cli.py", line 886, in before_run
    obj.package = MetaflowPackage(
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 109, in init_environment
    env_type, arch, req, base_env = self.extract_merged_reqs_for_step(
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 579, in extract_merged_reqs_for_step
    conda.virtual_packages,
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 225, in virtual_packages
    if "virtual_pkgs" in self._info:
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2172, in _info
    self._find_conda_binary()
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1716, in _find_conda_binary
    self._ensure_local_conda()
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1757, in _ensure_local_conda
    err = self._validate_conda_installation()
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1952, in _validate_conda_installation
    if "micromamba version" in self._info_no_lock:
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2178, in _info_no_lock
    self._cached_info = json.loads(self.call_conda(["info", "--json"]))
  File "/Users/tgallagher/Library/Python/3.9/lib/python/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 259, in call_conda
    if self._bins is None or self._bins[binary] is None:
KeyError: 'conda'

Oh I think I know.

ok, so I traced the code and your image doesn’t have

mamba

(if you didn’t change any of the defaults). There is a bug in the sense that the error message is not clear so I will push a fix for that shortly. The image this runs on needs to have one of

conda

,

mamba

or

micromamba

and you need to configure it as

METAFLOW_CONDA_DEPENDENCY_RESOLVER

(it defaults to

mamba

right now). So: • if your image has

mamba

, leave the config as is • if your image has

micromamba

, set

METAFLOW_CONDA_DEPENDENCY_RESOLVER=micromamba

• etc. The executable needs to be in the path (the code does

which

to find it.

note as well that

end

won’t have

clip

in the env (in your code).

you just resolved it with nothing else.

Small correction: it’s not finding conda on your local box (so even before going off to batch). For this use case you need mamba (or something like it) on both your local box and the remote one. You technically don’t need it on your local box if you don’t have any @conda or @pypi decorators (which you don’t really need here

You need a regular non conda flow and run —with batch

After that though you will need conda locally on your mac to do anything else involving conda.

The previous link I sent to install conda will work on your Mac too.

I'm trying to avoid using conda for the time being and the cutting edge extensions seem to depend on it in general.

I was just about to post an update. I did get it working.

What do you mean “avoid conda”? All pypi/conda decorators depend on conda (or some variation of it)

(I did find another bug in the process so this is good 🙂)

If anyone is wondering for future reference, this is one way to do it: • have micromamba locally on your machine • Use the following code (very closely adapted from the one above):

import os
import subprocess
import sys
import tempfile

from metaflow import FlowSpec, conda, step

class BuildClipWheel(FlowSpec):

    @conda
    @step
    def start(self):
        path = os.environ.get("PATH", "")
        home_path = os.environ.get("HOME", "~")
        path += ":%s/.local/bin" % home_path
        with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8") as req_file:
            req_file.write("clip @ git+<https://github.com/openai/CLIP.git@a1d071733d7111c9c014f024669f959182114e33>")
            req_file.flush()
            subprocess.check_call(
                [
                    sys.executable,
                    "-m",
                    "metaflow.cmd.main_cli",
                    "environment",
                    "resolve",
                    "-r",
                    req_file.name
                ],
                env={**os.environ, "CONDA_CHANNELS": "conda-forge", "PATH": path},
            )
        self.next(self.end)

    @step
    def end(self):
        print("All done")

if __name__ == "__main__":
    BuildClipWheel()

◦ This does use

@conda

on the resolving step and the reason for that is that you can then take advantage of Metaflow’s auto download of micromamba. ◦ You need to pass it the channels (most likely) because the installed micromamba may not have those channels configured ◦ The path variable makes sure that it can find the installed micromamba (Metaflow here installs it in ~/.local/bin) • Run using:

CONDA_CHANNELS=conda-forge METAFLOW_CONDA_DEPENDENCY_RESOLVER=micromamba python ./flow.py --environment=conda run --with batch

◦ Same reason to pass the channels (if your micromamba properly reports channels not need to do that). You use the

micromamba

resolver here too so that it can also be used on batch (else Metaflow will look for the default resolver which is

mamba

) Once that it done, the package will be built for linux-64 and you can then do stuff like

metaflow environment resolve --dry-run -r test.txt --arch linux-64

where

test.txt

includes the CLIP library and it will work fine (as well as including it in any

@pypi

decorator. (caveat: I did find a tiny bug in this process so this will work from 1.0.5 onwards — it will also give a better error message than the one you initially got).

clip is also a noarch package so technically, it can be built cross platform. I do plan on improving that aspect but am trying to find a way to do it “cheaply”