Hi there, I am reading the documentation on https://docs.metaflow.org/scaling/dependencies/internals. Is there a way to disable the below local <-> remote environment sync?

It ships the locally resolved environment for remote execution, even when the remote environment uses a different operating system and CPU architecture than the client (OS X vs. Linux).

I have also read https://outerbounds-community.slack.com/archives/C02116BBNTU/p1697359042901309?thread_ts=1697198947.028709&amp;cid=C02116BBNTU and I am still a bit confused what the workflow is when you're local hardware is different from your remote. Context: local is an Arm Mac developer, remote is Batch GPU multi-node. The below would resolve on the remote, but not locally, hence user is not able to launch flow.

@conda(libraries={"pytorch::pytorch-cuda": "12.1"})

There are two things here: storing the packages in a known fast location and resolving for the correct architecture. The uploading of packages is the first thing. For thr second thing, you should be able to get it to resolve properly even on thr Mac. Are you using the bleeding edge decorators or the standard ones? For the bleeding edge ones you can definitely configure it to “pretend” to have GPUs (and that should work even on the standard one), both will say that if you execute on batch you resolve on Linux and the bleeding edge ones also allow you to specify a glibc version. I can provide additional details in a bit if it helps (not at my desk). If you are using the bleeding edge ones, run with METAFLOW_DEBUG_CONDA=1 and I can also help further.

Thanks for your quick reply @dry-beach-38304. For context, we haven't been using the bleeding edge decorator (I take it this is it https://github.com/Netflix/metaflow-nflx-extensions), but will read through the documentation and try again. If you don't mind linking me to how we could "pretend" to have GPUs for the local resolve to pass (standard one https://docs.metaflow.org/api/step-decorators/conda or the bleeding edge)? edit: we are also happy to use the

@pypi

decorator if that simplifies the process.

the pypi decorator may work — (the GLIBC issues may show up there though depending on your setup). The typical way to use cuda is to set the env var CONDA_OVERRIDE_CUDA=12.1 (or whatever the highest version your kernel module supports). Note that for pytorch specifically though, the recommended approach is something like this:

pytorch pytorch-cuda=12.1 -c pytorch -c nvidia

so it means you need to add both the pytorch and nvidia channels. In the bleeding edge decorator, you can just add a

channels

argument to your

@conda

decorator (it’s a list that can contain

["pytorch", "nvidia"]

. In the standard decorator, I don’t think there is a way so you may have to add the channels directly to the configuration for micromamba (which is what the standard decorator uses — the bleeding edge ones can use conda, mamba or micromamba).

Thanks for the explanation. I will test it with the bleeding edge decorator.