Hey team to speed this up

Bootstrapping conda environment...(this could take a few minutes)

only thing I can do is set local conda config resolver to use "mamba" and make sure mamba is downloaded on machine right? Do we also need to create and activate the env with mamba and from there run the flow?

Just that - we are working on a much faster version as well 🙂

There is a faster version that is experimental that you can use as well. Install this one:

<https://pypi.org/project/metaflow-netflixext/>

Is it going to be merged into OSS metaflow so we get it via pip install metalfow at some point?

as @ancient-application-36103 said, a version of this is being worked on so it is likely that something like this will make it into the main

metaflow

package. One reason it has not yet is because I (at Netflix) can’t test all the configurations needed for OSS (it has exploded since our initial release of Metaflow and now includes 3 datastore backends and a multitude of ways to execute). It works great with my configuration internally and we have been testing it out but there may be kinks particularly around some of the other configurations (GS, Azure, etc). @cuddly-family-48123 recently tried it out on his setup with Azure and we worked out a few kinks (but it now works) but there may be some left depending on the exact configuration you use. It also optionally uses other OSS packages like

conda-lock

and

micromamba

which are newer and therefore may have some corner cases we haven’t seen yet (in fairness,

conda

still has weird corner case bugs) so we are trying to see if we can reduce the surface area of packages we depend on. All that being said, it works and although it requires one additional

pip install

, that should be the extent of it in terms of overheads so hopefully the barrier to try it out is low 🙂. And of course, any comment is most welcome. Thanks to @cuddly-family-48123 it now works on Azure for example 🙂.

Very awesome