Hello, would love to learn more about fast bakery image builds feature. Is this only available on the outerbounds platform? Also would love to dig up more articles setting metaflow on AWS, already looking at https://docs.metaflow.org/v/r/metaflow-on-aws Thank you!

Hey Yudi! Long time no see!

Would like to get in a meeting with you, we (Jam and Tea Studio) got invited as part of AWS AI accelerator program

Yes - fast bakery is an outerbounds-only feature.

Depending on what you need, you may get a lot of the functionality (reusable environments, etc) using the bleeding edge pypi/conda decorators. It’s definitely not the same thing but in terms of reuse and speed, you get a lot of the similar benefits.

Thanks @dry-beach-38304, yeah one of the questions I have is actually fully utilize micromamba, as Savin has mentioned here https://github.com/Netflix/metaflow/issues/497, with that I still find my metaflow workflow (locally) takes a bit of time to start up, this is in a micromamba environment without using conda or pypi decorator I haven't spend enough time digging into Metaflow documentation yet so may have missed some obvious steps. Thank you!

the extension allows you to select mamba, conda or micromamba. Micromamba is usually fairly fast so not sure where time is being spent. Once your environment is resolved (which may take a bit of time), after that, it should be almost instantaneous to launch.

correct - fast bakery doesn't use conda/mamba/micromamba/pip behind the scenes and is usually 1-2 orders of magnitude faster than conda/mamba/micromamba/pip (which open source metaflow and extensions rely on). a good amount of speed up comes because the environment resolution and creation happens in the cloud, another good amount happens because our solver and image builder implementation is custom built for speed. in addition, the built images are hosted on a blob store and not ecr, dockerhub etc. which are painfully slow to upload to and download from - so the tasks start faster. also, we employ a few heuristics and some secret sauce in building these image layers so that cache hit rate is maximised both when building these images as well as consuming them.