packaging/dependency management question for y'all: I'm hoping to migrate our metaflow infra onto pants (but you could sub bazel in here as well). Right now, pants can build a "pex", which is effectively the code + venv with all dependencies, bundled into one archive. Would it be possible to point the metaflow "bootstrap script" at a pex in s3 (as opposed to a python file)? If not possible today, any pointers on what would need to change in order to enable it?

Technically you could extend Metaflow to add another type of MetaflowEnvironment, i.e. have PantsEnvironment/BazelEnvironment next to CondaEnvironment. Like you pointed out there is a lot of subtlety related to different architectures

Ah, nice... I'll poke around to see if this is worth it, or just a huge pain.

Hey @freezing-soccer-84261 — yes, as @average-beach-28850 mentioned, you can definitely add another environment type that would do this for you. If all you want is the bootstrap script to extract this pex file (and not have to build it), you can create your own environment and override/extend any of the methods there: https://github.com/Netflix/metaflow/blob/master/metaflow/metaflow_environment.py. To create a new environment, you can use the metaflow extensions mechanism (see here: https://github.com/Netflix/metaflow-extensions-template). To see at least one example of an overriden environment, you can look here: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/mfextinit_netflixext.py (this has a lot of code to implement a different conda but in your case, it should be much more straightforward). Based on what you mention here, I don’t think this would be too hard for you to do, if it’s just a matter of fetching a pex and extracting it in some way, it should be fairly do-able. If you want to start having metaflow create the pex for you, that may be a tad bit tougher but may also be do-able. If you have quesitons, let us know.

IIRC it doesnt even needs to be extracted, its like a static binary with your flow.py and all deps including metaflow. Haven't used

pants

but --environment=bazel was a long time dream of mine :)

very interested if you get that up and running. Some people here are using bazel too so it would be pretty awesome!

Yeah, it's "static-ish" aka you just need python. But once you have python and the

pex

, you can just invoke it like a normal binary. You may even be able to create a multi-platform pex (e.g. {amd64, arm64} cross {linux, mac} pex) at the cost of extra storage space... Though I'll probably just start with linux+amd64. I'm probably going to start working on this in the next couple of weeks, so I'll keep y'all posted!

awesome! Thanks!!!

Ok, just starting to look into this.. One question: how does the "code bundle" get uploaded / generated by metaflow cli?

yep — that would bundle it up.

Do the

add_to_package

hooks in MetaflowEnvironment also get called before "deploying" the code? I assume yes, but just wanted to check

yes. Well, they basically should feed into the packaging of the code: https://github.com/Netflix/metaflow/blob/master/metaflow/package.py#L121