Hi everyone, I have a couple questions regarding dependency management in the case of a monorepo. What we have: • A monorepo with different projects that rely on a common set of dependencies defined in a pyproject.toml at the root • A CI/CD pipeline that builds a custom Docker image, installing all common dependencies using poetry, everytime a PR is approved • When a flow is deployed, using AWS Step function, it will uses the custom Docker image where the (global) virtual environment has already been created From my understanding (please correct me if I'm wrong), Metaflow will install custom sets of dependencies at run time, for a given flow, only when using the

@conda

and

@pypi

decorators. In our case, every flow that is run on AWS Batch, whether it's deployed or not, will use the custom Docker image, which creates two main pain points: • When adding new dependencies, the inability to run a flow with batch without first having updated the pyproject.toml on the main branch (so that the CI/CD triggers itself and the Docker image is re built, including freshly added packages) • The obvious bottlenecks it will create as more projects are created, depending on larger and probably most likely conflicting dependencies What I would like is/my questions are: • The ability to specify dependencies at the project level ◦ From what I've seen, I feel like the recommended way is essentially to use the

@conda

or

@pypi

decorators, is that correct? ◦ Is there an official alternative if for some reason someone would like to stick with poetry or uv? • If it's not yet possible, I was wondering what is everyone's opinion on eventually having an official decorator, let's say

@uv(pyproject="pyproject.toml")

or

@uv(lock="uv.lock")

, that would allow to specify the dependencies at the flow level while allowing the creation of the venv at runtime. ◦ Similar to, from the doc about netflix's metaflow extensions,

A more full-fledged environment command allowing you to resolve environments using external requirements.txt or environment.yml files as well as inspect and rehydrate environments used in any previously run step.

? In any case, happy to hear your thoughts and recommendations on that topic, thanks a lot in advance! PS: the spin feature looks amazing 🤩

hey @adamant-eye-92351 — let me make sure I understand your ask (because I am a little confused). Is your goal: • all Metaflow projects use the global repo’s

pyproject.toml

and

uv.lock

but can use it prior to having built the DOcker image (ie: not necessarily after having pushed a PR • OR, you want various Metaflow flows to have different sets of dependencies

Hi @dry-beach-38304, sorry for the confusion! -> I want various Metaflow flows to have different sets of dependencies in the case of a monorepo

ok, and you would ideally like to have their own

pyproject.toml

and

uv.lock

(not those of the monorepo but others within each project)?

Given my knowledge of the matter, I'm not sure if it would, in fact, be ideal, but yes that's the idea!

ok. So there are a few things: • There is an official

uv

support: https://docs.metaflow.org/scaling/dependencies/uv ◦ It should find the closest uv.lock from the flow file ◦ It comes with caveats though: ▪︎ It is not necessarily as scalable as the

@pypi

or

@conda

decorators ▪︎ Basically it ships around the

uv.lock

and

pyproject.toml

files to the remote node and then runs uv there to hydrate the environment. ◦ It works quite well if that is what you need. • We are working on a full-fledged

uv

decorator for the Netflix Metaflow extensions you pointed to. That decorator will be

@resolved_uv

and will allow you to point to lock/pyproject files as well but the difference with the previous approach is that it will create a fully reproducible and scalable metaflow environment (basically, it will skip the “resolution” phase instead relying on what uv figured out) but will do all the rest (caching, etc). This should be out soon. • A few other comments: ◦ Under the hood,

@conda

and

@pypi

use a variety of tools — for the Netflix extensions, in some situations, poetry is used as well as uv. But yes, it’s not just one tool so if you want to use just one tool, that’s not the right solution for you (the

@uv

decorator basically only uses uv so that may work for you) ◦ The Netflix extensions are not “official” in the sense that they are not part of what you get when you do

pip install metaflow

but they are as close to that as you are going to get 🙂 — we use them internally and this is just our way of putting out additional functionality that, for various reasons, we do not put in the main package. In short, if the Netflix extensions fit your bill (or will fit your bill when we have more full uv support) you should feel free to use it and rest assured that they will be kept functional.

Thanks a lot for the detailed response Romain! So if I understand correctly what you wrote.. Using uv with specific

pyproject.toml

and

uv.lock

within each project's folder should work in our case. • When developping a project, I can run a flow by specifying

environment=uv

and it will work in local but also with Batch • In the case of remote execution I can still call my custom image (on which I don't have to install dependencies beforehand) and uv will create the venv on the fly • When deploying to production, I can specify

environment=uv

as well so that each project comes with its own sets of dependencies • This would still hold if I have two custom images, let's say one for CPU jobs and the other for GPU ones? Some caveats are, as you mentionned, is that it might not be as scalable as

@conda

or

@pypi

(although I'm not sure of the implications of this) -> It does not create a fully reproducible and scalable metaflow environment like those decorators do. Eventually

@resolved_uv

from the Netflix Metaflow extensions will overcome these caveats so if we are keen to using it along side the official Metaflow this should be an appropriate approach. Apart from the uv integration, am I right to understand that this architecture would still work while relying solely on the

@conda

and

@pypi

decorator? Thanks again for the help!

that should be correct. The non-scalable point is that each node will be effectively using uv to download and install the dependencies which can result in lots of request for large jobs. Not sure what you mean by “this architecture would still work”.

Hey Romain, I took some time to test things out and I think everything works as intended on my end 🙂 Thanks again for these answers! Looking forward to seeing the

@resolved_uv()

decorator!