Hi, is it possible or planned to support uv for the virtual env besides pyenv and conda?

uv/pip/poetry are implementation details for @pypi and @conda. curious, besides any potential speed up, what’s the use case?

I’ll say its mostly a quality of life feature The latest versions of uv added several features besides speed: e.g., • 'uv init' to start a project and create the pyproject.toml • 'uv run' to execute the code with the .venv in the current directory without needing to explicitly activate the venv • 'uv add/sync' to manage dependencies It's similar to what Rye and Pixi offer, but it also has the older API support for the vanilla pip syntax. It seems to be gaining more traction in the community, and it's a bummer to have to move back to an older package manager in Metaflow.

You can use metaflow with uv/pixi etc. - there isn’t any limitation

If you mean about using it as a backend, I’ve been looking to add uv as a backend of the Netflix extension’s resolver but, for it to be an easy drop in, I am waiting on https://github.com/astral-sh/uv/issues/1442. I may go back and see if I can do it another way but for now that’s where I’m stuck. Internally, we use that to create the “report” that tells metaflow which packages are to be used and uv does not support it (yet). It’s mostly for speed though. I’ll also probably directly support pyproject.toml as specifications (right now it just accepts requirements.txt and environment.yml files). In terms of quality of life, I guess it depends what your workflow looks like: • for the

uv init

, there is no need for that with metaflow environments — just writing your flow will “init” the enviornment for you. It’s true there is no portable file for it. That’s something that could potentially be added. • for

uv run

, running a metaflow flow will run it in the proper enviroment (and it should be fast the second time around) • for

uv add

, it should also do it “automatically” when you try re-running the flow. The nextflix extensions has a bit more in terms of “commands” for environments so you can also check that out and see if that helps a bit.

Thanks a lot for the detail answers

@dry-beach-38304 supporting uv as a backend, and/or generally

pyproject.toml

in the extension would be a really great add, especially if it can support different

dependency-groups

. I hope that the above

uv

issue can be resolved soon! 🙏

thanks for the feedback/comment. It’s still on my list but haven’t made too too much progress.

👀 is this (and this) what I think it is?? blob excited

haha it is! we are working on a doc PR but very simply you can do

uv run python flow.py --environment=uv run --with kubernetes

a quick note: this copies over your uv environment but does not manage dependencies the way conda/pypi do. The idea is to improve it going forward. But just want to clarify that it doesn’t do the same thing (yet)

nice!! this is so cool everyone. thanks for all the work. FYI: It works locally for me. But with batch, it's failing with

OSError: Readme file does not exist: README.md

among other things. I wonder if that's because I also have a python package that it's wanting to install in editable mode (I normally symlink that package under my

flows

directory). Maybe a

--no-install-project

flag is needed. Just wanted to flag it here, happy to wait as this functionality develops further.

are you able to share a reproducible example?

Sure, see this repo. The problem appears to stem from the build system

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

which is needed to treat my code as a proper python package outside of Metaflow. If this is commented out, the uv dependencies build fine, and the package code is handled via a symlink.

@refined-insurance-51213 this is my dream ❤️ . I love your example Being able to 1. put the flow files wherever you want 2. treat the helpers as a proper python package so you don't need to add

src.

to your imports a. makes it so unit tests and flows can have the same import path b. and just generally is more consistent with Python standards (not putting

src.

in imports) 3. have those packages end up in the metaflow code package even if they aren't in a child directly of

flow.py

a. achieve this without having to build/publish the folder to PyPI or do a self-referential

git+https://...

statement (really overkill e.g. for 3 helper files)

It’s coming :) (at least components of it).

Following up: I was able to get it working both locally and with AWS Batch by 1. Removing the

readme = "README.md"

from the

pyproject.toml

(obviously 🙂) 2. Instead of symlinking the package name directly (here, called

metaflow_uv

), symlink

src

because that's what

uv

expects to find to install the local package in editable mode. So now the package is actually installed when the flow is running, and not hanging out in the local path of the flow itself. (Distinction without a difference in this case, but necessary for

uv

) The repo is updated.