Hi, I upgraded pydantic

version = "2.10.5"

->

version = "2.11.7"

and now seeing an error like below, is this a known issue?

File "/tmp/ray/session_2025-06-16_12-59-52_860923_1/runtime_resources/working_dir_files/_ray_pkg_e01e7abcca487ccc/metaflow/datastore/task_datastore.py", line 369, in load_artifacts
    yield name, pickle.loads(blob)
                ^^^^^^^^^^^^^^^^^^
AttributeError: 'FieldInfo' object has no attribute 'evaluated'

Hi everyone! I am new to MetaFlow and trying to understand how to deploy a flow using remote resources on different compute backends. In this tutorial, there is a flow with steps assigned either locally, on AWS Batch or on remote k8s. How is MetaFlow set up in this case? How does the config.json look like to support both AWS Batch and k8s? Also, in cases where this flow is deployed (for instance, on Argo), on which compute would each of these steps run? Thank you very much for your help!

hello, I'm trying to use the

metaflow-dev up

command... it seems to be mostly working, but it seems to keep asking for my password when it's starting all of the services. the weirder thing is that it keeps saying my password is wrong, so i'm not even sure what password it's trying to ask for (is it something related to the minikube/argo roles or something like that? I have no idea)- any thoughts on what I should be trying to fix this?

Hello I tried to apply metaflow.tf to our own internal infra, and I got the following error and really appreciate any pointer/help.

Hi everyone, I'm having a bit of a silly problem using metaflow as a dependency in a pycharm project. When trying to import things like

from metaflow import FlowSpec, step, kubernetes, retry

FlowSpec and step import fine, but I get Cannot find reference errors for kubernetes and retry. If I launch a python console or a notebook the imports work fine so it seems like an indexing issue. I've tried the typical invalidate caches, with no luck. I'd rather not just disable the warnings if possible. Has anyone had similar issues or have a solution? Many thanks

For anyone using the Metaflow Netflix Extensions — it’s been updated to 1.3.0 (compatible with the new packaging framework introduced in Metaflow 2.16.0). Apologies for taking so long to update, a bug in Mamba 2.3.1 was preventing the tests from running correctly (2.3.2 was released 2 days ago) and I wanted to make sure things were not too broken. Lots of bug fixes and a few new features. It is not compatible with Metaflow < 2.16.0. More features coming soon as well.

Hey everyone, There’s a lot of encouraging new developments (company has been mostly using Metaflow<2.12). I wonder if it is now possible with some reusable utility interfaces, to wrap any single function with a decorator to make it into a runnable flow without having to write a whole separate flow script. Particularly, we have been using Runner API in Airflow DAGs to run flows on AWS Batch. Furthermore, it makes sense as part of this to be able to unit test such functions. I’m encouraged by the latest Metaflow 2.18 blog post, but if the entry point is as simple as an RESTful endpoint say via FastAPI, it would be an encouraging push to update to newer Metaflow versions (with internal training).

conditionals are now GA

old thread but conditionals are now GA

old thread, but finally we got around to implementing it

Hey everyone, I have a question regarding cli command in metaflow. Is there a way to customize the cli command after the python file, like e.g.

myflow.py

. I want to create something like

python myflow.py data check

``python myflow.py data plan` next to the existing ones

python myflow.py run

``python myflow.py show` , any idea if it's possible to do this? I see it seems possible to use the metaflow-extension-template? And I also wonder if there's some other way to achieve the goal. like the customizing stepdecorators using a generator function. Thanks in advance!

👋🏻 Hey everyone! I’m an experienced AI Agent Developer open to new projects or full-time roles. I specialize in building autonomous agents using GPT-4, LangChain, AutoGen, CrewAI, and other advanced frameworks. What I Do: • Autonomous research & data-gathering bots • Multi-agent systems for delegation & collaboration • AI assistants with memory, planning & tool use • Trading bots, IVR agents, customer support agents & more Tech Stack: • Python, TypeScript, Go, C++ • LangChain,Langraph, AutoGen, ReAct, CrewAI • OpenAI, Claude, Hugging Face, Playwright, API integrations I'm especially interested in ambitious startups, Web3 projects, and next-gen AI tools. Feel free to reach out if you’re building something exciting — happy to chat!

does metaflow support configuring `concurrencyPolicy`for argo CronWorkflow, https://argo-workflows.readthedocs.io/en/latest/cron-workflows/?

I'm running into a problem when trying to resolving named environments with packages from a private pypi repository (AWS codeartifact). I want to use pip environment variables to pass the extra-index-url, e.g. through PIP_EXTRA_INDEX_URL. My understanding from other posts here is that what I'm trying to do should work. I've tried

PIP_EXTRA_INDEX_URL=<url_with_temporary_token> metaflow environment resolve -r requirements.txt --alias test_env

which fails already in the first step of resolving the environment. It just doesn't have access to the private repo and fails to resolve any private packages.

ERROR: Could not find a version that satisfies the requirement <private-package>==0.1 (from versions: none)
ERROR: No matching distribution found for <private-package>==0.1

Strangely, when creating a pip.conf that contains the extra-index-url it almost works. When running

PIP_CONFIG_FILE=pip.conf metaflow environment resolve -r requirements.txt --alias test_env

Metaflow is able to resolve the environment including the private packages and their dependencies, but in the step where it downloads the packages from the web, I get a

401 Client Error: Unauthorized for url:

for the private repo. It looks like when trying to download from the web it doesn't use the pip.conf anymore but instead tries to directly access the url prepared earlier in the process (without the token) and hence fails. I see that there is some auth handling here but this doesn't seem to do the thing that's necessary for my use case. I'm using metaflow==2.15.21 and metaflow-netflixext==1.2.3.

Hi everyone!! I am running

netflixoss/metaflow_metadata_service:v2.4.12

for the UI service startup. When running the command i get

"/opt/latest/bin/python3 -m services.ui_backend_service.ui_server": stat /opt/latest/bin/python3 -m services.ui_backend_service.ui_server: no such file or directory

. Do you know if the path of the container init script changed ? 😁

Hey All, while I work on testing the decorators and test it with Agentic workflow. I had some generic questions for the team. 1. Who are the primary users? Do Metaflow intend to cater them, or expand to other professions/fields in the future? 2. In which direction is Metaflow moving at this stage? What is the vision of the product?

hey Metaflow team, I wanted to see whether it's feasible to do something like version and track the packages built for each flow (not the runs but rather the code backing the different runs). Basically, I have an org where there are people who write code around statistics and other math functions but are not engineering savvy. With this in mind, I wanted to see whether I could make something like the following: 1. I define a somewhat abstract flow that has static inputs and outputs 2. Someone else would just give me a function that adheres to the static interface (a simple statistical function that takes some well-defined data inputs and produces some well-defined outputs) 3. I'm able to run the same higher level flow where it just handles some of the data input handling and the function output handling in a consistent way 4. I build some index of the packages build & run for the different custom functions given so that I can: a. Re-run that specific package with different data inputs potentially b. Produce some index of the packages and runs The main goal here is to try to close the gap between the more engineer-y interface of Metaflow and the technical capabilities of people who are more math/statistics focused... but I think there are a few things that are hard: 1. Dynamically creating a

FlowSpec

subclass such that one of the steps calls whatever random function gets thrown into the mix 2. Managing the underlying packages backing the different runs I might be way over-complicating this, so I would appreciate any thoughts or pointers! I'm happy to dig into the code and work through some of the internal APIs for this. I do realize there are security concerns with executing arbitrary functions in this manner, but I think that is manageable in the environment we work in

Hi all, I'm new to metaflow trying to help setup a poc environment. We already have Argo workflows deployed, and multiple kubernetes clusters in Azure and onprem. I don't really want to just run the pre-built terraform config as we already have many of the resources we need, included storage accounts etc. Can anyone point me to documentation on this? Should I be starting with the helm charts here? https://github.com/outerbounds/metaflow-tools/tree/master/charts/metaflow Many thanks!

In the event-trigger docs, there's a mention of setting an event-trigger via the config. But when I follow the example given, I get a Python error saying that the config symbol could not be found. Are the docs up to date? I also found https://github.com/outerbounds/config-examples/blob/83ece9c8e916f4d5f549e7bb717cbfca80b5b555/flow-level/toplevel.py#L15 which indicates using

config_expr

Hi, please clarify, maybe somebody have an example of connection metaflow with argo-workflows, without argo-events for this moment?

Hey all! Could you ballpark the cost of deploying Metaflow on GCP? For a deployment just sitting idle, that is. Thanks!

announcement speaker PSA: Outerbounds Office Hours We’ve got two great talks coming up: • Michael Bao (Netflix) – 📆 Aug 19 ⏰ 9am PT ◦ Building a scalable GPU training service with Metaflow. Learn how they migrated PyTorch Lightning scripts with zero model/data changes, added custom decorators for configurability, and deploy seamlessly via CI/CD. ◦ 📍 RSVP: http://lu.ma/office-hours-with-netflix • Gergely Daróczi (Spare Cores) – 📆 Sep 2 ⏰ 9am PT ◦ Right-sizing compute for AI/ML workloads. See their open-source

@track_resources

package in action, with interactive reports that optimize CPU/GPU usage and cut cloud costs—plus a live demo. ◦ 📍 RSVP: http://lu.ma/office-hours-with-spare-cores ✨ Why attend? Outerbounds Office Hours are a space to: • 💡 Learn practical strategies from real-world Metaflow use cases • 🤝 Connect directly with the Outerbounds team & ML/infra community • 🧰 Discover patterns you can reuse in your own workflows • 🚀 Stay inspired by how others are solving cutting-edge challenges 🙋 Want to present? Presenting is a chance to showcase your work, get feedback from peers, and build visibility in the ML/infra ecosystem. Reach out if you’d like to share your story in a future session!

Hey, do metaflow support cycles? My 2 cents says no as it is a DAG. I want to test a reasoning/feedback based AI Agent, with below system. This system covers both conditions and looping.

Hey 👋 I am building a POC for my company to demo an Analytics Processing Workflow using Metaflow. The Steps in the flow require access to an existing S3. If I run the flow by just doing

aws sso login

and then run the flow; Metaflow is able to pickup the AWS credentials ok. But I am having trouble when trying to run the same flow using the

metaflow-dev

stack. Because the pods can't pickup the aws credentials from the process when I run the flow in

metaflow-dev shell

. Wondering if anyone has been able to mount aws credential Secrets from a local shell into pods spun up by

metaflow-dev

. I am trying to set up a good DevExp for the engineering org. Thanks!

Hello, we're trying set up a metaflow production environment on EKS following this readme, and we have a few questions. In particular, we're wondering how we can set up the UI services, in addition to the argo, metadata services included in the example. Also came cross this resource for deploying metadata and UI but not sure if it's the recommended route - would someone be able to advise?

Hi, Is there a way to conditionally link steps. Say that I have some external condition that determines what downstream steps that should be executed. If the condition is fulfilled we run branch A and if it is not we run branch B. The workaround is to have branch A and B as separate flows that are triggered by the first steps, but Ideally I would like to do it in the same flow. Something like

class MyFlow(FlowSpec):

@step
def first_step(self):
   condition = os.getenv(CONDITION)

   if condition:
      self.next(step_a1)
   else:
      self.next(step_b1)

@step
def step_a1(self):
   self.next(step_a2)

@step
def step_b1(self):
   self.next(step_b2)

When I have tried something similar I usually get an error saying that some steps can't be reached from the start step. Is there a way to ignore those steps if the condition isn't met?

Hi, I had another question. Has anyone here managed to trigger a workflow based on a file landing in a specified folder in an S3 bucket? If so, what's the best practice to do it? I already added @trigger(event="file_uploaded") at the top of the flow, but the rest is where the jank starts. I basically start a master node in a Kubernetes cluster which runs a python script on startup. The python script is an infinite while loop which just looks if there is a file in the S3 bucket, and if so, publishes an ArgoEvent with the name "file_uploaded". My intuition tells me there's gotta be a less Neanderthal way of doing it, but I can't seem to figure it out. Any help would be much appreciated. Thanks in advance :)

we're building an extension following https://github.com/Netflix/metaflow-extensions-template/blob/master/README.md. and we'd like to do some validation when people needs to interact with k8s, e.g. via

--with kubernetes

or

argo-workflows

. Currently, we put the validation code in

config/mfextinit_org.py

, but that gets run even when user just tries to run the flow locally, which is annoying, esp. when the validation fails, and user gets confused why is k8s even relevant when they run the flow locally. What would be the recommended way to put such checks? An equivalent question is how to know if the user command needs to interact with k8s or not reliably from the extension?

Is there a way to disable pylint through env var or config instead of CLI? we enforce pylint for the whole repo, so feels unnecessary to run pylint for individual flow file. It's annoying to specify

--no-pylint

everytime user needs to run a flow.

?