Hello, I stumbled upon metaflow today and I was wondering if it supports heterogeneous clusters ? The use case goes as following: Let's say you want to process videos with a deep learning model on gpus. Decoding videos requires a lot of compute, and currently is the cheapest on arm cores with instances like c7g on aws. Deep learning model requires gpus to run. Using gpu instance only to both decode and process the video often lead to underprovisionning of cpu compute. You can dramatically reduce the cost by decoding videos on dedicated cpu instances and transfer the decoded video through a socket to gpu instances to be processed. This in turns means that you need to setup a heterogenous cluster with a hybrid workflow, one for cpu instance and one for gpu instance. Is this scenario supported by metaflow ?

Hi does Metaflow support GCP as a private pypi index?

when running cron argo workflows, is it possible to specify that concurrent runs are not allowed, meaning that if a run is already in progress and we have hit the next trigger time, it will not allow another run start?

. nvm solved

Hi, I have a flow that I want to trigger based on other flows using

@trigger_on_finish

. The flows that I want to base the trigger on can change so I would like to specify the flow using a command line argument (or some other way of supplying arguments). Basically I want to do something like this

parser = parser = argparse.ArgumentParser()
parser.add_argument("--flow")
args = parser.parse_args()

@trigger_on_finish(flow=args.flow)
class MyFlow(FlowSpec):
    @step
    def some_step(self)

I am deploying the pipelines to Airflow so I don't think I can use

Runner

to specify these arguments. When trying to run

python my_flow.py --flow flow_one airflow create my_dag.py

I get

error: unrecognized arguments: airflow create my_dag.py

Is there a way to specify arguments to flow decorators from outside of the python script?

I just wanted to ask if the custom decorators introduced work with argoworkflows ? I.e I can compose a flow and then deploy to argo accordingly or is it not supported yet ? Also can those custom decorators used with @kubernetes decorators ?

@square-wire-39606 Is there any update on the ability to integrate metaflow with a gitops / argocd deployment strategy? I saw your comment here and both you and @straight-shampoo-11124 were referencing a related update at various points (thread), but not sure what updates you were referencing? Like others, we would like to be using our metaflow with gitops (push workflow templates to gitops repo, argocd publishes), but given the need to run the

argo-workflows create

command, we instead have to run a metaflow sync client in each of our deployment envs, making metaflow a CD-pattern outlier.

is there a way to configure how many tasks get run at the same time with

foreach

?

My team is really excited about the new

uv

feature. How do we use it in conjunction with google artifact registries? We have some private python artifacts that we would use in our flows.

Hello quick question, if I have a docker image in a private docker repo, where do I need to share my credentials so it can be used by metaflow in @batch ? I only see the parameter image. I saw this old issue https://github.com/Netflix/metaflow/issues/319. Is it still the only way ? Is there a way with environment variable and secret ?

Hello, when using local resume is there a way to suppress dumping stdout/stderr to the terminal?

hi everyone, running into the following error on

metaflow==2.15.18

and

micromamba v1.5.10

haven’t seen this in a while, but saw some mentions about micromamba2 being a problem, however i’m on 1.5.10. any other things worth checking?

critical libmamba Download error (1) Unsupported protocol [<s3://metaflow-bucket/metaflow/conda_env/packages/conda/packages/conda/conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.1-hb9d3cd8_1.conda/libnsl-2.0.1-hb9d3cd8_1.tar.bz2/6786aa05708c1e034b22564380b19f75/libnsl-2.0.1-hb9d3cd8_1.tar.bz2>]
Protocol "s3" not supported

Hi we are seeing an bug with Metaflow

2.15.17

where objects in the config are not available in future steps when they are assigned as instance variables.

Friendly reminder that this is happening at 9am today! Join us for a deep dive into AI workflows at scale with Nick Schmeller from Lacework! Nick will share how his team uses Metaflow to power Lacework’s AI-driven security tools—highlighting Riskwatch, a system that analyzes huge volumes of open-source code to find real vulnerabilities. Expect insights into: 🔹 Scaling with dynamic fan-out 🔹 Custom decorators & artifact storage 🔹 Advanced features like memoization & input validation 📅 When: July 8 📍RSVP: lu.ma/office-hours-with-lacework

Hi. I'm new to Metaflow and I'm trying to get a grip on how environments are handled. Specifically I'm trying to do the following: I have a flow that creates an ML model and I want to be able to execute that flow locally (architecture osx-arm64). Afterwards I might want to use that model in a different flow. In that flow I'd like to use the same environment that was used to create it so that I can be sure that it works the same. This should ideally also work when that flow is deployed on a different architecture (linux-64). The simplest way for making environments easily reusable across flows seems to be via named environments from the netflix extensions package, but it doesn't seem to be possible to resolve two environments on different architectures and give them the same name (correct?). I.e. something like this doesn't result in both of them having the same alias:

metaflow environment resolve --arch osx-arm64 --arch linux-64 --force -r requirements.txt --alias my_test_env

There is the option of using

fetch_at_exec

to determine the name of the named environment at runtime. Then I could resolve the environments for both architectures and give them different aliases, e.g.

my_test_env:<arch>

, and define through runtime information, which is the correct one. I'm just wondering whether this is an appropriate way of doing it. Is there a better way?

Hey, is there a good way to make an ArgoEvent published in a test/user namespace not trigger e.g. the production flow but only flows in that same namespace?

Lineage update: This week, I'm going all on on this POC of to show data lineage between airflow and metaflow. If anyone here has ever used DataHub or OpenLineage instrumentation before, I'd love to talk here

Hi! Have a question regarding using

metaflow_ray

on argo workflows, I've installed JobSet to the same namespace with my argo workflows runner namespace, and had the permission added to my service account, but hitting some error at the moment about the webhook service path, is there anyway i can modify this value? can't seem to find it either from the metaflow configs or anywhere, thanks in advance!

Hi! In my k8s deployment, I have tried to change the datastore location, now I have this in my helm chart:

metaflow-ui:
  uiBackend:
    metaflowDefaultDatastore: "s3"
    metaflowDatastoreSysRootS3: "<s3://metaflow>"
    metaflowS3EndpointURL: "<http://minio.caic:9000>"

plus AWS_SECRET_* values through

env

. Previously I had DefaultDatastore set to "local" I have restarted both deployments for mataflow-ui. Now I'm running yet another flow, and in UI it still shows me Since this run uses local datastore instead of cloud services, some information may be incomplete. And of course there are no data in my minio bucket Is my s3 configuration correct? Do I need to drop the database or deploy the backend service too to see the effect?

Hello team! I was wondering if any of you had this error when loading cards from a flow:

URL/api/flows/ComponentDemoFlow/runs/4543/steps/start/tasks/222732/cards/blank/0490a0de7c024207af081c2187dcea5d?embed=true

returns

[Errno 2] No such file or directory: '/root/services/ui_backend_service/ui/index.html'

. • We are using a custom DNS behind a vpn in

http

• Recent versions for all services:

metaflow-ui.uiBackend.image.tag=2.4.13-2-g70af4ed

,

metaflow-service.image.tag=2.4.13-2-g70af4ed

,

metaflow-ui.uiStatic.image.tag=1.3.5-123-g95238f8-obp

Thank you!

Hello everyone, I setup metaflow on prem with argo events and workflows. I wanted to ask if we have something similar to sensors that react on events like for example uploading a file to s3 ? Also can I connect this on prem deployment to outerbounds UI ? I specifically mean this https://docs.outerbounds.com/outerbounds/connect-to-outerbounds/ The third option here ?

Hi, I hope this is the right channel for such question. I've deployed metaflow using helm chart to y k8s cluster (using https://github.com/outerbounds/metaflow-tools/tree/master/charts/metaflow). I connected it to my postgresql db, this parts seems to work fine. I have 3 services running, metaflow-service, -ui and -ui-static. So far so good. I've set

METAFLOW_DEFAULT_DATASTORE=local

as I did not find out how to set up credentials for s3/minio. Now, I port-forward the service (port 8080) to localhost and with

METAFLOW_SERVICE_URL=<http://localhost:8080/>

I'm able to run example flow; from the logs on service pod it seems it is connecting to remote server which is nice. I can even check the flow status on port 8083, so it seems this is logged corerctly OK, but now I cannot see anything in the UI. The UI (running through ingress) just shows

Error loading data

on empty dashboard page, nothing about my flow and nothing more specific about the error, just

Unknown error

/

generic-error

. I assume some part of my stack is misconfigured but I can't see any hints anywhere ... Is it complaining about connection to backend? Or is it backend complaning about actual access to datastore? I see nothing relevant in the logs...

I'm doing a walkthrough of my work's ML Platform tomorrow if anyone wants to come. Deets in thread...

hi team, we recently updated our metaflow version to 2.5.0 and we seem to have lost metadata around the ec2-instance-id and ec2-instance-type for our batch jobs. Was this intended or moved somewhere else?

<!here> @lively-lunch-9285 and his team at Pattern (they are the biggest seller on Amazon after Amazon!) will be discussing some cool stuff they have been hacking on top of Metaflow tomorrow. Join us if you are able to! Slack Conversation

Hello! First of all, thank you for your work — we're using Metaflow extensively to structure the results from our library, and it's been very helpful. In our current workflow, we call various functions from elsewhere in our code within a Metaflow flow. I was wondering: is there a way to automatically treat all functions called as Metaflow steps, without needing to explicitly annotate them with

@step

?

Hello team! I was wondering if it could ring a bell: we are trying to display cards in our metaflow flows but we are having in the metaflow ui pod the following error - the

/tmp

seems to be well used as I can see that some data have been written:

/tmp/metaflow_client/gs.gs:/argo-workflows-artifacts-pilot.ComponentDemoFlow/98

. Here is the error in the thread. Thank you!

Hi, while running workflows as stepfunctions, we wanted to use our own image for step executions which runs as non root user. But this fails with below error. is it possible to run as non root user or it has to be root always.

Hey! In my efforts to instrument metaflow with openlineage: • I'm storing a value on

self.

, e.g.

self.x

• and I want to access it on

current.

, e.g.

current.x

(accessing it in the same flow run) But for the life of me, I can't find a way to do that. Could anyone help me out?

Hi all, I'm getting this error from my Kubernetes pod on Metaflow:

The node was low on resource: ephemeral-storage.
Threshold quantity: 32204283974, available: 30875356Ki.
Container main was using 51664160Ki, request is 10240M, has larger consumption of ephemeral-storage.

To resolve this, would I need to increase disk space or tmpfs storage?