Hi. I'm new to Metaflow and I'm trying to get a grip on how environments are handled. Specifically I'm trying to do the following: I have a flow that creates an ML model and I want to be able to execute that flow locally (architecture osx-arm64). Afterwards I might want to use that model in a different flow. In that flow I'd like to use the same environment that was used to create it so that I can be sure that it works the same. This should ideally also work when that flow is deployed on a different architecture (linux-64). The simplest way for making environments easily reusable across flows seems to be via named environments from the netflix extensions package, but it doesn't seem to be possible to resolve two environments on different architectures and give them the same name (correct?). I.e. something like this doesn't result in both of them having the same alias:

metaflow environment resolve --arch osx-arm64 --arch linux-64 --force -r requirements.txt --alias my_test_env

There is the option of using

fetch_at_exec

to determine the name of the named environment at runtime. Then I could resolve the environments for both architectures and give them different aliases, e.g.

my_test_env:<arch>

, and define through runtime information, which is the correct one. I'm just wondering whether this is an appropriate way of doing it. Is there a better way?

Hey, is there a good way to make an ArgoEvent published in a test/user namespace not trigger e.g. the production flow but only flows in that same namespace?

Lineage update: This week, I'm going all on on this POC of to show data lineage between airflow and metaflow. If anyone here has ever used DataHub or OpenLineage instrumentation before, I'd love to talk here

Hi! Have a question regarding using

metaflow_ray

on argo workflows, I've installed JobSet to the same namespace with my argo workflows runner namespace, and had the permission added to my service account, but hitting some error at the moment about the webhook service path, is there anyway i can modify this value? can't seem to find it either from the metaflow configs or anywhere, thanks in advance!

Hi! In my k8s deployment, I have tried to change the datastore location, now I have this in my helm chart:

metaflow-ui:
  uiBackend:
    metaflowDefaultDatastore: "s3"
    metaflowDatastoreSysRootS3: "<s3://metaflow>"
    metaflowS3EndpointURL: "<http://minio.caic:9000>"

plus AWS_SECRET_* values through

env

. Previously I had DefaultDatastore set to "local" I have restarted both deployments for mataflow-ui. Now I'm running yet another flow, and in UI it still shows me Since this run uses local datastore instead of cloud services, some information may be incomplete. And of course there are no data in my minio bucket Is my s3 configuration correct? Do I need to drop the database or deploy the backend service too to see the effect?

Hello team! I was wondering if any of you had this error when loading cards from a flow:

URL/api/flows/ComponentDemoFlow/runs/4543/steps/start/tasks/222732/cards/blank/0490a0de7c024207af081c2187dcea5d?embed=true

returns

[Errno 2] No such file or directory: '/root/services/ui_backend_service/ui/index.html'

. • We are using a custom DNS behind a vpn in

http

• Recent versions for all services:

metaflow-ui.uiBackend.image.tag=2.4.13-2-g70af4ed

,

metaflow-service.image.tag=2.4.13-2-g70af4ed

,

metaflow-ui.uiStatic.image.tag=1.3.5-123-g95238f8-obp

Thank you!

Hello everyone, I setup metaflow on prem with argo events and workflows. I wanted to ask if we have something similar to sensors that react on events like for example uploading a file to s3 ? Also can I connect this on prem deployment to outerbounds UI ? I specifically mean this https://docs.outerbounds.com/outerbounds/connect-to-outerbounds/ The third option here ?

Hi, I hope this is the right channel for such question. I've deployed metaflow using helm chart to y k8s cluster (using https://github.com/outerbounds/metaflow-tools/tree/master/charts/metaflow). I connected it to my postgresql db, this parts seems to work fine. I have 3 services running, metaflow-service, -ui and -ui-static. So far so good. I've set

METAFLOW_DEFAULT_DATASTORE=local

as I did not find out how to set up credentials for s3/minio. Now, I port-forward the service (port 8080) to localhost and with

METAFLOW_SERVICE_URL=<http://localhost:8080/>

I'm able to run example flow; from the logs on service pod it seems it is connecting to remote server which is nice. I can even check the flow status on port 8083, so it seems this is logged corerctly OK, but now I cannot see anything in the UI. The UI (running through ingress) just shows

Error loading data

on empty dashboard page, nothing about my flow and nothing more specific about the error, just

Unknown error

/

generic-error

. I assume some part of my stack is misconfigured but I can't see any hints anywhere ... Is it complaining about connection to backend? Or is it backend complaning about actual access to datastore? I see nothing relevant in the logs...

I'm doing a walkthrough of my work's ML Platform tomorrow if anyone wants to come. Deets in thread...

hi team, we recently updated our metaflow version to 2.5.0 and we seem to have lost metadata around the ec2-instance-id and ec2-instance-type for our batch jobs. Was this intended or moved somewhere else?

<!here> @lively-lunch-9285 and his team at Pattern (they are the biggest seller on Amazon after Amazon!) will be discussing some cool stuff they have been hacking on top of Metaflow tomorrow. Join us if you are able to! Slack Conversation

Hello! First of all, thank you for your work — we're using Metaflow extensively to structure the results from our library, and it's been very helpful. In our current workflow, we call various functions from elsewhere in our code within a Metaflow flow. I was wondering: is there a way to automatically treat all functions called as Metaflow steps, without needing to explicitly annotate them with

@step

?

Hello team! I was wondering if it could ring a bell: we are trying to display cards in our metaflow flows but we are having in the metaflow ui pod the following error - the

/tmp

seems to be well used as I can see that some data have been written:

/tmp/metaflow_client/gs.gs:/argo-workflows-artifacts-pilot.ComponentDemoFlow/98

. Here is the error in the thread. Thank you!

Hi, while running workflows as stepfunctions, we wanted to use our own image for step executions which runs as non root user. But this fails with below error. is it possible to run as non root user or it has to be root always.

Hey! In my efforts to instrument metaflow with openlineage: • I'm storing a value on

self.

, e.g.

self.x

• and I want to access it on

current.

, e.g.

current.x

(accessing it in the same flow run) But for the life of me, I can't find a way to do that. Could anyone help me out?

Hi all, I'm getting this error from my Kubernetes pod on Metaflow:

The node was low on resource: ephemeral-storage.
Threshold quantity: 32204283974, available: 30875356Ki.
Container main was using 51664160Ki, request is 10240M, has larger consumption of ephemeral-storage.

To resolve this, would I need to increase disk space or tmpfs storage?

Is there a way to use

@schedule

with different parameters being passed into the flow? Is

Config

the only option? We have an existing setup where a single Flow is being called with different parameters and migrating it to use

Config

is non-trivial

Are there any recommendations about setting a TTL for Metaflow objects in S3 / other object stores? https://github.com/Netflix/metaflow/issues/21 only says that the TTL needs to be longer than the lifetime of the workflow, not sure if that has changed in the last six years. Any info would be great!

Is there a way to make the pylint checker that runs before a flow check all of the code in symlinked directories in addition to the flow itself?

I just discovered Marimo notebooks! Anyone use Marimo notebooks with MetaFlow yet? https://marimo.io/

Hey guys, is there any reason why a particular run-id does not show up on metaflow UI? I see the following logs, This run-id is a resumed argo-workflow.

Metaflow 2.15.14 executing flowTrain for user:j.kollipara
Validating your flow...                                                                                                                                                     
The graph looks good!│
Running pylint...                                                                                                                                                           
Pylint is happy! 
Workflow starting (run-id 1749756923772615):

I took a stab at logging lineage for a metaflow flow using OpenLineage and Marquez. The concept is super exciting.

Let me know if there is a more appropriate developer channel for this type of question.

message has been deleted

I am running into several initialization challenges with a custom step decorator that can run remote tasks in Azure Batch. I tried to use the batch_decoratory.py from metaflow.plugins.aws.batch module as a template (see attached code). However, when I use this in flowspec, I get errors such as: MyAzureBatchDecorator is not callable ImportError: cannot import name 'my_azure_batch' from 'custom_metaflow.plugins.decorators.my_azure_decorator' AttributeError: 'MyAzureDecorator' object has no attribute '__func__'. Did you mean: '__doc__'?

You know, the

python <flow>.py ...

cli has a lot going on. There are a lot of arguments and options you can pass in. We could totally make a VS Code extension that turns some of it into a push-button affair: • killing kubernetes pods • resuming (and from which step? 🤔 ) • running • choosing your environment • launching a debug session that stops if you set a breakpoint I'm just saying, there are a TON of subcommands. And I've been guilty of asking repeat questions on these, as well as fielding tons of questions about it all.

Hey good people, has anyone used API Gateway (with Lambda) to trigger a metaflow workflow? Due to nature of the read-only filesystem of lambda, I was facing a wall right now. If not, are there any other alternatives to get it triggered as an endpoint? Thanks in advance!

Hi, I am using "step-functions delete" command to delete the stepfunctions and other resources created as part of "step-functions create" command. Its deleting only the Step function and Event bridge rule associated with it. But not deleting the job definitions. Is there any reason behind this? We have a bunch of job definitions which are not used by any step functions and figuring out a way to delete those automatically.

Hi, how do you delete a flow completely from metaflow + argo workflow? Is it just removing the template? or something else?

Hi , is it normal for Metaflow to create local datastore in ECS container (via aws batch) when MF_DATASTORE=s3 is set to s3 ? Getting below error Creating local datastore in current directory (/opt/my_app/portfolio_calcs/.metaflow) Internal error Traceback (most recent call last): File "/opt/my_app/metaflow/metaflow/cli.py", line 554, in main start(auto_envvar_prefix="METAFLOW", obj=state) . . . . File "/opt/my_app/metaflow/metaflow/datastore/content_addressed_store.py", line 140, in load_blobs with open(file_path, "rb") as f: TypeError: expected str, bytes or os.PathLike object, not NoneType