Hey folks My team is looking to integrate Metaflow into our existing infrastructure. We are currently testing with the minimal deployment with ECS and AWS batch, but for a fully fledged production instance, we want to deploy everything within Kubernetes (Metaflow UI, Argo, Metaflow service, etc) What components will we need to deploy into kubernetes to interact with Argo? Will it only need to be the UI or just the Backend service? Any advise would help us significantly.

@bright-energy-75500 if you are looking for a very minimal deployment on kubernetes with argo-workflows, then the only other extra component needed would be the blob store - say an s3 bucket. The metadata service and the UI are optional components - which can also be hosted on this Kubernetes cluster

If those components are not deployed to the cluster, then the "local" instance of Metaflow on the developers machine would define the blob storage? Another question: Would decorators from the Metaflow workflow be passed to the argo workers for scheduling? ie. one workflow requires GPU instances that are tainted in the cluster and should only be used by workers that define them.

The blob storage would need to be configured ahead of time if you plan on running any workloads on kubernetes - since that is the primary mechanism of passing state from laptops to Kubernetes pods and amongst Kubernetes pods. Re:decorators - correct - all decorators are passed to argo. Metaflow guarantees that you will see the same execution behavior when you execute everything locally on the laptop; when you execute only a few steps locally on the laptop and the rest in the cloud; when you execute all the steps in the cloud from your laptop; and when you execute all the steps in the cloud from argo workflows/airflow/step-functions.

Decorators are passed, but will the taint defined in the decorator ensure that it is scheduled on a GPU tainted worker?

Yep!

For clarity as well. What benefits do we get for deploying a metaflow-service and metaflow-ui into the cluster alongside the Argo deployment?

With the metadata service, there is a single place where all the metadata is gathered, which allows you to access metaflow’s artifact store programmatically. Basically, in any other Python process you can ask information about any run, access the data generated during that run (code, data, models, logs, intermediate variables, lineage, dependencies etc.) - it’s a good way to get output from flows without worrying about storing those outputs by yourself. The UI service - as the name suggests - enables the metaflow UI - you can also use cards to get some UI visibility in lieu of deploying the UI.

excellant! Thank you!