We're in the process of considering kubernetes(EKS) and argo to run metaflow ML jobs. Currently we're running step-functions and AWS Batch. At the moment, we create multiple AWS Batch Queue jobs, and Computer environment to manage various workloads according to environment such as (dev, prod, gpu, cpu, team). This allow us to support different teams using a single metaflow platform. those with experience on kubernetes and argo, how does ArgoCD and EKS manage multiple queue and various compute environment ?

You can get pretty far without queueing semantics, by separating resources by node group (similar to compute environments) and using e.g. node selectors when submitting jobs so they are routed to the correct group

due to my lack of understandinf of ArgoCD, can i create multiple job queue in ArgoCD ? at the moment with AWS Batch that is manage to change values in METAFLOW_BATCH_JOB_QUEUE How does that look like for Argo ?

So at the fundamental level, Argo doesn't really care. At K8S level its possible to create several node groups (which roughly correspond to compute environments), and then when you submit your workloads you can tell K8S to schedule them on a specific node group. You can also enable autoscaling so those node groups scale separately

Thanks for the input for METAFLOW_KUBERNETES_TOLERATIONS and/or METAFLOW_KUBERNETES_NODE_SELECTOR That's something i was trying to find out 🙂 ..

In addition to what Oleg described, you could separate teams’ workloads providing a separate k8s namespace to each team. Then templates and workflows runs would be “local” to those k8s namespaces. We use this approach to isolate our clients workloads. Teams could configure their environments to use “their” k8s namespace with

KUBERNETES_NAMESPACE

config parameter or

--k8s-namespace

cli argument.

@average-beach-28850 Would

KUBERNETES_NODE_SELECTOR

and

KUBERNETES_TOLERATIONS

be the key names we'd use for setting the config? https://github.com/Netflix/metaflow/blob/2f9e443574de038a40f9af0d07b6fc71327dce8b/metaflow/metaflow_config.py#L248 Looks like can also be set in an attribute: https://github.com/Netflix/metaflow/pull/1207#issue-1477115895 or command line too: https://github.com/Netflix/metaflow/pull/1207#issuecomment-1339857278

yep that's the ones, but they are prefixed with METAFLOW_ in config

Ah, ok! I didn't know about needing the METAFLOW_ prefix yet. I'm new. 🙂 I found an explanation of the prefix here: https://github.com/Netflix/metaflow/blob/2f9e443574de038a40f9af0d07b6fc71327dce8b/metaflow/metaflow_config.py#L16

@User Resurrecting this discussion because we are running into the limits of the EKS scheduler for kubernetes. Currently we’re running 1000+ jobs and the scheduler seems to be struggling to keep up despite us having enough resources available in our nodegroups/limits Some of the things we are currently implementing: • lowering the TTL of the job objects due to demonstrated slowdown with 50k + jobs (current 7 days is too long) • having the datascientists manually collaborate to not schedule simultaneously How would I go about leveraging volcano with metaflow? I see you mentioned it in this thread but my guess is that it’s not just plug-and-play