The human-centric platform for production ML & AI

Outerbounds

Hi all! I was wondering if Kueue fits into the roadmap for integration with Metaflow? I see an old <https://github.com/Netflix/metaflow/issues/1513|issue> and and old <https://github.com/Netflix/metaflow/pull/1754|PR> so I was wondering if anyone is planning on coming back to this eventually.

hi mark! we (and many others) have been using kueue already with metaflow - particularly to create gang clusters with `@parallel`

also, there is an integration driven by g-research to integrate metaflow with armada actively in the works. it's getting demoed this week internally.

ah, I see. Are there any published examples along those lines (kueue with `@parallel` )?

i believe the PR that you linked highlights the labels/annotations that would be needed. the reason why the PR wasn't merged was because depending on your setup, your labels/annotations could be very different. internally we choose to apply these as policies at workload submission time rather than client side - which helps with modifying infrastructure without encroaching into user workloads