Hi all, I have a questions about defining priorities for using different nodes in K8s for different types of compute. For example I have 3 nodes and I’d like to run CPU loads on nodes in this order node1, node2, node3 but GPU loads run on node3, node2, node1. Is there a way to set these priorities using Metaflow or K8? I found this switch but not sure if this will allow to use more then one node, if default one is busy, will it run on another?

# Default node selectors to use by K8S jobs created by Metaflow - foo=bar,baz=bab
KUBERNETES_NODE_SELECTOR = from_conf("KUBERNETES_NODE_SELECTOR", "")

Another question, what happens if there are no resources in the cluster at the moment? Will it wait until resources will be available or just fails? Will queue work if I use Argo? And last one, can I set priorities for jobs so Argo manages which job to start first? Thanks!

👋🏼 this is an interesting scenario. Mind sharing the context behind it?

My nodes and not the same. Nodes with slower gpus should be prioritized for cpu only loads to keep nodes with faster gpu available for gpu loads.

makes sense - we have plans to prioritize certain types of nodes over others for specific workloads. That should be able to cover your use case.

Hi @ancient-application-36103 any idea on when this might be implemented?