Hi all, I have a questions about defining priorities for using different nodes in K8s for different types of compute. For example I have 3 nodes and I’d like to run CPU loads on nodes in this order node1, node2, node3 but GPU loads run on node3, node2, node1. Is there a way to set these priorities using Metaflow or K8? I found this switch but not sure if this will allow to use more then one node, if default one is busy, will it run on another?

# Default node selectors to use by K8S jobs created by Metaflow - foo=bar,baz=bab
KUBERNETES_NODE_SELECTOR = from_conf("KUBERNETES_NODE_SELECTOR", "")

Another question, what happens if there are no resources in the cluster at the moment? Will it wait until resources will be available or just fails? Will queue work if I use Argo? And last one, can I set priorities for jobs so Argo manages which job to start first? Thanks!