Hi 👋🏻 I have a question regarding submitting many jobs through

argo-workflows trigger

. I submitted about 30 at once, and each one said it was submitted and gave me a run-id, but many of them (about 20 of the 30) I can’t fine in our Metaflow UI or in Argo. How can I debug what’s going on? Is there a place besides the UI or argo that the submitted job is recorded or stored before it goes to argo or the UI?

Hi @mysterious-coat-51999 - do you have access to Kubernetes objects in your cluster?

they should be visible in the Argo UI under "Workflows" view but not necessarily in the Metaflow UI - you may need to click "Next page" to see all of them

Got it - I see them in argo UI now. Thank you! Here’s one that hasn’t started yet. I guess it could be many reasons why this job hasn’t started after 3+ hours? 🤔

a likely reason is that you don't have enough available capacity in your cluster

I think from what I can tell that’s exactly it… We will look into this further - thank you for your expertise and the help 🙂

Also, keep in mind that in addition to having capacity to run the workflow pods themselves, your argo-workflows server may also need more memory resources to process and schedule a larger volume of workflows simultaneously.