Hello all, what's the best/recommended practice on programmatically querying the pod names/ids involved in the execution of a workflow. Our metaflow set-up uses EKS + Argo running on top of on-demand and spot instances. We recently started looking into workflow cost tracking with OpenCost (w\ Prometheus) and wanted to validate the cost calculations programmatically. So far, I can query the pod names/ids via each Task.metadata attribute by filtering the list of metadata objects by 'field_name'. Also played around with making GET requests directly to the client API. Any idea where I can find documentation for making more sophisticated queries to the client API to apply the field_name filters? Or should I get the information through the Argo-workflows DB? (Any advice is welcomed) (Also super excited for the event-triggered workflows release over here 🤩 )

We have this same issue and use kubernetes labels to do this. I created a PR to allow adding kubernetes labels via, config, env vars, or the kubernetes decorator for both kubernetes and argo. https://github.com/Netflix/metaflow/pull/1236

Kubecost & AWS recently rolled out an integration for Kubecost on EKS - https://aws.amazon.com/blogs/containers/aws-and-kubecost-collaborate-to-deliver-cost-monitoring-for-eks-customers/

Hey Sergio, You could query K8s directly for getting this information. Metaflow adds annotations such as flow_name, run_id, step_name, task_id to the pods.

metaflow/flow_name: StarterFlow
    metaflow/run_id: "410"
    metaflow/step_name: start
    metaflow/task_id: "7099"
    metaflow/user: shri