https://outerbounds.com/ logo
Join Slack
Powered by
Unrelated to the above, we had a question about af...
# ask-metaflow
p
Unrelated to the above, we had a question about after-the-fact figuring out what instance type a given step (running in k8s via argo workflows) was running on. We've gotten more aggressive about GCing old pods but belatedly realized that the kubernetes annotations (like instance-type) on the pod were the only way to find that info. Is there some approved method for saving some k8s pod annotations along with a given step results?
✅ 1
I saw a 
METAFLOW_KUBERNETES_FETCH_EC2_METADATA=False
being set in our per-step config but couldn't find any documentation or source on it
The specific question we're trying to answer is that we noticed some steps in a big 
foreach
flow were completing much faster than the others, where we'd expect them to all be identical. We've mostly ruled out CPU rate limiting so thought the speed difference might be core clock speeds on different instance types, and that's when we realized we weren't capturing enough historical data about where a job ran (beyond saving the host node name the pod ran on via the argo workflows summary) and ALSO didn't have enough prometheus data to reverse from a node host name to the instance type.
I can improve our prometheus metric collections to give us that breadcrumb trail but I was wondering if there was an approved metaflow/k8s approach to solving the "what instance type did this pod run on?"
s
there are multiple ways to answer this - internally we have built an analytics layer that inspects and aggregates over all the pod attributes and allows us to understand long term perf trends related to pod placement strategies for any given flow - i am not aware of any out of the box tooling that can provide this analysis. you can also read the ec2-instance-metadata within the task if that's available to you and run a similar analysis on task metadata rather than pod attributes (which may miss any scheduling related detail though)
p
That makes sense. We could modify the tasks to read the metadata and collect the instance type in-task. I'll probably solve this from the infra side and work on improving our metric collection through prometheus so we have a way of working out what node type a given node was, since we can already tie the tasks to a node.
👍🏼 1
Thank you.
2 Views
Open in Slack
PreviousNext
Powered by