Hey team - we're running into some noisy neighbor problems with metaflow on our k8s cluster similar to what's described here: https://outerbounds.com/blog/whisper-kubernetes/ I was curious if anyone has any guides or tips on how best to right size step decorators to improve performance given that its a bit difficult at face value to identify which flow steps are not right-sized and exceeding resource usage with bursting. We currently set these already in the step decorators, but still are seeing noisy neighbor problems

This article covers this topic somewhat tangentially. Internally, we track resource usage for every single task, and aggregate that over multiple executions to derive opportunities for right sizing. I am curious how other folks have been dealing with this.

It looks like this is internal to Outerbounds hosted metaflow but how are you capturing cost / usage at the step level level? That's really where my question is getting to

we have systems that track requested resources and consumption for every workload which then gets aggregated across multiple executions of the same step to pinpoint offenders with a higher degree of precision. one can image that the same step might exhibit different consumption patterns when fed with different data - which is often the case with scheduled flows - but accounting for multiple executions allows us to generate trends and predict which steps are either over-provisioned (spend wastage) or under-provisioned (at risk of failure)

Is the resource utilization summary supported on the open source Metaflow UI?

Not today - it requires deeper integration into underlying infrastructure that is rather tenuous to support in open source given the wide variety of setups folks are running