hey everyone — does anyone have any tips for profiling code inside flows? it’s been a while, but from what i recall, cprofile does not play nicely with metaflow steps. at a cursory glance i also can’t seem to see anything in the docs that suggests there’s any native support for profiling steps (e.g. an

@profile

decorator, or anything that integrates with the ui). cheers!

👋 I don’t have a good answer, but adding my ➕ to hear any tips or strategies the community might have for profiling code within Metaflow steps. It would be great to learn how others are tackling this!

Hi! I've been tinkering with an experimental @step decorator related to this using a memory profiler called memray. It wraps the step function, like how this minimal example does, then adds cards and Metaflow artifacts containing the HTML files produced by memray. Here is the current implementation. There is some bug I've introduced so sometimes cards aren't getting rendered correctly in Metaflow GUI, but it reliably produces the HTML for a flamegraph, a table ranking the most expensive operations, and a histogram of the allocation sizes.

here is an example for GPU profiling and another one for evacuating metrics to products like datadog