Hi, what's the best way currently to visualize training level metrics, like loss/epoch inside Metaflow? We currently use Tensorboard, and events get stored in S3. For now, I can access Tensorboard locally, but was looking for a hosted solution, similar to Metaflow UI, to view metrics for all runs easily. Does Metaflow UI support Tensorboard? If not, what would be the best way to go about this? Kinda looking to have a wandb like UI, where its easy to view runs, metrics across users, but preferably using open source tools and inside Metaflow stack itself if possible

Metaflow can show you artifacts using cards which you can customize further. But I assume you want to aggregate results over different runs to compare metrics? In that case you might to store metrics over multiple runs somewhere and then pull it inside a card to just display it. I haven't done anything like this. So I don't know how easy all of this is.

Yeah, I was looking for a way to view multiple cards, compare, aggregate etc over them

Does Metaflow UI support Tensorboard?

It can, but we don't have any pre-made examples as far as I'm aware. Can do a quick POC we can build off of and get back to you later this week if helpful.

view runs, metrics across users...

Is the idea here to have a single view across many runs that appears in MFGUI (which currently only attaches cards at the step level of flows), or to aggregate data in the TensorBoard server across Metaflow runs/user namespaces? We are working on a feature that will directly address the dynamic plotting aspect, stay tuned!

If you could do a POC, that would be super helpful! Even without, any pointers on how to do that would be helpful as well And yeah, looking to aggregate data across runs/users - for the same flow, setting

namespace(None)

and visualizing and comparing loss curves, etc for all the runs, so we could see how different parameters of the run affect the loss and so on, something like the plot here And looking forward to the dynamic plotting feature release!

yep makes sense. I think unpacking the

events.out.tfevents.foo

files in Python an adding them to card is a partial solution. This repo is interesting guide. at the moment there isn't really support for aggregating cross-run visualizations in MFGUI - would doing this in a jupyter notebook suffice, or is the idea strongly tied to a wandb-ish dashboard view?