Hello all, Anyone have thoughts on the Metaflow experiment tracker that's already built-in? Is that sufficient for most Data Scientists? I've also been looking into more standalone experiment trackers like W&B, MLflow, Sagemaker Experiments, Aim (which has a beautiful UI), and Polyaxon. I see that Metaflow also has a third party integration with W&B. Any other plans to have integrations with other open source ML experiment tracking tools? W&B is nice, but would prefer open source.

hey Riley! interesting topic. In my view, it primarily comes down to preferences on the data science team and which interface they like to look at when interpreting results and deciding what to do after tasks like model selection and hyper-parameter comparisons. Another factor I find less important, but is important to some, is automated tracking with one-liners for common ML frameworks via commands like

mlflow.sklearn.autolog()

. I typically end up preferring the base APIs for experiment trackers over these integrations - or using Metaflow like

self.experiment_data = DataFrame(params=..., metrics=...)

- but some people don't like having things like

self.x = 1

or

mlflow.log_param('x', 1)

everywhere in their model code. Metaflow ecosystem has some tools for more advanced users to make dashboards out of cards + Metaflow UI. These require a bit more effort than tracking with wandb/comet/mlflow, which come with more out-of-the-box visualization features. A few times I have found the cards + Metaflow UI pattern nice because it brings data science interpretation layer closer to orchestration layer. This is nice for environments where context you can get around flows/namespaces across local and prod branches is useful to connect to the more model-oriented info that tends to be associated with experiment tracking. Curious to hear what experiment tracking problems you are looking to solve for, and how you end up navigating this decision!

I am 100% biased as I have spent many hours chatting with Gideon but comet is great. :-)

Thanks so much @crooked-jordan-29960 for the insight. Very helpful. We've decided to give the experiment tracker that is built-in to Metaflow a try and we'll see how the Data Scientists react to it and will collect their feedback. I'm aware that MLFlow has a very nifty

autolog

functionality that is compatible with all the major machine learning + deep learning framework. Not sure what their preference is. The core group of data scientists we are collaborating with are coming off of Neptune. They really enjoyed Neptune but unfortunately the license expired. They have also tried MLFlow and have had a positive experience with that as well, but they've only used MLFlow locally. Essentially, they just need a way to track & log their experiments. They want an experience VERY similar to Tensorboard which gives them insight into training metrics and such.

Riley, thanks for raising this topic. I learned about AIM from you, and it looks really interesting, but i'm not actually sure how it would work with metaflow unless the training is being done locally. looks like different storage options (S3) are on the roadmap though.

Thanks @fast-honey-9693. That's good to know. Lack of S3 storage options would be a dealbreaker for us.