I would like to “sell” metaflow to our devs who use CI/CD and github actions and started to “reimplemt” an MLOPs framework. Let us say as data scientist I created a model using metaflow (scheduled) steps for data engineering, hyper parameter optimisation etc. As it stands I currently hand over a pickle file, which is wrapped into FastAPI. I would like to help to automate this step with metaflow as well also involving CI/CD (?) and possibly even measure model degradation using some metric in metaflow and facilitate automatic deployment if threshold is met. If not hold back and send message or so. We also use AWS and Sagemaker but find the latter to restrictive … Is there some “hello world” example I could use to help to sell MetaFlow as THE (?) MLOps framework? Any pointers appreciated please. Thanks!

here is a toy demo of a CI process like this using outerbounds. the auth pieces will bedifferent for engineers who deploy open source metaflow themselves, but the end user code/workflow could be shaped like this template.

thanks! will have a look. btw would you call metaflow a mlops framework?

I think it is hard to define the boundary of MLOps frameworks in a way that satisfies everyone. A lot of features associated with MLOps frameworks like experiment tracking, hyperparameter tuning, model registry, dvc-style versioning etc. aren't built into open-source Metaflow. Metaflow has lower level foundations where you can, for example, build your own experiment tracking and model registry components and operate them in your own cloud at a lower price point than managed MLOps frameworks can sustain a business model on. So my view is Metaflow is more like a platform for building MLOps tools, which is most useful when you have the engineering skill + appetite and/or need to control more of the ML stack for cost or security reasons.

thanks eddie. yes mlops is of a fuzzy concept. reckon hyperparameter tuning is straightforward ootb with metaflow and experiment tracking too?

yep, you can build those things using metaflow's foreach tasks for hpo and artifact tracking + cards for experiment tracking. alternatively you could plug in a dedicated experiment tracker like wandb or comet

@happy-wolf-7852 we open sourced few reference implenentation with metaflow, including depoyment of model artifacts from metaflow using SM endpoints ->

<https://github.com/jacopotagliabue/post-modern-stack>

Thanks. We moved away from SM endpoints and use models pickled and FastAPI at the moment. Is there a close enough example out there?

this repo is sort of related to that use case. in the trees subdir it shows comparison of using pickle + fastAPI to serve a tree model from sklearn, compared to using fil+treelite and serving with NVidia's Triton inference server.

thanks all. will digest 😀

Jacopo's example is much more realistic, a great reference!

understood. I have not done ML in a while, but the great thing about SM endpoint (possibly, the ONLY great thing) is that you dont need to run anything else and can use Python to define the endpoint within the DAG, vs having an entire setup and model moving code to serve a GET request (if the model is indeed pickled I imagine is a simple one)

yeah thanks. tbh I just used ootb xgb and ootb sm endpoints to just get a walking skeleton up. things start to get bit more sophisticated - e.g. I use tailored loss function. this would require a different flavour of sm endpoints - at least

script mode

but even better

BOYC

. tbh I am quite glad we moved away from sm as it makes certain things more complicated (imho) and I quite like fastapi + docker whatever. however, as it stands, I would not know how to do ci/cd deployment stuff here. need to digest all the above. but thanks for you input! will also show to the proper devs.

I hate SM dont get me wrong, but for simple stuff (note: you can ship your own docker there as well with custom code etc) is cool that you can have everything, including serving, inside of Metaflow, with no dependencies on other teams or systems. Good luck!

Yes I came to that conclusion w.r.t. sm as well.