hey folks - had a question that I ran into for a project I'm working on with metaflow. What do you use for your Feature store when building ml applications and experimenting with multiple models in staging or prod? Does the feature store also need to be promoted the same way the champion model gets promoted to prod?

great question! There are many valid patterns, one of them being not using a feature store. It depends on your use case: The needs of LLMs and computer vision and different than, say, a forecasting model or a real-time fraud detection system. Can you share more about your use case?

Use case is one where features are a core part of the model. Simple description: embeddings created from documents as features for a prediction system, predicting the forecasted value of a entity (customer) in various horizons.

nice, thanks for sharing! How are the results being used - do you need some real-time features too?

Features are not real time. batch job. But features change and models update accordingly

makes sense. In cases like that one pattern is to have a library of feature encoders, which you can use consistently for training and inference. You can find an example of this pattern in Chapter 9 of the Metaflow book (code here). Metaflow packages the encoders with each deployment separately, so you can configure different sets of encoders, modify them, and develop new ones for new model variants without impacting the production version. When you promote a model to production, their specific configuration of encoders (and hence features) get promoted with them.

Thanks for the sharing the design pattern, will check the code out! Are there any limitations with the ability to experiment one set of feature encoders with a challenger model? Will it allow me to set sorta feature flags that I can use to crossover features for the prod model to another set of feature encoders? Would I need to decouple it from the existing inference pipeline to make sure I can crossover after a certain @event?

sure, that shouldn't be hard to do, but you have to implement the logic by yourself e.g. through some configuration