Hey guys, I have been working on building a RAG system with metaflow(leveraging conditions and loop). Metaflow became a great fit for ingestion and embedding. I am ingesting/parsing multiple types of documents and each parser becomes a step and can be processed in parallel. I want to know how will metaflow behave for retrieval. For retrieval I get the request through REST API and is processed through FASTAPI. I intend to replace it with metaflow, but not sure if metaflow is a great tool for scalable real time inference. Below is the link to the github, a system diagram should be enough to understand concept. https://github.com/patel-lay/contex-aware-search

outerbounds offers first class support for hosting apis, but meanwhile purely within the metaflow land, here is an example

Hey Savin, thanks for a quick reply. I have been using this example as my guidance. As you mention, this is within metaflow. I am interested in knowing whether metaflow could accept and process request through web-interface(REST API/socket) based. As per my understanding, metaflow is more of a platform rather than real time processing/quering system.

inferencing is not part of the OSS ecosystem as much. Companies that use Metaflow also typically integrate with some serving solution but there is no OSS ready-to-use solution. The way those work, typically, is that artifacts produced through training (like the model) are served through some other system. There was an old article about using Seldon for example: https://medium.com/@seldonhq/simple-production-ml-with-metaflow-and-seldon-707b1abf5552. I don’t know if it still works as is but just pointing to it in case this is the type of things you want to do.

@bland-garden-80695 the example I shared sets up an endpoint within a metaflow task and communicates the state through events. Not ideal but works quite well in practice

Thanks Savin and Romain.