My framework is the following: Raw Data (Quartr API) → Process Data for modeling (sentiment analysis) → Generate Inference → frontend Consumption I’m thinking on using Metaflow Artifacts and S3 Client is the following way: 1. Raw Data Collection (Quartr API) -> Use Built-in Artifacts 2. Processing for Modeling -> Use Built-in Artifacts 3. Writing Outputs (Inference) S3 client. (store them in S3) 4. Dashboard Consumption FastAPI backend Does it make sense to use s3 for data collection/ fetching as well?

do you need to access the data outside of metaflow?

Yes - S3 and External API (Quartr)

typically if you need to access the data outside of metaflow, it's easier to directly share s3 paths (vs needing to use the metaflow client)