Hey there, I have a flow that takes a few parameters and uses them to run some expensive queries to generate a relevant dataset. This dataset is then used in subsequent steps of the flow. I imagine I could use a combination of IncludeFile and GCS, and fall back to generating (and uploading) a dataset if none are found. Is this the preferred pattern for caching/dataset management with Metaflow?

yep sounds like a good plan! if the data is small you can assign it (or bytes) to

self.

variable and metaflow will serialize so it is accessible across steps. if the data is big then cloud storage is the way for caching.

Got it, appreciate your reply Eddie! The data might be small enough to assign it to `self`; my remaining hesitation there, however, would be the datasets becoming slightly less organized and accessible. Based on my understanding of the docs right now, it seems I’d have to iterate through all past runs to check their parameters in order to retrieve the dataset artifact, so it might be more straightforward to check the bucket at a parameterized file path in the cloud. Does that sound accurate to you?

Great q. I think that is a good mental model, will add my decision process for completeness. If the data is not changing over workflow runs, then

self.

introduces some cognitive overhead - organizing by Metaflow run ID. If the data is changing, I think a major feature of Metaflow is that it is pretty easy to index these data artifacts based on only a Metaflow run ID, and I don't need to open an AWS session or think about that layer much at all. You don't need to iterate over all runs, if you know which run produced the artifact you want

S3(run=metaflow.Run(run_id))

.