Hello everyone, I have a quick question on saving datasets as artifacts. I'm currently working with a large dataset (1 GB+) and want to save it as an artifact. In previous discussions on this topic, it is recommended that we save these dataset in S3. https://outerbounds-community.slack.com/archives/C02116BBNTU/p1622657805149500?thread_ts=1622655878.145800&cid=C02116BBNTU, https://outerbounds-community.slack.com/archives/C02116BBNTU/p1641247012312400?thread_ts=1641235340.310500&cid=C02116BBNTU. These discussions happened a while ago and just wanted understand whether this is still the recommended practice. Thanks again.

yep, that recommendation is still valid. The 1GB is not a strict limit. For instance, a Numpy matrix or a Pandas dataframe that takes 5GB of RAM can compress nicely as a regular artifact. Here's additional context why storing large datasets separately is a good idea: • You can get a much higher performance in downloading/uploading data, if you have multiple files instead of a single artifact. See e.g. this example for syncing full directories to/from S3 quickly. • You can have full control over serialization and compression, so you can feed data efficiently to your models etc. • You can read/write data from other systems outside Metaflow. Metaflow's internal artifacts are not suitable for cross-system communication. As long as you persist pointers to your own datasets as artifacts, you can get a complete lineage from source data to your workflows and models. Note that

S3(run=self)

can come in handy here, if you want to write results from Metaflow in a versioned manner.

Thanks @straight-shampoo-11124!