Hi, I recently asked about saving datasets as arti...
# ask-metaflow
c
Hi, I recently asked about saving datasets as artifacts. https://outerbounds-community.slack.com/archives/C02116BBNTU/p1680550636387349. I was previously working with pandas dataframes but after reading https://outerbounds-community.slack.com/archives/C02116BBNTU/p1638394866487100 and going through chapter 7 of the effective data science infrastructure book, decided to work with pyarrow tables. My question is when should you consider saving pyarrow tables as metaflow artifacts versus when to store them in S3 in a metaflow versioned bucket. I do see that in https://outerbounds.com/docs/chunk-df/, we can save pyarrow tables as artifacts but when should you refrain from doing that? Thanks again.
👍 1
1