Join Slack
Powered by
Hi, I recently asked about saving datasets as arti...
# ask-metaflow
c
chilly-midnight-51093
05/09/2023, 3:52 PM
Hi, I recently asked about saving datasets as artifacts.
https://outerbounds-community.slack.com/archives/C02116BBNTU/p1680550636387349
. I was previously working with pandas dataframes but after reading
https://outerbounds-community.slack.com/archives/C02116BBNTU/p1638394866487100
and going through chapter 7 of the effective data science infrastructure book, decided to work with pyarrow tables. My question is when should you consider saving pyarrow tables as metaflow artifacts versus when to store them in S3 in a metaflow versioned bucket. I do see that in
https://outerbounds.com/docs/chunk-df/
, we can save pyarrow tables as artifacts but when should you refrain from doing that? Thanks again.
👍 1
✅ 1
Open in Slack
Previous
Next