Has anyone used S3's intelligent tiering for their metaflow s3 bucket? Seems pretty trivial to set up and use which makes the skeptic in me suspicious if there’s something I missed considering.
✅ 1
s
square-wire-39606
04/10/2024, 11:16 AM
It should work but performance might be a bit off
q
quick-lighter-52296
04/10/2024, 11:20 AM
Surprisingly AWS guarantees milliseconds access times even for infrequent tiers (as long as you don’t go all the way down to deep archive — which is opt-in), and once accessed the objects are moved back to standard tier. So theoretically it doesn’t seem there would be any perf impact at all
s
square-wire-39606
04/10/2024, 11:22 AM
🥳 do share what the real world results look like on your end!
✅ 1
square-wire-39606
04/11/2024, 1:23 PM
Looks like Netflix’s metaflow bucket (which holds dozens of petabytes of metaflow artifacts) is on intelligent tiering without any issues