Is it possible to reconstruct a metadata database ...
# ask-metaflowe
elegant-beach-10818
09/21/2023, 4:19 PMIs it possible to reconstruct a metadata database just from the objects in S3? I don't have a need now, but I'm curious of the possibility in the worst-case disaster recovery scenario
✅ 1
v
victorious-lawyer-58417
09/21/2023, 6:17 PMby design it is doable for the most critical parts.
Tags are only in the db, so they are not recoverable but it shouldn’t affect functioning of Metaflow itself
e
elegant-beach-10818
09/21/2023, 7:11 PMWe’re starting to use tags for dataset/model versioning, so losing tags would impact our functional ability to use Metaflow in production until we recreated them. Good to know however, thanks for the answer
👍 1
v
victorious-lawyer-58417
09/21/2023, 7:28 PMmakes sense. Eg RDS provides plenty of options for making sure that the db stays highly available.
The largest Metaflow DBs I’m aware of have TBs of data and there haven’t been issues of data loss ever.
👍 1
c
cuddly-rocket-69327
09/22/2023, 9:42 PMHas anyone used MLFlow and propagated Metaflow tags to MLFlow?
cuddly-rocket-69327
09/22/2023, 9:47 PMGood idea about propagating Metaflow tags to datasets or Hive tables.
Integrating Metaflow well with Hive / datasets is the idea behind https://github.com/zillow/zdatasets Hive especially.
10 Views