https://outerbounds.com/ logo
Join Slack
Powered by
Hi Team. I have a question related to persistence ...
# ask-metaflow
f
Hi Team. I have a question related to persistence of features within Metaflow. Features annotated with 
self.
is stored in postgres DB instances or ECS? The hyperlinks for those features are captured in postgres and actual data is in S3/Object store. Is that correct?
✅ 1
Follow-up: Do you have any recommendations on how to discover features that were historically captured within metaflow? Thanks!
c
If you have some sort of consistent naming convention behind the Flows and the Project they belong to, you could create an API that leverages the Metaflow Client API to create a basic Offline Feature Store for feature discovery.
v
yeah, correct re: artifacts are stored in S3 or supported other datastore like ECS (here ECS refers to this S3-compatible storage product, not AWS ECS 🙂 ), metadata in the database
re: discovering historical artifacts, since all metadata about artifacts is stored in the DB, you can export a full log everything stored from the DB using a script like this https://gist.github.com/tuulos/dd80913552f6cfe87166ace1faa672a9
it'll be quite slow though to dump everything, if you have to go through many many runs. Accessing artifacts from a subset will be faster
👍 1
3 Views
Open in Slack
PreviousNext
Powered by