Hello all, I've seen some previous discussion on GDPR compliance, and came across this thread detailing a

metaflow garbage-collect

command (https://github.com/Netflix/metaflow/issues/530) We do have a use case where a user will want to delete some data. Maybe the user will want to delete a certain feature that cannot be used anymore. Then, we need to figure out what version of the model was using that designated feature and then we subsequently delete that version of the model, and then retrain without that particular feature. Any advice or stop-gap solution for facilitating this?

here's one approach, since it sounds like you want to delete the model artifact specifically. • store the features used to produce a model in an artifact (or maybe you can tag the run with a certain feature-set identifier) • persist models manually using

metaflow.S3(run=self)

and store the returned URL in an artifact • when a deletion request comes in, use the Client API to find all runs using the feature • retrieve the model URL from the runs • delete the model object • tag the run as "tainted", so you know it has been processed (just for bookkeeping) • mark the feature as disallowed or remove the feature altogether so it won't be used in the future

Got it, makes sense. Thanks @straight-shampoo-11124. What does this

• mark the feature as disallowed or remove the feature altogether so it won't be used in the future

entail? Are we permanently removing all traces of the data/feature in the artifacts through the client API? Manually in s3?

I don't know the specifics of your use case but if you need to be able to delete specific datapoints historically, I would handle it on the data warehouse side and store pointers to the data warehouse in Metaflow. For instance, if you extract data from your data warehouse in S3 as Parquet tables, you could store S3 URLs as artifacts, pointing at the parquet files. If a deletion request comes in, you delete the files used by the run. The fact that artifacts then point at missing files serve as a useful audit trail without revealing any sensitive data, should you ever need to analyze historical runs. You could wrap the logic in a simple access library / decorator of your own, so it's mostly transparent to the user.

Yup all of that makes sense, but was thinking more in terms of what if the user saves a snapshot (or multiple snapshots) of the training data that includes that particular feature. We could certainly guide our users to use an access library/decorator that is opinionated about this -- creating pointers to the data warehouse and that would most certainly resolve the issue. But I'm still not sure if that would resolve the issue if the user creates a pandas dataframe that reads in that S3 URL, then saves that as an artifact.

right, it's a universal problem of data governance which applies to all data systems: What if a user takes a data point X and stores its outside the governance boundary, like running a SQL query and saving the results as a CSV on a laptop which becomes impossible to track and delete practically. I'd encapsulate all sensitive data access in a custom library and design it in a way that prevents accidental data leakage, e.g. by not providing

to_pandas()

etc. If this is too draconian, another approach might be to prevent e.g. serialization of Pandas specifically. You should be able to do this by registering a custom serializer using `copyreg` that crashes with a clear warning for Pandas, which you can apply to sensitive workflows. All these are just safeguards for preventing accidental leakage. If you need hard guarantees, you can set up Metaflow to run in an environment without internet connectivity, so you can fully control data flows in and out.

Great wealth of information here! Thank you @straight-shampoo-11124. This is super helpful! Much obliged.

great question, btw! Data governance is hard 😅

It REALLY is