The human-centric platform for production ML & AI

Outerbounds

A small question: When we have a flow that need to load data from S3, and when we run that flow locally, I assume that Metaflow will download those data to local disk. If so, would the data be cleaned up / deleted after the flow completes, or would we need to delete those data files manually ourselves? Where would these data files be stored?

good question! If you use the context manager `with` with `metaflow.S3`, it'll clean up by itself:
```with S3() as s3:
    s3.get_many(...)```
this will delete all temporary files after `with` exits

if you don't use `with`, you can call `s3.close()` manually to delete any temp files

by default the temp files are stored in the current working directory, but you can change <https://docs.metaflow.org/api/s3#S3|it via the >`tmproot`<https://docs.metaflow.org/api/s3#S3| argument in the constructor>

in addition, if you run tasks in the cloud and set `use_tmpfs=True` in `@batch` or `@kubernetes`, it'll use the fast in-memory tmpfs for temporary storage <https://docs.metaflow.org/scaling/data#using-metaflows3-for-in-memory-processing|as explained here>