Hi, I would like to extend the S3 and S3Object classes in order to manipulate local files and S3 files seamlessly. The bucket and the local directory are not synced but if a file exits in the local directory, it exists in the S3 bucket. In other words, I would like to store the files on a local machine and reuse them later as S3Objects. I see some developments were already done with the classes Local and LocalObject. Do you have any recommendation or information about further developments?

here's an example that syncs all local files to an S3 location and vice versa - it's very fast too since it parallelizes syncing

Thank you for your answer ! I would like to use the S3Object and LocalObject seamlessly so as a same class. For example, a file exists in S3. The user download it once locally. The next time, the user wants to download the file from S3. It checks if it exists locally and use the local version. The main goals are to save download costs and to speed up the processes on local machines. The class will also be used outside a metaflow flow. (if possible)

makes sense. I don't have an existing snippet that does that but it shouldn't be hard. Just do

if os.path.exists(filename)

to check if the file exists locally and if it doesn't, load from S3