Hello, Are there any updates on this issue - Maintain dir structure with S3 client get methods? https://github.com/Netflix/metaflow/issues/66 Would be wonderful to have some functionality to download directories or easily move downloaded files to a specific location. I have checked the CV tutorials to find the best way to use S3 to download image datasets or other directories. But only found that a zip is downloaded and extracted or use libraries is used to get the dataset like Iris. We may have some defined structure like below. In my usecase it is some internal format for a video that we dont really zip in s3 for various reasons. It is extremely easy to download it with get_all. But since its not in the need structure or with the same file name, I find it bit inconvenient to create that structure from the url and move all files accordingly. Any tips are very much appreciated. Thank you!

main_directory/
...class_a/
......a_image_1.jpg
......a_image_2.jpg
...class_b/
......b_image_1.jpg
......b_image_2.jpg

Ah yes - it's been on our roadmap to find a solution for this. @victorious-lawyer-58417 is there any workaround on top of your mind?

yeah, it’s definitely doable - there’s no overhead when using the current methods. I can share an example later

That's nice to know. The reason is also that in many cases the code depends on the structure and the filenames with extension. One could use cli or boto but if it would be nice if one could do it in MF S3 client itself. Looking forward to your example!

sorry, I haven’t had a chance to do it yet but I’ll follow up by the end of day today 🙏

Awesome, that is very helpful.

today turned out to be busier than expected but I’ll follow up tomorrow - apologies for the delay 🙏

@straight-shampoo-11124 No worries Ville, its not a blocker for me but was hoping there is a nice inbuilt way to upload and download directories while preserving structure, right now, I am moving with os/path.rename using the key, uri information from the objects.

thanks for your patience! Here's an example how to synchronize full directories to/from S3 efficiently using

metaflow.S3

. (sharing with the channel since this is a FAQ) This parallelizes all uploads and downloads, so it should be very fast. When downloading files locally, there's no unnecessary local disk IO since all files are just moved to the right place.

Thats great, thank you! Not handling empty dir is expected for anyone I guess.