Hello, Are there any updates on this issue - Maintain dir structure with S3 client get methods?
https://github.com/Netflix/metaflow/issues/66
Would be wonderful to have some functionality to download directories or easily move downloaded files to a specific location.
I have checked the CV tutorials to find the best way to use S3 to download image datasets or other directories. But only found that a zip is downloaded and extracted or use libraries is used to get the dataset like Iris. We may have some defined structure like below. In my usecase it is some internal format for a video that we dont really zip in s3 for various reasons. It is extremely easy to download it with get_all. But since its not in the need structure or with the same file name, I find it bit inconvenient to create that structure from the url and move all files accordingly.
Any tips are very much appreciated. Thank you!
main_directory/
...class_a/
......a_image_1.jpg
......a_image_2.jpg
...class_b/
......b_image_1.jpg
......b_image_2.jpg