Hello, we have seen a weird but interesting thing: we have a folder with a lot of files (e.g.

hundred

of files) on

s3

. The size of the folder is the like of

hundred

of

GBs

. • We download the data with https://docs.metaflow.org/api/S3#S3.get_all • We modify something from the data • We upload the data again with https://docs.metaflow.org/api/S3#S3.put_files to a new s3 path. We noticed on a machine (with only 2 CPUs), downloading the data takes

30

mins but

upload

the data again takes only

5

mins!!!! How is that even possible? If any one nows why pls let me know here. Many thanks!

that sounds odd. Where are you downloading the data - to an EC2 instance, laptop, or something else?

Hi there, it is an EC2 instance… There should be no different region on the s3 path that I downloaded/uploaded. So this sounds weird ….

We also tried increasing number of CPUs to 16 but it still looks similar. We're running on AWS step functions with spot instances, so the only remaining suspect might be disk speed. Is it possible that it's related to disk speed? We're dowloading to /, which is on EBS.

Usually AWS EC2 have much much greater bandwidth on upload than download, so this might play a role to some degree

right, it sounds like an EBS disk speed issue. If you want to test if that’s the case, you could test with a large instance with enough memory to fit data in memory and use the

tmpfs

attribute to make the S3 client utilize it See details here https://outerbounds.com/blog/metaflow-fast-data/

Yes, thanks for the comments! We tried a bunch of things, and what resolved it sufficiently in the end was to switch to a larger instance 👍

great!