https://outerbounds.com/ logo
Join Slack
Powered by
Hello all, Are there any suggested strategies for...
# ask-metaflow
a
Hello all, Are there any suggested strategies for when to use 
use_tmpfs
? I tried using it in hopes that it would speed up s3 downloads, but didn't see any significant improvement. How big does the dataset have to be to see gains?
✅ 1
Example Code:
Copy code
from metaflow import FlowSpec, step, batch, retry, card, current, S3

class DataProcessingFlow(FlowSpec):

    @step
    def start(self):

        self.next(self.load_data)

    @batch(memory=30000, use_tmpfs=True)
    @step
    def load_data(self):
        import time

        start = time.time()
        with S3(s3root="<s3://my_bucket/test/sample_image_files/>") as s3:
            res = s3.get_all()

            total_size = sum([obj.size for obj in res])
            print(total_size)
            

        print(time.time() - start)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    DataProcessingFlow()
v
it doesn't improve speed if the dataset fits in memory already. It mainly helps to work around the issue that you can mount actual volumes on 
@batch
on the fly. Say, you want to load 128GB of data, you have 20GB of disk, and 200GB of RAM. The data can fit in RAM but without 
use_tmpfs
it'd annoyingly fail due to running out of disk space
🙌🏽 1
to increase S3 throughput in general, see this advice. If you don't see something close to the maximum network throughput of the instance (like 10-20Gbps on a large instance), let us know
a
Ah I see, thanks Ville! That's very helpful. Appreciate it.
👍 1
v
just for future reference for anyone curious about the topic - take a look at this blog article
❤️ 1
a
For 5GB of data, it took about 2 minutes to download, so that's about 0.33 Gbps? Instance type is c5.4xlarge and EBS volume is 100GB
v
5GB is not a lot, so the baseline latency reduces the throughput but 0.33Gbps sounds a bit low. Looking at the checklist here, is the data in the same region? is it one file or many files?
a
Yup data in the same region and it's many files. 50K files
v
ok, so just 100KB per file - that's the issue. Accessing an object in S3 has a relatively high baseline latency which adds up when accessing 50k objects. If you wanted to maximize throughput, 20-50 objects would be more optimal for 5GB of data
a
Got it. In our organization, most of our data would be image files or CAD drawings. So effectively, we will have millions of smaller files. Any suggestions for increasing throughput for this?
v
afaik there's not much you can do with S3 without changing the file layout. Many data lake systems run automatic compaction to pack files into larger objects to circumvent the problem. That's one approach
you could write such a compaction job in Metaflow, reading small files and writing larger chunks to another location to be consumed by subsequent consumers
❤️ 1
another option is to have more parallel tasks in a 
foreach
to consume tiny batches in parallel, but it would be effective only if you processed way more data than 5GB
a
Right, I was thinking of using 
foreach
as well but then there's overhead in terms of setting up the N batch instances.
v
right. 2mins isn't enough time to see the benefit
9 Views
Open in Slack
PreviousNext
Powered by