Currently we are processing huge files of size ~100-150 GBs on k8s using metaflow. But this involves copying data from S3 to the EC2 instance where pod is running. I saw the fast data blog but this requires that we should have big enough memory to hold all of this data in memory. Is there something intermediate between downloading all of data inside pod vs loading everything inside memory? I The only idea I can think of is to split such files in smaller chunk and then load it in memory?

Chunking the data is the way to go.

true! I see that downloading on the pod is actually not that slow (~10 mins) for 110 GBs.

if you have enough disk space on the pod, you can load the files to local pod disk without having to keep everything in memory. Depending on the format of the files, you may be able to decode them in a streaming fashion from disk, so that you don't need to keep everything in memory at once even when decoding data