Hi We have a use case in which we are using the Metaflow S3 client to download a large amount of data. It is too much to fit into RAM on a single node, so we are saving to disk. Wondering what the best way of using the client to achieve this result? Currently seems to be batching calls of

get_many

. Is that right? Also, a related question, is there a reason that this function returns a list and doesn't yield instead?

yeah, batching calls to

get_many

would do the trick

sorry, not sure I understand. The

_get

method seems to be an iterator. You are saying it will download all of the data regardless of that being an iterator?

full_path = f"s3://{s3_bucket}/{path}"
    with S3(s3root=full_path, tmproot=output_path) as s3:
        list_of_files = s3.list_paths()

    # not all the files are the same size so shuffling for uniform distribution among batch
    random.shuffle(list_of_files)

    batched = [
        list_of_files[i : i + sublist_size]
        for i in range(0, len(list_of_files), sublist_size)
    ]

    for i, batch in enumerate(batched):
        with S3(s3root=full_path, tmproot=output_path) as s3:
            [
                os.rename(s3obj.path, os.path.join(output_path, s3obj.key))
                for s3obj in s3.get_many(batch)
            ]

I have been running into issue with memory limits. To try and resolve this I implemented the batching strategy yet the memory is not cleared after the context manager has closed

Watch out when writing too much to disk, make sure your AWS EBS settings can handle it.