Hi
We have a use case in which we are using the Metaflow S3 client to download a large amount of data. It is too much to fit into RAM on a single node, so we are saving to disk.
Wondering what the best way of using the client to achieve this result? Currently seems to be batching calls of
get_many
. Is that right?
Also, a related question, is there a reason that this
function returns a list and doesn't yield instead?