Getting the following error when I am trying to run a multi-gpu training run using metaflow

botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role: Error retrieving metadata: Received non 200 response (429) from ECS metadata: You have reached maximum request limit.

The issue which I am facing: • The dataloader object is being called for each GPU process, which is resulting in calling the

get_many()

for each GPU (I am using torch.distributed.launch for multi GPU training), which is resulting in multiple calls and sometimes results in OOM erros.

there might be a way to structure your dataloader differently so that it doesn’t instantiate

S3()

so often. It re-fetches credentials every time the object is instantiated but you can avoid this by reusing the same

S3()

instance

Hi @victorious-lawyer-58417, I've tried loading 1500 images in a single get_many run unlike 10k images in my previous loader. I am still getting the same error.