Hi friends We re experiencing a problem when running many pa Outerbounds #ask-metaflow

Hi friends. We're experiencing a problem when runn...

user

01/30/2023, 2:51 PM

Hi friends. We're experiencing a problem when running many parallel instances of a flow that uploads files to S3. These flows are failing due to S3

SlowDown

errors while using Metaflow's

s3c.put_many

. I see that the client already is supposed to handle this, but I guess the if there are too many concurrent flows running, the backoff isn't sufficient. Is there a pattern you would recommend to work around this? Would it make sense to create a feature request to expose the number of retries as a kwarg?

✅ 1

user

01/30/2023, 2:52 PM

Copy code

Traceback (most recent call last):
File "/home/ray/venv.mapnik/lib/python3.8/site-packages/metaflow/datatools/s3op.py", line 223, in worker
s3.upload_file(url.local, url.bucket, url.path, ExtraArgs=extra)
File "/home/ray/venv.mapnik/lib/python3.8/site-packages/boto3/s3/inject.py", line 129, in upload_file
return transfer.upload_file(
File "/home/ray/venv.mapnik/lib/python3.8/site-packages/boto3/s3/transfer.py", line 285, in upload_file
raise S3UploadFailedError(
boto3.exceptions.S3UploadFailedError: Failed to upload ... to ...: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.
Task failed.

dry-beach-38304

01/30/2023, 3:25 PM

Which version of metaflow are you on? It should back off and also retry a lot more (and print something out).

user

01/30/2023, 4:09 PM

@dry-beach-38304

metaflow==2.7.18

dry-beach-38304

01/30/2023, 4:39 PM

That should be recent enough. Could you post the full execution log to gist or something so I can take a look. I’ll look at the config variables as well to see whr you can tweak. There are some. I’ll get back to you later today. It should handle it though so this is somewhat surprising.

square-wire-39606

01/31/2023, 10:28 PM

@User can you share the full execution log?

dry-beach-38304

01/31/2023, 10:55 PM

@square-wire-39606 : he DMd me yesterday. It was using an older version of metaflow prior to the fix. He was going to check why.

👍 1

Open in Slack

Previous Next