https://outerbounds.com/ logo
Join Slack
Powered by
Hey folks, I am trying to run a flow where I am up...
# ask-metaflow
a
Hey folks, I am trying to run a flow where I am uploading and fetching files from s3 using metaflow's S3 API call but I am getting error: 
botocore.exceptions.NoCredentialsError: Unable to locate credentials
. This happens only if I run with 
--max-workers
>1 but with 
--max-workers 1
all seems to work fine but that is not desirable for my use case. Can someone please help on this front?
✅ 1
If it helps I have attached the full error stack
Copy code
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] Error with S3 operation: Traceback (most recent call last):
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/metaflow/datatools/s3op.py", line 224, in worker
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] s3.upload_file(url.local, url.bucket, url.path, ExtraArgs=extra)
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/boto3/s3/inject.py", line 129, in upload_file
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] return transfer.upload_file(
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/boto3/s3/transfer.py", line 279, in upload_file
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] future.result()
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
2022-11-20 20:52:42.182 [42847/generate_upload_arr/6402925 (pid 22958)] return self._coordinator.result()
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] raise self._exception
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] return self._execute_main(kwargs)
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] return_value = self._main(**kwargs)
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/upload.py", line 694, in _main
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] return self._make_api_call(operation_name, kwargs)
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 662, in _make_api_call
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] http, parsed_response = self._make_request(
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 682, in _make_request
2022-11-20 20:52:42.183 [42847/generate_upload_arr/6402925 (pid 22958)] return self._endpoint.make_request(operation_model, request_dict)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] return self._send_request(request_dict, operation_model)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 132, in _send_request
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] request = self.create_request(request_dict, operation_model)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 115, in create_request
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] self._event_emitter.emit(event_name, request=request,
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] return self._emitter.emit(aliased_event_name, **kwargs)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] return self._emit(event_name, kwargs)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] response = handler(**kwargs)
2022-11-20 20:52:42.184 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/signers.py", line 90, in handler
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] return self.sign(operation_name, request)
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/signers.py", line 162, in sign
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] auth.add_auth(request)
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/auth.py", line 373, in add_auth
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] raise NoCredentialsError()
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] botocore.exceptions.NoCredentialsError: Unable to locate credentials
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] Traceback (most recent call last):
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/metaflow/datatools/s3op.py", line 224, in worker
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] s3.upload_file(url.local, url.bucket, url.path, ExtraArgs=extra)
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/boto3/s3/inject.py", line 129, in upload_file
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] return transfer.upload_file(
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/boto3/s3/transfer.py", line 279, in upload_file
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] future.result()
2022-11-20 20:52:42.185 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] return self._coordinator.result()
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] raise self._exception
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] return self._execute_main(kwargs)
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] return_value = self._main(**kwargs)
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/s3transfer/upload.py", line 694, in _main
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
2022-11-20 20:52:42.186 [42847/generate_upload_arr/6402925 (pid 22958)] return self._make_api_call(operation_name, kwargs)
2022-11-20 20:52:48.301 [42847/generate_upload_arr/6402927 (pid 22407)] nest_0_flight_30105
2022-11-20 20:53:39.730 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 662, in _make_api_call
2022-11-20 20:53:39.730 [42847/generate_upload_arr/6402925 (pid 22958)] http, parsed_response = self._make_request(
2022-11-20 20:53:39.730 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 682, in _make_request
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] return self._endpoint.make_request(operation_model, request_dict)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] return self._send_request(request_dict, operation_model)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 132, in _send_request
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] request = self.create_request(request_dict, operation_model)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/endpoint.py", line 115, in create_request
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] self._event_emitter.emit(event_name, request=request,
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] return self._emitter.emit(aliased_event_name, **kwargs)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] return self._emit(event_name, kwargs)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] response = handler(**kwargs)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/signers.py", line 90, in handler
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] return self.sign(operation_name, request)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/signers.py", line 162, in sign
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] auth.add_auth(request)
2022-11-20 20:53:39.731 [42847/generate_upload_arr/6402925 (pid 22958)] File "/opt/conda/lib/python3.8/site-packages/botocore/auth.py", line 373, in add_auth
2022-11-20 20:53:39.732 [42847/generate_upload_arr/6402925 (pid 22958)] raise NoCredentialsError()
2022-11-20 20:53:39.732 [42847/generate_upload_arr/6402925 (pid 22958)] botocore.exceptions.NoCredentialsError: Unable to locate credentials
2022-11-20 20:53:39.732 [42847/generate_upload_arr/6402925 (pid 22958)] s3op failed:
2022-11-20 20:53:39.732 [42847/generate_upload_arr/6402925 (pid 22958)] Unknown error
2022-11-20 20:53:39.732 [42847/generate_upload_arr/6402925 (pid 22958)] Worker process failed (exit code 8)
I am using 
metaflow 2.5.3
s
that's peculiar. Are you running this on your laptop or on a cloud workstation?
it seems that your credentials provider doesn't like too many concurrent requests for some reason 🤔
a
@straight-shampoo-11124, I am working on an EC2 instance with IAM role credentials.
s
what's the EC2 instance type? I am asking since the smallest instances may have hard time dealing with high levels of concurrency
a
Copy code
g5-48xlarge-1
that's a pretty big machine with 8 GPUs and about 748GB RAM.
it seems that your credentials provider doesn't like too many concurrent requests for some reason
shouldn't it have produced some 
large request rate
error or something, instead of 
NoCredentialsFound
?
@straight-shampoo-11124 I have tried breaking my upload request to 100 files at a time (using for loop) and the flow is still crashing.
Is there any information I can provide to debug this?
s
that's mysterious. Is it so that it fails immediately with 
--max-workers 2
or is it sporadic?
I presume you don't have any special AWS configuration but you are just relying on instance metadata for credentials?
a
Is it so that it fails immediately with 
--max-workers 2
or is it sporadic?
It fails bit sporadically. It works well for a while but then it crashes with above error stack
I presume you don't have any special AWS configuration but you are just relying on instance metadata for credentials?
Yes, I can configure rolling credentials but they would expire before the flow finishes.
s
have you tried it on any other instance by any chance?
a
yes, I tried it on couple of other instances too and they fail with same error
s
were they 
g5-48xlarge-1
too or some other types?
what you are trying to do seems totally typical and standard, so I am trying to figure out what's special about your situation that makes it fail 🤔
a
were they 
g5-48xlarge-1
too or some other types?
Nope, I tried that on a single GPU machine with 48GB RAM.
what you are trying to do seems totally typical and standard, so I am trying to figure out what's special about your situation that makes it fai
I am totally puzzled here too. We have flows that takes about 24 hours and they are constantly upload and fetch data using S3 api. Not sure what's happening here. The only big difference here is that we are looking to upload a much higher number of files at a time.
Maybe I am wrong; but I get a sense that error is misleading here to some other underlying issue.
s
right.. might be. Can you try executing 
ForeachFlow
listed here and see if you get errors with 
--max-workers=2
?
a
I actually did try something very simple like that and the flow worked fine. It just errors out with 
s3
API call in the step
s
just to clarify, a simple flow with a foreach worked with 
--max-workers=2
flawlessly?
a
yup
s
ok, so the difference between a simple flow (that works) and your flow (that doesn't) is that your flow interacts with S3 much more actively, right?
do you use 
metaflow.S3
explicitly or just many 
self.
artifacts?
a
I have temporary files that I am uploading to S3 bucket using the API call.
s
is it a lot of data in terms of bytes and/or many files?
we have some improvements coming up for the S3 client that may help here. @dry-beach-38304 - have you seen 
botocore.exceptions.NoCredentialsError: Unable to locate credentials
during your explorations?
d
I have not but we use a slightly different way of providing credentials internally so it is possible I did not run into this specifically due to this. let me look at the trace.
which version of boto/botocore are you using?
a
1.24.87
d
it’s old but not prehistoric. You can try: • updating it (who knows, it may just work) • use 
AWS_METADATA_SERVICE_NUM_ATTEMPTS
to increase the number of times boto will try to get the credentials. I was just googling around and found this: https://github.com/boto/botocore/issues/2117 but as @victorious-lawyer-58417 points out, this is usually a problem for small instances so it is a bit weird that it is hitting you here.
the upcoming S3 changes do not seem to address this issue although we could classify this type of error as intermittent. It’s a bit weird though because this is usually a fatal error so we would have to do something a little more funky to deal with it and properly distinguish a “this is an actual error” vs “something in S3 is intermittently failing”. The upcoming changes deal more with transient errors due to SlowDown responses (or other transient errors — transient as identified by AWS).
a
Thanks @dry-beach-38304! I am re-writing the flow to avoid sending 1000s of upload request using S3 API. I will get back to you if issue persists in the new flow too.
s
@abundant-furniture-62668 just to confirm, you have one 
S3.put_files
or 
S3.put_many
call that uploads 1000s of files?
a
I have 
s3.put_files
v
cool, and that 
s3.put_files
is inside some foreach task, so many 
put_files
get executed in parallel if you do 
--max-workers=2
or higher, right?
a
@straight-shampoo-11124, yes
v
makes sense. @dry-beach-38304 - this sounds similar to the issue you reported, right?
d
my issue was due to slowdown (so AWS server side crapping out). this error feels more client side (can’t even send to S3). We currently (and even with my patch) don’t consider those errors to be transient so they would still not be retried. We could change that but distinguishing a true transient one from a non transient one would be a bit trickier.
v
yeah, but in both cases the root cause is that uploading inside foreach causes an amplified load to S3
since we don't have any kind of rate limiting across tasks
d
yes. That is true. The weird thing here is that per client, the load shouldn’t be higher per client. That’s the part that is surprising to me.
foreach will increase the load on S3 for sure and there max-workers has definite impact but I am a little baffled that client side errors are sensitive to that.
s
it's X*K workers where 
--max-workers=X
and K is the number of parallel 
s3op
workers which is enough to overwhelm instance metadata
d
oh wait, this is running on a single machine. sorry, I stupidly assumed these were running on batch or something.
👍 1
s
@abundant-furniture-62668 anyways, thanks for reporting this! As you can see, we have been improving related issues recently 🙂 We can make sure that we address this case properly too
meanwhile one workaround is to micro-batch 
put_files
calls so that instead of uploading all files at once, upload e.g. 100 files at a time and then sleep for a few seconds (randomly) between calls to avoid overwhelming the instance another option is to distribute processing over multiple instances with 
@batch
or 
@kubernetes
which may help a bit
a
Thanks @straight-shampoo-11124 and @dry-beach-38304 for looking into this. We do not have @batch or @kubernetes enabled at the moment but I’m working on re writing the flow to not send such large amount of upload requests.
s
cool! Let us know how it goes. Also if it is ok with you, we may ask you to test an improved version once its out
a
Sure
👍 1
8 Views
Open in Slack
PreviousNext
Powered by