https://outerbounds.com/ logo
Join Slack
Powered by
Is it possible to use cached credentials when gett...
# ask-metaflow
e
Is it possible to use cached credentials when getting Metaflow artifacts? We have a flow with ~1000 splits on a foreach, and when running 
self.merge_artifacts(...)
we're seeing a 
botocore.exceptions.NoCredentialsError
which based on this thread is because we're hitting the AWS Instance Metadata Service too often and exceeding the throttle limits described here
v
(sorry for the delayed reply)
does it happen only when you use 
merge_artifacts
?
e
Yes. We verified this by putting the merge artifacts in a step of its own. We tried setting the AWS key/secret/session token environment variables right before the call to merge_artifacts but it hasn't resolved the issue
Wanted to follow up on this and see if anyone has had experience caching AWS credentials when making many calls that overwhelm the Instance Metadata Service
a
e
Yes I tried that as well. I don't think it's a communication error between the instance and the metadata server. My theory is that it hits the metadata service every time it tries to request an object from S3, and because we're trying to pull so many objects in a short period of time we're hitting the throttle limit described here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
We throttle queries to the IMDS on a per-instance basis, and we place limits on the number of simultaneous connections from an instance to the IMDS.
If you're using the IMDS to retrieve AWS security credentials, avoid querying for credentials during every transaction or concurrently from a high number of threads or processes, as this might lead to throttling. Instead, we recommend that you cache the credentials until they start approaching their expiry time. For more information about IAM role and security credentials associated with the role, see Retrieve security credentials from instance metadata.
I tried getting frozen credentials and setting environment variables like below and the problem persisted:
Copy code
import boto3
session = boto3.Session()
credentials = session.get_credentials()
a = credentials.get_frozen_credentials()
import os
os.environ['AWS_ACCESS_KEY_ID'] = a.access_key
os.environ['AWS_SECRET_ACCESS_KEY'] = a.secret_key
os.environ['AWS_SESSION_TOKEN'] = a.token
👀 1
@ancient-application-36103 Any other thoughts on this? The error popped up again today unfortunately
My first successful workaround is to use the environment variable 
DATATOOLS_DEFAULT_CLIENT_PARAMS
I save the following in a python file
Copy code
import boto3
session = boto3.Session()
c = session.get_credentials().get_frozen_credentials()
print('{{ "aws_access_key_id": "{}", "aws_secret_access_key": "{}", "aws_session_token": "{}" }}'.format(c.access_key,c.secret_key,c.token))
Then when running the flow I can do: 
METAFLOW_DATATOOLS_DEFAULT_CLIENT_PARAMS=$(python creds.py) python flow.py …
This has the disadvantage that my flow can’t be longer than remaining expiry time on the token. I see in the code that it creates a new boto3 session for each and every s3 url that needs downloading which is causing the issue. I’ll create an issue to track this and try to submit a PR next week with a proposed solution
11 Views
Open in Slack
PreviousNext
Powered by