Hi team, in order to query Athena and update the c...
# ask-metaflowc
clever-pencil-68951
03/08/2023, 7:21 PMHi team, in order to query Athena and update the catalog, a session has to be passed into the awswrangler SDK which will need to have a policy to allow it access to the AWS resources (athena,s3,glue). When running this from within the context of Metaflow, would I need to configure AWS CLI and setup the profile in that container to assume a role that has this policy attached, or can I just add this extra policy to an existing role as per the metaflow configuration (the json returned from the metaflow TF module). If I can attach to a role returned from the metaflow config which role will it be? Below are all the roles as created from the TF module (metaflow). I've highlighted the one I think I need to attach the extra policy too, but if you can point me in the right direction if will save trail and error time...
✅ 1
a
average-beach-28850
03/08/2023, 7:24 PMmetaflow-batch_s3_task_role-pocc
clever-pencil-68951
03/08/2023, 7:46 PM👍
clever-pencil-68951
03/09/2023, 11:16 PMHi Oleg quick question...Using the example from the documentation I see that no boto3 session is passed into the AWS SDK for pandas when performing a sql query against Athena, which implies that a default session is being used....where does this default configuration sit that allows connection to Athena/Glue/S3? Is this something that will have to added into the Docker file in order to configure this default credential so that the img going to ECR can run this flow and extract the data from Athena or will the session be based on the metaflow-batch_s3_task_role and so when I attach the extra policy of access to this role the AWSwrangler SDK will be able to authenticate successfully?
a
average-beach-28850
03/10/2023, 2:06 PMyes, the latter, on AWS Batch this code will run with the task role
c
clever-pencil-68951
03/14/2023, 6:59 AMHi Oleg, I've attached the additional policy to the task role in order for the flow application that runs under this role to have access to Athena and Glue (data) so the flow can use awswrangler to run SQL, plus write to and update the Glue catalog...however it seems like this package looks for a configured profile when it runs. Did you configure any profiles as part of the image that you push to ECR? Currently when I look into the log streams I'm getting no profile found error and the batch task fails on load
clever-pencil-68951
03/14/2023, 9:29 AMHi Oleg, hang on, I suspect the problem is between the keyboard and monitor..Will revert back after my investigation...
2 Views