Hello. I set up AWS resources using Terraform and now I'm working through the examples in the Effective Data Science Infrastructure book. When I get to the first one that uses anything in AWS, process_demo.py, I'm getting an error that says "S3 access denied" with no further explanation. I've verified that my workstation AWS credentials have full permissions for the metaflow s3 bucket and so does the METAFLOW_ECS_S3_ACCESS_IAM_ROLE. I'm not sure what else do. Any advice?

remind me does this flow run locally or on AWS batch?

I'm able to run it successfully locally but I encounter the error when I use with batch.

can you double check check if your local aws creds have both GetObject and ListBucket permissions for that bucket? Particularly this reminds me of some past issues when ListBucket was missing

I'm able to run aws s3 ls s3://le-metaflow-s3-dev just fine. Is there anything I might need to add to the metaflow config file or will it automatically check my aws config/credentials file?

yes it should pickup default creds just like aws cli 🤔

image.png

I think its supposed to be missing, since the run hasn't started, its just checking if its there or not. But you also get that Access Denied if the file is missing and you don't have ListBucket .

Sorry, I should have clarified. I could run without batch when I used the local store, not s3. When I have the metaflow config datastore set to s3, I still get this access denied issue when I run without batch.

Aha that narrows it down, gotta be something with local AWS creds. I wonder if somehow aws CLI is using a different set of credentials there

Okay, that would make sense. Using the CLI, get-user is coming back with what I was expecting. Anyway to check what's being used running the metaflow script?

can you try

aws sts get-caller-identity

in the same terminal where you run metaflow script and see if that matches what you'd expect?

aws sts get-caller-identity returns what I expect. process_demo has nothing dealing with AWS credentials

at the top of the flow, if you add

from metaflow import get_aws_client
print(get_aws_client("sts").get_caller_identity())

that prints the same thing, right?

Where is that import coming from? I'm getting an error from it.

from metaflow import get_aws_client

that bit?

yeah

hmm try

from metaflow.plugins.aws.aws_client import get_aws_client

Okay, that worked. The client is showing the role I expect and the s3 method is giving an error from the file not being found.

but then you get that S3AccessDenied later still ?

yeah

ok that's really weird then, maybe time for plan B https://outerboundsco.slack.com/archives/C02116BBNTU/p1700165895083919?thread_ts=1700146225.118289&cid=C02116BBNTU

That's what I started looking into, I was thinking there might be something weird going on with the aws config

great! no worries! we're always happy to help, if you hit this someone else could too

I’m running into similar S3 bucket access issues (also working through the Effective Data Science Infrastructure book). I’ve been using the AWS CloudFormation template instructions here, and I’ve attached a screenshot of the issue I’m running into. In this case, this Yelp dataset should be a publicly accessible S3 bucket, and running

aws s3 ls <s3://fast-ai-nlp/yelp_review_full_csv.tgz>

on my local machine has the following output (

2018-10-08 14:54:02  196146755 yelp_review_full_csv.tgz

) — which I assume means I have access? This issue happens when I run the flow on AWS Batch (the local run works perfectly fine when I’m grabbing the same dataset on S3). So my guess is that there’s something not working with Metaflow grabbing the local AWS security creds, but I can’t figure out how to fix this. I’ve looked through all the other S3 bucket access issue posts on this Slack, and tried a combination of different tactics to try to fix this: • Reinstalling aws-cli, Metaflow • Creating new AWS security access keys and reconfiguring these details in my local machine’s aws-cli — running

aws configure

and following all the instructions there ◦ I’m not doing any IAM roles because I’m new to this and wanted the least complexity, so just followed the access key instructions in the book — maybe this isn’t the right way to do this?) • Reconfiguring multiple CloudFormation stacks just to make sure it wasn’t some incorrect copy-paste during configuration ◦ I’m just doing

metaflow configure aws

after I’ve executed the CloudFormation template, and plugging in the corresponding service configurations My apologies if anything I wrote here is confusing — I’m very new to all of this stuff and trying to find my way through. Happy to provide any additional details that I’ve missed.