I’m running into similar S3 bucket access issues (also working through the Effective Data Science Infrastructure book). I’ve been using the AWS CloudFormation template instructions here, and I’ve attached a screenshot of the issue I’m running into. In this case, this Yelp dataset should be a publicly accessible S3 bucket, and running

aws s3 ls <s3://fast-ai-nlp/yelp_review_full_csv.tgz>

on my local machine has the following output (

2018-10-08 14:54:02  196146755 yelp_review_full_csv.tgz

) — which I assume means I have access? This issue happens when I run the flow on AWS Batch (the local run works perfectly fine when I’m grabbing the same dataset on S3). So my guess is that there’s something not working with Metaflow grabbing the local AWS security creds, but I can’t figure out how to fix this. I’ve looked through all the other S3 bucket access issue posts on this Slack, and tried a combination of different tactics to try to fix this: • Reinstalling aws-cli, Metaflow • Creating new AWS security access keys and reconfiguring these details in my local machine’s aws-cli — running

aws configure

and following all the instructions there ◦ I’m not doing any IAM roles because I’m new to this and wanted the least complexity, so just followed the access key instructions in the book — maybe this isn’t the right way to do this?) • Reconfiguring multiple CloudFormation stacks just to make sure it wasn’t some incorrect copy-paste during configuration ◦ I’m just doing

metaflow configure aws

after I’ve executed the CloudFormation template, and plugging in the corresponding service configurations My apologies if anything I wrote here is confusing — I’m very new to all of this stuff and trying to find my way through. Happy to provide any additional details that I’ve missed.