Hi all! Fairly new to metaflow and trying to figure out this issue running with batch: We are running into an issue where the EC2 instance that kicks off the scripts (so for example, if I ssh into the box and run python metaflow_test.py run --with batch from the box) it returns

Metaflow 2.8.0 executing test for user:ec2_test
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    S3 access denied:

This all kicks off correctly running off of local. Additionally, when checking the actual bucket using the amazon cli

aws s3 ls

using the ec2 default profile (which would be the same one running the test script) I am able to see the contents of the bucket I suspect that the issue lies within permissioning around the role with the ec2 instance (we can call it arn:ec2_test) and the other roles How do these roles assume each other? Is the arn:ec2_test assuming the role of "METAFLOW_ECS_S3_ACCESS_IAM_ROLE" etc locally, and then trying to connect to s3 for example? Or is the issue likely that permissions are okay and maybe we're missing something in a config file? The arn:ec2_test role has admin on the s3 bucket, so I don't believe that's the issue

@curved-jewelry-19125 what is the error stack that you see below

s3 access denied

?

Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    S3 access denied:
    <s3://metaflow-s3/metaflow/test_flow/231/_parameters/1152/0.attempt.json>

Is all that pops out, is there a --verbose flag, or something similar?

are you able to read that file directly using the aws cli?

That file doesn't actually get generated, so the run 231 doesn't appear in the s3 bucket, But when I try to read one of the files from another bucket (so run 216 which exists) using aws s3 ls, I'm able to read that bucket/file

are you able to make a put request?

yeah it has putObject permissions

the default role on the instance

gotcha. Yeah the default role on the ec2 instance has full access on both s3 and on aws batch, so not certain why its failing

are you able to make head calls as well?

@square-wire-39606 you genius, thank you. We had full s3 rights no problem, but we just copied a text file from the ec2 box to the metaflow bucket and it blocked us on KMS:GenerateDataKey