Hey folks! Running into an

S3 access denied

error when locally running a flow connected to a remote datastore:

Metaflow 2.8.1 executing SalaryTrainer for user:winstonvan
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    S3 access denied:
    <s3://metaflow-datastore/metaflow/flow/342/_parameters/11645/0.attempt.json>

We’ve set up a IAM user policy to access the metaflow datastore bucket below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketVersions",
        "s3:GetBucketTagging",
        "s3:GetBucketVersioning",
        "s3:GetObject*",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::metaflow-datastore",
        "arn:aws:s3:::metaflow-datastore/*"
      ]
    }
  ]
}

Is there a permission that we’re missing to allow metaflow to access our S3 datastore bucket?

can you add

HeadObject

as well

Hmm it seems like

s3:HeadObject

isn’t a valid permission, but looking at this stackoverflow thread, it seems like the head operation requires

ListBucket

permission, along with specifying the bucket and the objects within the buckets for the target resources. I already have this, so I’m not sure why it’s not working 😕

get, put and head

ah interesting, let me take a look

did it help?

I am experiencing the same issue while setting up a new Role to talk to AWS

@square-wire-39606 are these the right KMS permissions?

@square-wire-39606 sorry for the late reply — had to step out for a bit, but yep, it seemed like the original error was due to the

KMS:GenerateDataKey

permission, but after granting that permission, I’m now getting this 😭

if possible I'd start with

kms:*

to verify that it works and then work backwards to narrow it down if needed

ah got it, let me give decrypt and encrypt a try

another potential gotcha sometimes worth checking is if your AWS org is using SCP https://github.com/Netflix/metaflow/issues/1191 (this is not enabled by default, but used by some bigger companies to enforce org level policies)

the only SCP I'm seeing is one called FullAWSAccess

@average-beach-28850 I retried with blanket

kms:*

, still getting an S3 access denied error after the

Pylint is happy!

step (apparently on

0.attempt.json

creation, before

start

step)

it is correctly incrementing the run id, so it's obiously able to read the S3 bucket. I also adding the permissions from here

I ended up adding the

kms:GenerateDataKey

,

kms:Encrypt

, and

kms:Decrypt

permissions and everything works now 😄!! Without the

GenerateDataKey

permission, we get the s3 access denied before the start step, so it seems like it was necessary. Here’s our policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetBucketTagging",
        "kms:Decrypt",
        "kms:Encrypt",
        "s3:ListBucketVersions",
        "kms:GenerateDataKey",
        "s3:ListBucket",
        "s3:GetBucketVersioning",
        "s3:DeleteObject",
        "s3:GetObject*"
      ],
      "Resource": [
        "arn:aws:s3:::metaflow-datastore/*",
        "arn:aws:s3:::metaflow-datastore",
        "arn:aws:kms:xxx"
      ]
    }
  ]
}

I double checked all those permissions, still getting the error :c

I figured out the problem. I had been setting the Role in StepFunctions, assuming that Role would be used inside the docker container itself. I had to set my Role in the "Job role configuration" of the Batch Job Definition

I am still getting this s3 Access Denied error, on @batch, it looks like when Metaflow tries to read/download a conda dependency.

Access denied to URL: s3://[bucket_name]/metaflow/conda/conda.anaconda.org/conda-forge/linux-64/abseil-cpp-20210324.2-h9c3ff4c_0.tar.bz2/baa652d7d0da41d757a31d00b4ae2c38/abseil-cpp-20210324.2-h9c3ff4c_0.tar.bz2.

I deployed all infra from the Terraform project and verified that the IAM batch step role was provisioned correctly, with the get/put/listbucket permissions for the bucket, kms permissions… bucket and Batch job are in the same region. I’ve relaxed IAM permissions, testing with stars for s3 and kms perms. I’ve even tried adding s3 permissions to the batch execution service role on the compute environment. I’ve searched here for

MetaflowS3AccessDenied

and followed all recommendations I believe, but no luck. My local flow steps run fine, it only fails on @batch. (abseil is apparently a C++ package, must be a dep of something in my conda base:

@conda_base(python='3.9.7', libraries={'pandas': '1.4.4', 'boto3': '1.24.59', 'fsspec':'2022.10.0', 's3fs':'2022.10.0', 'botocore':'1.27.59', 'aiobotocore':'2.4.0','pyarrow':'8.0.0'})

The only other thing I can think of that could be a factor is I’m trying to use a custom image from ECR. When I try using a stock image with simple @resources, my job never starts because it doesn’t associate itself with a log group for some reason. But when I run with the custom image, it does pick up the correct log group. It runs but fails 14 seconds in consistently. Super confused here because I’m able to run the batch step in Ch. 5 of the tutorials.

last resort for me is usually trying to find that failed request in CloudTrail (you'd need to enable cloudtrail for S3 API requests)

Ok, I’ve narrowed it down a bit more and I do think this error is an issue with the custom ECR image I’m trying to use, e.g.

@batch(image='[ECR image URI]')

I was able to get my step running on a stock Python image without the error, so it must be something about the container that’s spun up on the image or a networking issue of some type around the task, where the privileges are stopping short of the task or container in this specific case.