Hi all - we're getting the following error when trying (for the first time) to run our terraform-aws-metaflow instance with batch:

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the DescribeJobQueues operation: User: arn:aws:iam::12345678910:user/our-ds-user is not authorized to perform: batch:DescribeJobQueues on resource:

Looking at the TF here and in our cluster, it doesn't seem like the

METAFLOW_ECS_S3_ACCESS_IAM_ROLE

is granted batch permissions? We are also setting

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

in our .env file to a custom IAM created to access a non-metaflow s3 bucket the flow needs to access. All flows run locally, hitting the metadata service and uploading artifacts to s3, all visible with the UI. Our

config_metaflow.json

looks like this:

{
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://our_bucket/metaflow>",
  "METAFLOW_DATATOOLS_S3ROOT": "<s3://our_bucket/data>",
  "METAFLOW_BATCH_JOB_QUEUE": "arn:aws:batch:us-east-1:12345678910:job-queue/metaflow-batch-queue",
  "METAFLOW_ECS_S3_ACCESS_IAM_ROLE": "arn:aws:iam::12345678910:role/metaflow-batch-s3-task-role",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://metaflow-nlb-123.elb.us-east-1.amazonaws.com/>",
  "METAFLOW_SERVICE_URL": "<https://metaflow-metadata-service.company.com>",
  "METAFLOW_SFN_STATE_MACHINE_PREFIX": "metaflow-"
}

Wondering how the batch job queue permissions should be working and if setting the AWS access key and secret creds in the .env is problematic? Thanks! cc: @green-analyst-32514 @billions-city-11284

METAFLOW_ECS_S3_ACCESS_IAM_ROLE is the role that tasks running on AWS Batch will use to access S3. The tasks run on AWS Batch; but the code running inside these tasks typically doesn't need access to AWS Batch API itself. The AWS credentials you use locally to launch the flow (that is, where you run

flow.py run

) do need Batch API access. I basically think of it as "my laptop AWS identity" vs "Batch task AWS identity".

When you say "I'd just extend their permissions rather than creating custom IAM user" do you mean you'd avoid using the access key and secret we created? If so, can you explain a bit more how you'd give the laptop access to batch?

One option is to create an IAM policy that allows to read/write that S3 bucket (and KMS key used to encrypt it if necessary). Attach that policy to METAFLOW_ECS_S3_ACCESS_IAM_ROLE and the IAM user / role that you use on your laptop. The latter part really depends on how you set up and distribute AWS credentials to employees, e.g. do you create an IAM User and give them static keys, or use AWS IAM Identity Center, some Okta integration etc.

ok got it - thanks for the clarification! One more question - do you all have docs of permissions the laptop identity needs? Didn't see it anywhere, but might have missed it

this is a good starting point https://github.com/outerbounds/metaflow-tools/blob/master/aws/terraform/metaflow/iam-custom-role.tf but you may not need sagemaker:* stuff, important ones are

batch:*

,

states:*

events:*