image.png

--with batch

gives me the same error in the

start

step

@straight-shampoo-11124: here’s how the stack currently stands (focus on the fargate service, ecs cluster, and the roles defined for the fargate service. I’ve attached the cdk definition for the cluster and fargate service. for the respective task role (

metaflowbatchs3taskrole68AA1286

) and execution role (

metaflowbatchserviceroleE707E0E1

), I’ve also attached the cdk synth yaml output for these roles below to match with the metaflow reference yaml from the Outerbounds site.

metaflowbatchs3taskrole68AA1286:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: <http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/AWSXRayDaemonWriteAccess
      Policies:
        - PolicyDocument:
            Statement:
              - Action: s3:ListBucket
                Effect: Allow
                Resource: arn:aws:s3:::*
                Sid: BucketAccessBatch
            Version: "2012-10-17"
          PolicyName: CustomS3ListBatch
        - PolicyDocument:
            Statement:
              - Action:
                  - s3:PutObject
                  - s3:GetObject
                  - s3:DeleteObject
                Effect: Allow
                Resource:
                  - arn:aws:s3:::*
                  - arn:aws:s3:::*/
                Sid: ObjectAccessBatch
            Version: "2012-10-17"
          PolicyName: CustomS3Batch
        - PolicyDocument:
            Statement:
              - Action:
                  - sagemaker:CreateTrainingJob
                  - sagemaker:DescribeTrainingJob
                Effect: Allow
                Resource:
                  Fn::Join:
                    - ""
                    - - "arn:"
                      - Ref: AWS::Partition
                      - ":sagemaker:"
                      - Ref: AWS::Region
                      - ":"
                      - Ref: AWS::AccountId
                      - :*
                Sid: AllowSagemakerCreateDescribe
              - Action:
                  - sagemaker:CreateModel
                  - sagemaker:CreateEndpointConfig
                  - sagemaker:CreateEndpoint
                  - sagemaker:DescribeModel
                  - sagemaker:DescribeEndpoint
                  - sagemaker:InvokeEndpoint
                Effect: Allow
                Resource:
                  Fn::Join:
                    - ""
                    - - "arn:"
                      - Ref: AWS::Partition
                      - ":sagemaker:"
                      - Ref: AWS::Region
                      - ":"
                      - Ref: AWS::AccountId
                      - :*
                Sid: AllowSagemakerDeploy
            Version: "2012-10-17"
          PolicyName: AllowSageMaker
        - PolicyDocument:
            Statement:
              - Action: iam:PassRole
                Condition:
                  StringEquals:
                    iam:PassedToService: <http://sagemaker.amazonaws.com|sagemaker.amazonaws.com>
                Effect: Allow
                Resource: "*"
                Sid: AllowPassRole
            Version: "2012-10-17"
          PolicyName: IAM_PASS_ROLE
        - PolicyDocument:
            Statement:
              - Action:
                  - dynamodb:PutItem
                  - dynamodb:GetItem
                  - dynamodb:UpdateItem
                Effect: Allow
                Resource:
                  Fn::Join:
                    - ""
                    - - "arn:"
                      - Ref: AWS::Partition
                      - ":dynamodb:"
                      - Ref: AWS::Region
                      - ":"
                      - Ref: AWS::AccountId
                      - :metaflow/
                Sid: items
            Version: "2012-10-17"
          PolicyName: DynamoDB
        - PolicyDocument:
            Statement:
              - Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Effect: Allow
                Resource: "*"
                Sid: AllowPutLogs
            Version: "2012-10-17"
          PolicyName: Cloudwatch
      RoleName: metaflow-stackBatchS3TaskRole
    Metadata:
      aws:cdk:path: metaflow-stack/metaflow-batch-s3-task-role/Resource

metaflowbatchserviceroleE707E0E1:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: <http://batch.amazonaws.com|batch.amazonaws.com>
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSBatchServiceRole
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/AWSXRayDaemonWriteAccess
      Path: /
      Policies:
        - PolicyDocument:
            Statement:
              - Action: iam:PassRole
                Condition:
                  StringEquals:
                    iam:PassedToService:
                      - <http://ec2.amazonaws.com|ec2.amazonaws.com>
                      - <http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>
                Effect: Allow
                Resource: "*"
                Sid: VisualEditor0
            Version: "2012-10-17"
          PolicyName: IAM_PASS_ROLE
        - PolicyDocument:
            Statement:
              - Action: iam:CreateServiceLinkedRole
                Condition:
                  StringEquals:
                    iam:AWSServiceName:
                      - <http://ecs.amazonaws.com|ecs.amazonaws.com>
                      - <http://autoscaling.amazonaws.com|autoscaling.amazonaws.com>
                Effect: Allow
                Resource: "*"
                Sid: VisualEditor3
            Version: "2012-10-17"
          PolicyName: iam_custom_policies
        - PolicyDocument:
            Statement:
              - Action: ec2:CreateTags
                Condition:
                  StringEquals:
                    ec2:CreateAction: RunInstances
                Effect: Allow
                Resource: "*"
                Sid: VisualEditor4
            Version: "2012-10-17"
          PolicyName: ec2_custom_policies
      RoleName: metaflow-stackBatchServiceRole
    Metadata:
      aws:cdk:path: metaflow-stack/metaflow-batch-service-role/Resource

as far as I can tell, the permissions look to line up. i also went through all the roles defined vs. the metaflow cloudformation yaml on the Outerbounds site and believe I have aligned them up, but still getting the same data store error

i’m wiping the stack and re-deploying again clean to re-verify

just tried to create using the reference cloudformation yaml with RDS removed, and encountering this error on the ECSFargateService. I only removed the RDSMasterInstance, the RDSSecret, and the environment variables in the ECS task definitions. I don’t believe this should have impacted the NLB creation, the yaml looks to line up correctly to me

Resource handler returned message: "Invalid request provided: CreateService error: The target group with targetGroupArn arn:aws:elasticloadbalancing:us-east-1:372645068752:targetgroup/metafl-NLBTa-6BOG8J5PQMLD/b9f4d662ce37e333 does not have an associated load balancer. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 7884e683-6e8d-4f0c-ba90-3be19c327990; Proxy: null)" (RequestToken: 21d04f78-9aa6-02bb-9ce5-ae7ab6d02484, HandlerErrorCode: InvalidRequest)

would it be worth it to still run this cloudformation exercise without RDS in this case?

it is possible that the error you are seeing is caused by something else but it's hard to know especially if your deployment is different than the default

odd, it triggered the same

Data store error

failure, i attached the cfn yaml i deployed for reference, but i am pretty sure the removals were minimal

and the parameters selected

we’ve deployed metaflow in a more strict AWS account using the python stack, but we wanted to move to typescript and also enable sagemaker, the api gateway, etc. as part of the transition, so i took @flaky-plumber-70709’s v1 and converted it to v2, but have continued to encounter odd behaviors i did not expect

i’m going to try something extreme and add the S3FullAccess managed policy to all of the IAM role’s that are created and see if that forces something to be able to access S3 correctly

the step 1 metadata goes to s3 when it runs from my local machine

but definitely no /2 dir

interesting, so i checked the s3 bucket and i do have a hello/ step folder with a 2/ directory for the last run i tried

image.png

but compared to start/1/, i only see the below files

where start/1/ has all of these because the step completed

since the error says

No completed attempts of the task was found for task 'HelloAWSFlow/1673375328209195/hello/2'

, is it looking for a file in particular that should be generated? i am feeling like this might not be an s3 issue but an execution of the step in batch issue

when i look at the failure logs, it just reports

exec /bin/bash: exec format *error*

which i’m not 100% what the exec error is related to

okay this is a good realization, the batch container image I was using seems to have been the culprit. if I want to build a custom batch image, is there a specific image base we should be building from? (this custom image currently works on our python stack, it is likely because i’m building from my local system

@straight-shampoo-11124: thank you for helping me step through this while it may have had something to do with permissions since I added a blanket S3 access to all roles, i’ll revert and see where it is no longer needed. thank you for the help!

hopefully it start working soon 🙂 Let me know how it goes

i always forget about the platform incompatibility when building on macos and deploying on aws ec2

@straight-shampoo-11124 looks like most of the problems are solved now. I’m encountering one more issue that seems to be a bit weird. when I am running

--metadata=local

the execution runs fine and i see the data being created for the runs in s3, but when i change it to

--metadata=service

, it doesn’t run at all, any ideas? the stack is pretty much identical to the cloudformation yaml, so not really sure where to begin, api gateway? nlb?

to use

--metadata=service

you need a few pieces in place: • Metaflow service deployed on ECS • RDS where the service stores its data • Proper VPC setup so all parts can talk to each other If you use our cloudformation/terraform templates, all these pieces will get set up automatically

firstly, can you see the service running on ECS?

and the RDS service is stood up as well

ECS services here

and the task configuration for the metadata service looks as i would expect based on the configuration in the cdk

the service looks to possibly be going through the API endpoint based on the values i’m using in the config provided exported the cdk deployment following Bryan Galvin’s example

"METAFLOW_SERVICE_INTERNAL_URL": "<http://metaf-metaf-1RRPUVW2887Z1-4b6250904057ed29.elb.us-east-1.amazonaws.com/>",
  "METAFLOW_SERVICE_URL": "<http://metaf-metaf-1RRPUVW2887Z1-4b6250904057ed29.elb.us-east-1.amazonaws.com/api>",

@straight-shampoo-11124 your pointers helped now that you’ve helped me better understand some of the infra connections here. I tried using the public ip of the metadata service in the service url as

{publicip}:8080

and that seems to have worked perfectly and the helloaws flow ran without issues. For some reason I cannot connect to it through the elb url as above or the private ip, so some networking to be figured out. i think one part i wasn’t totally familiar with was the API gateway, what does the APIGW service with regard to metaflow?