Quick question - does anyone know what could be causing this error? Is it some permissions issue with the s3 bucket? If it is, interestingly, the bucket policy gives root access to the account id and I have authenticated to AWS

2023-03-13 10:58:10.815 [8/train_xgb/18 (pid 5326)] Metadata request (/flows/GradientBoostedTreesFlow/runs/8/steps/train_xgb/tasks/18/artifact) failed (code 403): {"message":"Forbidden"}

that seems to be related to the metadata service, not S3

Oh ok. The reason I surmised it could be s3 related was because it looked like some sort of Artifact issue. Some of the basic Hello World type examples seem to work no problem.

strange. Do you have some custom auth set up for the metadata service?

Only the API key and a lambda authorizer but if that was an issue the flow wouldn't have started up at all.

is the issue reproducible?

Yes - I've been able to isolate the issue 🙂. I created a notebook environment on a private VPC connecting to the internal metadata service url, and no issues with any of the flows. So that informed that there's nothing wrong with the Metaflow services themselves. So it was definitely the API Gateway that we configured yet again. Currently, removing layers one at a time - WAF, Lambda Authorizer, API Key, DNS, etc.

ok, hopefully you’ll find the root cause

Yup it's the WAF. Once I removed it, things ran fine.

great!