Hey all - i am running into some issues with metaflow that i could use some help with, but im not really sure. After some time of iterating on a flow, i get the following error:

> python hellworld.py run
Metaflow 2.7.19 executing HelloAWSFlow for user:sderosa
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Metaflow service error:
    Metadata request (/flows/HelloAWSFlow) failed (code 403): {"message":"Forbidden"}

if i remove my

~/.metaflowconfig

, i can run the helloworld flow no problem. Then when i copy it back, i get the same

403

error. I can

aws s3 cp

without any issues. Any idea whats going on ?

this means that you can’t hit your metadata service. What does your

.metaflowconfig

have? It may have something related to

SERVICE_AUTH_KEY

or

SERVICE_HEADERS

.

hm, i have neither of those. I have the following populated:

"METAFLOW_DEFAULT_DATASTORE"
"METAFLOW_DEFAULT_METADATA"
"METAFLOW_DATASTORE_SYSROOT_S3"
"METAFLOW_DATATOOLS_S3ROOT"
"METAFLOW_BATCH_JOB_QUEUE"
"METAFLOW_BATCH_INF_JOB_QUEUE"
"METAFLOW_ECS_S3_ACCESS_IAM_ROLE"
"METAFLOW_SERVICE_URL"
"METAFLOW_SERVICE_INTERNAL_URL"
"METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE"
"METAFLOW_SFN_IAM_ROLE"
"METAFLOW_SFN_DYNAMO_DB_TABLE"
"METAFLOW_ECS_FARGATE_EXECUTION_ROLE"
"METAFLOW_UI_URL"

I am not quite sure. NOt 100% familiar with the setup. @average-beach-28850 probably has a better idea (so tagging him if he is around).

@average-beach-28850 if you get a sec to look at this I'd greatly appreciate it 🙂

hmm does it still happen on and off? if you deployed it using cloudformation template from here https://github.com/outerbounds/metaflow-tools/tree/master/aws/cloudformation it would normally set up auth using a key, so im surprised you don't have it in your config

yes this is still happening on/off. hitting the metadata service url i get:

> curl -v <https://bg758e3hbg.execute-api.us-east-2.amazonaws.com/api/>
*   Trying 10.255.0.165:443...
* TCP_NODELAY set
* Connected to <http://bg758e3hbg.execute-api.us-east-2.amazonaws.com|bg758e3hbg.execute-api.us-east-2.amazonaws.com> (10.255.0.165) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.<http://execute-api.us-east-2.amazonaws.com|execute-api.us-east-2.amazonaws.com>
*  start date: Sep  2 00:00:00 2022 GMT
*  expire date: Aug 31 23:59:59 2023 GMT
*  subjectAltName: host "<http://bg758e3hbg.execute-api.us-east-2.amazonaws.com|bg758e3hbg.execute-api.us-east-2.amazonaws.com>" matched cert's "*.<http://execute-api.us-east-2.amazonaws.com|execute-api.us-east-2.amazonaws.com>"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET /api/ HTTP/1.1
> Host: <http://bg758e3hbg.execute-api.us-east-2.amazonaws.com|bg758e3hbg.execute-api.us-east-2.amazonaws.com>
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Server: Server
< Date: Wed, 25 Jan 2023 21:27:51 GMT
< Content-Type: application/json
< Content-Length: 23
< Connection: keep-alive
< x-amzn-RequestId: 47b7b605-cac6-4798-8289-25c8e1fd78a6
< x-amzn-ErrorType: ForbiddenException
< x-amz-apigw-id: fUXBtGBbiYcFYCQ=
< 
* Connection #0 to host <http://bg758e3hbg.execute-api.us-east-2.amazonaws.com|bg758e3hbg.execute-api.us-east-2.amazonaws.com> left intact
{"message":"Forbidden"}%

does it ever succeed with curl? i think you need to put

METAFLOW_SERVICE_AUTH_KEY

in your metaflow config

yes it passes when this issue doesnt exist. i dont understand why this issue happens sometimes and not others. I'll try adding that to the config.

yes that does it sometimes, depending on a VPN

Hi team I'm running into the same issue. After deploying the AWS stack with TF using your metaflow module, the output json written locally I consume and perform a local-exec calling a python script using boto3 client to take the input coming directly from key "METAFLOW_SERVICE_AUTH_KEY" and programmatically I alter it with the value to reflect the key like this "METAFLOW_SERVICE_AUTH_KEY": "NzWX4K3bED8XduP9yvrok1gchtSHNY8s6065AL7A" I then point to this config doing "metaflow configure import FILE". I then to the step function create command like this: python3 flows/segmentation_training_flow.py --with retry --with batchimage=metaflow batch pocv1 step-functions create --namespace poc. And specify the custom image pushed to ECR. however I get metadata service error (1) and (2) when I look at the state machine code even though I indicate a particular image it still uses a default image