Hi All, new to MetaFlow here and trying to do setup the most vanilla AWS setup here as well. This was the post I referred to for some debugging. I have an extremely similar approach to getting things up. I followed the CloudFormation steps with no changes, and then configured my terminal using

metaflow configure aws

, copying and pasting all the values i see in the outputs to what was asked in the terminal. Still, I'm having this issue. Any help would be greatly appreciated 🙂:

honson@Aisan:~/metaflow$ python3 flow.py run
Metaflow 2.8.0 executing HelloFlow for user:honson
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Metaflow service error:
    Metadata request (/flows/HelloFlow) failed (code 403): {"message":"Forbidden"}

Note: I believe the resolution was this thread was a VPN issue. I'm doing this for more of a personal learning experience, so there shouldn't be any networking or work perm related stuff causing these errors I believe.

have you tried the

curl

command from that thread? that should help debug

Hi @narrow-lion-2703, thanks for following up. Regarding your

curl

command: https://outerbounds-community.slack.com/archives/C02116BBNTU/p1632414121331100?thread_ts=1632408385.327700&cid=C02116BBNTU This was what was returned to me, with the URL and api key redacted:

honson@Aisan:~/metaflow$ curl -v -H 'x-api-key: <REDACTED>' <METAFLOW_SERVICE_URL>/flows/HelloWorld
*   Trying 52.85.61.86:443...
* Connected to <METAFLOW_SERVICE_URL> (52.85.61.86) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.<http://execute-api.us-east-1.amazonaws.com|execute-api.us-east-1.amazonaws.com>
*  start date: Jan 25 00:00:00 2023 GMT
*  expire date: Feb 23 23:59:59 2024 GMT
*  subjectAltName: host "<METAFLOW_SERVICE_URL>" matched cert's "*.<http://execute-api.us-east-1.amazonaws.com|execute-api.us-east-1.amazonaws.com>"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x560f6bea7e80)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /api/flows/HelloWorld HTTP/2
> Host: <METAFLOW_SERVICE_URL>
> user-agent: curl/7.81.0
> accept: */*
> x-api-key: <REDACTED>
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 403
< content-type: application/json
< content-length: 23
< date: Tue, 28 Feb 2023 02:19:11 GMT
< x-amzn-requestid: 6221948e-c7f9-4448-a55d-971d655336e8
< x-amzn-errortype: ForbiddenException
< x-amz-apigw-id: <READCTED>
< x-cache: Error from cloudfront
< via: 1.1 <http://3f65d34f6010e326e59d2f311de6e202.cloudfront.net|3f65d34f6010e326e59d2f311de6e202.cloudfront.net> (CloudFront)
< x-amz-cf-pop: EWR53-P1
< x-amz-cf-id: jaM4v91IhjHdM1rXVdr8f0_DFVGu_oauyp61nQ_E3siw1YitQl2NCg==
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host <METAFLOW_SERVICE_URL> left intact

I think you forgot to redact one instance of api key there. But it looks like its not the actual api key, its api key ID (if it looks like 10 random characters)

I think you forgot to redact one instance of api key there.

redacted, thanks

yes its kind of hidden in the docs https://github.com/outerbounds/metaflow-tools/tree/master/aws/cloudformation#additional-configuration

this right?

yes that's the ID but not the key itself, if you go to aws console for api gateway its here:

Ah, the command worked! Can you also link to where I can access the api key from the dashboard for reference?

yes please, easier to keep track of what questions are open/closed this way :)

Yes!

the key cannot be accessed from Metaflow UI. You can only get it via AWS Console or awscli command from the readme

Just to be clear, any of the [VARIABLE_NAMES] I just match with the

outputs

on CloudFormation right? i.e.,

[METAFLOW_BATCH_JOB_QUEUE]

,

[METAFLOW_ECS_S3_ACCESS_IAM_ROLE]

yes those are metaflow config variable names

yes those are metaflow config variable names

Gotcha, I was confused because every other config was following this schema, but this API key had this nuance to it.

ah yes I can see how that description can be confusing

@average-beach-28850 regarding your screenshot of

Resource Groups & Tag Editor

, I don't seem to have the same layout as you to access my API key. I was able to access it via the CLI command, but I was trying to find the "AWS Console" equivalent. Any ideas? I left the CloudFormation yaml the same, without any chances. Perhaps this API was automatically generated for me?

should be under https://us-west-2.console.aws.amazon.com/apigateway/home (but you may need to switch region to where you deployed the template)

Found it! Thank you so much @average-beach-28850! This was very helpful. I've posted questions in new threads in #C02116BBNTU. Hope you're able to help me out there. I really appreciate your time!