https://outerbounds.com/ logo
Join Slack
Powered by
Hi, I’m trying to setup Metaflow in my company’s i...
# ask-metaflow
m
Hi, I’m trying to setup Metaflow in my company’s internal AWS. I’ve got a API Gateway authorisation type AWS_IAM that’s needed for me to create the api gateway. However , my admin role doesn't have execute-api:Invoke permission so it’s blocking all requests. If i try to make a manual request with the authorisation header in postman it’s fine, but through metaflow I just keep running into a Missing Authentication Token error. I’ve tried enabling the APIBasicAuth to use the API key and try to bypass but same issue. I’m a data scientist and not much of a cloud expert so running out of ideas of what to try to fix this. Has anyone encountered something similar? How did you fix?
f
hey Coraline! so sounds like apigw is set up with IAM auth not api key auth, so even though you have the key its going to fail since its expecting your request to be on the up and up. If you want to use basic auth you need authorization = none w the api key auth to work on its own
was this deployed w the metaflow terraform module?
the missing auth token error is kind of misleading here, its not talking about your api key its about the IAM signature its expecting from you, so it not like you had the wrong key. You just can't do IAM + key auth and have it work
m
Hi Bryan, it was deployed with the cloudformation template. Ok so I can take out the API Basic Auth then, but with only IAM i’ve been trying all day and I’ve also been getting the same error Missing Auth Token
Enabling API Basic Auth was a bit of a last ditch effort
And I don’t think I can have the API Gateway without IAM, my entreprise setup is not letting me.
f
yeah you don't have the iam permissions so the basic auth won't work - that combo is impossible basically. No auth + key is what the terraform I linked to does and is probably the way to go unless your infra is locked down. The token it's talking about isn't what it sounds like, its technically accurate but that error isn't the most clear. The token its talking about is a whole other thing w aws sig4
lemme look up the outerbounds cloudformation template and show you where to look
so the default here I think is auth type = None + key
you have key enabled and fetched it right? maybe I'm silly and missed that part
think you'll need to phone an (infra) friend on this one, your options are: 1. swap out auth = None + keep the key auth 2. keep IAM, get infra friend to grant 
execute-api:Invoke
, but that is not ideal
big takeaway here is that IAM + apikey are mutually exclusive and can't work
m
Ok that’s helpful, yeah debugging with claude code led me to Sig4 too, that’s where I figured that if I passes the authentification header in a postman request it’s fine. I’ll try to figure out if I can get the infra people to let me create an Api Gateway without IAM
f
ahh yeah - can see how that can be super frustrating with how confident CC is. Its not wrong but context
🫠 1
yeah its the easiest way to go - otherwise it may mess with using the metaflow client (depending on how your org handles aws creds)
you can point them to those two links that I sent, both versions had that going on
👍 1
thankyou 1
m
While I’m on it, how does it work with the UI? I’m also getting ”waiting for connection” on that which I guess was due to the IAM + no execute-api:Invoke, but if I go the API route do I need to switch anything up on the UI?
f
that could be a number of things with the network. Do you know if you used the outerbounds cloudformation template or is this an artifact from another time?
cloudformation is about the worst to debug
wait, no
it is the worst to debug
🥲 1
m
I used the cloudformation one, but I also had to frankenstein it quite a bit to fit the architecture
Glad to hear that, it’s been very painful on my side to debug 😄
f
haha it truly is the monster then - but there are tools to help out if it’s something obvious, like https://github.com/aws-cloudformation/cfn-lint. Though it can be wily to tune down. If you’re in vscode you should get the extension it’s pretty good for obv issues
😮 1
but cdk is just infinitely better since its operating at a much higher level and can be managed in Python etc. i think @lively-lunch-9285 has a Python metaflow cdk construct floating around somewhere if this gets too frustrating
👍 1
m
That’s great to know, will explore cfn-lint. I’ll try to make the API gateway work first and hope that fixes the UI, it’s been a 10 days of trying to make this setup work with the internal architecture, and for a non-cloud person, it’s been a steeeep learning curve
Ok so it looks like I already have execute-api:Invoke permission, and: aws apigateway test-invoke-method --rest-api-id {id} --resource-id {id} --http-method GET --path-with-query-string "/ping" --region eu-north-1 does reach the NLB, and send back a pong. But I still get Metadata request (/flows/DataPrepFlow) failed (code 403): {"message":"Missing Authentication Token"} when testing a flow out.
f
yep that’s the IAM thing from earlier
which means you need aws creds to use the metaflow client w this setup, which may be a high bar depending on your org
m
I’m running this from my aws profile
I’m on Metaflow 2.18.3 also in case that’s relevant
f
maybe you didn’t get the key from the cfn template?
m
Which key? I do have "METAFLOW_SERVICE_USE_AWS_CLIENT": "true", "METAFLOW_SERVICE_AUTH_KEY": "" (since not enabling API Basic Auth)
f
can you share your Frankenstein monster?
👍 1
m
I saw this earlier, but this mentions browser issues, mine is first just the apigateway in the code itself
that's my template
metaflow-cfn-template.yml
f
so we could debug this sure, however de d you’d still be tweaking a yaml cloudformation template. And that will no doubt cause frustration and possibly tears, which isn’t your fault it’s the tool. Cdk though is OO language of your choice, the thing w apigw would be handled intuitively by you thanks to the IDE superpowers of having that typed completion
it’s fun even
I’ll still help you out in a bit but it’s doesn’t have to suck so bad hah
😮 1
ohh and Claude rocks at it because it’s popular mature and typed
😮 1
Copy code
from aws_cdk import aws_apigateway as apigw
  from aws_cdk import aws_iam as iam

  api = apigw.RestApi(self, "MetaflowApi",
      rest_api_name="metaflow-service"
  )

  # method with IAM auth
  resource = api.root.add_resource("myresource")
  resource.add_method("GET",
      authorization_type=apigw.AuthorizationType.IAM
      # api_key_required is ignored when using IAM
  )

  # grant invoke permissions to a role
  role = iam.Role(self, "ApiInvokerRole", ...)
  api.grant_invoke(role)
all that IAM heartache replaced with a least privilege grant_invoke method, that your IDE knows about
I'll help you w the monster in a bit promise
m
That’s super good to know about, i’ll definitely check it out, I’m not much of a cloud specialist so I just followed the Outerbounds instructions and hoped for the best
f
so I 100% recommend you cease and desist w the cloudformation, however, this should do it according to my robot anyways. I just can't look at cfn yaml. Thought that I could but I can't. The changes made were:
Copy code
1. added `APIAuthorizationType` parameter can be NONE or AWS_IAM. When AWS_IAM you need sigv4 signing (issue you've been seeing for days) 

2. updated apigw methods from AWS_IAM to ref of the parameter, before it was hardcoded so even if you added api key it would do nothing

3. output clarified on metaflow URL, `METAFLOW_SERVICE_URL` you can skip auth for api gateway and hit the nlb direct like
metaflow-cfn-template.yml
cfn-lint found a bunch of stuff but wouldn't have caught this issue (unless my settings are just tuned lame) but its easy for claude to address lint issues so if you must use cloudformation add linting up in there
👍 1
m
updated the stack with this template but still Metaflow service error: Metadata request (/flows/DataPrepFlow) failed (code 403): {"message":"Missing Authentication Token"} (setting APIAuthorizationType to AWS_IAM, chatting with my AWS team, it's a requirement for the api gateway)
My AWS team is asking why we need API Gateway, they believe I could just connect to the Load Balancer the UI uses and go from there
Which is similar to what your lint tool says too
f
yeah for the metaflow client you can bypass apigw with the nlb with 
METAFLOW_SERVICE_URL
set
but the metaflow service as you've deployed it is api gateway + rds postgres - which is what manages the metadata for your runs, the metaflow client is you when you're getting past run info or any of those kinds of operations in python
m
ah so directly going through the NLB won't let me see past runs?
f
give me a min and I'll give you a better answer, imagine the service / client stuff isn't the most clear
🙏 1
sorry that was a bit over a min, had a surprise recruiter screen that wasn't on my calendar. Hopefully this clears things up, requests go something like this when metaflow is deployed on aws: client -> apigw -> NLB -> metadata service the api gateway is the public-facing external endpoint and is how you (the client) interact with metaflow, it forwards traffic to the private networked parts of the infrastructure, so can't work without it I didn't catch this yesterday with the robot answer to the cloudformation template update but the only way you can bypass things using the NLB by setting 
METAFLOW_SERVICE_URL
to its dns is if you ( the client) is if the request was within the network, so like from a sagemaker notebook. So its kind of a hack and doesn't work for your usecase unless you are using metaflow from within the vpc. The root of this problem is that IAM setting, its expecting your requests to be signed with aws sigv4, which I don't think the metaflow client is setup for, so I am pretty sure that this will not work period w that unless the client is updated (not 100% here, firing from the hip on this one). Is this apigw IAM auth thing an absolute requirement here? it can just forward traffic to the nlb, IAM over the basic auth doesn't really harden things and there isn't anything a malicious actor could do other than interact with the metadata api
m
unfortunately yes, apigw IAM auth is required from what the cloud people are telling me, They think I'm somehow directly connected to the VPC from their network model:
load balancer will on enterprise , it should be routable and have alias dns name to reach from our network and cross account VPC subnets routable
38 Views
Open in Slack
PreviousNext
Powered by