https://outerbounds.com/ logo
Join Slack
Powered by
Hi again all, Unfortunately, our organization has...
# ask-metaflow
a
Hi again all, Unfortunately, our organization has disabled VPC Links across the entire org citing that misconfiguration of it has led to problems in the past and also huge spend. As a result, the tf deployment failed prompting that I don't have user permission to create a VPC Link resource in AWS. Any suggestions or guidance on how one can modify the API Gateway code such that VPC Links aren't necessary?
✅ 1
v
interesting. @average-beach-28850 any ideas?
a
not sure if thats an option, but if you have some way to connect to resources inside your VPC (like VPN) you won't need API Gateway at all
a
That's a good point. I think we should be able to connect to the resources in the VPC using VPN. So the metadata service would just use the alb that's created within the VPC? Is that correct?
a
NLB, yes
check out this picture, one thing is missing in the picture is that there is an NLB between api gateway and ECS for metaflow service
👌 1
if you can connect to resources inside VPC, you don't need API GW
a
Why does one service use ALB and one use API Gateway?
a
Good question. I think it would be possible to use an ALB instead of NLB there as well. But generally humans interact with UI in the browser, and programmatic metaflow API goes to Metadata Service. API Gateway provides auth for those API calls, and IIRC an NLB is needed to have api gateway route requests inside the VPC
ALB for UI is nice because it understands HTTP so it can route to UI backend or UI frontend server based on HTTP path. It also integrates nicely with Cognito for browser auth
a
Oh ok thank you. That makes complete sense
For context, It's this resource that fails:
Copy code
resource "aws_api_gateway_vpc_link" "this" {
  name        = "${var.resource_prefix}vpclink${var.resource_suffix}"
  target_arns = [aws_lb.this.arn]

  tags = var.standard_tags
}
But you are suggesting that the entire api-gateway.tf can be removed and instead users just need to use this:
Copy code
output "METAFLOW_SERVICE_INTERNAL_URL" {
  value       = "http://${aws_lb.this.dns_name}/"
  description = "URL for Metadata Service (Accessible in VPC)"
}
This might work, but not using https is not ideal
a
its possible to modify it to listen on SSL, but you'd have to create a SSL cert and possibly have to create a DNS (sub) domain somehow. Its not too hard but its not part of the template because many organizations already have some infrastructure to manage certs and domain names in a certain way
thankyou 1
a
A couple other things I noticed as well - There are 2 
<http://ec2.tf|ec2.tf>
files in the 
metadata-service
, and 
ui
submodules. Would it make sense to rename these to 
<http://lb.tf|lb.tf>
? I don't think any `ec2`instances are created in this infrastructure stack. Also the 
metaflow_profile_json
configuration output is missing the 
UI_URL
a
good point re:naming. Metaflow config doesn't have UI URL because Metaflow CLI doesn't use it anywhere, though I think I had a PR somewhere to show it in the console when you do flow.py run
also I completely forgot 🤦‍♂️ that there is a ready-to-merge PR from @narrow-garden-54875 pending for quite a while to make api gateway optional
🎉 1
a
@average-beach-28850 - that is perfect!
2 Views
Open in Slack
PreviousNext
Powered by