Hi, we are getting this error in a new metaflow deployment on EKS

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<http://dasdasd.execute-api.AWS_REGION.amazonaws.com|dasdasd.execute-api.AWS_REGION.amazonaws.com>', port=443): Max retries exceeded with url: /api/ping (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x109cee970>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

It's bit weird that another older metaflow deployment works.

^^your AWS_REGION thingy must be a valid region isnt it? you just need to fill in the placeholder in your modularized template/.tf ? it seems not getting substituted well .. isnt it @fast-vr-44972?

That was my attempt to hide the region. It is set to a valid value

put a sleep before a line in either of your tasks/steps that try reaching out to the HOST above- • if it is through a pod, "exec" (i.e get inside it), realise if the HOST is ping-able, i.e nc -v host port works, check the /etc/hosts entry, try resolving it by comparing it to the "other older metaflow deployment that works" • if it is outside K8s cluster, well try the same activity, check if /etc/hosts entries are similar (from the lowest common ancestor node/container in the entire CI/CD pipeline before the error!) ^^ I mean above is just a general attempt at troubleshooting, because otherwise, one'd need a lot more to know how to troubleshoot/help, vis-a-vis WHAT FLOW/DEPLOYMENT ARE YOU RUNNING AND HOW?

This error is thrown when we even try to create a flow using Argo workflow

How about debugging it, say in Pycharm IDE? set breakpoints to where any exceptions may be thrown- it will take you right at the line/call wherein such request is being made to Argo. Compare how things change when you hit the same breakpoint while running a-flow-that-just-works!

Can you check the metadata service? Seems that when trying to check its health the service doesn’t respond back even after multiple retries.

Yeah but I am not sure why it's not responding. This is a new

dev

env after modularizing our metaflow terraform. We have another

prd

env which uses the same module and it's working. They even share the same

vpc

. The only difference is

prd

was created a while ago and

dev

is new. While comparing the metadata ECS service between the two, I also don't see any issue. The underlying task status is RUNNING and the load balancers are HEALTHY

Could you check the logs of the metadata service through your monitoring tool?

So I see the last log from 20th Sept i.e. 10 days ago for the

dev

metadata and then it stopped. Although no idea why. On the other hand

prd

metadata service has more recent logs. The one change we added on 20th Sept was putting out metaflow UI behind our VPN. Although both

prd

and

dev

are behind VPN. So not sure why only

dev

should be affected by it and not

prd