Hi all We are running metaflow on EKS and are able to access all the services successfully from inside the cluster. I am wanting to do some local development work (i.e. access metaflow from my mac). I set all the same environment variables as what is set on my EKS pods however I keep landing up with this error:

Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Could anyone point me to sample config/setup on how to do this correctly?

Hey Niklas.. when you say

access all the services successfully from inside the cluster

, do you mean accessing the Metaflow metadata service? It's possible that the metadata service was accessible only from within the cluster. Also, can you share what you set as

METAFLOW_SERVICE_URL

?

Hi @User, yes that is correct. I am accessing the metaflow meta data service from inside the cluster. I am setting the

METAFLOW_SERVICE_URL

to

"<http://prefix-metaflow-metadata-microservice-svc.namespace.svc.cluster.local>"

From within the EKS cluster I only set the following environment variables:

METAFLOW_DATASTORE_SYSROOT_S3="<s3://bucket/flows>"
METAFLOW_DEFAULT_METADATA="service"
METAFLOW_DATATOOLS_S3ROOT="<s3://bucket/data>"
METAFLOW_SERVICE_URL="<http://prefix-metaflow-metadata-microservice-svc.namespace.svc.cluster.local>"
METAFLOW_DEFAULT_DATASTORE="s3"

and I am able to access the meta data service/ When I set the exact same environment variables locally I land up with the connection error. We use cillium network policies and I have confirmed that no traffic is being blocked or dropped. I was hoping that maybe this is just a configuration error.

Thanks for sharing this! The endpoint

prefix-metaflow-metadata-microservice-svc.namespace.svc.cluster.local

is only accessible from within the EKS cluster. There are two options to make this service accessible from your local laptop: 1. You can port-forward the service locally. See this for instructions. You can then set the

METAFLOW_SERVICE_URL

to

<http://127.0.0.1>:<port-number>

and run metaflow. 2. Create a loadbalancer for the service. See these instructions. This will give you a loadbalancer endpoint that you can set as the

METAFLOW_SERVICE_URL

and not have to worry about port-forwarding everytime. However, this option also has some security and cost implications. It will be a bit more involved to get this production ready. I'd suggest starting with the port-forwarding option since it's safe and you can confirm that the system works end-to-end as expected.

Thank you so much @User! The port forwarding idea is genius. I’m going to test it now

👍 ... Let me know how it goes or if there are more questions

Success!