Hi. I'm testing a workflow against a kubernetes cluster, where I leveraged the

outerbounds/metaflow-tools/k8s/helm/metaflow

helm chart. I have a minio running in the same cluster and modified the

forward_metaflow_ports.py

script to forward that minio-api to

localhost:9000

... so I pass

METAFLOW_S3_ENDPOINT_URL:<http://localhost:9000>

when running a flow. However, that also gets passed through to the kubernetes pod ... and so the job errors out with

fatal error: Could not connect to the endpoint URL: "<http://localhost:9000/minio-metaflow-bucket/metaflow/>...

in the pod logs ... since the pod knows nothing about

localhost:9000

. Instead of

localhost:9000

, I'm going to need

minio.default.svc.cluster.local:9000

for the pod to communicate back to my minio in the cluster ... I tried adding

METAFLOW_S3_ENDPOINT_URL=minio.default.svc.cluster.local:9000

as part of secret (including access key and secret for minio) on k8s, leveraging the

METAFLOW_KUBERNETES_SECRETS

variable , but that doesn't seem to get picked up in the pod. Could someone suggest how I might resolve this? Thanks alot.

The

S3_ENDPOINT_URL

would be needed to bootstrap the kubernetes pod container. You can set

METAFLOW_S3_ENDPOINT_URL

in your metaflow config and it will be picked up appropriately.

Thanks @square-wire-39606... still struggling a bit with this to get this to work. For whatever reason, the cluster internal address for minio is not getting picked-up ... with this

localhost:9000

always prevailing. This is the error from the pod/container each time:

fatal error: Could not connect to the endpoint URL: "<http://localhost:9000/minio-metaflow-bucket/metaflow/TestFlow/data/ac/acd20682dbc23e3aaf7cc1397afe85aae3ae460a>

My metaflow config file ~/.metaflowconfig/config_k8s-helm-civo looks like this:

{
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_KUBERNETES_NAMESPACE": "default",
  "METAFLOW_KUBERNETES_SERVICE_ACCOUNT": "default",
  "METAFLOW_SERVICE_INTERNAL_URL": "<http://localhost:8083/api>",
  "METAFLOW_SERVICE_URL": "<http://localhost:8080>",
  "METAFLOW_KUBERNETES_SECRETS": "s3-metaflow-secret",
  "METAFLOW_DEFAULT_DATASTORE": "s3",
  "AWS_ACCESS_KEY": "metaflow",
  "AWS_SECRET_ACCESS_KEY": "metaflow",
  "METAFLOW_S3_ENDPOINT_URL": "<http://minio.default.svc.cluster.local:9000>",
  "METAFLOW_DATASTORE_SYSROOT_S3": "<s3://minio-metaflow-bucket/metaflow>",
  "METAFLOW_DATATOOLS_SYSROOT_S3": "<s3://minio-metaflow-bucket/metaflow/data>",
  "METAFLOW_CONDA_DEPENDENCY_RESOLVER": "mamba"
}

I'm then calling metaflow with:

METAFLOW_S3_ENDPOINT_URL=<http://localhost:9000> METAFLOW_PROFILE=k8s-helm-civo AWS_PROFILE=minio-metaflow METAFLOW_KUBERNETES_SECRETS=s3-metaflow-secret CONDA_CHANNELS=anaconda,conda-forge python test-flow.py --environment=conda run --with kubernetes

For containers to spawn, having

METAFLOW_S3_ENDPOINT_URL=<http://localhost:9000>

in the command above seems to be a necessity (recall, I'm port-forwarding this port 9000 from the cluster ... along with 8080, 8083 and 3000) otherwise the flow doesn't run ... but I realise there's an inconsistency with what I set for this variable in the config file ... but it's still not clear to me how I get

<http://minio.default.svc.cluster.local:9000>

to be used in the container. I have also tried adding this as a field in the

s3-metaflow-secret

on the cluster, but that doesn't appear to help.

@wooden-state-97148 gotcha - can you access

minio

locally using http://minio.default.svc.cluster.local:9000?

@square-wire-39606... not locally .. but from within a pod/container on the cluster.