https://outerbounds.com/ logo
Join Slack
Powered by
Hi folks, I'm getting the `Service token file does...
# ask-metaflow
b
Hi folks, I'm getting the 
Service token file does not exist
error below when trying to deploy a flow to Argo (running on an AWS EKS cluster) from a Bitbucket Cloud pipeline. I don't get errors when running the same command locally, the Bitbucket pipeline has the same 
~.metaflowconfig/config.json
file as I have locally, and also it's able to run 
kubectl
commands connecting to the cluster OK.... Any ideas what's going on here?
Copy code
$ python main.py --production argo-workflows create --only-json
Metaflow 2.7.19 executing DummyFlow for user:bitbucket
Project: myproj, Branch: prod
Validating your flow...
    The graph looks good!
Deploying myproj.prod.dummyflow to Argo Workflows...
    Internal error
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/metaflow/cli.py", line 1172, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/usr/local/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows_cli.py", line 156, in create
    token = resolve_token(
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows_cli.py", line 382, in resolve_token
    workflow = ArgoWorkflows.get_existing_deployment(name)
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/argo/argo_workflows.py", line 201, in get_existing_deployment
    workflow_template = ArgoClient(
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/argo/argo_client.py", line 16, in __init__
    self._kubernetes_client = KubernetesClient()
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/kubernetes/kubernetes_client.py", line 30, in __init__
    self._refresh_client()
  File "/usr/local/lib/python3.10/site-packages/metaflow/plugins/kubernetes/kubernetes_client.py", line 37, in _refresh_client
    config.load_incluster_config()
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 121, in load_incluster_config
    try_refresh_token=try_refresh_token).load_and_set(client_configuration)
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 54, in load_and_set
    self._load_config()
  File "/usr/local/lib/python3.10/site-packages/kubernetes/config/incluster_config.py", line 73, in _load_config
    raise ConfigException("Service token file does not exist.")
kubernetes.config.config_exception.ConfigException: Service token file does not exist.
Digging a bit in the code I think the reason this is failing is that the Bitbucket pipeline must be running in a Kubernetes cluster (because the 
KUBERNETES_SERVICE_HOST
environment variable is set pika detective), and metaflow assumes that if we're in a pod, we're running in the same cluster we want to deploy the flow to, because:
Copy code
if os.getenv("KUBERNETES_SERVICE_HOST"):
    # We are inside a pod, authenticate via ServiceAccount assigned to us
    config.load_incluster_config()
else:
    # Use kubeconfig, likely $HOME/.kube/config
    # TODO (savin):
    #  1. Support generating kubeconfig on the fly using boto3
    #  2. Support auth via OIDC - <https://docs.aws.amazon.com/eks/latest/userguide/authenticate-oidc-identity-provider.html>
    config.load_kube_config()
That code may need some rethinking, but in the meantime perhaps an easy/backwards-compatible improvement (not really a fix, because this situation isn't limited to running things in a CI) would be:
Copy code
if os.getenv("KUBERNETES_SERVICE_HOST") and not os.getenv("CI"):
    config.load_incluster_config()
else:
    config.load_kube_config()
Any thoughts? I'd be happy to open a PR if useful 🙂
a
yep! we would love to get a PR!
🚀 1
u
@bulky-fireman-68939 another option would be to check if a 
KUBECONFIG
env var is set. That should take precedence over 
KUBERNETES_SERVICE_HOST
. In your CI pipeline, you will have to setup the 
KUBECONFIG
of the target cluster anyway. AFAIK, most K8s tools support 
KUBECONFIG
. So even in Metaflow we could:
Copy code
if os.getenv("KUBECONFIG"):
   config.load_kube_config()
elif os.getenv("KUBERNETES_SERVICE_HOST"):
    # We are inside a pod, authenticate via ServiceAccount assigned to us
    config.load_incluster_config()
else:
     # raise an exception since neither KUBECONFIG exists nor are we running inside a cluster.
     raise
WDYT?
b
@User That looks like a more general approach - Thanks! I'll create a PR with that change soon 🙂
🚀 https://github.com/Netflix/metaflow/pull/1716 🚀 @User @square-wire-39606
153 Views
Open in Slack
PreviousNext
Powered by