https://outerbounds.com/ logo
Join Slack
Powered by
Hey Metaflowers! I have deployed a metaflow kuber...
# ask-metaflow
i
Hey Metaflowers! I have deployed a metaflow kubernetes cluster along with the UI using Terraform. I can submit kubernetes jobs and they complete successfully, however, the UI does not show any jobs at all
āœ… 1
One thing that sticks out is that the backendui service has its METAFLOW_SERVICE_URL set to localhost.... should that actually match the Metaflow Metadata Service URL for the deployment?
i
Did you edited the .metaflowconfig/config.json?
i
I'm talking about the task definition of the uibackend service container
i
Yes, you need to set the path to the UI in the config,json
Copy code
{
  "METAFLOW_SERVICE_SECRET_KEY": "XXXXX",
  "METAFLOW_DEFAULT_METADATA": "service",
  "METAFLOW_SERVICE_URL": "<http://XX.XX.XX.XX:8080/>",
  "METAFLOW_DEFAULT_DATASTORE": "gs",
  "METAFLOW_DATASTORE_SYSROOT_GS": "bucket path"
}
i
Is one of those the UI?
i
Btw you need to connect to backend on 8080, not to UI
i
Does the UI container provide both the metadata service and the ui service? I have them running separately currently:
i
I'm not sure about terraform script, I've deployed manually. In my setup 2 pods: 1) database, 2) backend on 8080 + ui on 8083
what services do you have on cluster 
kubectl get svc
?
i
Copy code
$ kubectl get svc
NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
argo-events-webhook-eventsource-svc   ClusterIP   10.100.213.44   <none>        12000/TCP                             22h
eventbus-default-js-svc               ClusterIP   None            <none>        4222/TCP,6222/TCP,7777/TCP,8222/TCP   22h
kubernetes                            ClusterIP   10.100.0.1      <none>        443/TCP                               24h
i
sorry, I have very different setup. I'm afraid cannot help (
i
We have deployed using the Terraform Argo example and then added the UI module to that... so the database is AWS RDS and Metadata/UI services are running in ECS........ Are you running those services from within kubernetes?
i
Yes, I use custom GKE setup without Argo
i
Okay, thanks for taking a look
i
Look for the backend with port 8080 by default
šŸ‘€ 1
i
Yeah, I See the metadata service serves both an 8080 and an 8082
Do you think it may be serving the UI there on port 8080?
b
some hopefully helpful answers on what goes on with the services: ā€¢ the metadata service container usually runs two things ā—¦ metadata service API on port 8080 ā—¦ migration service on port 8082 ā€¢ the UI service is a separate deployment, defaulting to port 8083 ā—¦ internally it also hosts a read-only metadata service, which is why you see an env var with localhost value. This is so that we can use a Metaflow client for some data fetching under the hood without reimplementing things all service code is bundled into the one published docker image. The deployment templates simply define different env vars and a command to run on the image to distinguish which service to launch.
šŸ‘€ 1
as to why your runs are not showing up in the UI if they are successfully executing on Kubernetes, can you verify that you have configured the metadata service in your metaflow profile correctly? If the service was unreachable then the execution should be erroring out, so I' m suspecting no data is ending up in the metadata service. The relevant values for Metaflow config should be
Copy code
{
    ...
    "METAFLOW_DEFAULT_DATASTORE": "s3",
    "METAFLOW_DEFAULT_METADATA": "service",
    "METAFLOW_SERVICE_URL": "url-to-your-metadata-service",
}
i
@brave-flag-76472 you are a lifesaver. All this time it was the 
METAFLOW_DEFAULT_METADATA=service
that I needed. I find it weird that kubernetes steps require the use of the metadata service for the code package, but that doesn't cause the job to show in the UI
Open in Slack
PreviousNext
Powered by