https://outerbounds.com/ logo
Join Slack
Powered by
metaflow-dev up failed at Postgresql - Generating ...
# ask-metaflow
c
metaflow-dev up failed at Postgresql - Generating local authentication configuration Hi all. I ran 
metaflow-dev up
on a newly setup Ubuntu: 22.04.5 LTS machine, which then minikube starts a couple of services including postgresql. However, the postgresql is always restarting, resulting in failure of downstream services. Does anyone face this before?
Copy code
Successfully loaded Tiltfile (15.435355288s)
uncategorized │
uncategorized │ Initial Build
uncategorized │ STEP 1/1 — Deploying
uncategorized │      Applying YAML to cluster
uncategorized │      Updating "jobsets.jobset.x-k8s.io" failed: CustomResourceDefinition.apiextensions.k8s.io "jobsets.jobset.x-k8s.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
uncategorized │      Attempting to create or replace
uncategorized │      Updating "jobsets.jobset.x-k8s.io" succeeded!
uncategorized │      Objects applied to cluster:
uncategorized │        → jobset-system:namespace
uncategorized │        → jobsets.jobset.x-k8s.io:customresourcedefinition
uncategorized │        → jobset-mutating-webhook-configuration:mutatingwebhookconfiguration
uncategorized │        → minio-sa:serviceaccount
uncategorized │        → jobset-controller-manager:serviceaccount
uncategorized │        → metaflow-service:serviceaccount
uncategorized │        → metaflow-ui:serviceaccount
uncategorized │        → jobset-leader-election-role:role
uncategorized │        → jobset-manager-role:clusterrole
uncategorized │        → jobset-metrics-reader:clusterrole
uncategorized │        → jobset-proxy-role:clusterrole
uncategorized │        → jobset-full-access:clusterrole
uncategorized │        → jobset-leader-election-rolebinding:rolebinding
uncategorized │        → jobset-manager-rolebinding:clusterrolebinding
uncategorized │        → jobset-metrics-reader-rolebinding:clusterrolebinding
uncategorized │        → jobset-proxy-rolebinding:clusterrolebinding
uncategorized │        → default-jobset-binding:clusterrolebinding
uncategorized │        → minio:configmap
uncategorized │        → jobset-manager-config:configmap
uncategorized │        → minio:secret
uncategorized │        → minio-secret:secret
uncategorized │        → postgresql:secret
uncategorized │        → jobset-webhook-server-cert:secret
uncategorized │        → jobset-validating-webhook-configuration:validatingwebhookconfiguration
uncategorized │
uncategorized │      Step 1 - 1.65s (Deploying)
uncategorized │      DONE IN: 1.65s
uncategorized │
        minio │
        minio │ Initial Build
        minio │ STEP 1/1 — Deploying
minio-post-j… │
minio-post-j… │ Initial Build
minio-post-j… │ STEP 1/1 — Deploying
   postgresql │
   postgresql │ Initial Build
   postgresql │ STEP 1/1 — Deploying
minio-post-j… │      Applying YAML to cluster
        minio │      Applying YAML to cluster
   postgresql │      Applying YAML to cluster
minio-post-j… │      Objects applied to cluster:
minio-post-j… │        → minio-post-job:job
minio-post-j… │
minio-post-j… │      Step 1 - 0.04s (Deploying)
minio-post-j… │      DONE IN: 0.04s
minio-post-j… │
jobset-contr… │
jobset-contr… │ Initial Build
jobset-contr… │ STEP 1/1 — Deploying
jobset-contr… │      Applying YAML to cluster
        minio │      Objects applied to cluster:
        minio │        → minio-console:service
        minio │        → minio:service
        minio │        → minio:deployment
        minio │
        minio │      Step 1 - 0.06s (Deploying)
        minio │      DONE IN: 0.06s
        minio │
   postgresql │      Objects applied to cluster:
   postgresql │        → postgresql-hl:service
   postgresql │        → postgresql:service
   postgresql │        → postgresql:statefulset
   postgresql │
   postgresql │      Step 1 - 0.07s (Deploying)
   postgresql │      DONE IN: 0.07s
   postgresql │
jobset-contr… │      Objects applied to cluster:
jobset-contr… │        → jobset-controller-manager-metrics-service:service
jobset-contr… │        → jobset-webhook-service:service
jobset-contr… │        → jobset-controller-manager:deployment
jobset-contr… │
jobset-contr… │      Step 1 - 0.04s (Deploying)
jobset-contr… │      DONE IN: 0.04s
jobset-contr… │
minio-post-j… │ [minio-make-user]
minio-post-j… │ [minio-make-bucket]
minio-post-j… │
minio-post-j… │ Attaching to existing pod (minio-post-job-2kp29). Only new logs will be streamed.
   postgresql │
   postgresql │ Attaching to existing pod (postgresql-0). Only new logs will be streamed.
jobset-contr… │
jobset-contr… │ Attaching to existing pod (jobset-controller-manager-8dd446b54-7lxtf). Only new logs will be streamed.
        minio │
        minio │ Attaching to existing pod (minio-5c6cbc86c9-db9tm). Only new logs will be streamed.
   postgresql │ WARNING: Detected container restart. Pod: postgresql-0. Container: postgresql.
   postgresql │ postgresql 07:02:55.09
   postgresql │ postgresql 07:02:55.09 Welcome to the Bitnami postgresql container
   postgresql │ postgresql 07:02:55.09 Subscribe to project updates by watching <https://github.com/bitnami/containers>
   postgresql │ postgresql 07:02:55.19 Submit issues and feature requests at <https://github.com/bitnami/containers/issues>
   postgresql │ postgresql 07:02:55.19
   postgresql │ postgresql 07:02:55.39 INFO  ==> ** Starting PostgreSQL setup **
   postgresql │ postgresql 07:02:55.79 INFO  ==> Validating settings in POSTGRESQL_* env vars..
   postgresql │ postgresql 07:02:55.89 INFO  ==> Loading custom pre-init scripts...
   postgresql │ postgresql 07:02:55.89 INFO  ==> Initializing PostgreSQL database...
   postgresql │ postgresql 07:02:56.19 INFO  ==> pg_hba.conf file not detected. Generating it...
   postgresql │ postgresql 07:02:56.19 INFO  ==> Generating local authentication configuration
   postgresql │ WARNING: Detected container restart. Pod: postgresql-0. Container: postgresql.
   postgresql │ postgresql 07:08:07.97
   postgresql │ postgresql 07:08:07.97 Welcome to the Bitnami postgresql container
   postgresql │ postgresql 07:08:08.05 Subscribe to project updates by watching <https://github.com/bitnami/containers>
   postgresql │ postgresql 07:08:08.05 Submit issues and feature requests at <https://github.com/bitnami/containers/issues>
   postgresql │ postgresql 07:08:08.06
   postgresql │ postgresql 07:08:08.26 INFO  ==> ** Starting PostgreSQL setup **
   postgresql │ postgresql 07:08:08.57 INFO  ==> Validating settings in POSTGRESQL_* env vars..
   postgresql │ postgresql 07:08:08.67 INFO  ==> Loading custom pre-init scripts...
   postgresql │ postgresql 07:08:08.77 INFO  ==> Initializing PostgreSQL database...
   postgresql │ postgresql 07:08:09.07 INFO  ==> pg_hba.conf file not detected. Generating it...
   postgresql │ postgresql 07:08:09.16 INFO  ==> Generating local authentication configuration
   postgresql │ WARNING: Detected container restart. Pod: postgresql-0. Container: postgresql.
   postgresql │ postgresql 07:13:25.06
   postgresql │ postgresql 07:13:25.15 Welcome to the Bitnami postgresql container
   postgresql │ postgresql 07:13:25.16 Subscribe to project updates by watching <https://github.com/bitnami/containers>
   postgresql │ postgresql 07:13:25.16 Submit issues and feature requests at <https://github.com/bitnami/containers/issues>
   postgresql │ postgresql 07:13:25.16
   postgresql │ postgresql 07:13:25.46 INFO  ==> ** Starting PostgreSQL setup **
   postgresql │ postgresql 07:13:25.85 INFO  ==> Validating settings in POSTGRESQL_* env vars..
   postgresql │ postgresql 07:13:25.96 INFO  ==> Loading custom pre-init scripts...
   postgresql │ postgresql 07:13:26.05 INFO  ==> Initializing PostgreSQL database...
   postgresql │ postgresql 07:13:26.35 INFO  ==> pg_hba.conf file not detected. Generating it...
   postgresql │ postgresql 07:13:26.36 INFO  ==> Generating local authentication configuration
Further information from 
kubectl
indicates that the container is failing
Copy code
$ kubectl describe pod/postgresql-0
Name:             postgresql-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             minikube/192.168.49.2
Start Time:       Fri, 26 Dec 2025 14:41:04 +0800
Labels:           <http://app.kubernetes.io/component=primary|app.kubernetes.io/component=primary>
                  <http://app.kubernetes.io/instance=postgresql|app.kubernetes.io/instance=postgresql>
                  <http://app.kubernetes.io/managed-by=tilt|app.kubernetes.io/managed-by=tilt>
                  <http://app.kubernetes.io/name=postgresql|app.kubernetes.io/name=postgresql>
                  <http://apps.kubernetes.io/pod-index=0|apps.kubernetes.io/pod-index=0>
                  controller-revision-hash=postgresql-799b6c58c5
                  <http://helm.sh/chart=postgresql-12.5.6|helm.sh/chart=postgresql-12.5.6>
                  <http://statefulset.kubernetes.io/pod-name=postgresql-0|statefulset.kubernetes.io/pod-name=postgresql-0>
                  tilt.dev/pod-template-hash=70df629db4ea6c3703fd
Annotations:      <none>
Status:           Running
IP:               10.244.0.17
IPs:
  IP:           10.244.0.17
Controlled By:  StatefulSet/postgresql
Containers:
  postgresql:
    Container ID:   <docker://6466c279e8c73a6feb6df61fcaa0f75e0f272c60bc331762a7074afe0ac7d25>1
    Image:          <http://docker.io/bitnamilegacy/postgresql:15.3.0-debian-11-r7|docker.io/bitnamilegacy/postgresql:15.3.0-debian-11-r7>
    Image ID:       <docker-pullable://bitnamilegacy/postgresql@sha256:cc301eef743685f4f69d1d719853988e8a9650c90fd9521f4742ce400b3fdf6a>
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 26 Dec 2025 15:49:53 +0800
      Finished:     Fri, 26 Dec 2025 15:49:59 +0800
    Ready:          False
    Restart Count:  18
    Limits:
      cpu:     100m
      memory:  256Mi
    Requests:
      cpu:      50m
      memory:   128Mi
    Liveness:   exec [/bin/sh -c exec pg_isready -U "metaflow" -d "dbname=metaflow" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/sh -c -e exec pg_isready -U "metaflow" -d "dbname=metaflow" -h 127.0.0.1 -p 5432
] delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                        false
      POSTGRESQL_PORT_NUMBER:               5432
      POSTGRESQL_VOLUME_DIR:                /bitnami/postgresql
      PGDATA:                               /bitnami/postgresql/data
      POSTGRES_USER:                        metaflow
      POSTGRES_POSTGRES_PASSWORD:           <set to the key 'postgres-password' in secret 'postgresql'>  Optional: false
      POSTGRES_PASSWORD:                    <set to the key 'password' in secret 'postgresql'>           Optional: false
      POSTGRES_DB:                          metaflow
      POSTGRESQL_ENABLE_LDAP:               no
      POSTGRESQL_ENABLE_TLS:                no
      POSTGRESQL_LOG_HOSTNAME:              false
      POSTGRESQL_LOG_CONNECTIONS:           false
      POSTGRESQL_LOG_DISCONNECTIONS:        false
      POSTGRESQL_PGAUDIT_LOG_CATALOG:       off
      POSTGRESQL_CLIENT_MIN_MESSAGES:       error
      POSTGRESQL_SHARED_PRELOAD_LIBRARIES:  pgaudit
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9f58v (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-9f58v:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason   Age                 From     Message
  ----     ------   ----                ----     -------
  Warning  BackOff  6s (x327 over 69m)  kubelet  Back-off restarting failed container postgresql in pod postgresql-0_default(889c3a70-1208-4bf7-b9a7-79f87fb815ae)
7 Views
Open in Slack
PreviousNext
Powered by