https://outerbounds.com/ logo
Join Slack
Powered by
Hi team, Sorry - one more question. We have Metaf...
# ask-metaflow
a
Hi team, Sorry - one more question. We have Metaflow on Kubernetes w/ Argo Workflows set up now, and everything seems to be working fine now except for the flows disappearing from the Argo UI after completion or error. Is this expected behaviour? Could it be because of archiving?
✅ 1
l
Hey Riley, do you have any type of 
ttl
configured on your argo workflows? Or any custom cleanup in place? These could contribute to this behavior. One thing to check would be if you have the 
Workflow
objects around in your k8s cluster when things disappear from the UI.
👀 1
a
Thanks for the response @limited-tomato-18674! Let me double check if the 
Workflow
objects are still present
Yup they seem to be present:
Copy code
(miniforge3-23.11.0-0) hunr@TORC02F70GAMD6T amps % kubectl get workflows -n cagepart
NAME                    STATUS      AGE    MESSAGE
demoforeachflow-2qzkp   Succeeded   58m    
demoforeachflow-jx54p   Succeeded   21h    
demoforeachflow-lljfb   Succeeded   19h    
demoforeachflow-sx4vp   Succeeded   21h    
parameterflow-6zmsk     Succeeded   63m    
parameterflow-c6xj2     Error       22h    
parameterflow-f58zg     Error       22h    
parameterflow-fthqb     Error       24h    
parameterflow-fzgqb     Error       24h    
parameterflow-hl5pr     Error       22h    
parameterflow-htfj5     Succeeded   19h    
parameterflow-jlxnc     Succeeded   21h    
parameterflow-kkbth     Succeeded   21h    
parameterflow-kn698     Error       3d1h   
parameterflow-mjk6z     Succeeded   19h    
parameterflow-nggnx     Error       27h    
parameterflow-pqgrc     Error       22h    
parameterflow-qj7cs     Error       24h    
parameterflow-s4fwf     Succeeded   22h    
parameterflow-tn8lj     Succeeded   59m    
parameterflow-vtmrf     Succeeded   19h
l
Hmm can you check in the devtools if the request to fetch workflows is succeeding? You should see a request like:
Copy code
https://<your-host>/api/v1/workflows/<your-namespace>?listOptions.limit=50&nameFilter=Contains&fields=metadata,items.metadata.uid,items.metadata.name,items.metadata.namespace,items.metadata.creationTimestamp,items.metadata.labels,items.metadata.annotations,items.status.phase,items.status.message,items.status.finishedAt,items.status.startedAt,items.status.estimatedDuration,items.status.progress,items.spec.suspend
m
Do you have an archive database set up for your argo workflow instance? If not, then you'll only see the workflows for as long at they are persisted on the cluster
this 1
a
Yup we have an archive database set up.
m
what version of Argo are you on?
a
We are on version 
0.45.4
m
of argo workflows?
a
Yup
m
That is super old, no?
we are on 3.5.11
a
Oh sorry 
0.45.4
is the helm chart target revision
m
Prior to version 3 Argo Workflows use to show the archived workflows in the archive page. Since version 3 it was merged into the main page, which use to be just the workflows on the cluster
right, we don't use helm with our install
You can see the version of Argo on the UI
a
Oh ok that does make sense because I was looking for the archive page but couldn't find it
We are on 
3.6.2
m
um.......and you are sure you are hooked up to the archive ok?
a
Well I checked the controller logs and seems to have connected to RDS
m
What does your workflow controller configmap look like? Tangentially, this looks like an argo problem that is not specific to metaflow.
a
Copy code
Namespace:    argo-workflows
Labels:       app=workflow-controller
              <http://app.kubernetes.io/component=workflow-controller|app.kubernetes.io/component=workflow-controller>
              <http://app.kubernetes.io/instance=argo-workflows-controller|app.kubernetes.io/instance=argo-workflows-controller>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=argo-workflows-cm|app.kubernetes.io/name=argo-workflows-cm>
              <http://app.kubernetes.io/part-of=argo-workflows|app.kubernetes.io/part-of=argo-workflows>
              <http://argocd.argoproj.io/instance=amps-c-uw2-argo-workflows-controller|argocd.argoproj.io/instance=amps-c-uw2-argo-workflows-controller>
              <http://helm.sh/chart=argo-workflows-0.45.4|helm.sh/chart=argo-workflows-0.45.4>
Annotations:  <http://argocd.argoproj.io/tracking-id|argocd.argoproj.io/tracking-id>:
                amps-c-uw2-argo-workflows-controller:/ConfigMap:argo-workflows/argo-workflows-controller-workflow-controller-configmap

Data
====
config:
----
artifactRepository:
  archiveLogs: true
  s3:
    bucket: argo-artifacts
    endpoint: <http://s3.amazonaws.com|s3.amazonaws.com>
    insecure: false
    keyFormat: "argo-artifacts/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{pod.name}}"
    region: us-west-2
    useSDKCreds: true
persistence:
  archive: true
  archiveTTL: 30d
  postgresql:
    database: argowfdb
    host: <HOST_NAME>
    passwordSecret:
      key: POSTGRESQL_PASSWORD
      name: argo-postgres-config
    port: 5432
    tableName: argo_workflows
    userNameSecret:
      key: POSTGRESQL_USERNAME
      name: argo-postgres-config
nodeEvents:
  enabled: true
workflowEvents:
  enabled: true


BinaryData
====
m
It looks correct to me. I think your best bet it to raise the issue in the CNCF slack channel for Argo Workflows. Maybe also check the github issues in Argo Workflows. Maybe it is a regression issue. I think 3.6.2 is pretty new. Maybe it has some bugs.
You could also try a 3.5.x version.
a
Thanks - appreciate it. Will raise this w/ them
m
cool
a
Thanks for your help @mammoth-rainbow-82717! I raised a Github Issue, and in the meantime, downgraded to 3.5.11 and now I can see my workflows in the UI again:
m
Awesome!
3 Views
Open in Slack
PreviousNext
Powered by