Beginner question about Argo! Having some trouble with Argo on Azure Kubernetes Service. I deployed K8s with the Metaflow template. When I submit flow templates to argo with

argo-workflows create

Metaflow says everything is fine. Problem is that the templates don't appear on the Argo UI which I forwarded with the usual port forward script.

argo template list

does show these flows, in the

default

namespace (is this the same namespace Metaflow refers to when creating Argo flows, because there I see

production:myflow-0-tvwa

?). Any pointers? I also tried to trigger the flows with

argo-workflows trigger

. Metaflow happily submits this. Nothing on the Argo or Metaflow UI comes up.

argo list

does show the flows, in pending state. I'm thinkin either my pods don't have enough resources (8gb memory nodes) or metaflow and my argo cli are not talking to the same server as the UI.. but I shouldn't have any local instance that would mess things up.

You can always check what happens in k8s cluster directly with •

kubectl get wftmpl

to check workflow templates (flows) •

kubectl get wf

to check workflows (runs)

Problem is that the templates don't appear on the Argo UI

It happens to me often and usually it's because I didn't select a proper namespace

Thanks! I'm not able to change the namespace. I do see my flow template with

kubectl get wftmpl

though.

this is strange ... But at least it explains why you don't see templates since UI always points to the

argo

namespace

I'm probably missing something basic in my setup

Argo UI which I forwarded with the usual port forward script

I usually don't use port-forwarding but a feature of the

argo

cmd tool to "expose" UI:

argo server --auth-mode=server --secure=false

That works, now I can find the flows 🤯

probably there is some security config on argo side.

argo

command actually starts a server on a local computer as far as I understand instead of using a server from a cluster

Yeah! Now I just need to figure out why these flows stay stuck in pending state.

Flows can be stuck because you requested resources that may not be available. You mentioned about 8GB memory on the nodes. Not all of it is available for the pods. Some of it is used by K8s for running daemonsets, etc. Similar is the case with CPU. Can you check with

kubectl get pods -n default -o wide

and then if say, pod

pod-1

is

Pending

, you can do a

kubectl describe pod pod-1 -n default

? The

describe

commands output could show you the reasons for

Pending

Thanks for the tips @User! The argo workflows don't appear in my kubectl pod list :S so

kubectl get pods

doesn't show anything out of the ordinary. I see the pending state with

argo list

Can you run

argo get <wf-name> -n default

? This should also provide additional insights.

This looks weird though (kubectl describe nodes)

Namespace                   Name                                           CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                           ------------  ----------  ---------------  -------------  ---
  argo                        argo-server-78f47df69f-wjg9d                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  argo                        httpbin-69794bbd74-bk5j7                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  argo                        minio-647b67bc44-649kc                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  argo                        postgres-869f7fbd7f-2c4zj                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  argo                        workflow-controller-b99cbc8bf-txn2j            0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m

kubectl describe nodes

provides info about the node (azure instance) . The list above is the list of pods running on the node you described

Here's one output from

argo get ...

Name:                etlflow-lkm72
Namespace:           default
ServiceAccount:      unset
Status:              Pending
Created:             Fri Dec 02 11:11:00 +0200 (8 hours ago)
Progress:

This workflow does not have security context set. You can run your workflow pods more securely by setting it.
Learn more at <https://argoproj.github.io/argo-workflows/workflow-pod-security-context/>

Good.. this doesn't seem like a resources problem. There might be a missing field in the global defaults. Let me check something and get back to you

Do you know what version of Argo was installed on this K8s cluster? From the image you posted above, it looks like the version is

untagged

. There was a bug about the security context that was fixed in argo in v3.3.0.

Version is specified in the top left corner under the Argo logo.

argo version

shows 3.3.0, and that is what I see if I use the UI via

argo server --auth-mode=server --secure=false

If I use metaflow's port forward script with

--include-argo

then it shows untagged in the UI

We had an offline discussion about this. However, argo-workflows didn't run after some of the fixes. • Confirmed that the Argo version was 3.3.0. • Confirmed that the fix for the problem above indeed went into v3.3.0 • Added default securityContext in the argo workflow defaults. Restarted the workflow controller and re-submitted the workflow template. But the workflow was still pending..

Joining the convo late... The azure TF templates currently use this command to install Argo Workflows (official quick start):

kubectl apply -n argo -f <https://raw.githubusercontent.com/argoproj/argo-workflows/master/manifests/quick-start-postgres.yaml>

Unfortunately, we are pulling the latest master version of this file, PLUS the content of the YAML file also refers to all kinds of

:latest

images which also float forwards all the time. Argo Workflows appears to have updated their quickstart instructions recently - it's a different set of steps now. https://argoproj.github.io/argo-workflows/quick-start/#install-argo-workflows In particular, it is now always pre-pinned. I think we need to migrate our Azure (and GCP) TF templates to invoke THAT instead. Line here needs changing. I will post an GH issue for this for now, it will take some testing to validate it all works fine.

FYI: https://github.com/outerbounds/metaflow-tools/issues/21

Awesome, thanks! I'll try out the new template when it's ready.

We did some additional debugging today and were eventually able to run flows using Argo on the Kubernetes cluster in Azure. Here's a summary of what we did. The main problem was that the Metaflow config had a default Kubernetes service account and namespace (both set to

default

). As a result the Argo workflows were being created incorrectly and therefore were not running. This was changed to the correct service account and namespace (both set to

argo

). A good change (but maybe not needed for the problem above) was to pin the version of the argo workflows controller to v3.4.1. It was earlier set to

latest

making it variable as newer versions of Argo would be released. A similar change to the

argo-server

COULD be done. But wasn't in the session today. •

kubectl edit deployment argo-server -n argo

• Change the

image

tag such that it is set to

v3.4.1

• Save + quit In a previous debugging session, the workflows configMap was edited to set the default securityContext (as per this). However, this was not needed. In fact, it was causing failures because the METAFLOW_INIT_SCRIPT can try to create dirs in

/

of the container requiring

root

privileges. This change was reverted in the configMap. At the end of this, argo workflows created using metaflow were able to run successfully. Let me know if anything was missed... /cc @alert-florist-32940 @User

I pinned the argo server version as well, everything seems to work 😅 Thanks for all the helps!

wuhoo!!! 🚀