https://outerbounds.com/ logo
Join Slack
Powered by
[This is a different question than the one above] ...
# ask-metaflow
f
[This is a different question than the one above] I have installed 
argo-workflows
using the standard helm chart. I can create workflows (templates) and I can trigger the wf from the command line and from the 
argowf-ui
However, I am having some issues with getting the logs in the 
argowf-ui
I can get the logs with argo logs <wf-name>, but in the ui the logs tab is always empty (even if the pods are still alive). Is this something that you guys have observed before? if you have any pointer on what to look at I would really appreciate it. Thanks
✅ 1
l
Hi Alexander, could you help me with the following questions? 1. What namespace are your task pods running in? 2. What namespace is argo-workflows controller and server running in? Don’t have a concrete hypothesis yet, this is to confirm that the cluster permissions are setup properly
f
Hi, they are running in different namespaces, but I can create and trigger the flows. Should they be in the same namespace or that is just nicer?
l
Different namespaces is fine, we just need to make sure that the permissions are setup correctly. Do you have an 
argo-workflows
service account that argo-workflows is using?
f
yeah, the one for the server looks like this (created by helm)
and the controller
l
For the server - is it associated to a Role or a ClusterRole? Since your pods are executing in a different namespace, we need to make sure that its using a ClusterRole
f
It is just a role with a rolebinding in the argo namespace
l
Ah I see, then something like this should fix your problem - (make sure the namespace, service account names etc are correct for your setup)
Copy code
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: argo-ui-log-viewer
rules:
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: argo-ui-log-viewer
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: ClusterRole
  name: argo-ui-log-viewer
subjects:
- kind: ServiceAccount
  name: argo-workflows
  namespace: argo-workflows
(Since a 
Role
only allows all provided access in the same namespace)
f
I see, do I have to do something else with the controller svcaccnt? or just binding the server svcaccnt to that cluster role would do?
l
I think just binding it to the server svcaccnt should be sufficient. Curious to know what we discover here 🙂
f
actually, what would you recommend: 1) keep Argo and metaflow in different namespaces or 2) have Argo deployed in the same namespace than metaflow pods?
l
It could be a matter of preference but I prefer keeping it in different namespaces. Its a nice logical isolation for ephemeral metaflow task pods and other long running pods in the cluster. Also keeps the door open for some namespace level primitives like ResourceQuotas, NetworkPolicies, ClusterPolicies etc if you decide to go down these roads one day
👍 1
Did the logs thing work?
f
I am on it, had to log in again (Denmark time 😅)
😅 1
It did not work 🤔
I see this in k9s
Copy code
Setting up task environment.
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

albumentations 1.3.1 requires opencv-python-headless>=4.1.1, which is not installed.
lightning 2.0.8 requires fastapi<2.0,>=0.92.0, but you'll have fastapi 0.88.0 which is incompatible.
lightning 2.0.8 requires pydantic<2.2.0,>=1.7.4, but you'll have pydantic 2.3.0 which is incompatible.
Downloading code package...
Code package downloaded.
Task is starting.
This flow was deployed at 2023-09-11T17:10:12.816903 by Alexander
The alpha used in this run is 0.7
Unable to publish Argo Event (metaflow.DeploymentInfoFlow.start): unknown url type: 'None'
time="2023-09-11T18:37:50.088Z" level=info msg="sub-process exited" argo=true error="<nil>"
time="2023-09-11T18:37:50.089Z" level=info msg="no need to save parameter - on overlapping volume: /mnt/out/task_id" argo=true
Stream closed EOF for metaflow/deploymentinfoflow-c7spc-start-1690402011 (main)
but nothing in argo ui
l
Shot in the dark…can you also apply this to the controller svcaccnt?
f
yeah, 2 mins
I am triggering one of the old flows, should I create a new flow? could that be the reason?
l
If its a permissions issue then looking at an old vs new flow should have no impact
f
I applied it to the controller as well, but still not able to see anything in the ui
l
Is the behavior same for old flows vs new flows?
f
I only have one flow template at the moment, and I have triggered it 7 times, all of which showed the same behavior
Do you think we need the verb 
watch
in the clusterRole? or 
get
is enough?
l
get
should be enough thinking spin
f
would the 
svcaccnt
that I use to authenticate to the argowf-ui have to have access to the logs?
l
Are you using 
client/sso
auth mode?
f
- --auth-mode=client
l
Oh very interesting, yes I think the svcaccnt you’re using to authenticate needs to have the access to 
get
logs in that case
f
so, just adding a binding to the clusterRole that we applied before?
l
Yeah - I believe that should be sufficient
f
ok, let me give it a shot
Success!!! I guess I do not need the binding to the svcaccnts of the server and the controller, but just to the svcaccnt for auth
l
Awesome!!
I guess I do not need the binding to the svcaccnts of the server and the controller
Yes
f
Sweet, thanks a lot 🙏
👍 1
@thankful-father-61351 👀
3 Views
Open in Slack
PreviousNext
Powered by