Hey, <@U01N9QPNV0E> <@U01U2JMQW5A> :wave:
Question...
# ask-metaflowb
bitter-orange-8782
08/29/2023, 11:18 AMHey, @User @square-wire-39606 👋
Question regarding the workflow-controller.
We are seeing a pod that accumulate memory much more than the other pods and i'm trying to understand that behaviour.
Could it be that when we are running a flow (lets say we have 3 steps: start, something, end), the workflow controller orchestrate it on a single pod or should it scale?
✅ 1
d
dazzling-van-84908
08/29/2023, 11:23 AMIn general, can a flow be scaled on more than one pod per service? Or is it 1 pod per 1 flow policy?
a
ancient-application-36103
08/29/2023, 2:15 PM@bitter-orange-8782 @dazzling-van-84908 every task is a different pod. Every step in metaflow can result in one or more than one task (in the case of foreach scope). Each pod is independently scheduled and managed
d
dazzling-van-84908
08/29/2023, 2:18 PMThanks @ancient-application-36103. I just want to emphasize that we're talking about the pods of the Metaflow service. Not the ones that actually execute the tasks on K8s.
a
ancient-application-36103
08/29/2023, 2:20 PMIt’s totally up to you how many pods for the metaflow service you would want to run. The only complication is when you have to orchestrate database upgrades - in that scenario you have to ensure that all service pods also update in a lock step sequence together
s
square-wire-39606
08/29/2023, 2:48 PMalso want to point out that metaflow service is not orchestrating the workflow execution
b
bitter-orange-8782
08/29/2023, 3:00 PMReg. Each pod is independently scheduled and managed.
The one that will orchestrate that when running with argo-workflows is the workflow-controller that we have deployed on our end right?
(depending if the above answer is yes)
what i'm trying to ask is if i have 2 pods of workflow-controller, will they share the responsibility of scheduling and managing the pods of the same run (my flow)?
s
square-wire-39606
08/29/2023, 3:49 PMThe workflow controllers for argo are different than the metaflow service. Regarding running 2 pods of workflow-controller and have them share responsibilities, that functionality is not available in argo-workflows out-of-the-box (you can only run a hot stand-by). Outerbounds has a different mechanism for distributing workloads on argo workflows through this workflow controller, so we are able to work around bottlenecks presented by the open source set up of workflow controller.
b
bitter-orange-8782
08/29/2023, 4:52 PMOk, gotcha, thank you! 🙏
bitter-orange-8782
08/30/2023, 3:17 PMHi @square-wire-39606
I have some followup question regarding the workflow-controller and offloading large workflows to postgres.
As suggested (according to the documentation) we have configured the workflow-contorller to offload the flow to postgres if it exceed the 1MB limitation from etcd.
We have successfully connected to that postgres and query the
argo_workflowss
square-wire-39606
09/01/2023, 2:09 AMStoring large blobs in postgres requires careful thinking - the absolute theoretical limit is quite high (afaik measured in GBs/TBs) but you are likely to run into issues much earlier depending on your query patterns. etcd has a 3MB limit so it is likely that that's where the error originated from.
square-wire-39606
09/01/2023, 2:11 AMIf you are routinely hitting this limit, you might run into issues with the Argo UI too.
square-wire-39606
09/01/2023, 2:15 AMOne mechanism to run this flow as-is without any infra changes would be to look at fields in your argo workflow object that can be pruned away
square-wire-39606
09/01/2023, 2:15 AMotherwise, you can create workflows-of-workflows using metaflow's event triggering to split the work
b
bitter-orange-8782
09/04/2023, 6:11 PMok, understood.
the fields in the argo workflow can be pruned using the yaml?
regarding workflow of workflows event triggering, it requires an event "topic" in argo, right?
9 Views