Hi Team, I was using the terraform-aws-metaflow to deploy eks clusters. I found there is no horizontal pod autoscaler in aws. In order to have eks autoscale pods for the metaflow deployment, is there a way/document we can have the metaflow scripts running on eks and enable the hpa on eks automatically? Thank you!

@great-computer-6335 - HPA is used for scaling out services. when you use @kubernetes - we run a pod that runs till completion - there isn't any utility for HPA there. Maybe I am misunderstanding your question?

@ancient-application-36103 thank you for your reply! My understanding is that the hpa has to target to specific deployment. So once the metaflow script runs as a deployment, we need to enable the hpa on eks. I guess under the hood, metaflow framework runs a metaflow script as deployment in a pod through calling eks api, so why can't we enable hpa at the same time when metaflow calls eks api to run the script in a pod/deployment? I mean metaflow framework probably should support this by allowing users to specify a parameter when run the script. I am super new to metaflow, let me know if my understanding is correct, thank you!

@great-computer-6335 metaflow runs your task as individual pods and not deployments

So do you mean we can specify how many pods we can run for that script through specifying the number of cpu and gpu?

number of pods are controlled by the

foreach

attribute

I see

this line in

def start

-

self.next(self.a, foreach='titles')

- instructs metaflow to launch a pod for every element in title

Got it, thanks! Does that mean we cannot dynamically scale up and down for the script we run?

the number of pods are dynamic

hmm, but that is defined in the script, right?

you could do -

from metaflow import FlowSpec, step, kubernetes, parameter

class ForeachFlow(FlowSpec):
    num_titles = Parameter(num_titles)
    @step
    def start(self):
        self.titles = range(num_titles)
        self.next(self.a, foreach='titles')

    @kubernetes(cpu=2, gpu=1)
    @step
    def a(self):
        self.title = '%s processed' % self.input
        self.next(self.join)

    @step
    def join(self, inputs):
        self.results = [input.title for input in inputs]
        self.next(self.end)

    @step
    def end(self):
        print('\n'.join(self.results))

if __name__ == '__main__':
    ForeachFlow()

yeah, the autoscaler will bring more nodes for scaling up, but HPA is for scaling up on pods level. So what we wanted is HPA 🙂

metaflow takes care of that

You mean "services" is like metaflow service we installed on eks?

correct!

When we check the aws eks cluster, the HPA page shows nothing, is that normal?

you would have to enable it for the services

yeah

Hmm yeah I am not sure how that page works. Applying HPA to metaflow service should be similar to any other service..

Do you mean we have to manually apply HPA to the metaflow service after the cluster is setup?

the terraform template is a starter template and is not intended to cover all varieties of deployment. we usually recommend folks to write their own templates taking inspiration from the current template

That sounds fair.

with metaflow you can submit thousands (and millions) of pods

E.g. clients specify 1 pod to run a script but it is actually not enough, the job then fails and the client has to specify more pods in the script and re-run, right?

depends - you can scale up as well as scale out

OK. I will take a look. Thank you!

no problem! let us know if you have any more questions!