https://outerbounds.com/ logo
Join Slack
Powered by
Hi All I have a question with regard to the inabi...
# ask-metaflow
m
Hi All I have a question with regard to the inability to set resource limits on Kubernetes-based workflows through Metaflow. I have asked several times and the general feedback is that this should be done through a policy agent. I wonder whether anyone can provide me with a reference for such an approach? I have investigated mutations through open policy agent gatekeeper. However, it seems it is not possible, as far as I can tell, to set a resource limit to be equal to the resource request. In particular, there is not a generic solution for using information in the current object in a mutation. It seems it is possible to set the limit to some standard default. However, that is not going to suffice for all workflows, so this feels like it will not be a scalable solution. Thanks in advance!
✅ 1
f
Not sure why this isn't supported in the 
kubernetes
decorator itself though.
m
I did make a Github issue on this point, but there was a preference on the Metaflow side against it. IIRC it adds complexity to the user experience, or something like this. I can't remember the exact reasons, to be honest. I'm not adverse to trying the policy agent approach. It is just not clear how to actually do it in practice.
a
re: mutations - you could use a kyverno policy. we use that internally to apply validation checks, enforce priorities and heuristics to set requests & limits. re: enabling limits client side - there is no technical reason not to enable it, it is more of a user experience consideration. ideally, one would want to set requests and limits the same for memory and disk but leave cpu limits floating. this may or may not work for everyone - but could be an option that is exposed through the config. as a proposal - setting 
METAFLOW_KUBERNETES_QOS=burstable
could unset limits, 
METAFLOW_KUBERNETES_QOS=guaranteed
could set requests=limits, 
METAFLOW_KUBERNETES_QOS=optimum
could set requests=limits for everything but cpu. i am yet to come across a use case where requests and limits would be set to non-zero yet different values for running ML workloads.
m
Copy code
re: mutations - you could use a kyverno policy. we use that internally to apply validation checks, enforce priorities and heuristics to set requests & limits.
Do you have an example, or can you point me to a reference on how you can use kyverno to set a resource limit to match the resource request of the pod? I don't seem to be able to see how one can use the current object as part of the mutation, but maybe I am missing something obvious. I know it is definitely not possible with gatekepper
> re: enabling limits client side - there is no technical reason not to enable it, it is more of a user experience consideration. ideally, one would want to set requests and limits the same for memory and disk but leave cpu limits floating. this may or may not work for everyone - but could be an option that is exposed through the config. as a proposal - setting 
METAFLOW_KUBERNETES_QOS=burstable
could unset limits, 
METAFLOW_KUBERNETES_QOS=guaranteed
could set requests=limits, 
METAFLOW_KUBERNETES_QOS=optimum
could set requests=limits for everything but cpu. > > i am yet to come across a use case where requests and limits would be set to non-zero yet different values for running ML workloads. This is effectively the proposal we originally agreed upon and for which I subsequently made a PR. However, I wasn't able to make much progress in getting a review on it on the Metaflow side and you suggested using a policy agent instead.
p
https://kyverno.io/policies/other/add-default-resources/add-default-resources/ is a mutating policy that adds defaults for resource requests if they're not listed. This would be a starting point for "find pods in the metaflow namespace with requests set but not limits and set the limit to match the request". I thought I had a more specific example policy for just that, but I can't find it now.
a
i was going to share the same link ^
p
FWIW, I went down the path of "I have to make every metaflow job have the best possible QoS by matching resource requests==limit" but eventually realized that setting just the request and avoiding scheduling jobs that burst beyond their requests on the same nodes seems to be good enough to avoid getting pods killed due to memory pressure.
💯 1
a
one big benefit of using a policy agent is that you can apply these policies to all the workloads in one shot without requiring every user to not only upgrade metaflow but also redeploy argo-workflow templates. it should theoretically make it easier to assess the impact of this change organization wide.
m
> FWIW, I went down the path of "I have to make every metaflow job have the best possible QoS by matching resource requests==limit" but eventually realized that setting just the request and avoiding scheduling jobs that burst beyond their requests on the same nodes seems to be good enough to avoid getting pods killed due to memory pressure. My personal experience is that we get sporadic instances in which rogue (i.e. badly configured) workflows cause issues for a large number of nodes. It happened to us again this week, for example. As far as I am aware, this is one of the main reasons that limits exist in the first place. IMO this is very easy for users to do this by mistake and I think it is best to have guardrails in place to prevent it.
> https://kyverno.io/policies/other/add-default-resources/add-default-resources/ is a mutating policy that adds defaults for resource requests if they're not listed. This would be a starting point for "find pods in the metaflow namespace with requests set but not limits and set the limit to match the request". I thought I had a more specific example policy for just that, but I can't find it now. Thanks for the link. It still seems to be setting static mutations, and not mutations based on the resource itself, though. I don't think that is a viable solution for what I want to do. Anyway, thanks again. I'll have a further look at the kyveno docs and see if it is possible.
a
something of this sorts should work (haven't tested) -
Copy code
apiVersion: <http://kyverno.io/v1|kyverno.io/v1>
kind: ClusterPolicy
metadata:
  name: set-limits-equal-to-requests
spec:
  rules:
    - name: set-limits-equal-to-requests
      match:
        resources:
          kinds:
            - Pod
      mutate:
        foreach:
          - list: "spec.containers"
            context:
              - name: requestsCpu
                variable: "{{ element.resources.requests.cpu }}"
              - name: requestsMemory
                variable: "{{ element.resources.requests.memory }}"
            patchStrategicMerge:
              spec:
                containers:
                  - (name): "{{ element.name }}"
                    resources:
                      limits:
                        cpu: "{{ requestsCpu }}"
                        memory: "{{ requestsMemory }}"
            preconditions:
              all:
                - key: "{{ element.resources.requests.cpu }}"
                  operator: Exists
                - key: "{{ element.resources.requests.memory }}"
                  operator: Exists
                - key: "{{ element.resources.limits.cpu }}"
                  operator: NotExists
                - key: "{{ element.resources.limits.memory }}"
                  operator: NotExists
m
cool, thanks! Yeah, reading through the docs in more detail, I see you can reference the request object, as in this example. I think this is enough for me to scope a solution. Thanks!
Open in Slack
PreviousNext
Powered by