Sorry, an additional (completely unrelated) question. It seems that all Kubernetes related functionality in Metaflow, e.g., Argo Workflows, relies on resource requests only and there is no use of limits. Ideally we would like to be able to at least specify limits on resources. Is there any appetite in Metaflow to allow for the specification of limits in Kubernetes?

@mammoth-rainbow-82717 great point - ideally we would only like to support limits for specific attributes. one way to introduce them could be through a flag in metaflow config. what do you think of that idea?

Hey Thanks for the response!

correct

Cool. Yeah, it makes sense to me. Would be happy to make an issue for it, if you think it is worth recording.

totally!

Also happy to have a stab at if, if you are open to that. 😄 No problem, if not. 🙂

that will be great!

ok, cool! I'll make the issue now and then pick up the work soon.

@mammoth-rainbow-82717 Did you create an issue on this? (Found it) While this might solve a few peoples problem incl. Thomas, I think it is still not sufficient for most peoples needs. Request are “I need at least X”, limits are “I want this pod to use at most Y”, These should not really be exclusive. When using kubernetes, often you want to specify both of these. So I am a little worried that the solution suggested by @ancient-application-36103, will move us to a worse position for making a good solution in the future, because then we end up having this flag that then needs to be deprecated.

Hi, I have made a PR with the proposed approach. Would be great to get a review, if you have time. There is an alternative approach in a related PR and some open comments on the issue too btw. Both are linked in the PR. Thanks!

@mammoth-rainbow-82717 thanks for the reminder! one proposal here is to introduce a

qos

argument -

guaranteed

,

burstable

and

besteffort

- similar in spirit to what kubernetes enables today. this will avoid the complexity of having to think about requests and limits (and what do they even mean for gpus)

Hi Thanks for the response. Slightly confused by the suggestion though. Unless I am missing something, the docs suggest that

qos

is automatically assigned (by Kubernetes) depending on the limit/request specification of the pod. Am I missing something here? For context, one of our main use cases for limits would be to put limit ranges in place for our namespaces, which requires all pods to have limits in place. One could try to set a default limit, but there is no guarantee it will be less than the request. This would lead to pods that can't be scheduled.

Agree with @mammoth-rainbow-82717 I don't think qos is the solution here because we may not know the request & limit at declaration time - pushing this decision to the step's decorator fits more use cases

Indeed, this week we've had more issues related to not being able to set limits. Would love this to get prioritised.