The human-centric platform for production ML & AI

Outerbounds

Hi all, we recently upgraded our k8s cluster to GKE autopilot 1.29 and all of a sudden all of our workflows are not working with the defaults.
After a lot of debugging, it seems to be related with the fact that autopilot has changed the way it calculates the resource allocations, making the base requests of `4096` for ram and `10240` for disk too large (by fractions... but it fails to schedule the pods).

My question is, where can I change the <https://docs.metaflow.org/scaling/remote-tasks/kubernetes|default configurations>? to be slightly lower, without having to specify for all of our flows, so our clusters will accept the base requests?

UPDATE: I found it in
`~/miniforge-pypy3/lib/python3.9/site-packages/metaflow/plugins/kubernetes/kubernetes_decorator.py`

is there a way to replicate this same configuration change for argo-workflows?

UPDATE: I was able to confirm that once the defaults are changed in the decorator, it follows through to Argo.