The human-centric platform for production ML & AI

Outerbounds

Is it possible to launch Metaflow steps with fractional GPUs? We have Multi-Instance GPU (MIG) set up. A k8s equivalent to:
```resources:
  limits:
    nvidia.com/mig-1g.24gb: "1"
  requests:
    nvidia.com/mig-1g.24gb: "1"```

GPT suggests using Kyverno as an admission controller to modify k8s pods containing keyword labels and inject MIG resource that way.

<@U08D5N08BRA> (obligitory not an Outerbounds employee) Are you using Argo Workflows on k8s?
If so, this is where the resources get set - <https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/kubernetes/kubernetes_jobsets.py#L674|github.com/Netflix/metaflow/blob/…/kubernetes_jobsets.py#…>
This is where the environment/conf variables get sucked in <https://github.com/Netflix/metaflow/blob/f8295b1d5a54c22cd78760bbdfe0346985002fdd/metaflow/metaflow_config.py#L415|github.com/Netflix/metaflow/blob/…/metaflow_config.py#…>
And this is where the kubernetes decorator is defined - <https://github.com/Netflix/metaflow/blob/f8295b1d5a54c22cd78760bbdfe0346985002fdd/metaflow/plugins/kubernetes/kubernetes_decorator.py#L54|github.com/Netflix/metaflow/blob/…/kubernetes_decorator.py#…>
You might consider forking, adding a param called something like `shared_gpu` and `shared_gpu_type`. You can then use decorators or your metaflow config to control if it's a shared GPU workload or not. This has been something we're after, so if you end up going this route...consider opening a PR? :slightly_smiling_face: