Hi everyone! I've had some success setting up my cloud/onprem hybrid multicluster metaflow, but I'm hitting one last thing that seems it should be easy. If I do

docker run -it --rm myimage:latest /usr/bin/nvidia-smi

I get

/usr/bin/nvidia-smi not found

. But, if I run

docker run -it --rm --gpu all myimage:latest /usr/bin/nvidia-smi

I get the proper output. When I run my nvidia-smi from within the step below, I get the

/usr/bin/nvidia-smi no found

message.... how do I influence metaflow to pass the

--gpu

parameter?

@kubernetes(
        image="myimage:latest",
        gpu=1,
        tolerations=[{"key": "<http://node-role.kubernetes.io/control-plane|node-role.kubernetes.io/control-plane>", "operator": "Exists", "effect": "NoSchedule"}],
    )
    @resources(gpu=1)
    @step
    def nvidia_smi(self):
        """Make sure that we can run nvidia-smi."""
        subprocess.run(["/usr/bin/nvidia-smi", "-q"])
        self.next(self.end)

how is the kubernetes cluster setup? usually we rely on

nvidia-device-plugin-for-gpu

to pass the

gpu

flag

I had followed the nvidia-device-plugin instructions, but it never would advertise the GPU's as available so I've added that extended resource manually using the kubectl proxy. Possibly there is an issue at that layer

nice! did it work finally?

Yeah, just the install and its working perfectly

great!