https://outerbounds.com/ logo
Join Slack
Powered by
I'm running into an issue with `@kubernetes` CUDA ...
# ask-metaflow
c
I'm running into an issue with 
@kubernetes
CUDA and 
pypi
The code works on AWS Batch. I've deployed k8s and the nvidia plugin for pods (https://github.com/NVIDIA/k8s-device-plugin/tree/main/deployments/helm/nvidia-device-plugin) my pod can see the GPU using 
nvidia-smi
however I'm getting the following error.
Copy code
@pypi_base(
    python="3.11.7",
    packages={"torch": "2.2.1", "torchmetrics": ""},
    extra_indices=["<https://download.pytorch.org/whl/cu118>"],
)
Copy code
@kubernetes(cpu=1, memory=6000, gpu=1, shared_memory=6000)
@environment(vars={"LD_LIBRARY_PATH": "/tmp", "NVIDIA_DRIVER_CAPABILITIES": "compute,utility"})
@step
Copy code
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory
I've also tried using the docker image 
pytorch/pytorch:2.2.1-cuda11.8-cudnn8-runtime
same error.
✅ 1
c
My guess is that the issue is compatibility between other dependencies and the CUDA toolkit. Suggested paths to try: • use 
@kubernetes(image='pytorch/pytorch:2.2.1-cuda11.8-cudnn8-runtime')
without @pypi • if you want use custom torch versions, use @conda and install compatible version of cuda toolkit
c
I should also mention I'm using the netflix extension from @dry-beach-38304
@crooked-jordan-29960 I will try that but I really want the environment to be controlled by metaflow, as opposed to having to maintain docker images with our environment. I'm the worst at maintaining docker images 😆
c
Is there a specific reason you need to be setting LD_LIBRARY_PATH=/tmp? This changes where the dynamically linked .so files are searched for. Where ever the cuda runtime file 
libcudart.so
is, should be on LD_LIBRARY_PATH path, otherwise CUDA app cannot start because it fails to link to where you (by way of installing pytorch) installed these linux dependencies.
d
You shouldn’t have to set ld lib path. The extension will take care of it for you. I am out today but I’ll check in a bit.
c
That was a great thought but same error.
d
have you tried with the conda decorator instead of pypi? It may better install some of those deps (worth a try if you haven’t).
c
I thouht pypi with the extension was conda under the hood anyway
d
it is and it isn’t 🙂. I mean it does use conda to create a base environment but then uses pip to install packages in that environment. The conda one installs the conda packages which may include more libraries.
c
Could it be I setup k8s wrong because the same code works fine with AWS Batch. Is there some special docker image or AMI I should use that has CUDA
d
that part I am not work. What image is used with batch?
c
I never specified one, I'll take a look
c
this repo contains some examples tested on k8s
c
Awesome thank you. From what I'm seeing you do what you said before use a custom image without pypi or conda env?
For future reference. I'm not sure what was wrong exactly. Something in my conda environment went wrong, to the point where I was getting timeout errors on lock files when building environments. I eventually completely purged conda and everything and setup from scratch. Once I had I ran the flow again and it worked correctly. I know thats not exactly helpful but thats what made it work. Thanks to everyone who helped. 🙂 Notes Make sure your node_group has the right AMI type and you've setup nvidia-pod-plugin (https://github.com/NVIDIA/k8s-device-plugin/tree/main/deployments/helm/nvidia-device-plugin)
Copy code
gpu_group = {
      ami_type  = "AL2_x86_64_GPU" <--------- GPU has cuda

      desired_capacity = 0
      max_capacity     = 1
      min_capacity     = 0

      instance_types = ["g4dn.xlarge"]

      update_config = {
        max_unavailable_percentage = 50
      }
      kubelet_extra_args = "--node-labels=nvidia.com/gpu=true"
}
d
hum — I need to improve this stuck thing at least. Not sure what happend with the resolved envs though. That is weird
17 Views
Open in Slack
PreviousNext
Powered by