Hey everyone this is basic question but I have been struggli Outerbounds #ask-metaflow

Hey everyone, this is basic question but I have be...

hallowed-art-27766

06/29/2023, 2:24 PM

Hey everyone, this is basic question but I have been struggling with having my gpu recognized in a metaflow flow running locally. I am able to recognize my gpu locally (outside of a metaflow flow) using

torch.cuda.is_available()

but inside a flow the statement returns False. Any suggestions? Thanks.

✅ 1

user

06/29/2023, 4:01 PM

Hey Jack .. 👋 .. Are you running your flow on Kubernetes?

hallowed-art-27766

06/29/2023, 4:03 PM

Hey shri, I am running locally on my own machine.

ancient-application-36103

06/29/2023, 7:11 PM

can you share a reproducible example?

hallowed-art-27766

06/30/2023, 1:15 AM

Copy code

from metaflow import (FlowSpec, step, FlowSpec, conda, conda_base, step)

@conda_base(python='3.10.9')
class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.my_var = 'hello world'
        self.next(self.a)

    @conda(libraries={
        'pytorch': '2.0.1',
        'cudatoolkit': '11.7.0',
    })
    @step
    def a(self):

        import torch
        import sys
        import os
        from subprocess import call
        print(os.popen("nvidia-smi").read())
        print('__Devices')
        call(["nvidia-smi", "--format=csv",
              "--query-gpu=index,name,driver_version,memory.total,memory.used,memory.free"])
        print(os.popen("nvcc --version").read())

        # See if pytorch picks up the GPUs
        print('__Python VERSION:', sys.version)
        print('__pyTorch VERSION:', torch.__version__)
        print('__CUDA VERSION', torch.version.cuda)
        print('__CUDNN VERSION:', torch.backends.cudnn.version())
        print('__Is CUDA available:', torch.cuda.is_available())
        print('__Number CUDA Devices:', torch.cuda.device_count())
        print('Active CUDA Device: GPU', torch.cuda.current_device())
        print('Available devices ', torch.cuda.device_count())
        print('Current cuda device ', torch.cuda.current_device())
        print(f"GPU count: {torch.cuda.device_count()}")

        self.next(self.end)

    @step
    def end(self):
        print('the data artifact is still: %s' % self.my_var)

if __name__ == '__main__':
    LinearFlow()

flat-television-23413

06/30/2023, 1:54 AM

Did you try without using

cudatoolkit

@conda

? PyTorch should ship with its own CUDA runtime, so long as the driver is installed on the machine (which it clearly is in your case)

flat-television-23413

06/30/2023, 1:57 AM

If that doesn't work, can you paste the results when you run these two commands outside of the flow: •

nvidia-smi

•

nvcc --version

hallowed-art-27766

06/30/2023, 2:14 AM

thanks Eddie, I have removed the cudatoolkit library and got the same error

hallowed-art-27766

06/30/2023, 2:15 AM

Metaflow 2.9.7 executing LinearFlow for user:jack Validating your flow... The graph looks good! Running pylint... Pylint is happy! Bootstrapping conda environment...(this could take a few minutes) 2023-06-29 220451.543 Workflow starting (run-id 5269): 2023-06-29 220454.308 [5269/start/81123 (pid 141269)] Task is starting. 2023-06-29 220503.328 [5269/start/81123 (pid 141269)] Task finished successfully. 2023-06-29 220506.212 [5269/a/81124 (pid 141622)] Task is starting. 2023-06-29 220512.628 [5269/a/81124 (pid 141622)] Thu Jun 29 220512 2023 2023-06-29 220512.639 [5269/a/81124 (pid 141622)] +---------------------------------------------------------------------------------------+ 2023-06-29 220512.639 [5269/a/81124 (pid 141622)] | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] |-----------------------------------------+----------------------+----------------------+ 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | | | MIG M. | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] |=========================================+======================+======================| 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | 0 NVIDIA GeForce RTX 2070 ... Off | 000000000100.0 On | N/A | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | N/A 50C P5 12W / 80W | 94MiB / 8192MiB | 15% Default | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | | | N/A | 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] +-----------------------------------------+----------------------+----------------------+ 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] +---------------------------------------------------------------------------------------+ 2023-06-29 220512.640 [5269/a/81124 (pid 141622)] | Processes: | 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] | GPU GI CI PID Type Process name GPU Memory | 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] | ID ID Usage | 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] |=======================================================================================| 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] | 0 N/A N/A 2835 G /usr/lib/xorg/Xorg 93MiB | 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] +---------------------------------------------------------------------------------------+ 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] __Devices 2023-06-29 220512.641 [5269/a/81124 (pid 141622)] index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB] 2023-06-29 220512.649 [5269/a/81124 (pid 141622)] 0, NVIDIA GeForce RTX 2070 with Max-Q Design, 535.54.03, 8192 MiB, 94 MiB, 7879 MiB 2023-06-29 220512.649 [5269/a/81124 (pid 141622)] nvcc: NVIDIA (R) Cuda compiler driver 2023-06-29 220512.719 [5269/a/81124 (pid 141622)] <flow LinearFlow step a> failed: 2023-06-29 220514.410 [5269/a/81124 (pid 141622)] Internal error 2023-06-29 220514.411 [5269/a/81124 (pid 141622)] Traceback (most recent call last): 2023-06-29 220514.411 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/cli.py", line 1172, in main 2023-06-29 220514.412 [5269/a/81124 (pid 141622)] start(auto_envvar_prefix="METAFLOW", obj=state) 2023-06-29 220514.412 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/core.py", line 829, in call 2023-06-29 220514.412 [5269/a/81124 (pid 141622)] return self.main(args, kwargs) 2023-06-29 220514.888 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/core.py", line 782, in main 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] rv = self.invoke(ctx) 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/core.py", line 1259, in invoke 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] return _process_result(sub_ctx.command.invoke(sub_ctx)) 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/core.py", line 1066, in invoke 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] return ctx.invoke(self.callback, ctx.params) 2023-06-29 220514.889 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/core.py", line 610, in invoke 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] return callback(args, kwargs) 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/_vendor/click/decorators.py", line 21, in new_func 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] return f(get_current_context(), args, kwargs) 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/cli.py", line 581, in step 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] task.run_step( 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/task.py", line 586, in run_step 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] self._exec_step_function(step_func) 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] File "/tmp/tmp3ht1ojhf/metaflow/task.py", line 60, in _exec_step_function 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] step_function() 2023-06-29 220514.890 [5269/a/81124 (pid 141622)] File "/home/jack/ml/ava-mcmt/linearflow.py", line 34, in a 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] print('Active CUDA Device: GPU', torch.cuda.current_device()) 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] File "/home/jack/anaconda3/envs/metaflow_LinearFlow_linux-64_78865ec46b5ad83ff49bbb3321019ab27997cacd/lib/python3.10/site-packages/torch/cuda/__init__.py", line 674, in current_device 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] _lazy_init() 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] File "/home/jack/anaconda3/envs/metaflow_LinearFlow_linux-64_78865ec46b5ad83ff49bbb3321019ab27997cacd/lib/python3.10/site-packages/torch/cuda/__init__.py", line 239, in _lazy_init 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] raise AssertionError("Torch not compiled with CUDA enabled") 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] AssertionError: Torch not compiled with CUDA enabled 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] Copyright (c) 2005-2022 NVIDIA Corporation 2023-06-29 220514.891 [5269/a/81124 (pid 141622)] Built on Tue_May__3_184952_PDT_2022 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] Cuda compilation tools, release 11.7, V11.7.64 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] Build cuda_11.7.r11.7/compiler.31294372_0 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __Python VERSION: 3.10.9 (main, Jan 11 2023, 152140) [GCC 11.2.0] 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __pyTorch VERSION: 2.0.1 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __CUDA VERSION None 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __CUDNN VERSION: None 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __Is CUDA available: False 2023-06-29 220514.892 [5269/a/81124 (pid 141622)] __Number CUDA Devices: 0 2023-06-29 220515.189 [5269/a/81124 (pid 141622)] Task failed. 2023-06-29 220515.402 Workflow failed. 2023-06-29 220515.403 Terminating 0 active tasks... 2023-06-29 220515.403 Flushing logs... Step failure:

flat-television-23413

06/30/2023, 2:36 AM

How did you install pytorch locally? With pip? As far as I'm aware using drivers > 12.0 is not officially supported yet (see this issue), and the conda package hasn't been updated in two months. Another idea: If you only intend to run this locally, I think you may be able to remove

@conda_base

and

@conda

and run with the same versions of these dependencies you mentioned were working outside of the flow.

hallowed-art-27766

06/30/2023, 7:29 PM

Ok I finally got it to work. Here are the versions that worked for me:

Copy code

python==3.8.10
cuda==11.7 (11.5 inside the flow)
nvidia driver==515
cudnn=8.5 (8.3 inside the flow)
metaflow==2.7.1 (metaflow version can affect whether my gpu is seen or not)
pytorch==1.11.0

(I had to re-install conda and delete .metaflow folder)

hallowed-art-27766

06/30/2023, 7:30 PM

Thanks for letting me know cuda 12 is not fully supported.

flat-television-23413

06/30/2023, 7:55 PM

Sure thing, nice sleuthing!

✅ 1

ancient-application-36103

06/30/2023, 8:32 PM

@hallowed-art-27766 can you expand on

(metaflow version can affect whether my gpu is seen or not)

ancient-application-36103

06/30/2023, 8:32 PM

were you seeing different results keeping everything the same but changing the metaflow version?

hallowed-art-27766

07/01/2023, 3:29 AM

I have tested it by changing the metaflow version to

2.9.1

and it still recognized my gpu so I take that back. Apologies. I was under the impressiong that metaflow version was a cause for my issue but it seems not.

Open in Slack

Previous Next