Hi team, I am facing issues with using pytorch wit...
# ask-metaflowt
thousands-dusk-37800
12/05/2023, 7:06 AMHi team, I am facing issues with using pytorch with Azure Kubernetes.
Attached a screenshot of the error that I am seeing.
My hunch is that my cluster supports CPU only and conda is installing the GPU version of pytorch because of which its failing somewhere. This is the dependency that I am using
@conda_base(libraries={“pytorch:pytorch”“2.1.1", “pytorch:torchvision”“0.16.1", “pytorch:cpuonly”“*”}, python=“3.8”)
And this works just fine locally.
thousands-dusk-37800
12/05/2023, 7:07 AMMaybe my hunch is wrong because my local machine also doesn’t have a GPU.
Also I tried to run simple flows on the same Kubernetes cluster and they seem to be working fine
m
most-telephone-75272
12/05/2023, 8:35 AMthat's the same error I had here https://outerbounds-community.slack.com/archives/C02116BBNTU/p1700592774256399
most-telephone-75272
12/05/2023, 8:36 AMcould be other things, tl;dr is that metaflow generations the conda.manifest locally and creates a hash with the dependencies in your case
a2f1123ff9e658amost-telephone-75272
12/05/2023, 8:37 AMsomewhere down the line the hash is not the same, this could be because the python version is handled different as happened to me above of maybe the architecture (you generate on for linux-64 and you load the manifest somewhere else)
most-telephone-75272
12/05/2023, 8:38 AMmost-telephone-75272
12/05/2023, 8:39 AMthings like platform and python version are put into the hash
👍 1
d
dry-beach-38304
12/05/2023, 9:29 AMCould you try the bleeding edge version of the decorators: http://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators. In some cases, they deal with these cases better. If it doesn’t work either, let me know.
👍 1
s
square-wire-39606
12/05/2023, 3:27 PM@thousands-dusk-37800 what version of metaflow are you on?
m
most-telephone-75272
12/05/2023, 4:42 PM@thousands-dusk-37800 probably changing python from
python=3.8python=3.8.12🙌 1
t
thousands-dusk-37800
12/05/2023, 5:48 PMThanks folks for the help. @aloof-army-35605 changing the python version worked!
10 Views