https://outerbounds.com/ logo
Join Slack
Powered by
As a very simple example, here is some code:
# ask-metaflow
f
As a very simple example, here is some code:
✅ 1
d
should be — you may have to install tensorflow-gpu though.
f
Isn’t tensorflow gpu deprecated? “https://pypi.org/project/tensorflow-gpu/”
If run my code, I get 0 GPU’s available. When I remove the @conda decorators, and just run the code w/o --environment, I get the correct number of GPUs.
Using --environment=conda creates a blank slate it seems, I even have to reinstall tensorflow, even though the docker image should already have it. In addition to wiping tensorflow it also seems to remove any instructions to use the gpu, although tensorflow should do this automatically it seems.
d
A few quick notes: • yes, using 
@conda
will create a “clean slate”. It is meant to isolate you from the underlying image’s environment in a way (it’s not additive) ◦ if you have a docker image with everything you need on it, by all means, use that and don’t use 
@conda/@pypi
• if you use 
@conda
, packages are taken from the conda world (so not pypi). In particularl, I don’t think 
tensorflow-gpu
in the conda world (https://anaconda.org/conda-forge/tensorflow-gpu) is deprecated. It does seem to depend on regular tensorflow but also a cuda environment which may help with it not finding GPUs 🙂
f
After using @conda, I got complaints regarding lack of cuda drivers. Ultimately I think I’ll go with installing via pip once the machine is built. Couldn’t get over the @pip issue of needing the ‘promise’ library and the @conda issue of removing CUDA instructions. Thanks for all of the help though!
d
could you let me know what environment you are trying to support. 
@conda
should not remove the CUDA instructions and you should be able to build a conda environment with CUDA libraries and then install promise on top of it. No promises or anything but I can then try to resolve the environment for you and see if it works properly (if you have a test flow to confirm whether it works or not that would be helpful too)
f
I am trying to use this image from Docker 
image="<http://docker.io/tensorflow/tensorflow:latest-gpu|docker.io/tensorflow/tensorflow:latest-gpu>
.
Here is a sample program:
from metaflow import FlowSpec, step, batch, conda, conda_base, pypi
@conda_base(python='3.12')
class dummy_test_pypi(FlowSpec):
@batch(gpu=1, image="<http://docker.io/tensorflow/tensorflow:latest-gpu|docker.io/tensorflow/tensorflow:latest-gpu>", queue="job-queue-gpu-metaflow",)
# @pypi(packages={'keras-cv': '0.9.0', 'tensorflow': '2.16.1'})
@conda(libraries={'keras-cv': '0.9.0', 'tensorflow': '2.16.1'})
@step
def start(self):
import tensorflow as tf
print("tensorflow" + tf.__version__)
import keras_cv
print("keras_cv" + keras_cv.__version__)
import sys
print("Python version")
print(sys.version)
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
self.next(self.end)
@step
def end(self):
print("All done. \n\n Congratulations!\n")
return
if __name__ == '__main__':
dummy_test_pypi()
When using @pypi the issue is the promise library, when using @conda the issue is the program can’t detect a GPU
Feel free to use < for library/Python versions, they don’t necessarily have to be the exact versions listed
This is the CloudFormation template I am using, although this should not be the issue
metaflow-cfn-gpu.yaml
2 Views
Open in Slack
PreviousNext
Powered by