One more question… does metaflow ever build images...
# ask-metaflows
square-accountant-83677
04/04/2023, 8:47 PMOne more question… does metaflow ever build images?
✅ 1
square-accountant-83677
04/04/2023, 8:50 PMI need to make sure I’m not pushing my company’s docker images to dockerhub or something
a
ancient-application-36103
04/04/2023, 9:04 PMWe don’t build any images on the fly
s
square-accountant-83677
04/04/2023, 9:05 PMAwesome
square-accountant-83677
04/04/2023, 9:06 PMSo how do I get my libraries to sync over? Like if I’m working on a thing locally, how do I execute that remotely?
v
victorious-lawyer-58417
04/04/2023, 9:09 PMtwo approaches:
• Create and use an existing custom Docker image. Metaflow doesn't create or push any images automatically.
• Use
@conda👍 1
a
ancient-application-36103
04/04/2023, 9:10 PMAlso, any python file in the same directory as your flow automatically gets shipped to AWS Batch
👍 1
s
square-accountant-83677
04/04/2023, 9:11 PMOh that’s awesome!
v
victorious-lawyer-58417
04/04/2023, 9:12 PMdependency management can be a gnarly topic, so don't hesitate to ask if you have any issues with it
s
square-accountant-83677
04/04/2023, 9:16 PMWill do
👍 1
square-accountant-83677
04/04/2023, 9:17 PMI’m seeing you suggest using os to pip install libraries that aren’t in conda
square-accountant-83677
04/04/2023, 9:17 PMIs that right?
square-accountant-83677
04/04/2023, 9:18 PMSpecifically, I’m trying to install jax with gpu support
square-accountant-83677
04/04/2023, 9:18 PMIn batch
v
victorious-lawyer-58417
04/04/2023, 9:27 PMgetting CUDA drivers match with a JAX version can cause headaches (hopefully NVidia will make their official JAX image public soon)
meanwhile you can try using the Nvidia Docker image and install JAX on the fly as described here. Let us know if you can't get it working easily, so we can look into it more
👍 1
2 Views