is there a way to reference extra libraries (eithe...
# ask-metafloww
wooden-dusk-90720
09/13/2023, 10:05 PMis there a way to reference extra libraries (either local directories or 3rd party libraries) using the metaflow pyspark plugin?
ETA: I see it's marked as a TODO here. Guess it doesn't work right now?
wooden-dusk-90720
09/13/2023, 10:14 PMWill local files work if we pass
--py-files=METAFLOW_CODE_URLsparkSubmitParametersv
victorious-lawyer-58417
09/14/2023, 6:19 AMprobably not out of the box. Would you need to include local py files or 3rd party modules?
w
wooden-dusk-90720
09/14/2023, 6:27 AMyes, we want to include both local files and 3rd party modules
wooden-dusk-90720
09/15/2023, 12:58 AMI figured out how to do it with the use of a custom docker image and passing a bunch of extra arguments to the spark config. I'll send you a PR once I clean up the code. Would be nice to do it without the custom image -- maybe you'll have some ideas
v
victorious-lawyer-58417
09/15/2023, 1:55 AMtotally. I should be able to share some ideas by the end of the week
w
wooden-dusk-90720
09/15/2023, 4:25 AMhere it is: https://github.com/outerbounds/metaflow-pyspark/pull/1
thinking about it some more, it might be possible to get rid of the docker image by using
venv-packspark.archives3 Views