Hi, if we want to use the <metaflow-pyspark> plugi...
# ask-metafloww
wooden-dusk-90720
08/04/2023, 11:35 PMHi, if we want to use the metaflow-pyspark plugin, does it need to be in declared in the flow dependencies (ie
@conda@pip_base✅ 1
v
victorious-lawyer-58417
08/05/2023, 12:13 AMno, extensions get included automatically
victorious-lawyer-58417
08/05/2023, 12:14 AMare you going to use Spark on EMR Serverless?
w
wooden-dusk-90720
08/05/2023, 12:22 AMA colleague is going to try it out. Are there dragons?
v
victorious-lawyer-58417
08/05/2023, 12:25 AMEMR Serverless should work out of the box. Elsewhere be dragons 🐉
victorious-lawyer-58417
08/05/2023, 12:25 AM(it's just that spark config works differently in different environments - we can add support as needed)
👍 1
w
wooden-dusk-90720
08/05/2023, 6:29 PMis anyone using Metaflow with Java/Scala? I'm curious how it's setup -- like is there a separate build system and Metaflow only uses the published jars?
v
victorious-lawyer-58417
08/05/2023, 6:31 PMyou could call Java/Scala apps like any other external binary e.g. if you had an uber-jar that you could call on the CLI
6 Views