is there a reason the

@pyspark

decorator needs a

job

parameter? couldn't it use the code inside the step as the job?

good question! A challenge with that approach is that it wouldn’t need only the code inside the step but also the whole module surrounding it for imports etc The module contains Metaflow imports, so the whole Metaflow would need to be shipped etc Also the pyspark step wouldn’t behave like other Metaflow steps - you couldn’t refer to self. etc, so it would only look like a step but not behave like one, which is a bit confusing