Hi team, I am very new to metaflow and the demo in the sandbox is great! I was wondering if it is possible to run data processing steps in Java in metaflow too? For example maybe something similar to the BashOperator in airflow?

Hi Enyi ... 👋 .. Glad you liked the demo! Metaflow is currently available in python and R. You can always invoke the java processing from within python using the

subprocess

module. Something like:

@step
    def my_step(self):
        cmd = 'java -cp "pathToJar;pathToCommandJar" com.package.helloworld.HelloWorldCommand'
        subprocess.run(shlex.split(cmd))
        self.next(self.my_next_step)

Fab! Thanks for the prompt response Shri 🙂

Happy to help! Can you share some details about your setup? What are the Java based data processing steps for?

I am thinking of using Metaflow with AWS. Another question about it actually: it seems like metaflow is nicely integrated with AWS, if my workflows are deployed to AWS Step Function, would I still need the metaflow service for tracking the metadata? The Java steps are mainly existing data processing step we have.

The metaflow service is required to track of all the runs, the tasks within each run as well as the artifacts produced by the steps.

@busy-florist-83700 - the metadata service is technically optional when you are deploying the workflow on Step Functions - but you will lose the ability to track the flow with the Metaflow UI or use the Metaflow Client

Makes sense, thanks both!