hey MF folks! I'm thinking of cleaning up a bit and PR the prefect guys with this

<https://github.com/jacopotagliabue/metaflow-as-prefect-task/blob/main/orchestration/MetaflowShellTask.py>

which is a MF shell task supporting either a local flow file, or - mimicking other Prefect classes - a remote github repo containing the target MF flow. Quick q: 1. Do you guys think it's helpful to get a first stab at it, with some minimal functionalities up and running? 2. If yes, do you want me to send you my WIP code, or is something on your roadmap to support "officially", so you want to handle it yourself?

@worried-mechanic-36312 this looks great! Let me set up prefect on my end and give it a try. How are you using this shell task today?

mind u, I just tested it with prefect cloud 😉 if u have prefect cloud, this quick script will work to run the sample code

<https://github.com/jacopotagliabue/metaflow-as-prefect-task/blob/main/orchestration/prefect_cloud_main.py>

(ask if anything is unclear)

@worried-mechanic-36312 I have noticed a few folks indeed pursue that path with Airflow etc. in the community and it's a great start - but it should be possible for us to augment your existing DAG with a metaflow DAG.

oh, so you're thinking of mapping

MF steps -> P UI tasks

and so having

@retry

and all that to work (through some config) for Prefect like they know work in vanilla MF? That's cool, but also way more complex that what I was trying to do, which is letting prefect run metaflow as a command line tool and rely on metaflow own code for its execution (similar to dbt prefect task, which launch DBT but then relies on DBT intenral code to execute the SQL grapg)

Yep, you are indeed correct that it's way more complex 🙂 We would still want to make sure that we are providing appropriate hooks within Metaflow so that running Metaflow flow inside a Prefect step is rather straightforward and users don't have to peer inside our implementation to get it working.

yeah, let me chat a bit with the prefect guys to get their contribution guideline, and then I'll draft up something simple and dbt-like to start

<https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask>

, and share it with you for your review - sounds good? I'll try to work on this when I have some free time here and there in the next few weeks 😉

Sounds good!