Hey everyone, There’s a lot of encouraging new developments (company has been mostly using Metaflow<2.12). I wonder if it is now possible with some reusable utility interfaces, to wrap any single function with a decorator to make it into a runnable flow without having to write a whole separate flow script. Particularly, we have been using Runner API in Airflow DAGs to run flows on AWS Batch. Furthermore, it makes sense as part of this to be able to unit test such functions. I’m encouraged by the latest Metaflow 2.18 blog post, but if the entry point is as simple as an RESTful endpoint say via FastAPI, it would be an encouraging push to update to newer Metaflow versions (with internal training).

are you looking for something of the nature of

@metaflowize
def foo(...)

Thanks for the quick reply @square-wire-39606 .I think it’s on the same lines but I can’t find this metaflowize in the docs To flesh this idea further I mean:

@metaflowize
@batch(cpu,memory)
def foo(…) -> pl.DataFrame

Then directly calling foo will run the function in AWS Batch provided that the necessary config is added to Metaflow profile.

yep - that decorator doesn't exist today. we do have some work underway to enable users to create their own syntax to generate flows

We use the Runner API to refer to a path to a flow.py script. I wonder if it is possible to directly put in an imported flow class instead, it looks hackable but if it’s on the roadmap it would be duplicates effort

unfortunately much of that discussion is buried in threads in private slack channels and docs. i can check if some of those are ready to be surfaced.

Much appreciated @ancient-application-36103 🙏🏼

how would this work if

foo

has some arguments?

it could be potentially mapped as parameters?

it would have to come from s3 or something.. similarly the output would have to be serialized

Something that piggy backs on the client API I suppose? Like if run.successful: output = run.variable

To summarize what you would possibly like (I think we have 90% of the pieces there already): • you want a decorator that would take a function

foo(a, b) -> c

and run it in a Metaflow flow with two steps (minimum metaflow flow) where the

start

step would effectively execute the foo function and the

end

step would be a no-op • you want to be able to apply other Metaflow flow or step level decorators to this function (like the

batch

you showed, or

pypi

, etc. • you want input of that function to map to

Parameter

(for simple types — we don’t currently support too complex things but thinking about it) • you want the output of that function to be stored in an artifact like

run.output_variable

or something like that. • ideally, you would like the

Runner

to take the class directly (the class formed through the

@metaflowize

decorator Did I capture what you are looking for accurately?