are any of you building out metaflow pipelines from config files that dynamically write the code for you? does anyone have any examples that are publicly available?

hi Emily 👋 One (slightly contrived) example is Metaflow's own test suite that generates Metaflow flows based on config files (see here)

@straight-shampoo-11124 the feedback i got from my data scientist stakeholders is that they want to be able to configure an entire metaflow pipeline from a simple json or yaml config without actually jumping into all the code i've written to make the pipeline work for them. does that help?

(highlighting this answer since it is a FAQ) If you have a well-defined workflow that data scientists don't need to change often, you can certainly make it all config-driven. Take a look at this simple example that reads a config file like this:

{
  "model": "svm",
  "dataset": "wine"
}

In this case the config is in JSON but t could be YAML or anything else as well. It defines what data and model family to use and based on this info, trains and evaluates a model using a predefined workflow. In a more complex real-life case you could have

model_

and

dataset_

functions be plugins that data scientists can contribute by themselves (see e.g. this example from the Metaflow book for inspiration), and you can have pluggable feature encoders too to ensure offline/online consistency. You still get all the benefits of Metaflow in this example: Configs for every run get persisted automatically (thanks to

IncludeFile

, as well as all the other artifacts). You can test it at scale and deploy as usual.