I haven't taken a look at the new Metaflow plugin design. Does it support loading Python packages dynamically? Maybe any package that begins

metaflow_

? The reason I need this is to support a dynamic way to "decorate" a Metaflow step with Kubernetes "tolerations, affinity" etc based on a

@resource("gpu": 1, "accelorator": "nvidia-tesla-p100")

that could render the "tolerations, affinity" or "label, nodeSelector" based on the plugin specific to the Kubernetes deployment. I don't want a Zillow specific way to be the default

interesting. You'd still need to list all custom decorators explicitly in your

metaflow_custom

but those decorators will become available for all users of

zillow_metaflow

or whatever you want to call it

@accelerator("nvidia-tesla-p100")
@resource("gpu")

looks nice.. Does Metaflow now have a mechanism to dynamically load Python modules like

zillow_metaflow

?

your users would

pip install zillow_metaflow

that is a custom package that includes custom plugins and then the upstream

metaflow

as a dependency

Would it need to be in STEP_DECORATORS https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/__init__.py#L69 ?

Yes. I can share the doc I wrote about custom plugins in a bit but yes, add stuff in STEP_DECORATORS. Currently only supports a single customization. The package it looks for is called metaflow_custom so your zillow_metaflow package installs that package and depends on an upstream metaflow. For your users it looks like a single metaflow package with all your extras and defaults.

Hey @broad-photographer-33842 have you shared the doc on custom plugins publicly yet? I'd love some help getting this supported for my org