i'm struggling a bit with writing my own metaflow extensions. Is it possible to import multiple packages somehow that extend metaflow? I also use netflix-ext for using conda, and next to that I'd like to add my own decorators. But that is where it goes wrong. For some reason, it cannot find the decorators I use from netflix-ext anymore at this point. Any ideas @dry-beach-38304?

yes, you can have multiple extension packages out of curiosity, could you implement your custom decorators using the new custom decorator mechanism? It’s much simpler than extensions The only downside is that you don’t get access to all internals of Metaflow but many decorators don’t need to do that anyways

What @victorious-lawyer-58417 said. If you do need access to the internal hooks (basically, the user level decorators allow you to do things before and after a step runs while the internal decorators have a lot more hooks and access to more internal objects), a good place to start is METAFLOW_DEBUG_EXT=1 which will print out what metaflow is trying to load.

what I actually intend to do is create a decorator that is a variation on @schedule, so I need access to the schedule. I'll check out to debug the extension first to see what's going on

okie — lmk. Note for the

schedule

decorator, it doesn’t actually do anything itself but it is used by other things to change their behavior. If you look at the

schedule

decorator, it just sets

self.schedule

which is then used in argo like so for example:

schedule = _self_.flow._flow_decorators.get("schedule")

. To modify stuff like that without actually changing the users (ie: argo), you may have to “pretend” to be a schedule decorator. You can definitely do it with an extension and create a new decorator (as long as its name is

schedule

it will override the built-in one) but you can also use

pre_mutate

on

FlowMutators

that can add a

schedule

decorator with your own custom

cron

string (since that is what Argo expects for example). I am not sure what exactly what you want to do but if it boils down to constructing a cron string from different arguments than the ones given in

schedule

, that may be the easiest way.

we're using it with stepfunctions. Essentially what i'm trying to do is create a schedule based on the deployment environment. So I think a `FlowMutator`might totally be valid. All I need is to modify the flow's state essentially based on some env variable configuration

you would be able to do that yes. something like:

class MyMutator(FlowMutator):
  def pre_mutate(self, mutable_flow):
    my_env_var = os.environ.get("...")
    # Do some fun stuff
    cron_string = ...
    mutable_flow.add_decorator(schedule, cron=cron_string, duplicates=mutable_flow.OVERRIDE)

or something like that.

Ok very cool! Thanks for the explanation! 🙂

You can then use is using:

@MyMutator
class MyFlow(FlowSpec):
...

You can also totally hide if from users by using either the BaseFlow pattern (and having your users derive from

YourBaseFlow

isntead of

FlowSpec

or by setting

DEFAULT_DECOSPECS

Will do 🙏

also this example might be relevant from the neighboring thread