Hello Team, I'm working on deploying a metaflow flow using AWS Step Functions. The flow is already scheduled and running successfully. I want to deploy the exact same flow twice with: • Different input parameters (

alpha = 0

or

alpha = 1

) • Different project names, based on

alpha

(e.g.,

my_flow_0

and

my_flow_1

) • Different schedules based on the

alpha

value The goal is to make the deployment dynamic, where the

@project

name and

@schedule

are automatically set based on

alpha

(and also to avoid code duplication). I have tried to use

config_expr

utility as it is useful to make flows configurable, but I can't yet create some custom

@project

and

@schedule

decorators that solve this challenge. Any idea about how to solve that challenge ?

you can now configure

@project

and

@schedule

through a config file. Did you get it working? creating a custom

@project

/

@schedule

is possible too. I can show you examples if the simple config approach didn't work

So yes I understand how to configure

@project

and

@schedule

using a config file, when the config is simple :

@schedule(cron=config_expr('config.schedule_cron.alpha_0'))
@project(name=config_expr('config.project_name_prefix'))
class MyFlow(FlowSpec):
    config = Config("config", default="./config/config_test.yaml", parser=yaml.safe_load)

    @step
    def start(self):
        <http://logging.info|logging.info>('Start VanillaFlow')
        self.next(self.step1)

    @step
    def step1(self):
        <http://logging.info|logging.info>('Step 1')
        self.next(self.end)

    @step
    def end(self):
        <http://logging.info|logging.info>(f"End")

if __name__ == '__main__':
    MyFlow()

where config_test.yaml looks like:

alpha: 'alpha_0'
project_name_prefix: 'project_'
schedule_cron:
  alpha_0: '0 12 ? * FRI *'
  alpha_1: '0 18 ? * MON *'

Next step for me is to have a dynamic input in

@schedule

and

@project

, based on

alpha

value. if alpha = 'alpha_0', then : • get the corresponding cron :

config.schedule_cron.alpha_0

• create a project_name like

f"{config.project_name_prefix}_{config.alpha}"

Same logic if alpha = 'alpha_1'. Here is my

custom_schedule

decorator:

def custom_schedule():
    def decorator(f):
        alpha = config_expr('config.alpha')
        cron_key = f'config.schedule_cron.{alpha}'
        return schedule(cron=config_expr(cron_key))(f)
    return decorator

But I got that a syntax error :

config.schedule_cron.<metaflow.user_configs.config_parameters.DelayEvaluator object at 0x106a178d0>

because maybe

config_expr

treats the entire expression as a literal string. So to conclude that would be the dynamic configuration that I want :

alpha = config_expr('config.alpha')
cron_expr = config_expr(f'config.schedule_cron.{alpha}')
project_name = config_expr(f'{config_expr("config.project_name_prefix")}_{alpha}')

@schedule(cron=cron_expr)
@project(name=project_name)
class MyFlow(FlowSpec):
    config = Config("config", default="./config/config_test.yaml", parser=yaml.safe_load)

I have also had a look at that repo that gives config examples, but I haven't been able to find a solution. https://github.com/outerbounds/config-examples any suggestions ?