Hello is it possible to schedule deploy to step functions a Outerbounds #ask-metaflow

Hello - is it possible to schedule/deploy to step...

gifted-musician-70702

01/31/2023, 5:41 PM

Hello - is it possible to schedule/deploy to step functions a flow and provide parameters via

... step-functions create --my-param hi

so that every-time the flow is executed based on

@schedule

it gets the same params that I have specified? I won't be using

step-functions trigger

because I am scheduling.

✅ 1

square-wire-39606

01/31/2023, 6:00 PM

not today - although you could read from a config file within deploy-time parameters to achieve a similar outcome.

gifted-musician-70702

01/31/2023, 6:35 PM

@square-wire-39606 OK - I have three sets of parameters per environment (dev, staging, prod) - is there a place I can read the environment and then load config using

IncludeFile

based on that value?

ancient-application-36103

01/31/2023, 6:52 PM

You could do that in the

start

step

✅ 1

ancient-application-36103

01/31/2023, 6:52 PM

or you could use

branch

in the context for deploy time parameters

gifted-musician-70702

01/31/2023, 6:53 PM

Oh and read the env from environment vars

ancient-application-36103

01/31/2023, 6:54 PM

yep

gifted-musician-70702

01/31/2023, 6:54 PM

I can specify env var upon step create?

ancient-application-36103

01/31/2023, 6:54 PM

something of the order of

python flow.py --branch test step-functions create

ancient-application-36103

01/31/2023, 6:55 PM

and then within deploy time parameters read the value from your config file that pertains to the value of

context.branch

gifted-musician-70702

01/31/2023, 7:20 PM

My setup is a little different. I have three metaflow deployments in three accounts corresponding to env. Is there a way to pass env variable in via cli command

python flow.py --env-vars {'env': 'dev'} step-functions create

type thing?

ancient-application-36103

01/31/2023, 7:21 PM

you can pass environment variables to metaflow task by using the

@environment

decorator, if that's what you are looking for

ancient-application-36103

01/31/2023, 7:22 PM

What is the end goal that you are trying to achieve?

ancient-application-36103

01/31/2023, 7:23 PM

happy to jump on a quick call!

gifted-musician-70702

01/31/2023, 7:23 PM

I have a flow that can be deployed and scheduled via step funcs to three separate metaflow deployments and each has different set of parameters (configuration)

ancient-application-36103

01/31/2023, 7:26 PM

https://docs.metaflow.org/production/coordinating-larger-metaflow-projects#custom-branches-for-production-deployment

bland-energy-98079

05/09/2023, 5:21 PM

@square-wire-39606 is this something that would be deployed in the near future, the ability to pass in parameters when creating step function with a schedule? It would be incredibly nice feature to have

ancient-application-36103

05/09/2023, 5:30 PM

@bland-energy-98079 the default values of the parameters are passed when creating the step function. you can rely on deploy time functions to tweak the values of parameters that are submitted

icy-psychiatrist-43391

08/07/2023, 12:22 PM

Another simple solution that works for me (and is compatible with my constraints) is to specify the parameters in the EventBridge rule.

2 Views

Open in Slack

Previous Next