Hello all, I'm new to Metaflow and trying to understand the interaction between the recent

Config

and

Step Function

generation. The goal is to be able to change the Config for a Flow dynamically based on the input to the Step Function State Machine. I've noticed that when I define a

Config

on a flow, it ends up generating an environment configuration (

METAFLOW_FLOW_CONFIG_VALUE

) for the Step Function. I'm struggling to understand how the generated ,

kv.

-prefixed entry is meant to be used. If I try to invoke a

run

on the flow with

METAFLOW_FLOW_CONFIG_VALUE

in-lined, I run into

Could not load expected configuration values from the CONFIG_PARAMETERS file. This is a Metaflow bug. Please contact support.

I've tried tracing and debugging the code around this but as far as I can tell this is hardcoded based on

user_configs.config_parameters.CONFIG_FILE

. Would anyone be able to provide an example of how

Config

is meant to be used in relation to Step Functions? Thank you!

Configs are meant to be frozen at deploy time so it is not meant to be dynamic at runtime (you may be able to use Parameters for that). I am therefore not 100% sure what you mean about “dynamically based on the input to the Step Function”. If you mean that configs should change for each instance of the sfn graph then that is not what configs are for. You can, however, use configs to deploy multiple SFNs thta have slighlty different setups).

If you mean that configs should change for each instance of the sfn graph then that is not what configs are for

ah ok thank you for clarifying this. So the premise is that for a given SFN that gets deployed. The resources for a given SFN are expected to be statically declared at creation-time then?

yes. You have: • Config: statically locked at deploy time; can use in decorators and anywhere else • Parameters: locked in place when the run starts; cannot be used in decorators but can be used in the code. What we don’t have is something locked at run time that can be used in decorators (that is significantly harder 🙂 )

got it, so its not possible to have use a single SFN with variable resource configuration, for example, since resources have to be assigned via decorators

correct. Internally we have been exploring this but for SFN (and others), these values are basically written in the configuration so hard to change dynamically.

I see, so is the current state effectively that you have • 1 Config -> 1 SFN (both statically defined)

The use is trhough the CLI using

myflow.py --config <config-file> step-functions create

or

myflow.py --config-value <inline string of the config value> step-functions create

. Internally, when we then run on a remote machine, we pass a special value to config (the famous

kv.

which tells metaflow: “the value of this config is actually stored in some meta data file you have in your code package — go read from there).