Hello Team I am trying to trigger an AWS

StepFunction

with a different docker image then the one that was defined at deployment. I am wondering if this is possible? (more context in the 🧵)

I have a docker image that is defined in the metaflow config file under

"METAFLOW_BATCH_CONTAINER_IMAGE": DEPLOY_IMAGE

then I run

python my_workflow.py step-functions create

. This successfully deploys the AWS

StepFunction

and when I run it it runs on the

DEPLOY_IMAGE

. I tried running

python my_workflow --with batch:image=RUNTIME_IMAGE stepfunctions trigger

to see if I could override the image that was used at deploy time. The job ran but completely ignored the new image. Is there a way to override the docker image that is defined in the config file when we trigger an previously deployed workflow?

you would have to redeploy

So there is no way to override the image at runtime?

the image is embedded in the definition of the state machine

Are there any established patterns for specifying the image to use at runtime without calling

step-functions create

?

The background on this is

step-functions create

requires privileged credentials in our broader AWS environment and we'd like to have to deploy a generic flow once (or rarely) in a CI/CD service. We're hoping we can parameterize the container image (somehow) at runtime without having to redeploy.

I know, for example, in AWS StepFunctions you can reference input variables inside the stepfunction definition. I'm wondering if there's some way we can have a static stepfunction definition that defines the image such that it reads from that input variable dynamically