👋Hello, Since April 26th, 2024, I've been encountering an error while creating state machine for Metaflow orchestration. It looks like AWS has implemented a new syntax check validation for definitions prior to creating a state machine (detailed in the AWS changelog here). Specifically, this issue arises only when using the

--use-distributed-map

functionality. The error is related to how

item_reader

parses the JSON parameters. To work around this issue, I generated the JSON definition using the

--with retry step-functions create --use-distributed-map --only-json

, manually adjusted the JSON as needed, and then created it using the AWS CLI. Originally, the problematic line in Json was:

.parameter("Key.$", "$.Body.ResultFiles.SUCCEEDED.[0].Key")

and I changed it to:

.parameter("Key.$", "$.Body.ResultFiles.SUCCEEDED[0].Key")

This tweak allowed me to successfully create/update the state machine. Here’s where the below error seems to be coming from: step_functions.py.

Bootstrapping virtual environment(s) ...
Virtual environment(s) bootstrapped!
AWS Step Functions error:
InvalidDefinition("An error occurred (InvalidDefinition) when calling the CreateStateMachine operation: Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The value must be a valid Reference Path or a valid intrinsic function call at /States/review_videos_Map/ItemReader/Parameters/Key.$'")`

Could anyone help verify if this is a universal issue with the current AWS update or if there are better solutions? Thanks for your insights!