Join Slack
Powered by
Hi I noticed <https://aws.amazon.com/blogs/aws/ste...
# ask-metaflow
p
plain-baker-2104
12/04/2022, 5:55 PM
Hi I noticed
https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/
removes step function parallel branch limits of 40 at time and 25000 events. Are there any plans to use this in metaflow? Currently we have to split a single run into 32 separate metaflow runs (1 orchestrator run, calling 1 preprocessing run followed by 30 parallel processing runs).
✅ 1
Open in Slack
Previous
Next