Hi, having an issue whereby when I'm using Metaflow with batch (2000+ tasks) and reaching the AWS API limit as Metaflow is constantly trying to keep up to date with every task - is there a way to reduce the update rate/run it in a sort of detached mode? Has anyone run into this issue?

Hi! One way to avoid that would be to use step-functions integration but that would reduce the concurrency (a limitation of step-functions) to 50-60 batch jobs at a time.

Hi @square-wire-39606, @bulky-gpu-70936 and I work together so same issue, we need far far more running concurrently

@melodic-train-1526 does your issue relate to the following PR? https://github.com/Netflix/metaflow/pull/634

I see. It's very likely that at that scale you will run into this limit with many other fixes that we have been thinking of including the above PR. A full proof solution would be to integrate with AWS Batch array jobs to farm out your compute.

Going to try cloning the forked repo and seeing whether that PR will help!