stale-scientist-29745
06/29/2023, 6:58 PM20 tasks to run and I set --max-workers = 10. I add @batch tag to run all tasks as separate AWS Batch jobs.
With that I assume metaflow will launch 10 EC2 instances and run the first 10 jobs on parallel. Meanwhile, the last 10 batch jobs will be in the queue and if any of those first 10 jobs is finished, its corresponding running EC2 instance will be terminated before a new EC2 instance will be launching to take one new job from the queue. Is it a correct understanding of what happens with metaflow in this setting?
Another question, in the above setting, is it possible that metaflow assigns 10 batch jobs to 1 single EC2 machine instead and runs those 10 batch jobs at the same time on the same machine? That is something we barely observed but we think we did see them once, so I just wanted to ask here if this could be the case or something off with our code? Many thanks!