stale-scientist-29745
06/29/2023, 6:58 PM20
tasks to run and I set --max-workers = 10
. I add @batch
tag to run all tasks as separate AWS
Batch jobs.
With that I assume metaflow
will launch 10
EC2
instances and run the first 10
jobs
on parallel. Meanwhile, the last 10
batch jobs will be in the queue and if any of those first 10 jobs is finished, its corresponding running EC2 instance will be terminated
before a new EC2 instance will be launching to take one new job from the queue. Is it a correct understanding of what happens with metaflow in this setting?
Another question, in the above setting, is it possible that metaflow
assigns 10
batch jobs to 1 single EC2 machine instead and runs those 10 batch jobs at the same time
on the same machine? That is something we barely observed but we think we did see them once, so I just wanted to ask here if this could be the case or something off with our code? Many thanks!