Hi there, Let me assume I have `20` tasks to run a...
# ask-metaflow
s
Hi there, Let me assume I have
20
tasks to run and I set
--max-workers = 10
. I add
@batch
tag to run all tasks as separate
AWS
Batch jobs. With that I assume
metaflow
will launch
10
EC2
instances and run the first
10
jobs
on parallel. Meanwhile, the last
10
batch jobs will be in the queue and if any of those first 10 jobs is finished, its corresponding running EC2 instance will be
terminated
before a new EC2 instance will be launching to take one new job from the queue. Is it a correct understanding of what happens with metaflow in this setting? Another question, in the above setting, is it possible that
metaflow
assigns
10
batch jobs to 1 single EC2 machine instead and runs those
10 batch jobs at the same time
on the same machine? That is something we barely observed but we think we did see them once, so I just wanted to ask here if this could be the case or something off with our code? Many thanks!
1