https://outerbounds.com/ logo
Join Slack
Powered by
Hi there, Let me assume I have `20` tasks to run a...
# ask-metaflow
s
Hi there, Let me assume I have 
20
tasks to run and I set 
--max-workers = 10
. I add 
@batch
tag to run all tasks as separate 
AWS
Batch jobs. With that I assume 
metaflow
will launch 
10
EC2
instances and run the first 
10
jobs
on parallel. Meanwhile, the last 
10
batch jobs will be in the queue and if any of those first 10 jobs is finished, its corresponding running EC2 instance will be 
terminated
before a new EC2 instance will be launching to take one new job from the queue. Is it a correct understanding of what happens with metaflow in this setting? Another question, in the above setting, is it possible that 
metaflow
assigns 
10
batch jobs to 1 single EC2 machine instead and runs those 
10 batch jobs at the same time
on the same machine? That is something we barely observed but we think we did see them once, so I just wanted to ask here if this could be the case or something off with our code? Many thanks!
✅ 1
a
Hi @stale-scientist-29745! Depending on the resource requirements ascribed to the AWS Batch job - one or more jobs can be packed into a single EC2 instance. If you use large instances, it is likely that multiple jobs will execute on the same instance.
--max-workers=10
will ensure that only 10 jobs are active at any given time - they may or may not run on 10 EC2 instances depending on the resources required by each of them.
is it possible that 
metaflow
assigns 
10
batch jobs to 1 single EC2 machine instead and runs those 
10 batch jobs at the same time
on the same machine
Totally possible~
s
wow. I did not know that. Two following-up questions: • Could you point me to a place where I can find documents about resource requirements? • I assume we mostly want to run 10 jobs on 10 workers instead of 10 jobs on a single worker. How do you often avoid this issue?
a
resource requirements for a metaflow task are specified using 
@batch(cpu=...,memory=...,gpu=...)
- https://docs.metaflow.org/scaling/remote-tasks/introduction
If you want to run only one task on a single instance, you can set your resource requirements high enough to ensure that no other job can be placed on that instance.
s
by resource you meant RAM/CPU? How about GPU? The last question I promised. Plus how does metaflow know how much resource it needs for a single task?
a
yes - RAM/CPU/GPU.
s
But knowing how much those things a task need is tricky to know in advance, no?
a
you can ask as many questions as would like. it helps many other folks who may chance upon this thread when they search for similar issues
✅ 2
correct - knowing ahead of time can be tricky. we recommend starting out with some sensible largish defaults to get the workflow running successfully and then profiling it to identify areas of improvements to reduce the cloud spend.
this is an example of how you can access GPU utilization. depending on how you have configured your deployment - you might be able to leverage some observability tooling to get the needed insights. Outerbounds platform happens to come with this support out of the box.
s
You are the best @square-wire-39606! Thank you so much for the very useful answer!
thankyou 1
8 Views
Open in Slack
PreviousNext
Powered by