https://outerbounds.com/ logo
Join Slack
Powered by
Hi all, I had a question about monitoring GPU Uti...
# ask-metaflow
a
Hi all, I had a question about monitoring GPU Utilization, GPU Memory, CPU Utilization and Memory for an AWS Batch Job? I read in a different thread someone indicating not to use cloud-watch agents to do this because it will result in the EC2 instances not shutting down. That would have been my approach. What would be the best way to get these metrics into Cloud Watch? Thanks in advance
✅ 1
v
you are in luck! We have a fun solution to that question. Drop in gpu_profile.py next to your flow file, add 
@gpu_profile
to your GPU steps and you'll see cards like this:
💯 5
(it uses 
nvidia-smi
to measure utilization so you need that in the image, but that's usually installed by default in all CUDA-enabled images anyways)
a
This is awesome! I'm going to give this a try now. Thank you very much for this.
v
let us know how it goes! We'll actually publish a blog article tomorrow featuring this for the first time publicly, so great timing 🙂 If folks find it useful, we might include it in Metaflow by default 🤔
❤️ 4
a
Niiice! Yes - this will be super useful. I was going to use CloudWatch since that's what Sagemaker training jobs use, but this will do nicely. It is however missing other heuristics like CPU utilization and I want to see if my training job is using all the CPUs available in the instance because I'm trying to pinpoint why my AWS Batch Job is 1 minute slower than the equivalent Sagemaker training job that's running the same algorithm, same data, sample batch size, etc.
v
yep, we should add similar stats for CPU/RAM too!
a
Thanks again. Excited to give this a try!
among us party 1
f
@victorious-lawyer-58417 this is sweet! thanks for sharing ❤️ just threw this onto an inference pipeline and worked like a charm! 2 minor nits/feedback: • will fail if there's not an existing 
@card
decorator on the step • needs to be 
@gpu_profile()
rather than 
@gpu_profile
even when using defaults
among us party 2
v
great feedback, thanks! The first issue is surprising. Need to look into it and fix it
definitely let us know if there are other features that you'd find useful. You can take a look at the code too - it's not too complicated so PRs are welcome too 😉
🙌 1
a
The plots didn't generate for me. I got the following error:
Copy code
Key axes.edgecolor: #666 does not look like a color arg
v
interesting. Seems like an issue with the 
matplotlib
version.
at least 
"matplotlib": "3.7.0"
should work
I moved it to this nicer repo: https://github.com/outerbounds/metaflow-gpu-profile please open issues there so we'll know to fix them
a
Thank you Ville. You were right - switched to 
matplotlib: 3.5.3
, and works like a charm now.
🤗 1
Thanks again Ville for this nifty decorator. I've adapted it a bit and added CPU and RAM for more heuristics
Copy code
MONITOR = """
set -e;
while kill -0 {pid} 2>/dev/null;
do
  gpu=$(nvidia-smi \
    --query-gpu=pci.bus_id,timestamp,utilization.gpu,memory.used,memory.total \
    --format=csv,noheader,nounits | sed 's/,/, /g');
  cpu=$(top -n 1 -b | grep "Cpu(s)" | awk '{{printf("%.1f", $2+$4)}}');
  ram=$(free -m | awk 'NR==2{{printf("%d, %d", $3, $2)}}');
  echo "$gpu, $cpu, $ram";
  sleep {interval};
done
""".replace(
    "\n", " "
)
I'm trying to understand why Sagemaker Training is on average 1.5 minutes faster than AWS Batch at completing 1 epoch for one of our customer's models. Using your decorator, I generated the below graphs - (first two are AWS Batch, the last one is Sagemaker Training) Looks like CPU utilization for Sagemaker Training is a lot more steady than AWS Batch. Also looks like GPU utilization on AWS Batch oscillates a lot more than Sagemaker Training and dips to zero quite often. Any thoughts on this? Could this be what's causing the slight uptick in execution time for AWS Batch training jobs vs. Sagemaker Training?
🙌 1
v
this is super awesome, great job! 🤩
thankyou 1
how are the instances sizes on Batch vs. Sagemaker? Smaller instances would definitely exhibit more erratic behavior
a
The same instance types. Batch is using 
p2.xlarge
and Sagemaker is using 
ml.p2.xlarge
.
v
are there other tasks running concurrently? Other runs or tasks from branches? Noisy neighbors could cause variable performance what you described above
a
It's a simple flow 
start
-> 
train
-> 
end
. The only thing that the Metaflow flow is doing that Sagemaker isn't doing is redirecting 
tqdm
to logs. But that's it I believe.
v
one thing you could try is to run it on a p2.xlarge outside batch (just 
flow.py run
locally on an ec2 instance) if it still behaves erratically (unlikely), it could be something in the code. If it behaves fine, it’s something about your batch setup
(here’s one case study about typical multi tenant issues)
a
Thanks Ville. Good idea. I'll give that a try.
v
let me know how it goes. There shouldn’t be any fundamental reason for it behaving differently
f
I'd also double check they're doing the same underlying unit of work – from the shared screenshots it looks like the metaflow step execution lasted ~1.5hrs compared to sagemaker running for ~10hrs? The charts could also be deceiving depending on what resolution cloudwatch is collecting/displaying the data, e.g. if it's aggregating each 5min of metrics into a single data point on the shared graph, making it seem more consistently utilized than it really is Unrelated to all that, I'd also just call out that there's really no reason to be using a 
p2
instance nowadays. Those nvidia K80s came out in 2014 and are a really bad price/performance value. Compared to something like the 
g5
instances (the "value" oriented modern GPU instances) that use nvidia A10Gs, and cost effectively the same amount. Comparing p2.xl (K80) vs g5.xl (A10G): • K80 can't go beyond CUDA 10 (many new optimizations for transformers require CUDA 11+) • A10G has 24GB VRAM compared to 12GB, at 2x the speed (600GB/s GDDR6 vs 288GB/s GDDR5) • A10G has 4x number of tensor cores compared to the K80's older CUDA cores, making the A10G have ~16x higher Tflops in raw fp32 compute performance • If you care about power, the A10G does all that with ~40% less consumption • g5 has 10Gb/s networking compared to 1Gb/s • g5 has 20% faster CPU clock on-demand g5.xl is $1/hr, compared to ml.p2xl at $1.1/hr, so you'll spend less money to get significantly more performance 😛 The A10Gs are even within striking distance to the p3's V100 GPUs in terms of raw performance, and win by a large margin in terms of price/performance there too. Lastly, there's way more spot availability of A10Gs if you're able to take advantage of that. We have some chonky pipelines that can spin up spot clusters of 100s of those A10G GPUs and it's wild how much bang for your buck you can get out of them 🚀
thankyou 2
v
yep, great points! I made this chart a while back that clearly shows that 
g5
are pretty much the sweet spot currently in AWS (unless you need tons of VRAM)
❤️ 1
💯 1
p
instances come in handy if you need to do distributed training across GPUs, thanks to NVLink
f
where's my fav 
g5.48xl
with 8 A10Gs?!?!
v
it's there 🙂
🔥 1
it shows price per GPU, so larger instances get interleaved with smaller ones. The larger instances are relatively more expensive since they pack more oomph on the other dimensions - more CPU cores, more RAM, more networking, so while price per GPU may seem higher, they might well perform better overall
f
nice! yea I noticed the included single GPU/large RAM instance flavors like g5.16xl
v
right - so many dimensions to consider
f
minor nit - this looks like "Peak TFlops / GPU" rather than summed across # GPUs
v
that's right - good point, the legend could be clearer
the total TFlops across GPUs is a more complicated figure, because NVLink etc.
also a minor detail is that some of these instances don't exist, practically speaking - good luck finding a 
p4
currently 😛
f
lol yea those p4s are still being rationed last I checked, and are only available via services like AWS Batch, where they can better rotate availability (outside of reserve instances)
😿 1
they're fun to play with for sure, and yea outside of training IMO they don't make a ton of sense unless you need to shard your model across GPUs just to fit it into VRAM also fun just to watch how much power you can squeeze out of a single instance over time - 8 A100s on a single p4d.24xl and it’s performance is equivalent to ~24 V100s lol https://lambdalabs.com/gpu-benchmarks
v
oh yeah, my bet is that the models will get massively optimized while instances/GPUs keep getting more powerful
📈 1
a
Okay I ran a different example -- one that I could control the experiment a bit better given that the previous example was a customer model and I can't be 100% sure that the configuration between Sagemaker training and AWS Batch were exactly the same. This new example had a smaller delta in run-time, as you can see below, so that's good! Sagemaker Training was 9% faster than AWS Batch. Looks like average GPU utilization is mostly consistent between Sagemaker Training and AWS Batch. Although CPU utilization for Sagemaker Training is able to overclock above 100% which is intriguing...
5 Views
Open in Slack
PreviousNext
Powered by