Hi all, Happy to share that we got Elastic Fabric Adapter integration working with Metaflow! This will allow us to spin up HPC Clusters on Batch using Metaflow!

@acoustic-van-30942 great job once again! Was wondering what benefits are there to using it? Does it provide a benefit if we only perform large scale distributed multi-node multi-gpu training jobs?

Hi Yudhiesh. Thanks so much! I've yet to run any benchmarking (because it would certainly be costly!), but it should be beneficial for large scale multi-node multi-gpu training jobs, especially ones that require 100's of nodes, where inter-node connectivity is the bottleneck. We are mainly interested in NCCL+EFA, which is only compatible with H100, A100, and V100 GPUs. But I think others have used EFA for mpi runs.

Thanks! Got it yeah, looking forward to seeing this! Might be beneficial as well for inferencing if the model is so big that it requires a multi-node multi-gpu setup.