Hey, I have a question about the

@pytorch_parallel

, my compute environment for GPU training jobs currently only supports a

g5.8xlarge

meaning I can only perform single GPU training jobs. If I use the

@pytorch_parallel

I should be able to perform model training leveraging multiple of these instances just as in a

foreach

but in an unbounded manner? Would it be better to initially just use a

g5.24xlarge

which has 4 GPU’s as I would not need to leverage the

@pytorch_parallel

which is quite experimental? Also, there would be more overhead in I/O communication between the smaller

g5

instances compared to the larger instance. Are there any issues with the current implementation of the

@pytorch_parallel

and are there any examples that I can refer to?

Hey Yudhiesh, Instance type If you can access the

g5.24xlarge

instances and your model(s) of interest fit, that is how I would start. By this estimate, you can fit up to 12B param model into memory of each GPU on that instance, and could do distributed data parallel approach on models at least up to that size. There is more communication overhead to use batch multi-node with N smaller instances, relative to one big instance with M processors. Example Here is an example from the tests. @acoustic-van-30942 also has shared a more realistic example here thankyou Issues There is at least one known issue that @acoustic-van-30942 surfaced around getting logs (e.g., TQDM progress bar) to work nicely. It isn't a pytorch_parallel issue, but it is an annoying limitation for the long run times of the jobs that require pytorch_parallel. There is a similar issue with tensorflow_parallel - we didn't resolve core issue, but here is a workflow organization suggestion to mitigate the core of the problem. Alternatively, I'd suggest writing intermediate results to TensorBoard in S3. This thread may be handy if you use PyTorch Lightning environments. There were a few CUDA compatibility issues with newer PyTorch versions discussed and some conversation around the communication mechanism (NCCL vs. GLOO) in the multi-node case.

@flat-television-23413 great thanks for this! I will stick to single-GPU training for now and get that right then. I will also do some experiments and try it out with the MLflow integration as I already have that deployed.

of course, let us know if you hit any roadblocks and/or if you can share any learnings 🙂