DistributedDataParallel

[Clean1224]

First of all, I would like to thank you for your code. I have noticed that you have implemented multi-GPU training using DataParallel in the PyTorch framework. However, using DataParallel can lead to low GPU utilization. I was wondering if you could provide a tutorial on how to use DistributedDataParallel for parallel training？

[adalca]

I'll ping @balakg and @ahoopes who know a lot more about the pytorch vxm implementation than I do. Don't know enough details about pytorch parallelism to comment meaningfully.