Skip to content

Experiments to accelerate GPU device for PyTorch training

License

Notifications You must be signed in to change notification settings

hinofafa/torch_accelerator

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

torch_accelerator

Experiments to boost GPU device to PyTorch training

Todo

  1. Automatic Mixed Precision FP16
  2. Activate Tensor Core
  3. Distribution of params on Tensorboard

Reference

  1. Tensor Core https://www.nvidia.com/en-us/data-center/tensor-cores/

  2. Turing Structure https://www.nvidia.com/en-us/design-visualization/technologies/turing-architecture/

  3. Tips for optimization Blog https://developer.nvidia.com/blog/optimizing-gpu-performance-tensor-cores/

  4. Memory limited layer https://docs.nvidia.com/deeplearning/performance/dl-performance-memory-limited/index.html

  5. Automatic Mixed Precision for Deep Learning https://developer.nvidia.com/automatic-mixed-precision

  6. Mixed Precision Training https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html

  7. Introducing Native Pytorch amp for faster training on nvidia gpus https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/

  8. Training Neural Networks with Tensor Cores PPT https://nvlabs.github.io/eccv2020-mixed-precision-tutorial/files/dusan_stosic-training-neural-networks-with-tensor-cores.pdf

  9. Nvidia Developer Blog https://developer.nvidia.com/deep-learning

  10. Nvidia Deep Learning Examples https://developer.nvidia.com/deep-learning-examples

  11. Using Nsight Compute or Nvprof to show mixed precision use in deep learning https://developer.nvidia.com/blog/using-nsight-compute-nvprof-mixed-precision-deep-learning-models/

  12. Cuda Pro Tip: nvprof is your handy universal gpu profiler https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/

  13. PyTorch tutorial Automatic Mixed Precision https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html

Releases

No releases published

Packages

No packages published