SEARCH SESSIONS

Search All
 
Refine Results:
 
Year(s)

SOCIAL MEDIA

EMAIL SUBSCRIPTION

 
 

GTC On-Demand

Tag Filter(s): AI and DL Research
Presentation
Media
Training Neural Networks with Mixed Precision: Theory and Practice
Paulius Micikevicius (NVIDIA)
We'll cover the theory and practice for training DNNs with Tensor Cores, introduced for AI processing with the Volta GPU architecture. Tensor Cores provide up to 120 TFlops throughput, mixing operations on IEEE half- and single-precision floats. In ...Read More
We'll cover the theory and practice for training DNNs with Tensor Cores, introduced for AI processing with the Volta GPU architecture. Tensor Cores provide up to 120 TFlops throughput, mixing operations on IEEE half- and single-precision floats. In the theory portion of the talk, we'll review the half-precision format, values that arise in DNN computations, and techniques that maximize utilization of fp16 format by these values. Techniques include loss-scaling, master weights, and choosing the proper precision for a given operation. In the practice portion of this talk, we'll survey various models that have been trained in mixed precision, matching the accuracy of fp32 training sessions while using the same hyperparameters. Models include various architectures (feed forward, recurrent, generative) as well as cover diverse tasks (image, speech, and language processing). We'll also provide network design and training guidelines to maximize speed when using Tensor Cores.  Back
 
Keywords:
AI and DL Research, Algorithms and Numerical Techniques, GTC Silicon Valley 2018 - ID S8923
Streaming:
Download: