SEARCH SESSIONS

Search All
 
Refine Results:
 
Year(s)

SOCIAL MEDIA

EMAIL SUBSCRIPTION

 
 

GTC ON-DEMAND

AI & Deep Learning Research
Presentation
Media
Scaling Machine Learning through Decentralization, Quantization, and Structured Sparsity
Abstract:
In this session, participants will get a taste of state-of-the-art techniques for scaling Deep Learning on GPU clusters. We present SuperML, a general and efficient communication layer for machine learning, which can scale neural network training to hundreds of GPU nodes. SuperML builds on three main ideas: decentralization, which allows algorithms to converge without a centralized coordinator (parameter server) or all-to-all communication, communication quantization, which significantly speeds up point-to-point messaging, and structured sparsity, by which SuperML induces model updates which only have a limited number of non-zero entries. From the technical perspective, SuperML provides a new implementation of the classic MPI standard, re-designed and re-implemented to provide efficient support for quantization and sparsity. We illustrate the performance characteristics of SuperML on CSCS Piz Daint, Europe's most powerful supercomputer, and on Amazon EC2, improving upon other highly optimized implementations such as CrayMPI and NVIDIA NCCL.
 
Topics:
AI & Deep Learning Research, Accelerated Data Science, HPC and Supercomputing
Type:
Talk
Event:
GTC Silicon Valley
Year:
2018
Session ID:
S8668
Streaming:
Download:
Share: