BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210402T160104Z
LOCATION:Track 4
DTSTART;TZID=America/New_York:20201119T130000
DTEND;TZID=America/New_York:20201119T133000
UID:submissions.supercomputing.org_SC20_sess178_pap216@linklings.com
SUMMARY:Convolutional Neural Network Training with Distributed K-FAC
DESCRIPTION:Paper\n\nConvolutional Neural Network Training with Distribute
 d K-FAC\n\nPauloski, Zhang, Huang, Xu, Foster\n\nTraining neural networks 
 with many processors can reduce time-to-solution; it is challenging, howev
 er, to maintain convergence and efficiency at large scales. The Kronecker-
 factored Approximate Curvature (K-FAC) was recently proposed as an approxi
 mation of the Fisher Information Matrix that can be used in natural gradie
 nt optimizers. We investigate here a scalable K-FAC design and its applica
 bility in convolutional neural network (CNN) training at scale. We study o
 ptimization techniques such as layer-wise distribution strategies, inverse
 -free second-order gradient evaluation, and dynamic K-FAC update decouplin
 g to reduce training time while preserving convergence. We use residual ne
 ural networks (ResNet) applied to the CIFAR-10 and ImageNet-1k datasets to
  evaluate the correctness and scalability of our K-FAC gradient preconditi
 oner. With ResNet-50 on the ImageNet-1k dataset, our distributed K-FAC imp
 lementation converges to the 75.9% MLPerf baseline in 18–25% less time tha
 n does the classic stochastic gradient descent (SGD) optimizer across scal
 es on a GPU cluster.\n\nTag: Data Analytics, Compression, and Management, 
 Linear Algebra, Machine Learning, Deep Learning and Artificial Intelligenc
 e\n\nRegistration Category: Tech Program Reg Pass
END:VEVENT
END:VCALENDAR