BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210402T160548Z
LOCATION:Track 3
DTSTART;TZID=America/New_York:20201112T164500
DTEND;TZID=America/New_York:20201112T171500
UID:submissions.supercomputing.org_SC20_sess209_ws_mlhpce107@linklings.com
SUMMARY:Deep Generative Models that Solve PDEs: Distributed Computing for 
 Training Large Data-Free Models
DESCRIPTION:Workshop\n\nDeep Generative Models that Solve PDEs: Distribute
 d Computing for Training Large Data-Free Models\n\nBotelho, Joshi, Khara, 
 Rao, Sarkar...\n\nRecent progress in scientific machine learning (SciML) h
 as opened up the possibility of training novel neural network architecture
 s that solve complex partial differential equations (PDEs). Several (nearl
 y data free) approaches have been recently reported that successfully solv
 e PDEs, with examples including deep feed forward networks, generative net
 works, and deep encoder-decoder networks. However, practical adoption of t
 hese approaches is limited by the difficulty in training these models, esp
 ecially to make predictions at large output resolutions (greater or equal 
 to 1024 x 1024). \n\nHere we report on a software framework for data paral
 lel distributed deep learning that resolves the twin challenges of trainin
 g these large SciML models – training in reasonable time as well as distri
 buting the storage requirements. Our framework provides several out of the
  box functionality including (a) loss integrity independent of number of p
 rocesses, (b) synchronized batch normalization, and (c) distributed higher
 -order optimization methods.\n\nWe show excellent scalability of this fram
 ework on both cloud as well as HPC clusters, and report on the interplay b
 etween bandwidth, network topology and bare metal vs cloud. We deploy this
  approach to train generative models of sizes hitherto not possible, showi
 ng that neural PDE solvers can be viably trained for practical application
 s. We also demonstrate that distributed higher-order optimization methods 
 are 2-3 times faster than stochastic gradient-based methods and provide mi
 nimal convergence drift with higher batch-size.\n\nRegistration Category: 
 Workshop Reg Pass
END:VEVENT
END:VCALENDAR