[IDA ML Seminar] IDA ML Seminar - Hans Kersting: The Beneficial Role of Stochastic Noise in SGD, December 7, 15:15-16:15

[Sourabh Balgi]

Welcome to the last IDA Machine Learning Seminar of 2022 on December 7, 15:15-16:15.

Hans Kersting<https://hanskersting.github.io/>, SIERRA Team, INRIA and Ecole Normale Superieure, Paris

Title: The Beneficial Role of Stochastic Noise in SGD

Abstract: The data sets used to train modern machine-learning models are often huge, e.g., millions of images. This makes it too expensive to compute the true gradient over all data sets. In each gradient descent (GD) step, a stochastic gradient is thus computed over a subset ("mini-batch") of data. The resulting stochastic gradient descent (SGD) algorithm, and its variants, is the main workhorse of modern machine learning. Until recently, most machine-learning researchers would have preferred to use GD, if they could, and considered SGD only as a fast approximation to GD. But new research suggests that the stochasticity in SGD is part of the reason why SGD works so well. In this talk, we investigate multiple theories on the advantages of the noise in SGD, including better generalization in flatter minima ('implicit bias') and faster escapes from difficult parts of the landscapes (such as saddle points and local minima). We highlight how correlating noise can help optimization and zoom in on the question which noise structure would be optimal for SGD.

Related reading: Anticorrelated Noise Injection for Improved Generalization<https://proceedings.mlr.press/v162/orvieto22a.html> , Explicit Regularization in Overparametrized Models via Noise Injection<https://arxiv.org/pdf/2206.04613.pdf>

Location: Zoom (https://liu-se.zoom.us/j/69011766298) and also live-streamed at Ada Lovelace, House B, https://www.ida.liu.se/department/location/search.en.shtml?keyword=ada


The list of future seminars in the series is available at: http://www.ida.liu.se/research/machinelearning/seminars/.
You can subscribe to the seminar series' calendar using this ics link: https://outlook.office365.com/owa/calendar/4d811ae47ce446f58d11a7c2f50a7ed8@ad.liu.se/0f5253d7bc7841248c71eb4c28eb2d668927992292494627279/calendar.ics<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office365.com%2Fowa%2Fcalendar%2F4d811ae47ce446f58d11a7c2f50a7ed8%40ad.liu.se%2F0f5253d7bc7841248c71eb4c28eb2d668927992292494627279%2Fcalendar.ics&data=05%7C01%7Csourabh.balgi%40liu.se%7C8b9707f4b5794b4d191b08da8d088666%7C913f18ec7f264c5fa816784fe9a58edd%7C0%7C0%7C637977364986504271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4WpD%2BBo%2Bl5n4OKAUPrZD6QyPdMKrhhX0R5Snk7TsK%2FM%3D&reserved=0>

Welcome!
IDA Machine Learning Group
Linköping University
[https://cdn.imbox.io/tickets/1407/incoming/a04eb1441e9e3712aa5f5d6109fa4032/image001.png]
Department of Computer and Information Science
s-581 83 Linköping
Please visit us at liu.se<https://liu.se/>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.liu.se/pipermail/ml-seminars/attachments/20221130/5d331106/attachment.html>