<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Welcome to the last IDA Machine Learning Seminar of 2022 on <b>
December 7, 15:15-16:15.</b><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:black"><a href="https://hanskersting.github.io/"><span style="color:#225785;text-decoration:none">Hans Kersting</span></a>, SIERRA Team, INRIA and Ecole Normale Superieure,
Paris<br>
</span></b> <o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:black">Title: The Beneficial Role of Stochastic Noise in SGD<o:p></o:p></span></b></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><i>Abstract: The data sets used to train modern machine-learning models are often huge, e.g., millions of images. This makes it too expensive to compute the true gradient over all data sets. In each gradient descent (GD) step, a stochastic
gradient is thus computed over a subset ("mini-batch") of data. The resulting stochastic gradient descent (SGD) algorithm, and its variants, is the main workhorse of modern machine learning. Until recently, most machine-learning researchers would have preferred
to use GD, if they could, and considered SGD only as a fast approximation to GD. But new research suggests that the stochasticity in SGD is part of the reason why SGD works so well. In this talk, we investigate multiple theories on the advantages of the noise
in SGD, including better generalization in flatter minima ('implicit bias') and faster escapes from difficult parts of the landscapes (such as saddle points and local minima). We highlight how correlating noise can help optimization and zoom in on the question
which noise structure would be optimal for SGD.<o:p></o:p></i></p>
<p class="MsoNormal"><i><o:p> </o:p></i></p>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:black;background:#E8E8EE">Related reading: </span><a href="https://proceedings.mlr.press/v162/orvieto22a.html"><b><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:#225785;text-decoration:none">Anticorrelated
Noise Injection for Improved Generalization</span></b></a><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:black;background:#E8E8EE"> , </span><a href="https://arxiv.org/pdf/2206.04613.pdf"><b><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:#225785;text-decoration:none">Explicit
Regularization in Overparametrized Models via Noise Injection</span></b></a><i><o:p></o:p></i></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><b>Location:</b> Zoom (<a href="https://liu-se.zoom.us/j/69011766298">https://liu-se.zoom.us/j/69011766298</a>) and also live-streamed at Ada Lovelace, House B,
<a href="https://www.ida.liu.se/department/location/search.en.shtml?keyword=ada" target="_BLANK">
https://www.ida.liu.se/department/location/search.en.shtml?keyword=ada</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The list of future seminars in the series is available at: <span lang="SV">
<a href="http://www.ida.liu.se/research/machinelearning/seminars/" target="_BLANK"><span lang="EN-US">http://www.ida.liu.se/research/machinelearning/seminars/</span></a></span>.<o:p></o:p></p>
<p class="MsoNormal">You can subscribe to the seminar series' calendar using this ics link:
<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office365.com%2Fowa%2Fcalendar%2F4d811ae47ce446f58d11a7c2f50a7ed8%40ad.liu.se%2F0f5253d7bc7841248c71eb4c28eb2d668927992292494627279%2Fcalendar.ics&data=05%7C01%7Csourabh.balgi%40liu.se%7C8b9707f4b5794b4d191b08da8d088666%7C913f18ec7f264c5fa816784fe9a58edd%7C0%7C0%7C637977364986504271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4WpD%2BBo%2Bl5n4OKAUPrZD6QyPdMKrhhX0R5Snk7TsK%2FM%3D&reserved=0" target="_BLANK">
https://outlook.office365.com/owa/calendar/4d811ae47ce446f58d11a7c2f50a7ed8@ad.liu.se/0f5253d7bc7841248c71eb4c28eb2d668927992292494627279/calendar.ics</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Welcome!<o:p></o:p></p>
<p class="MsoNormal">IDA Machine Learning Group<o:p></o:p></p>
<p class="MsoNormal">Linköping University<o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="3" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="100%" style="width:100.0%;border:none;border-bottom:solid black 1.0pt;padding:0cm 0cm 3.75pt 3.75pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><img border="0" width="170" height="44" style="width:1.7708in;height:.4583in" id="_x0000_i1025" src="https://cdn.imbox.io/tickets/1407/incoming/a04eb1441e9e3712aa5f5d6109fa4032/image001.png"><o:p></o:p></p>
</td>
</tr>
<tr>
<td width="100%" style="width:100.0%;padding:3.75pt 3.75pt 3.75pt 3.75pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif">Department of Computer and Information Science</span></b><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
s-581 83 Linköping<br>
Please visit us at </span><a href="https://liu.se/" title="liu.se"><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">liu.se</span></a><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>