Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias, in particular transitioning between kernel or feature learning.

翻译：尽管统计学习理论为理解监督学习提供了稳健的框架，但深度学习的许多理论方面仍不清晰，尤其是不同架构在使用基于梯度的方法训练时如何产生归纳偏置。本讲座旨在概述从学习理论角度理解深度学习时出现的主要问题。在简要回顾统计学习理论和随机优化后，我们讨论了良性过拟合背景下的隐式偏置。进而概述镜像下降算法，展示如何在给定学习问题的参数空间与对应函数空间之间来回转换，以及如何通过度量张量表示学习问题的几何结构。基于该框架，我们详细研究了线性对角网络在多种回归任务中梯度下降的隐式偏置，揭示了损失函数、初始化参数尺度及网络深度如何导致不同形式的隐式偏置，特别是核学习与特征学习之间的转换。