Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias, in particular transitioning between kernel or feature learning.

翻译：尽管统计学习理论为理解监督学习提供了稳健的框架，但深度学习的许多理论方面仍不清晰，尤其是不同架构在使用基于梯度的方法训练时如何产生归纳偏差。本讲座旨在概述从学习理论角度理解深度学习时出现的主要问题。在简要回顾统计学习理论和随机优化后，我们讨论良性过拟合背景下的隐式偏差。随后转向镜像下降算法的一般描述，展示如何在给定学习问题的参数空间与对应函数空间之间来回转换，以及学习问题的几何结构如何通过度量张量表示。基于这一框架，我们详细研究线性对角网络在不同回归任务中梯度下降的隐式偏差，揭示损失函数、初始化参数尺度与网络深度如何导致多种形式的隐式偏差，特别是在核学习与特征学习之间的转换。