In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of supervised machine learning algorithms is introduced. The method relies on the notion of \emph{gaps}, which characterize the variation of the expected empirical risk (when either the model or dataset is kept fixed) with respect to changes in the probability measure on the varying parameter (either the dataset or the model, respectively). This distinction results in two classes of gaps: Algorithm-driven gaps (fixed dataset) and data-driven gaps (fixed model). In general, the method relies on two central observations: $(i)$~The generalization error is the expectation of an algorithm-driven gap or a data-driven gap. In the first case, the expectation is with respect to a measure on the datasets; and in the second case, with respect to a measure on the models. $(ii)$~Both, algorithm-driven gaps and data-driven gaps exhibit closed-form expressions in terms of relative entropies. In particular, algorithm-driven gaps involve a Gibbs probability measure on the set of models, which represents a supervised Gibbs algorithm. Alternatively, data-driven gaps involve a worst-case data-generating (WCDG) probability measure on the set of data points, which is also a Gibbs probability measure. Interestingly, such Gibbs measures, which are exogenous to the analysis of generalization, place both the supervised Gibbs algorithm and the WCDG probability measure as natural references for the analysis of supervised learning algorithms. All existing exact expressions for the generalization error of supervised machine learning algorithms can be obtained with the proposed method. Also, this method allows obtaining numerous new exact expressions, which allows establishing connections with other areas in statistics.

翻译：本文引入了间隙方法，这是一种为监督机器学习算法的泛化误差推导基于信息度量的闭式表达式的技术。该方法依赖于“间隙”的概念，该概念刻画了期望经验风险（当模型或数据集之一固定时）相对于变化参数（分别为数据集或模型）的概率测度变化的变动。这一区分产生了两类间隙：算法驱动间隙（固定数据集）和数据驱动间隙（固定模型）。总体而言，该方法依赖于两个核心观察：$(i)$ 泛化误差是算法驱动间隙或数据驱动间隙的期望。在第一种情况下，期望是关于数据集上的测度；在第二种情况下，是关于模型上的测度。$(ii)$ 算法驱动间隙和数据驱动间隙均展现出基于相对熵的闭式表达式。具体而言，算法驱动间隙涉及模型集合上的吉布斯概率测度，该测度代表了一个监督吉布斯算法。或者，数据驱动间隙涉及数据点集合上的最坏情况数据生成（WCDG）概率测度，该测度也是一个吉布斯概率测度。有趣的是，这些在泛化分析中作为外生变量的吉布斯测度，使得监督吉布斯算法和WCDG概率测度成为监督学习算法分析的自然参照。所有现有的监督机器学习算法泛化误差的精确表达式均可通过所提方法获得。此外，该方法允许获得众多新的精确表达式，从而能够建立与统计学其他领域的联系。