We present a general approach, based on exponential inequalities, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error as well as bounds on its tail probability, for both the PAC-Bayesian and single-draw scenarios. Specifically, for the case of sub-Gaussian loss functions, we obtain novel bounds that depend on the information density between the training data and the output hypothesis. When suitably weakened, these bounds recover many of the information-theoretic bounds available in the literature. We also extend the proposed exponential-inequality approach to the setting recently introduced by Steinke and Zakynthinou (2020), where the learning algorithm depends on a randomly selected subset of the available training data. For this setup, we present bounds for bounded loss functions in terms of the conditional information density between the output hypothesis and the random variable determining the subset choice, given all training data. Through our approach, we recover the average generalization bound presented by Steinke and Zakynthinou (2020) and extend it to the PAC-Bayesian and single-draw scenarios. For the single-draw scenario, we also obtain novel bounds in terms of the conditional $\alpha$-mutual information and the conditional maximal leakage.

翻译：我们提出一种基于指数不等式的通用方法，用于推导随机学习算法的泛化误差界。利用该方法，我们分别给出了PAC-贝叶斯和单次采样场景下平均泛化误差界及其尾部概率界。具体而言，对于亚高斯损失函数情形，我们获得了依赖于训练数据与输出假设之间信息密度的新边界。通过适当弱化，这些边界可恢复文献中现有的诸多信息论界。我们还将所提出的指数不等式方法扩展到Steinke与Zakynthinou（2020）最近引入的框架中——该框架下学习算法依赖于训练数据的一个随机选择子集。针对该设定，我们基于给定全部训练数据时输出假设与决定子集选择的随机变量之间的条件信息密度，给出了有界损失函数的边界。通过该方法，我们恢复了Steinke与Zakynthinou（2020）提出的平均泛化界，并将其推广至PAC-贝叶斯和单次采样场景。针对单次采样场景，我们还基于条件$\alpha$-互信息和条件最大泄漏获得了新边界。