We consider the problem of estimating the mean of a sequence of random elements $f(X_1, \theta)$ $, \ldots, $ $f(X_n, \theta)$ where $f$ is a fixed scalar function, $S=(X_1, \ldots, X_n)$ are independent random variables, and $\theta$ is a possibly $S$-dependent parameter. An example of such a problem would be to estimate the generalization error of a neural network trained on $n$ examples where $f$ is a loss function. Classically, this problem is approached through concentration inequalities holding uniformly over compact parameter sets of functions $f$, for example as in Rademacher or VC type analysis. However, in many problems, such inequalities often yield numerically vacuous estimates. Recently, the \emph{PAC-Bayes} framework has been proposed as a better alternative for this class of problems for its ability to often give numerically non-vacuous bounds. In this paper, we show that we can do even better: we show how to refine the proof strategy of the PAC-Bayes bounds and achieve \emph{even tighter} guarantees. Our approach is based on the \emph{coin-betting} framework that derives the numerically tightest known time-uniform concentration inequalities from the regret guarantees of online gambling algorithms. In particular, we derive the first PAC-Bayes concentration inequality based on the coin-betting approach that holds simultaneously for all sample sizes. We demonstrate its tightness showing that by \emph{relaxing} it we obtain a number of previous results in a closed form including Bernoulli-KL and empirical Bernstein inequalities. Finally, we propose an efficient algorithm to numerically calculate confidence sequences from our bound, which often generates nonvacuous confidence bounds even with one sample, unlike the state-of-the-art PAC-Bayes bounds.

翻译：我们考虑估计随机元素序列 $f(X_1, \theta)$ , ..., $f(X_n, \theta)$ 均值的问题，其中 $f$ 是固定标量函数，$S=(X_1, \ldots, X_n)$ 是独立随机变量，$\theta$ 是可能依赖于 $S$ 的参数。此类问题的一个例子是估计在 $n$ 个样本上训练的神经网络的泛化误差，其中 $f$ 为损失函数。经典方法通过均匀适用于紧参数集（例如Rademacher型或VC型分析）的浓度不等式处理该问题。然而，在许多问题中，此类不等式往往产生数值上无效的估计。近年来，PAC-Bayes框架因其常能给出数值上非平凡界的优势而被视为此类问题的更优替代方案。本文中，我们展示了如何进一步优化：通过改进PAC-Bayes界的证明策略，能够获得**更紧**的保证。我们的方法基于**抛硬币投注**框架，该框架利用在线赌博算法的遗憾保证推导出已知数值最紧的时间一致浓度不等式。具体而言，我们首次推导出基于抛硬币投注方法的PAC-Bayes浓度不等式，该不等式对所有样本容量同时成立。通过**松弛**该不等式，我们以闭合形式获得了包括Bernoulli-KL和经验Bernstein不等式在内的多项已有结果，从而证明了其紧性。最后，我们提出一种高效算法，用于从我们的界中数值计算置信序列——与现有最优PAC-Bayes界不同，该算法即使在仅有一个样本的情况下也能生成非平凡的置信界。