We present a unified framework for deriving PAC-Bayesian generalization bounds. Unlike most previous literature on this topic, our bounds are anytime-valid (i.e., time-uniform), meaning that they hold at all stopping times, not only for a fixed sample size. Our approach combines four tools in the following order: (a) nonnegative supermartingales or reverse submartingales, (b) the method of mixtures, (c) the Donsker-Varadhan formula (or other convex duality principles), and (d) Ville's inequality. Our main result is a PAC-Bayes theorem which holds for a wide class of discrete stochastic processes. We show how this result implies time-uniform versions of well-known classical PAC-Bayes bounds, such as those of Seeger, McAllester, Maurer, and Catoni, in addition to many recent bounds. We also present several novel bounds. Our framework also enables us to relax traditional assumptions; in particular, we consider nonstationary loss functions and non-i.i.d. data. In sum, we unify the derivation of past bounds and ease the search for future bounds: one may simply check if our supermartingale or submartingale conditions are met and, if so, be guaranteed a (time-uniform) PAC-Bayes bound.

翻译：我们提出了一个统一框架，用于推导PAC-贝叶斯泛化界。与以往大多数相关文献不同，本文中的界具有任意时刻有效性（即时间一致性），这意味着它们适用于所有停止时间，而不仅限于固定样本量。该方法依次结合了四种工具：（a）非负上鞅或逆下鞅，（b）混合方法，（c）唐斯克-瓦拉德汉公式（或其他凸对偶原理），以及（d）维莱不等式。主要结果是一个适用于广泛离散随机过程的PAC-贝叶斯定理。我们展示了该结果如何推导出经典PAC-贝叶斯界（如Seeger、McAllester、Maurer和Catoni的界）以及许多近期界的时间一致性版本，同时提出了若干新界。该框架还放宽了传统假设，特别考虑了非平稳损失函数和非独立同分布数据。总之，我们统一了以往界的推导过程，并简化了未来界的探索：只需验证上鞅或下鞅条件是否满足，即可获得（时间一致的）PAC-贝叶斯界。