Standard Bayesian learning is known to have suboptimal generalization capabilities under model misspecification and in the presence of outliers. PAC-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data is affected by outliers. In recent work, PAC-Bayes bounds - referred to as PAC$^m$ - were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC$^m$ ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of model misspecification and outliers.

翻译：标准贝叶斯学习在模型误设定和存在异常值的情况下，已知具有次优的泛化能力。PAC-贝叶斯理论表明，贝叶斯学习所最小化的自由能准则是在采样分布未被异常值污染的假设下，对吉布斯预测器（即从后验中随机抽取的单一模型）泛化误差的一个上界。这一观点为贝叶斯学习在模型误设定（需进行集成）及数据受异常值影响时的局限性提供了理论依据。近期研究中，推导出了称为PAC$^m$的PAC-贝叶斯界限，用于引入考虑集成预测器性能的自由能度量，从而在误设定下获得增强性能。本文提出了一种新颖的鲁棒自由能准则，该准则将广义对数评分函数与PAC$^m$集成界限相结合。所提出的自由能训练准则能够产生预测分布，从而同时抵消模型误设定和异常值的有害影响。