We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online, continual learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled methods and guidance on when practitioners should prefer one approach over the other.

翻译：我们重新审视贝叶斯集成这一经典问题，致力于解决在线持续学习场景中贝叶斯模型最优组合的学习挑战。为此，我们通过新颖的经验贝叶斯视角重新阐释了贝叶斯模型平均（BMA）和贝叶斯堆叠等现有方法，从而揭示了BMA的局限性与缺陷。进一步受到在线优化理论的启发，我们提出了在线贝叶斯堆叠（OBS）方法，该方法通过优化预测分布的对数评分来自适应地组合贝叶斯模型。本工作的核心贡献在于建立了OBS与投资组合选择之间的新联系，将贝叶斯集成学习与一个经过深入研究的丰富理论框架相衔接，该框架提供了高效算法和完备的遗憾分析。我们进一步厘清了OBS与在线BMA之间的关系，证明二者优化的是相关但不同的损失函数。通过理论分析与实证评估，我们明确了OBS优于在线BMA的应用场景，并为实践者提供了关于何时应优先选择某种方法的原则性指导和依据。