We investigate the extent to which offline demonstration data can improve online learning. It is natural to expect some improvement, but the question is how, and by how much? We show that the degree of improvement must depend on the quality of the demonstration data. To generate portable insights, we focus on Thompson sampling (TS) applied to a multi-armed bandit as a prototypical online learning algorithm and model. The demonstration data is generated by an expert with a given competence level, a notion we introduce. We propose an informed TS algorithm that utilizes the demonstration data in a coherent way through Bayes' rule and derive a prior-dependent Bayesian regret bound. This offers insight into how pretraining can greatly improve online performance and how the degree of improvement increases with the expert's competence level. We also develop a practical, approximate informed TS algorithm through Bayesian bootstrapping and show substantial empirical regret reduction through experiments.

翻译：我们研究了离线示范数据在何种程度上能够改善在线学习。人们自然期望有所改进，但问题在于如何改进以及改进幅度多大？我们证明，改进程度必然取决于示范数据的质量。为获得可迁移的见解，我们聚焦于多臂老虎机问题中的汤普森采样算法，将其作为典型在线学习算法与模型展开研究。示范数据由具备特定能力水平（本文提出的新概念）的专家生成。我们提出一种信息型汤普森采样算法，通过贝叶斯规则以连贯方式利用示范数据，并推导出依赖于先验分布的贝叶斯遗憾界。该结果揭示了预训练如何大幅提升在线性能，以及改进程度如何随专家能力水平增强而提升。我们还通过贝叶斯自助法开发了一种实用的近似信息型汤普森采样算法，并通过实验展示了显著的实证遗憾值降低。