We consider nonstationary multi-armed bandit problems where the model parameters of the arms change over time. We introduce the adaptive resetting bandit (ADR-bandit), a bandit algorithm class that leverages adaptive windowing techniques from literature on data streams. We first provide new guarantees on the quality of estimators resulting from adaptive windowing techniques, which are of independent interest. Furthermore, we conduct a finite-time analysis of ADR-bandit in two typical environments: an abrupt environment where changes occur instantaneously and a gradual environment where changes occur progressively. We demonstrate that ADR-bandit has nearly optimal performance when abrupt or gradual changes occur in a coordinated manner that we call global changes. We demonstrate that forced exploration is unnecessary when we assume such global changes. Unlike the existing nonstationary bandit algorithms, ADR-bandit has optimal performance in stationary environments as well as nonstationary environments with global changes. Our experiments show that the proposed algorithms outperform the existing approaches in synthetic and real-world environments.

翻译：我们研究了臂的模型参数随时间变化的非平稳多臂赌博机问题。引入自适应重置赌博机（ADR-bandit），一种利用数据流文献中自适应窗口技术的赌博机算法类。首先，我们为自适应窗口技术所得估计量的质量提供了新保证，这些保证本身具有独立研究价值。进一步地，我们在两种典型环境中对ADR-bandit进行有限时间分析：突变环境（变化瞬时发生）和渐变环境（变化逐步发生）。我们证明，当突变或渐变以协调方式（称为全局变化）发生时，ADR-bandit具有近似最优性能。同时表明，在假设全局变化的情况下无需强制探索。与现有非平稳赌博机算法不同，ADR-bandit在平稳环境和具有全局变化的非平稳环境中均达到最优性能。实验表明，所提算法在合成与真实环境中均优于现有方法。