The multi-armed bandit (MAB) problem is a classical problem that models sequential decision-making under uncertainty in reinforcement learning. In this study, we propose a new generalized upper confidence bound (UCB) algorithm (GWA-UCB1) by extending UCB1, which is a representative algorithm for MAB problems, using generalized weighted averages, and present an effective algorithm for various problem settings. GWA-UCB1 is a two-parameter generalization of the balance between exploration and exploitation in UCB1 and can be implemented with a simple modification of the UCB1 formula. Therefore, this algorithm can be easily applied to UCB-based reinforcement learning models. In preliminary experiments, we investigated the optimal parameters of a simple generalized UCB1 (G-UCB1), prepared for comparison and GWA-UCB1, in a stochastic MAB problem with two arms. Subsequently, we confirmed the performance of the algorithms with the investigated parameters on stochastic MAB problems when arm reward probabilities were sampled from uniform or normal distributions and on survival MAB problems assuming more realistic situations. GWA-UCB1 outperformed G-UCB1, UCB1-Tuned, and Thompson sampling in most problem settings and can be useful in many situations. The code is available at https://github.com/manome/python-mab.

翻译：多臂老虎机（MAB）问题是强化学习中模拟序贯决策不确定性的经典问题。本研究通过广义加权平均扩展代表性MAB算法UCB1，提出新型广义上置信界算法（GWA-UCB1），并针对不同问题设定给出有效方案。GWA-UCB1对UCB1中探索与利用的平衡进行了双参数泛化，仅需对UCB1公式进行简单修改即可实现，因此可轻松应用于基于UCB的强化学习模型。在初步实验中，我们针对双臂随机MAB问题，分别研究了用于对比的简单广义UCB1（G-UCB1）和GWA-UCB1的最优参数。随后，在臂奖励概率服从均匀或正态分布的随机MAB问题中，以及更贴近现实场景的生存MAB问题中，验证了采用已研究参数的算法性能。GWA-UCB1在多数问题设定下优于G-UCB1、UCB1-Tuned和Thompson采样，可广泛应用于各类场景。代码详见https://github.com/manome/python-mab。