The multi-armed bandit (MAB) model is one of the most classical models to study decision-making in an uncertain environment. In this model, a player chooses one of $K$ possible arms of a bandit machine to play at each time step, where the corresponding arm returns a random reward to the player, potentially from a specific unknown distribution. The target of the player is to collect as many rewards as possible during the process. Despite its simplicity, the MAB model offers an excellent playground for studying the trade-off between exploration versus exploitation and designing effective algorithms for sequential decision-making under uncertainty. Although many asymptotically optimal algorithms have been established, the finite-time behaviors of the stochastic dynamics of the MAB model appear much more challenging to analyze, due to the intertwine between the decision-making and the rewards being collected. In this paper, we employ techniques in statistical physics to analyze the MAB model, which facilitates the characterization of the distribution of cumulative regrets at a finite short time, the central quantity of interest in an MAB algorithm, as well as the intricate dynamical behaviors of the model. Our analytical results, in good agreement with simulations, point to the emergence of an interesting multimodal regret distribution, with large regrets resulting from excess exploitation of sub-optimal arms due to an initial unlucky output from the optimal one.

翻译：多臂赌博机（MAB）模型是研究不确定环境下决策问题的最经典模型之一。在该模型中，玩家在每个时间步从$K$个可能的赌博机臂中选择一个进行游戏，所选臂会从某个未知分布中返回随机奖励。玩家的目标是在整个过程中收集尽可能多的奖励。尽管模型简单，MAB为研究探索与利用的权衡以及设计不确定性下序列决策的有效算法提供了绝佳平台。虽然已有大量渐近最优算法被建立，但由于决策过程与奖励收集之间的相互耦合，MAB模型随机动力学的有限时间行为分析难度显著增加。本文采用统计物理学方法分析MAB模型，这不仅有助于刻画MAB算法核心量——累积遗憾在有限短时间内的分布特征，还能揭示模型复杂的动力学行为。我们的解析结果与模拟高度吻合，揭示了有趣的多峰遗憾分布现象：当最优臂在初始阶段产生不利输出时，对次优臂的过度利用会导致巨大遗憾。