Much of the literature on optimal design of bandit algorithms is based on minimization of expected regret. It is well known that designs that are optimal over certain exponential families can achieve expected regret that grows logarithmically in the number of arm plays, at a rate governed by the Lai-Robbins lower bound. In this paper, we show that when one uses such optimized designs, the regret distribution of the associated algorithms necessarily has a very heavy tail, specifically, that of a truncated Cauchy distribution. Furthermore, for $p>1$, the $p$'th moment of the regret distribution grows much faster than poly-logarithmically, in particular as a power of the total number of arm plays. We show that optimized UCB bandit designs are also fragile in an additional sense, namely when the problem is even slightly mis-specified, the regret can grow much faster than the conventional theory suggests. Our arguments are based on standard change-of-measure ideas, and indicate that the most likely way that regret becomes larger than expected is when the optimal arm returns below-average rewards in the first few arm plays, thereby causing the algorithm to believe that the arm is sub-optimal. To alleviate the fragility issues exposed, we show that UCB algorithms can be modified so as to ensure a desired degree of robustness to mis-specification. In doing so, we also provide a sharp trade-off between the amount of UCB exploration and the tail exponent of the resulting regret distribution.

翻译：大量关于赌博机算法最优设计的文献基于期望遗憾最小化。众所周知，在特定指数族上最优的设计能够实现对数增长的期望遗憾（随臂拉动次数增长），其速率由赖-罗宾斯下界决定。本文证明，当使用此类优化设计时，相应算法的遗憾分布必然具有极重的尾部，具体表现为截断柯西分布。此外，对于$p>1$，遗憾分布的$p$阶矩增长速度远快于多对数增长，尤其达到总臂拉动次数的幂次增长。我们进一步展示，优化UCB赌博机设计在另一层面上同样脆弱：即使问题存在轻微误设，遗憾增长速度也可能远快于传统理论预测。我们的论证基于标准测度变换思想，表明遗憾超过预期的最可能情形是：最优臂在最初几次拉动中回报低于平均值，导致算法误判该臂为次优。为缓解所揭示的脆弱性问题，我们证明可通过修改UCB算法确保对误设具有期望的鲁棒性。在此过程中，我们给出了UCB探索量与遗憾分布尾部指数之间的精确权衡关系。