We study the distribution of regret in stochastic multi-armed bandits and episodic reinforcement learning through a unified framework. We formalize a distributional regret bound as a probabilistic guarantee that holds uniformly over all confidence levels $δ\in (0,1]$, thereby characterizing the regret distribution across the full range of $δ$. We present a simple UCBVI-style algorithm with exploration bonus $\min\{c_{1,k}/N, c_{2,k}/\sqrt{N}\}$, where $N$ denotes the visit count and $(c_{1,k},c_{2,k})$ are user-specified parameters. For arbitrary parameter sequences, we derive general gap-independent and gap-dependent distributional regret bounds, yielding a principled characterization of how the parameters control the trade-off between expected performance, tail risk, and instance-dependent behavior. In particular, our bounds achieve optimal trade-offs between expected and distributional regret in both minimax and instance-dependent regimes. As a special case, for multi-armed bandits with $A$ arms and horizon $T$, we obtain a distributional regret bound of order $\mathcal{O}(\sqrt{AT}\log(1/δ))$, confirming the conjecture of Lattimore & Szepesvári (2020, Section 17.1) for the first time.

翻译：我们通过一个统一框架研究随机多臂老虎机和回合制强化学习中的遗憾分布。我们将分布遗憾界形式化为一个概率保证，该保证在所有置信水平$δ\in (0,1]$上一致成立，从而刻画了$δ$全范围内的遗憾分布。我们提出一种简单的UCBVI风格算法，其探索奖励为$\min\{c_{1,k}/N, c_{2,k}/\sqrt{N}\}$，其中$N$表示访问次数，$(c_{1,k},c_{2,k})$是用户指定的参数。对于任意参数序列，我们推导出通用的与间隙无关和与间隙相关的分布遗憾界，从而原则性地刻画了参数如何控制期望性能、尾部风险以及实例依赖行为之间的权衡。特别地，我们的界在极小极大和实例依赖两种机制下均实现了期望遗憾与分布遗憾之间的最优权衡。作为特例，对于有$A$个臂和时域$T$的多臂老虎机，我们得到了阶为$\mathcal{O}(\sqrt{AT}\log(1/δ))$的分布遗憾界，首次证实了Lattimore & Szepesvári（2020，第17.1节）的猜想。