We study $K$-armed bandit problems where the reward distributions of the arms are all supported on the $[0,1]$ interval. It has been a challenge to design regret-efficient randomized exploration algorithms in this setting. Maillard sampling~\cite{maillard13apprentissage}, an attractive alternative to Thompson sampling, has recently been shown to achieve competitive regret guarantees in the sub-Gaussian reward setting~\cite{bian2022maillard} while maintaining closed-form action probabilities, which is useful for offline policy evaluation. In this work, we propose the Kullback-Leibler Maillard Sampling (KL-MS) algorithm, a natural extension of Maillard sampling for achieving KL-style gap-dependent regret bound. We show that KL-MS enjoys the asymptotic optimality when the rewards are Bernoulli and has a worst-case regret bound of the form $O(\sqrt{\mu^*(1-\mu^*) K T \ln K} + K \ln T)$, where $\mu^*$ is the expected reward of the optimal arm, and $T$ is the time horizon length.

翻译：我们研究当各臂的奖励分布均支持于 $[0,1]$ 区间时的 $K$ 臂老虎机问题。在此设定下，设计遗憾高效的随机探索算法一直是一个挑战。Maillard 采样~\cite{maillard13apprentissage} 作为汤普森采样的一个有吸引力的替代方案，最近被证明在次高斯奖励设定下能达到具有竞争力的遗憾保证~\cite{bian2022maillard}，同时保持闭合形式的动作概率，这对于离线策略评估非常有用。在本文中，我们提出了 Kullback-Leibler Maillard 采样（KL-MS）算法，这是 Maillard 采样的一种自然扩展，旨在实现 KL 风格的间隙相关遗憾界。我们证明，当奖励为伯努利分布时，KL-MS 具有渐近最优性，并且其最坏情况遗憾界形式为 $O(\sqrt{\mu^*(1-\mu^*) K T \ln K} + K \ln T)$，其中 $\mu^*$ 是最优臂的期望奖励，$T$ 是时间范围长度。