We study $K$-armed bandit problems where the reward distributions of the arms are all supported on the $[0,1]$ interval. It has been a challenge to design regret-efficient randomized exploration algorithms in this setting. Maillard sampling~\cite{maillard13apprentissage}, an attractive alternative to Thompson sampling, has recently been shown to achieve competitive regret guarantees in the sub-Gaussian reward setting~\cite{bian2022maillard} while maintaining closed-form action probabilities, which is useful for offline policy evaluation. In this work, we propose the Kullback-Leibler Maillard Sampling (KL-MS) algorithm, a natural extension of Maillard sampling for achieving KL-style gap-dependent regret bound. We show that KL-MS enjoys the asymptotic optimality when the rewards are Bernoulli and has a worst-case regret bound of the form $O(\sqrt{\mu^*(1-\mu^*) K T \ln K} + K \ln T)$, where $\mu^*$ is the expected reward of the optimal arm, and $T$ is the time horizon length.

翻译：我们研究臂的奖励分布均支持在[0,1]区间上的$K$臂赌博机问题。在此设定下，设计遗憾高效的随机探索算法一直是一个挑战。Maillard采样~\cite{maillard13apprentissage}作为汤普森采样的有吸引力的替代方案，近期已被证明在次高斯奖励设定中能实现具有竞争力的遗憾保证~\cite{bian2022maillard}，同时保持闭式动作概率，这对离线策略评估十分有用。本文中，我们提出Kullback-Leibler Maillard采样（KL-MS）算法，这是Maillard采样的自然扩展，旨在实现KL形式的间隙依赖遗憾界。我们证明，当奖励服从伯努利分布时，KL-MS具有渐近最优性，且其最坏情况遗憾界形如$O(\sqrt{\mu^*(1-\mu^*) K T \ln K} + K \ln T)$，其中$\mu^*$为最优臂的期望奖励，$T$为时间步长。