In this paper, we address the stochastic contextual linear bandit problem, where a decision maker is provided a context (a random set of actions drawn from a distribution). The expected reward of each action is specified by the inner product of the action and an unknown parameter. The goal is to design an algorithm that learns to play as close as possible to the unknown optimal policy after a number of action plays. This problem is considered more challenging than the linear bandit problem, which can be viewed as a contextual bandit problem with a \emph{fixed} context. Surprisingly, in this paper, we show that the stochastic contextual problem can be solved as if it is a linear bandit problem. In particular, we establish a novel reduction framework that converts every stochastic contextual linear bandit instance to a linear bandit instance, when the context distribution is known. When the context distribution is unknown, we establish an algorithm that reduces the stochastic contextual instance to a sequence of linear bandit instances with small misspecifications and achieves nearly the same worst-case regret bound as the algorithm that solves the misspecified linear bandit instances. As a consequence, our results imply a $O(d\sqrt{T\log T})$ high-probability regret bound for contextual linear bandits, making progress in resolving an open problem in (Li et al., 2019), (Li et al., 2021). Our reduction framework opens up a new way to approach stochastic contextual linear bandit problems, and enables improved regret bounds in a number of instances including the batch setting, contextual bandits with misspecifications, contextual bandits with sparse unknown parameters, and contextual bandits with adversarial corruption.

翻译：本文研究随机上下文线性赌臂问题，其中决策者收到一个上下文（从分布中随机抽取的动作集合）。每个动作的期望回报由该动作与未知参数的内积指定。目标是在多次动作选择后，设计一种算法使其学习到的策略尽可能接近未知的最优策略。该问题被认为比线性赌臂问题更具挑战性，后者可视为具有固定上下文的上下文赌臂问题。令人惊讶的是，本文证明随机上下文问题可以像线性赌臂问题一样被解决。具体而言，当上下文分布已知时，我们建立了一个新颖的归约框架，将每个随机上下文线性赌臂实例转化为一个线性赌臂实例。当上下文分布未知时，我们提出一种算法，将随机上下文实例转化为一系列带有小误差的线性赌臂实例，并实现与解决带误差线性赌臂实例的算法几乎相同的最坏情况遗憾界。作为结果，我们的方法为上下文线性赌臂问题给出了 $O(d\sqrt{T\log T})$ 的高概率遗憾界，推动了(Li 等, 2019) 和 (Li 等, 2021) 中开放问题的解决。该归约框架为处理随机上下文线性赌臂问题开辟了新途径，并在多个场景中实现了更优的遗憾界，包括批量设置、带误差的上下文赌臂、稀疏未知参数的上下文赌臂，以及对抗性噪声下的上下文赌臂。