Recent work shows that when contexts are drawn i.i.d., linear contextual bandits can be reduced to single-context linear bandits. This ``contexts are cheap'' perspective is highly advantageous, as it allows for sharper finite-time analyses and leverages mature techniques from the linear bandit literature, such as those for misspecification and adversarial corruption. However, this reduction crucially relies on the independence of contexts and does not extend to settings with temporally correlated (e.g., Markovian) contexts, which arise frequently in practice. Motivated by applications with temporally correlated availability, we extend this perspective to linear bandits with Markovian context processes, where the action set evolves via an exogenous Markov chain. Our main contribution is a reduction that applies under uniform geometric ergodicity. We construct a stationary surrogate action set to solve the problem using a standard linear bandit oracle, employing a delayed-update scheme to control the bias induced by the nonstationary conditional context distributions. We further provide a phased algorithm for unknown stationary distributions that learns the surrogate mapping online. In both settings, we obtain a high-probability worst-case regret bound matching that of the underlying linear bandit oracle in sufficiently fast mixing regimes. We then validate our results on a real-world instance, where we show practical gains over a LinUCB baseline.

翻译：最近的研究表明，当上下文独立同分布（i.i.d.）时，线性上下文赌博机可以简化为单上下文线性赌博机。这种“上下文廉价”的视角极具优势，因为它允许进行更精确的有限时间分析，并利用线性赌博机文献中的成熟技术（例如针对模型误设和对抗性 corrupt 的技术）。然而，这种简化严重依赖于上下文的独立性，无法推广到具有时间相关性（例如马尔可夫性）上下文的环境中，而这类环境在实践中经常出现。受具有时间相关性可用性的应用启发，我们将这一视角扩展到具有马尔可夫性上下文过程的线性赌博机，其中动作集通过一个外生马尔可夫链演变。我们的主要贡献是在均匀几何遍历条件下给出了一种简化方法。我们构建了一个平稳的代理动作集，以便使用标准线性赌博机预言机来解决问题，并采用延迟更新方案来控制由非平稳条件上下文分布引起的偏差。此外，针对未知的平稳分布，我们提供了一种分阶段算法，该算法可在线学习代理映射。在这两种设置中，我们获得了高概率的最坏情况遗憾界，在足够快的混合速度下该界与底层线性赌博机预言机的界相匹配。然后，我们通过一个真实世界实例验证了结果，展示了相较于 LinUCB 基线的实际性能提升。