We introduce a latency-aware contextual bandit framework that generalizes the standard contextual bandit problem, where the learner adaptively selects arms and switches decision sets under action delays. In this setting, the learner observes the context and may select multiple arms from a decision set, with the total time determined by the selected subset. The problem can be framed as a special case of semi-Markov decision processes (SMDPs), where contexts and latencies are drawn from an unknown distribution. Leveraging the Bellman optimality equation, we design the contextual online arm filtering (COAF) algorithm, which balances exploration, exploitation, and action latency to minimize regret relative to the optimal average-reward policy. We analyze the algorithm and show that its regret upper bounds match established results in the contextual bandit literature. In numerical experiments on a movie recommendation dataset and cryogenic electron microscopy (cryo-EM) data, we demonstrate that our approach efficiently maximizes cumulative reward over time.

翻译：我们提出了一种延迟感知上下文赌博机框架，该框架推广了标准上下文赌博机问题，其中学习者在动作延迟下自适应地选择臂并切换决策集。在此设定中，学习者观察上下文，并可从决策集中选择多个臂，总时间由所选子集决定。该问题可被构建为半马尔可夫决策过程（SMDPs）的一个特例，其中上下文和延迟是从一个未知分布中抽取的。利用贝尔曼最优方程，我们设计了上下文在线臂过滤（COAF）算法，该算法平衡了探索、利用和动作延迟，以最小化相对于最优平均奖励策略的遗憾。我们分析了该算法，并证明其遗憾上界与上下文赌博机文献中的既定结果相匹配。在电影推荐数据集和冷冻电子显微镜（cryo-EM）数据的数值实验中，我们证明了我们的方法能有效地随时间最大化累积奖励。