We study contextual bandits in the presence of a stage-wise constraint (a constraint at each round), when the constraint must be satisfied both with high probability and in expectation. Obviously the setting where the constraint is in expectation is a relaxation of the one with high probability. We start with the linear case where both the contextual bandit problem (reward function) and the stage-wise constraint (cost function) are linear. In each of the high probability and in expectation settings, we propose an upper-confidence bound algorithm for the problem and prove a $T$-round regret bound for it. Our algorithms balance exploration and constraint satisfaction using a novel idea that scales the radii of the reward and cost confidence sets with different scaling factors. We also prove a lower-bound for this constrained problem, show how our algorithms and analyses can be extended to multiple constraints, and provide simulations to validate our theoretical results. In the high probability setting, we describe the minimum requirements for the action set in order for our algorithm to be tractable. In the setting that the constraint is in expectation, we further specialize our results to multi-armed bandits and propose a computationally efficient algorithm for this setting with regret analysis. Finally, we extend our results to the case where the reward and cost functions are both non-linear. We propose an algorithm for this case and prove a regret bound for it that characterize the function class complexity by the eluder dimension.

翻译：我们研究了存在阶段约束（每轮约束）情况下的上下文赌博机问题，其中约束必须同时以高概率和期望形式满足。显然，期望约束设定是高概率约束的松弛形式。我们从线性情形开始，假设上下文赌博机问题（奖励函数）和阶段约束（成本函数）均为线性。针对高概率和期望两种设定，我们分别提出了基于上置信界（UCB）的算法，并给出了T轮遗憾界。我们的算法通过创新的缩放因子策略，对奖励和成本置信集的半径采用不同比例缩放，从而平衡探索与约束满足。我们进一步证明了该约束问题的下界，展示了算法与分析如何扩展至多约束场景，并通过仿真验证理论结果。在高概率设定中，我们阐述了动作集满足算法可计算性的最低要求。在期望约束设定中，我们专门将结果推广至多臂赌博机，并提出了具有遗憾分析的高计算效率算法。最后，我们将结论扩展至奖励函数和成本函数均为非线性的情形，提出了相应算法并给出了以埃尔金德维数表征函数类复杂度的遗憾界。