We study an online setting, where a decision maker (DM) interacts with contextual bandit-with-knapsack (BwK) instances in repeated episodes. These episodes start with different resource amounts, and the contexts' probability distributions are non-stationary in an episode. All episodes share the same latent conversion model, which governs the random outcome contingent upon a request's context and an allocation decision. Our model captures applications such as dynamic pricing on perishable resources with episodic replenishment, and first price auctions in repeated episodes with different starting budgets. We design an online algorithm that achieves a regret sub-linear in $T$, the number of episodes, assuming access to a \emph{confidence bound oracle} that achieves an $o(T)$-regret. Such an oracle is readily available from existing contextual bandit literature. We overcome the technical challenge with arbitrarily many possible contexts, which leads to a reinforcement learning problem with an unbounded state space. Our framework provides improved regret bounds in certain settings when the DM is provided with unlabeled feature data, which is novel to the contextual BwK literature.

翻译：本研究探讨一种在线决策环境，其中决策者（DM）在重复的决策周期中与具有背包约束（BwK）的情景多臂老虎机实例进行交互。每个决策周期以不同的资源初始量开始，且情景的概率分布在单个周期内呈现非平稳特性。所有决策周期共享同一个潜在转换模型，该模型根据请求的情景特征与资源分配决策生成随机结果。我们的模型可应用于多种实际场景，例如具有周期性补货机制的易逝资源动态定价问题，以及不同初始预算条件下重复进行的首价拍卖场景。我们设计了一种在线算法，该算法在总周期数 $T$ 上实现亚线性遗憾度，其前提是能够访问可实现 $o(T)$ 遗憾度的\emph{置信区间预言机}——此类预言机在现有情景老虎机文献中已有成熟实现。我们通过技术手段克服了可能情景数量任意多所导致的挑战，该挑战原本会引发状态空间无界的强化学习问题。当决策者获得未标注特征数据时，我们的框架在特定情境下能够提供更优的遗憾度边界，这为情景BwK研究领域带来了新的突破。