We study a nonstationary bandit problem where rewards depend on both actions and latent states, the latter governed by unknown linear dynamics. Crucially, the state dynamics also depend on the actions, resulting in tension between short-term and long-term rewards. We propose an explore-then-commit algorithm for a finite horizon $T$. During the exploration phase, random Rademacher actions enable estimation of the Markov parameters of the linear dynamics, which characterize the action-reward relationship. In the commit phase, the algorithm uses the estimated parameters to design an optimized action sequence for long-term reward. Our proposed algorithm achieves $\tilde{\mathcal{O}}(T^{2/3})$ regret. Our analysis handles two key challenges: learning from temporally correlated rewards, and designing action sequences with optimal long-term reward. We address the first challenge by providing near-optimal sample complexity and error bounds for system identification using bilinear rewards. We address the second challenge by proving an equivalence with indefinite quadratic optimization over a hypercube, a known NP-hard problem. We provide a sub-optimality guarantee for this problem, enabling our regret upper bound. Lastly, we propose a semidefinite relaxation with Goemans-Williamson rounding as a practical approach.

翻译：本文研究一类非平稳赌博机问题，其奖励同时取决于动作和潜在状态，后者由未知的线性动态系统控制。关键之处在于，状态动态也依赖于动作，这导致了短期奖励与长期奖励之间的权衡。针对有限时间范围$T$，我们提出一种探索-承诺算法。在探索阶段，通过随机Rademacher动作实现对线性动态系统马尔可夫参数的估计，这些参数刻画了动作-奖励关系。在承诺阶段，算法利用估计参数设计出面向长期奖励的优化动作序列。所提算法实现了$\tilde{\mathcal{O}}(T^{2/3})$的遗憾上界。我们的分析处理了两个关键挑战：从时间相关的奖励中学习，以及设计具有最优长期奖励的动作序列。针对第一个挑战，我们通过双线性奖励的系统辨识问题，给出了近乎最优的样本复杂度与误差界。针对第二个挑战，我们证明了该问题等价于超立方体上的不定二次优化问题——一个已知的NP难问题。我们为此问题提供了次优性保证，从而推导出遗憾上界。最后，我们提出采用半定松弛配合Goemans-Williamson舍入作为实际求解方法。