In the realm of Reinforcement Learning (RL), online RL is often conceptualized as an optimization problem, where an algorithm interacts with an unknown environment to minimize cumulative regret. In a stationary setting, strong theoretical guarantees, like a sublinear ($\sqrt{T}$) regret bound, can be obtained, which typically implies the convergence to an optimal policy and the cessation of exploration. However, these theoretical setups often oversimplify the complexities encountered in real-world RL implementations, where tasks arrive sequentially with substantial changes between tasks and the algorithm may not be allowed to adaptively learn within certain tasks. We study the changes beyond the outcome distributions, encompassing changes in the reward designs (mappings from outcomes to rewards) and the permissible policy spaces. Our results reveal the fallacy of myopically minimizing regret within each task: obtaining optimal regret rates in the early tasks may lead to worse rates in the subsequent ones, even when the outcome distributions stay the same. To realize the optimal cumulative regret bound across all the tasks, the algorithm has to overly explore in the earlier tasks. This theoretical insight is practically significant, suggesting that due to unanticipated changes (e.g., rapid technological development or human-in-the-loop involvement) between tasks, the algorithm needs to explore more than it would in the usual stationary setting within each task. Such implication resonates with the common practice of using clipped policies in mobile health clinical trials and maintaining a fixed rate of $\epsilon$-greedy exploration in robotic learning.

翻译：在强化学习领域，在线强化学习常被概念化为一个优化问题，即算法与未知环境交互以最小化累积遗憾。在静态环境中，可以获得强有力的理论保证（如亚线性$\sqrt{T}$遗憾界），这通常意味着收敛到最优策略并停止探索。然而，这些理论设置往往过度简化了现实强化学习实现中的复杂性——任务会顺序到达且任务间存在显著变化，算法可能无法在特定任务中自适应学习。我们研究超越结果分布的变化，涵盖奖励设计（从结果到奖励的映射）和可行策略空间的变化。研究结果揭示了短视地在每个任务中最小化遗憾的谬误：即便结果分布保持不变，在早期任务中获得最优遗憾率可能导致后续任务出现更差遗憾率。为实现跨所有任务的最优累积遗憾界，算法必须在早期任务中进行过度探索。这一理论洞见具有重要实践意义，表明由于任务间未预判的变化（如快速技术发展或人机交互介入），算法需要在每个任务中比常规静态设置进行更多探索。该结论与移动健康临床试验中使用截断策略、机器人学习中保持固定$\epsilon$-贪心探索率的常见实践相呼应。