Traditional reinforcement learning usually assumes either episodic interactions with resets or continuous operation to minimize average or cumulative loss. While episodic settings have many theoretical results, resets are often unrealistic in practice. The infinite-horizon setting avoids this issue but lacks non-asymptotic guarantees in online scenarios with unknown dynamics. In this work, we move towards closing this gap by introducing a reset-free framework called the periodic framework, where the goal is to find periodic policies: policies that not only minimize cumulative loss but also return the agents to their initial state distribution after a fixed number of steps. We formalize the problem of finding optimal periodic policies and identify sufficient conditions under which it is well-defined for tabular Markov decision processes. To evaluate algorithms in this framework, we introduce the periodic regret, a measure that balances cumulative loss with the terminal law constraint. We then propose the first algorithms for computing periodic policies in two multi-agent settings and show they achieve sublinear periodic regret of order $\tilde O(T^{3/4})$. This provides the first non-asymptotic guarantees for reset-free learning in the setting of $M$ homogeneous agents, for $M > 1$.

翻译：传统强化学习通常假设智能体与环境进行片段式交互（带有重置机制）或持续运行以最小化平均损失或累积损失。虽然片段式设定在理论上已有诸多成果，但重置机制在实践中往往不切实际。无限时域设定避免了这一问题，但在动态未知的在线场景中缺乏非渐近性保证。本研究通过引入一种称为周期性框架的无重置框架来弥合这一差距，其目标是寻找周期性策略：这类策略不仅最小化累积损失，还能在固定步数后将智能体返回到初始状态分布。我们形式化了寻找最优周期性策略的问题，并确定了在表格型马尔可夫决策过程中该问题定义良好的充分条件。为在此框架下评估算法，我们引入了周期性遗憾，这是一种平衡累积损失与终端律约束的度量。随后，我们提出了首个用于计算两种多智能体设定下周期性策略的算法，并证明其实现了阶为 $\tilde O(T^{3/4})$ 的次线性周期性遗憾。这为 $M$ 个同质智能体（$M > 1$）场景下的无重置学习提供了首个非渐近性保证。