Off-policy learning from multistep returns is crucial for sample-efficient reinforcement learning, but counteracting off-policy bias without exacerbating variance is challenging. Classically, off-policy bias is corrected in a per-decision manner: past temporal-difference errors are re-weighted by the instantaneous Importance Sampling (IS) ratio after each action via eligibility traces. Many off-policy algorithms rely on this mechanism, along with differing protocols for cutting the IS ratios to combat the variance of the IS estimator. Unfortunately, once a trace has been fully cut, the effect cannot be reversed. This has led to the development of credit-assignment strategies that account for multiple past experiences at a time. These trajectory-aware methods have not been extensively analyzed, and their theoretical justification remains uncertain. In this paper, we propose a multistep operator that can express both per-decision and trajectory-aware methods. We prove convergence conditions for our operator in the tabular setting, establishing the first guarantees for several existing methods as well as many new ones. Finally, we introduce Recency-Bounded Importance Sampling (RBIS), which leverages trajectory awareness to perform robustly across $λ$-values in an off-policy control task.

翻译：从多步回报中进行离策略学习对于样本高效的强化学习至关重要，但抵消离策略偏差而不加剧方差具有挑战性。经典方法以逐决策方式修正离策略偏差：通过资格迹，过去的时序差分误差在每次动作后由瞬时重要性采样（IS）比率重新加权。许多离策略算法依赖此机制，并结合不同的截断IS比率方案以对抗IS估计量的方差。遗憾的是，一旦迹被完全截断，其影响便不可逆转。这推动了同时考虑多个过去经验的信用分配策略的发展。这些轨迹感知方法尚未得到广泛分析，其理论依据仍不确定。本文提出一种可同时表达逐决策与轨迹感知方法的多步算子。我们在表格设定下证明了该算子的收敛条件，为多种现有方法及新方法提供了首个理论保证。最后，我们引入近期有界重要性采样（RBIS），该方法利用轨迹感知能力在离策略控制任务中实现跨λ值的鲁棒性能。