Offline meta reinforcement learning (OMRL) has emerged as a promising approach for interaction avoidance and strong generalization performance by leveraging pre-collected data and meta-learning techniques. Previous context-based approaches predominantly rely on the intuition that maximizing the mutual information between the task and the task representation ($I(Z;M)$) can lead to performance improvements. Despite achieving attractive results, the theoretical justification of performance improvement for such intuition has been lacking. Motivated by the return discrepancy scheme in the model-based RL field, we find that maximizing $I(Z;M)$ can be interpreted as consistently raising the lower bound of the expected return for a given policy conditioning on the optimal task representation. However, this optimization process ignores the task representation shift between two consecutive updates, which may lead to performance improvement collapse. To address this problem, we turn to use the framework of performance difference bound to consider the impacts of task representation shift explicitly. We demonstrate that by reining the task representation shift, it is possible to achieve monotonic performance improvements, thereby showcasing the advantage against previous approaches. To make it practical, we design an easy yet highly effective algorithm RETRO (\underline{RE}ining \underline{T}ask \underline{R}epresentation shift in context-based \underline{O}ffline meta reinforcement learning) with only adding one line of code compared to the backbone. Empirical results validate its state-of-the-art (SOTA) asymptotic performance, training stability and training-time consumption on MuJoCo and MetaWorld benchmarks.

翻译：离线元强化学习（OMRL）通过利用预收集数据与元学习技术，已成为一种兼具交互规避能力与强泛化性能的潜在方法。现有基于情境的方法主要依赖直觉：最大化任务与任务表征之间的互信息（$I(Z;M)$）可提升性能。尽管取得了可观结果，但此类直觉对应的性能提升理论依据仍不明确。受基于模型的强化学习中回报差异方案的启发，我们发现最大化$I(Z;M)$可理解为在给定最优任务表征的条件下，持续提升策略期望回报的下界。然而，该优化过程忽略了两次连续更新间的任务表征漂移，可能导致性能提升失效。针对此问题，我们采用性能差异界框架显式考虑任务表征漂移的影响。论证表明，通过约束任务表征漂移，可实现单调性能提升，从而展现相较于先前方法的优势。为便于实践，我们设计了简洁高效的算法RETRO（情境型离线元强化学习中任务表征漂移的约束），相较于基线仅需增加一行代码。在MuJoCo与MetaWorld基准上的实验结果验证了其渐进性能达到当前最优（SOTA），同时具备训练稳定性与更低的训练耗时。