Memory Gym introduces a unique benchmark designed to test Deep Reinforcement Learning agents, specifically comparing Gated Recurrent Unit (GRU) against Transformer-XL (TrXL), on their ability to memorize long sequences, withstand noise, and generalize. It features partially observable 2D environments with discrete controls, namely Mortar Mayhem, Mystery Path, and Searing Spotlights. These originally finite environments are extrapolated to novel endless tasks that act as an automatic curriculum, drawing inspiration from the car game ``I packed my bag". These endless tasks are not only beneficial for evaluating efficiency but also intriguingly valuable for assessing the effectiveness of approaches in memory-based agents. Given the scarcity of publicly available memory baselines, we contribute an implementation driven by TrXL and Proximal Policy Optimization. This implementation leverages TrXL as episodic memory using a sliding window approach. In our experiments on the finite environments, TrXL demonstrates superior sample efficiency in Mystery Path and outperforms in Mortar Mayhem. However, GRU is more efficient on Searing Spotlights. Most notably, in all endless tasks, GRU makes a remarkable resurgence, consistently outperforming TrXL by significant margins.

翻译：Memory Gym提出了一种独特的基准测试，旨在检验深度强化学习智能体的能力，特别是对比门控循环单元（GRU）与Transformer-XL（TrXL）在记忆长序列、抗噪声干扰及泛化方面的表现。该基准包含三个部分可观测、离散控制的2D环境，即"迫击炮混乱"（Mortar Mayhem）、"神秘路径"（Mystery Path）与"灼热聚光灯"（Searing Spotlights）。这些原为有限环境的场景被扩展为新颖的无限任务，其类似游戏"我装满包裹"的自动课程机制，不仅有助于评估效率，更对检验基于记忆智能体的方法有效性具有独特价值。鉴于公开可用的记忆基线稀缺，我们贡献了一种基于TrXL与近端策略优化的实现方案，该方案利用滑动窗口方法将TrXL作为情景记忆模块。在有限环境的实验中，TrXL在"神秘路径"中展现出更优的样本效率，并在"迫击炮混乱"中表现突出；然而在"灼热聚光灯"中，GRU效率更高。值得注意的是，在所有无限任务中，GRU实现了显著复苏，以大幅优势持续超越TrXL。