Sequential modeling has demonstrated remarkable capabilities in offline reinforcement learning (RL), with Decision Transformer (DT) being one of the most notable representatives, achieving significant success. However, RL trajectories possess unique properties to be distinguished from the conventional sequence (e.g., text or audio): (1) local correlation, where the next states in RL are theoretically determined solely by current states and actions based on the Markov Decision Process (MDP), and (2) global correlation, where each step's features are related to long-term historical information due to the time-continuous nature of trajectories. In this paper, we propose a novel action sequence predictor, named Mamba Decision Maker (MambaDM), where Mamba is expected to be a promising alternative for sequence modeling paradigms, owing to its efficient modeling of multi-scale dependencies. In particular, we introduce a novel mixer module that proficiently extracts and integrates both global and local features of the input sequence, effectively capturing interrelationships in RL datasets. Extensive experiments demonstrate that MambaDM achieves state-of-the-art performance in Atari and OpenAI Gym datasets. Furthermore, we empirically investigate the scaling laws of MambaDM, finding that increasing model size does not bring performance improvement, but scaling the dataset amount by 2x for MambaDM can obtain up to 33.7% score improvement on Atari dataset. This paper delves into the sequence modeling capabilities of MambaDM in the RL domain, paving the way for future advancements in robust and efficient decision-making systems.

翻译：序列建模在离线强化学习（RL）中已展现出卓越的能力，其中决策Transformer（DT）是最具代表性的方法之一，并取得了显著成功。然而，RL轨迹具有区别于传统序列（如文本或音频）的独特性质：（1）局部相关性，即RL中的下一状态理论上仅由当前状态和动作根据马尔可夫决策过程（MDP）决定；（2）全局相关性，由于轨迹的时间连续性，每一步的特征都与长期历史信息相关。本文提出了一种新颖的动作序列预测器，命名为Mamba决策者（MambaDM），其中Mamba因其对多尺度依赖关系的高效建模，有望成为序列建模范式的有前景的替代方案。具体而言，我们引入了一种新颖的混合器模块，该模块能够熟练地提取并整合输入序列的全局和局部特征，有效捕捉RL数据集中的相互关系。大量实验表明，MambaDM在Atari和OpenAI Gym数据集上实现了最先进的性能。此外，我们通过实验研究了MambaDM的缩放定律，发现增加模型规模并不会带来性能提升，但将MambaDM的数据集规模扩大2倍，可在Atari数据集上获得高达33.7%的分数提升。本文深入探讨了MambaDM在RL领域的序列建模能力，为未来构建鲁棒且高效的决策系统铺平了道路。