Transformers are neural network models that utilize multiple layers of self-attention heads and have exhibited enormous potential in natural language processing tasks. Meanwhile, there have been efforts to adapt transformers to visual tasks of machine learning, including Vision Transformers and Swin Transformers. Although some researchers use Vision Transformers for reinforcement learning tasks, their experiments remain at a small scale due to the high computational cost. Experiments conducted at a large scale, on the other hand, have to rely on techniques to cut the costs of Vision Transformers, which also yield inferior results. To address this challenge, this article presents the first online reinforcement learning scheme that is based on Swin Transformers: Swin DQN. Swin Transformers are promising as a backbone in neural networks by splitting groups of image pixels into small patches and applying local self-attention operations inside the (shifted) windows of fixed sizes. They have demonstrated state-of-the-art performances in benchmarks. In contrast to existing research, our novel approach is reducing the computational costs, as well as significantly improving the performance. We demonstrate the superior performance with experiments on 49 games in the Arcade Learning Environment. The results show that our approach, using Swin Transformers with Double DQN, achieves significantly higher maximal evaluation scores than the baseline method in 45 of all the 49 games ~92%, and higher mean evaluation scores than the baseline method in 40 of all the 49 games ~82%.

翻译：Transformer是一种利用多层自注意力头的神经网络模型，在自然语言处理任务中展现出巨大潜力。同时，研究者们也在尝试将Transformer适配至机器学习的视觉任务，包括Vision Transformer和Swin Transformer。尽管部分研究者已将Vision Transformer应用于强化学习任务，但受限于高昂的计算成本，其实验规模始终较小。而大规模实验又不得不依赖降低Vision Transformer成本的技术，这会导致性能下降。为解决这一难题，本文提出首个基于Swin Transformer的在线强化学习方案：Swin DQN。Swin Transformer通过将图像像素组分割成小 patches，并在固定大小的（移位）窗口内执行局部自注意力运算，展现出作为神经网络骨干网络的巨大潜力，已在多项基准测试中达到最优性能。与现有研究不同，我们的创新方法不仅降低了计算成本，还显著提升了性能。我们在Arcade学习环境的49个游戏中进行实验验证了其优越性。结果表明，采用Swin Transformer与Double DQN结合的方法，在全部49个游戏中有45个（约92%）达到显著高于基线方法的最大评估分数，在全部49个游戏中有40个（约82%）达到高于基线方法的平均评估分数。