Despite recent progress in reinforcement learning (RL) from raw pixel data, sample inefficiency continues to present a substantial obstacle. Prior works have attempted to address this challenge by creating self-supervised auxiliary tasks, aiming to enrich the agent's learned representations with control-relevant information for future state prediction. However, these objectives are often insufficient to learn representations that can represent the optimal policy or value function, and they often consider tasks with small, abstract discrete action spaces and thus overlook the importance of action representation learning in continuous control. In this paper, we introduce TACO: Temporal Action-driven Contrastive Learning, a simple yet powerful temporal contrastive learning approach that facilitates the concurrent acquisition of latent state and action representations for agents. TACO simultaneously learns a state and an action representation by optimizing the mutual information between representations of current states paired with action sequences and representations of the corresponding future states. Theoretically, TACO can be shown to learn state and action representations that encompass sufficient information for control, thereby improving sample efficiency. For online RL, TACO achieves 40% performance boost after one million environment interaction steps on average across nine challenging visual continuous control tasks from Deepmind Control Suite. In addition, we show that TACO can also serve as a plug-and-play module adding to existing offline visual RL methods to establish the new state-of-the-art performance for offline visual RL across offline datasets with varying quality.

翻译：尽管基于原始像素数据的强化学习近年来取得了进展，但样本效率低下的问题仍是一个重大障碍。先前研究试图通过构建自监督辅助任务来应对这一挑战，旨在向智能体学习到的表征中注入与控制相关的信息，用于未来状态预测。然而，这些目标往往不足以学习到能够表征最优策略或价值函数的表征，且通常仅考虑中小规模抽象离散动作空间的任务，因而忽视了连续控制中动作表征学习的重要性。本文提出TACO（时间动作驱动对比学习）——一种简洁而强大的时间对比学习方法，可促进智能体同时获取潜在状态与动作表征。TACO通过优化当前状态-动作序列联合表征与对应未来状态表征之间的互信息，同步学习状态与动作表征。理论分析表明，TACO能够学习到包含充分控制信息的表征，从而提升样本效率。在在线强化学习中，TACO在Deepmind控制套件的九个高难度视觉连续控制任务上，经过百万次环境交互步数后平均性能提升40%。此外，TACO可作为即插即用模块集成至现有离线视觉强化学习方法中，在不同质量的离线数据集上均能建立离线视觉强化学习的最新性能基准。