Inherently, robotic manipulation tasks are history-dependent: leveraging past context could be beneficial. However, most existing Vision-Language-Action models (VLAs) have been designed without considering this aspect, i.e., they rely solely on the current observation, ignoring preceding context. In this paper, we propose HAMLET, a scalable framework to adapt VLAs to attend to the historical context during action prediction. Specifically, we introduce moment tokens that compactly encode perceptual information at each timestep. Their representations are initialized with time-contrastive learning, allowing them to better capture temporally distinctive aspects. Next, we employ a lightweight memory module that integrates the moment tokens across past timesteps into memory features, which are then leveraged for action prediction. Through empirical evaluation, we show that HAMLET successfully transforms a state-of-the-art VLA into a history-aware policy, especially demonstrating significant improvements on long-horizon tasks that require historical context. In particular, on top of GR00T N1.5, HAMLET achieves an average success rate of 76.4% on history-dependent real-world tasks, surpassing the baseline performance by 47.2%. Furthermore, HAMLET pushes prior art performance from 64.1% to 66.4% on RoboCasa Kitchen (100-demo setup) and from 95.6% to 97.7% on LIBERO, highlighting its effectiveness even under generic robot-manipulation benchmarks.
翻译:本质上,机器人操作任务具有历史依赖性:利用过往上下文信息可能大有裨益。然而,现有的大多数视觉-语言-动作模型(VLA)在设计时并未考虑这一方面,即它们仅依赖于当前观测,而忽略了先前的上下文。在本文中,我们提出HAMLET,一个可扩展的框架,用于使VLA在动作预测时关注历史上下文。具体来说,我们引入时刻令牌(moment tokens),它紧凑地编码每个时间步的感知信息。其表示通过时间对比学习进行初始化,使其能够更好地捕捉时间上具有区分性的方面。随后,我们采用一个轻量级记忆模块,将过去时间步的时刻令牌整合为记忆特征,进而用于动作预测。通过实证评估,我们证明HAMLET成功地将一个先进VLA转化为历史感知策略,尤其是在需要历史上下文的长程任务中展现出显著提升。具体而言,在GR00T N1.5基础上,HAMLET在依赖历史的真实世界任务中实现了76.4%的平均成功率,超越基线性能47.2%。此外,HAMLET将先前最优方法在RoboCasa Kitchen(100演示设置)上的性能从64.1%提升至66.4%,在LIBERO上从95.6%提升至97.7%,即使是在通用机器人操作基准下也凸显了其有效性。