Many dexterous manipulation tasks are non-markovian in nature, yet little attention has been paid to this fact in the recent upsurge of the vision-language-action (VLA) paradigm. Although they are successful in bringing internet-scale semantic understanding to robotics, existing VLAs are primarily "stateless" and struggle with memory-dependent long horizon tasks. In this work, we explore a way to impart both spatial and temporal memory to a VLA by incorporating a language scratchpad. The scratchpad makes it possible to memorize task-specific information, such as object positions, and it allows the model to keep track of a plan and progress towards subgoals within that plan. We evaluate this approach on a split of memory-dependent tasks from the ClevrSkills environment, on MemoryBench, as well as on a challenging real-world pick-and-place task. We show that incorporating a language scratchpad significantly improves generalization on these tasks for both non-recurrent and recurrent models.

翻译：许多灵巧操作任务本质上具有非马尔可夫特性，然而在当前兴起的视觉-语言-动作范式研究中，这一事实尚未得到充分关注。尽管现有VLA模型成功地将互联网规模的语义理解引入机器人领域，但它们本质上是"无状态"的，难以处理依赖记忆的长时程任务。本研究探索通过融入语言草稿板机制，为VLA模型同时赋予空间与时间记忆能力。该草稿板使模型能够记忆任务特定信息（如物体位置），并支持其追踪任务计划及监测子目标完成进度。我们在ClevrSkills环境中的记忆依赖任务子集、MemoryBench基准以及具有挑战性的真实世界抓放任务上评估该方法。实验表明，语言草稿板的引入能显著提升非循环与循环模型在此类任务上的泛化性能。