State ambiguity is common in robotic manipulation. Identical observations may correspond to multiple valid behavior trajectories. The visuomotor policy must correctly extract the appropriate types and levels of information from the history to identify the current task phase. However, naively extending the history window is computationally expensive and may cause severe overfitting. Inspired by the continuous nature of human reasoning and the recoding of working memory, we introduce PAM, a novel visuomotor Policy equipped with Adaptive working Memory. With minimal additional training cost in a two-stage manner, PAM supports a 300-frame history window while maintaining high inference speed. Specifically, a hierarchical frame feature extractor yields two distinct representations for motion primitives and temporal disambiguation. For compact representation, a context router with range-specific queries is employed to produce compact context features across multiple history lengths. And an auxiliary objective of reconstructing historical information is introduced to ensure that the context router acts as an effective bottleneck. We meticulously design 7 tasks and verify that PAM can handle multiple scenarios of state ambiguity simultaneously. With a history window of approximately 10 seconds, PAM still supports stable training and maintains inference speeds above 20Hz. Project website: https://tinda24.github.io/pam/

翻译：状态模糊性在机器人操作中普遍存在。相同的观测可能对应多个有效的行为轨迹。视觉运动策略必须从历史中正确提取适当类型和层次的信息以识别当前任务阶段。然而，简单地扩展历史窗口计算成本高昂，并可能导致严重的过拟合。受人类推理的连续性及工作记忆重编码机制的启发，我们提出了PAM——一种配备自适应工作记忆的新型视觉运动策略。通过两阶段训练方式以最小额外成本，PAM支持300帧历史窗口同时保持高推理速度。具体而言，分层帧特征提取器生成运动基元和时间消歧两种不同表征。为获得紧凑表征，采用具有范围特定查询的上下文路由器来生成跨多历史长度的紧凑上下文特征。同时引入历史信息重建的辅助目标，确保上下文路由器作为有效的瓶颈。我们精心设计了7项任务，验证PAM能同时处理多种状态模糊性场景。在约10秒的历史窗口下，PAM仍支持稳定训练并保持20Hz以上的推理速度。项目网站：https://tinda24.github.io/pam/