Generalizable robot manipulation requires not only task-level reasoning over unseen scenes, but also reliable grounding of visual plans into embodiment-specific execution. To bridge this gap, we propose VICX (Video generation and In-Context eXecution), a decoupled closed-loop manipulation framework. In VICX, a frozen video generation model produces vision-language-conditioned high-level visual plans, while a Video-to-Trajectory In-Context Operator Network (V2T-ICON) serves as the task-agnostic interface that grounds these plans into executable robot-state trajectories. To improve execution generalization, V2T-ICON operates on segmentation-extracted arm-only frame observations and uses retrieved image-state pairs as in-context prompts, allowing a robust and generalizable visual-to-state mapping at inference time without parameter updates. Experiments on Meta-World show that VICX supports cross-task generalization, closed-loop self-correction, and cross-embodiment transfer, demonstrating dual generalization across both task semantics and robot execution. The project webpage can be found here: https://scaling-group.github.io/vicx/.
翻译:摘要:通用机器人操作不仅需要在未见场景中进行任务级推理,还需要将视觉计划可靠地落地到具体本体的执行中。为弥合这一差距,我们提出VICX(视频生成与上下文执行),一种解耦的闭环操作框架。在VICX中,冻结的视频生成模型产生视觉-语言条件化的高层视觉计划,而视频到轨迹的上下文操作网络(V2T-ICON)作为任务无关的接口,将这些计划落地为可执行的机器人状态轨迹。为提高执行泛化能力,V2T-ICON基于分割提取的仅机械臂帧观测,并利用检索的图像-状态对作为上下文提示,从而在推理时无需参数更新即可实现鲁棒且可泛化的视觉到状态映射。Meta-World上的实验表明,VICX支持跨任务泛化、闭环自我纠错以及跨本体迁移,展示了在任务语义和机器人执行两个层面的双重泛化能力。项目网页见:https://scaling-group.github.io/vicx/。