Generalist robots should be able to understand and follow user instructions. Despite providing a powerful architecture for mapping open-vocabulary language instructions to robot actions, current vision-language-action (VLA) models struggle to follow fine-grained commands. One cause for this is a lack of semantic diversity and language grounding in existing robot datasets and, specifically, a lack of fine-grained task diversity for similar observations. To address this, we present a novel method to augment existing robot datasets by leveraging vision-language models to create counterfactual labels. By augmenting existing datasets with these labels, we increase the diversity and granularity of language grounding for robot datasets, ultimately improving the language-following capabilities of VLAs. We evaluate the resulting model's ability to follow language instructions, ranging from simple object-centric commands to complex referential tasks, by conducting vision-language navigation experiments in 3 different indoor and outdoor environments. Our experiments show that counterfactual relabeling (without additional data collection) significantly improves instruction-following in VLA policies, outperforming state-of-the-art methods and doubling the success rate compared to VLAs trained on unaugmented data. We also evaluate our method for manipulation VLAs and find a similar gain in performance on tasks with distractors.

翻译：通用型机器人应能理解并遵循用户指令。尽管当前视觉-语言-动作（VLA）模型为将开放词汇语言指令映射至机器人动作提供了强大架构，但其在细粒度指令跟随方面仍存在困难。这一问题的成因之一是现有机器人数据集缺乏语义多样性与语言基础关联性，特别是相似观测场景下的细粒度任务多样性不足。为此，我们提出一种利用视觉-语言模型生成反事实标签来增强现有机器人数据集的新方法。通过使用这些标签扩充现有数据集，我们提升了机器人数据集语言基础关联的多样性和粒度，最终改进了VLA模型的语言跟随能力。为评估所得模型遵循语言指令的能力（从简单物体中心指令到复杂指代任务），我们在3个不同的室内外环境中开展了视觉-语言导航实验。实验表明，反事实重标注（无需额外数据收集）显著提升了VLA策略的指令跟随性能，其表现优于现有最优方法，且相较于在未增强数据上训练的VLA模型，成功率实现翻倍。我们还评估了该方法在操作型VLA模型上的效果，发现在含干扰物任务中同样获得了类似的性能提升。