Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception--action mappings. To address this challenge, we propose \textbf{AffordanceVLA}, a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation to establish a more precise and robust perception--action mapping. Specifically, we progressively model manipulation priors through three complementary components: 1) \textbf{Which2Act} for object-centric grounding via visual latent prediction to suppress distractions; 2) \textbf{Where2Act} for 2D interaction localization via affordance map estimation; and 3) \textbf{How2Act} for 3D geometric reasoning to guide manipulation policies. These affordance cues provide spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture with specialized experts and train the model using a three-stage training strategy with a progressive data curriculum. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on simulation and real-world demonstrate that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
翻译:视觉-语言-动作(VLA)模型利用预训练视觉-语言模型(VLM)丰富的世界知识,实现了遵循指令的机器人操作。然而,VLM语义空间与具身控制策略之间的结构性失配往往阻碍了精确感知-动作映射的学习。为解决这一挑战,我们提出**AffordanceVLA**——一个统一框架,通过引入结构化可供性预测作为面向任务的中间表征,建立更精确、更鲁棒的感知-动作映射。具体而言,我们通过三个互补组件逐步建模操作先验:1)**Which2Act**,通过视觉潜变量预测实现以物体为中心的定位以抑制干扰;2)**Where2Act**,通过可供性图估计实现二维交互定位;以及3)**How2Act**,通过三维几何推理指导操作策略。这些可供性线索提供了具有空间基础、语义条件化和动作耦合的中间表征,从而自然桥接了视觉、语言和动作。我们将这些模块集成到一种混合Transformer(MoT)架构中,配备专门化专家模块,并采用三阶段训练策略及渐进式数据课程对模型进行训练。为克服机器人数据集中密集可供性标注的稀缺性,我们还开发了一种鲁棒的自动化数据增强流水线。在仿真和真实环境中的大量实验表明,AffordanceVLA在多种操作场景下均取得了优异性能。