Effectively integrating diverse sensory modalities is crucial for robotic manipulation. However, the typical approach of feature concatenation is often suboptimal: dominant modalities such as vision can overwhelm sparse but critical signals like touch in contact-rich tasks, and monolithic architectures cannot flexibly incorporate new or missing modalities without retraining. Our method factorizes the policy into a set of diffusion models, each specialized for a single representation (e.g., vision or touch), and employs a router network that learns consensus weights to adaptively combine their contributions, enabling incremental of new representations. We evaluate our approach on simulated manipulation tasks in {RLBench}, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion, where it significantly outperforms feature-concatenation baselines on scenarios requiring multimodal reasoning. Our policy further demonstrates robustness to physical perturbations and sensor corruption. We further conduct perturbation-based importance analysis, which reveals adaptive shifts between modalities.

翻译：有效整合多样化的感知模态对于机器人操作至关重要。然而，典型的特征拼接方法往往并非最优：在接触丰富的任务中，视觉等主导模态可能会淹没触觉等稀疏但关键的信号，且单一架构无法在不重新训练的情况下灵活纳入新的或缺失的模态。我们的方法将策略分解为一组扩散模型，每个模型专用于单一表征（如视觉或触觉），并采用一个学习共识权重的路由网络来自适应地结合它们的贡献，从而支持新表征的增量式引入。我们在{RLBench}的模拟操作任务以及真实世界任务（如遮挡物体抓取、手中勺子重定向和拼图插入）上评估了我们的方法。在需要多模态推理的场景中，该方法显著优于特征拼接基线。我们的策略进一步展示了对物理扰动和传感器损坏的鲁棒性。我们还进行了基于扰动的重要性分析，揭示了模态间的自适应切换。