The Vision Transformer (ViT) architecture has established its place in computer vision literature, however, training ViTs for RGB-D object recognition remains an understudied topic, viewed in recent literature only through the lens of multi-task pretraining in multiple vision modalities. Such approaches are often computationally intensive, relying on the scale of multiple pretraining datasets to align RGB with 3D information. In this work, we propose a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains for 3D object recognition, focusing on fusing RGB and depth representations encoded jointly by the ViT. Compared to previous works in multimodal Transformers, the key challenge here is to use the attested flexibility of ViTs to capture cross-modal interactions at the downstream and not the pretraining stage. We explore which depth representation is better in terms of resulting accuracy and compare early and late fusion techniques for aligning the RGB and depth modalities within the ViT architecture. Experimental results in the Washington RGB-D Objects dataset (ROD) demonstrate that in such RGB -> RGB-D scenarios, late fusion techniques work better than most popularly employed early fusion. With our transfer baseline, fusion ViTs score up to 95.4% top-1 accuracy in ROD, achieving new state-of-the-art results in this benchmark. We further show the benefits of using our multimodal fusion baseline over unimodal feature extractors in a synthetic-to-real visual adaptation as well as in an open-ended lifelong learning scenario in the ROD benchmark, where our model outperforms previous works by a margin of >8%. Finally, we integrate our method with a robot framework and demonstrate how it can serve as a perception utility in an interactive robot learning scenario, both in simulation and with a real robot.

翻译：视觉Transformer（ViT）架构已在计算机视觉文献中确立其地位，然而，针对RGB-D物体识别任务训练ViT仍是一个研究不足的课题，近期文献仅从多任务预训练的角度在多视觉模态中进行探讨。此类方法通常计算量巨大，依赖多个预训练数据集的规模来对齐RGB与三维信息。在本工作中，我们提出一种简单而有效的策略，用于在RGB-D领域迁移预训练ViT以实现三维物体识别，重点在于融合由ViT联合编码的RGB与深度表征。与以往多模态Transformer研究相比，核心挑战在于利用ViT已验证的灵活性，在下游任务阶段而非预训练阶段捕获跨模态交互。我们探讨了何种深度表征能在精度方面取得更优效果，并比较了在ViT架构中对齐RGB与深度模态的早期融合与晚期融合技术。在华盛顿RGB-D物体数据集（ROD）上的实验结果表明，在RGB到RGB-D的迁移场景中，晚期融合技术优于多数常用的早期融合方法。基于我们的迁移基线，融合ViT在ROD数据集上达到了95.4%的top-1准确率，创下该基准测试的最新最优结果。我们进一步展示了在合成到真实视觉适应以及ROD基准中的开放式终身学习场景中，使用多模态融合基线相较于单模态特征提取器的优势，在该场景下我们的模型以超过8%的幅度优于先前工作。最后，我们将该方法与机器人框架集成，并展示了其在交互式机器人学习场景中作为感知工具的应用潜力，仿真实验与真实机器人实验均验证了其有效性。