Aiming to advance AI agents, large foundation models significantly improve reasoning and instruction execution, yet the current focus on vision and language neglects the potential of perceiving diverse modalities in open-world environments. However, the success of data-driven vision and language models is costly or even infeasible to be reproduced for rare modalities. In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space. Specifically, the modality-specific lens is tuned to project any-modal signals to an intermediate embedding space, which are then processed by a strong ViT with pre-trained visual knowledge. The encoded representations are optimized toward aligning with the modal-independent space, pre-defined by off-the-shelf foundation models. ViT-Lens-2 provides a unified solution for representation learning of increasing modalities with two appealing advantages: (i) Unlocking the great potential of pretrained ViTs to novel modalities effectively with efficient data regime; (ii) Enabling emergent downstream capabilities through modality alignment and shared ViT parameters. We tailor ViT-Lens-2 to learn representations for 3D point cloud, depth, audio, tactile and EEG, and set new state-of-the-art results across various understanding tasks, such as zero-shot classification. By seamlessly integrating ViT-Lens-2 into Multimodal Foundation Models, we enable Any-modality to Text and Image Generation in a zero-shot manner. Code and models are available at https://github.com/TencentARC/ViT-Lens.

翻译：为推动AI智能体发展，大规模基础模型显著提升了推理与指令执行能力，但当前集中于视觉与语言模态的研究忽略了在开放世界环境中感知多样模态的潜力。然而，数据驱动的视觉与语言模型的成功经验，对于稀缺模态而言成本高昂甚至难以复制。本文提出ViT-Lens-2，通过利用预训练ViT感知新模态并将其对齐至预定义空间，实现高效的全模态表示学习。具体而言，模态专用透镜通过微调将任意模态信号投影至中间嵌入空间，进而由具备预训练视觉知识的强健ViT处理。编码后的表示被优化以对齐至基于现成基础模型预定义的模态无关空间。ViT-Lens-2为日益增多的模态提供统一的表示学习方案，具有两大优势：（i）高效利用数据机制，充分释放预训练ViT在新模态上的巨大潜力；（ii）通过模态对齐与共享ViT参数实现新兴下游能力。我们将ViT-Lens-2定制化应用于3D点云、深度、音频、触觉与脑电图（EEG）的表征学习，并在零样本分类等多项理解任务中刷新最优结果。通过将ViT-Lens-2无缝集成至多模态基础模型，我们实现了任意模态到文本与图像的零样本生成。代码与模型已开源：https://github.com/TencentARC/ViT-Lens。