The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. After pretraining on VIDAL-10M, we outperform ImageBind by 5.8% R@1 on the MSR-VTT dataset with only 15% of the parameters in the zero-shot video-text retrieval task. Beyond this, our LanguageBind has greatly improved in the zero-shot video, audio, depth, and infrared understanding tasks. For instance, LanguageBind surpassing InterVideo by 1.9% on MSR-VTT, 8.8% on MSVD, 6.3% on DiDeMo, and 4.4% on ActivityNet. On the LLVIP and NYU-D datasets, LanguageBind outperforms ImageBind with 23.8% and 11.1% top-1 accuracy. Code address: https://github.com/PKU-YuanGroup/LanguageBind.

翻译：视频-语言（VL）预训练在多个下游任务中取得了显著进展。然而，当前的VL预训练框架难以扩展到视觉和语言之外的多模态（N模态，N≥3）。为此，我们提出LanguageBind，将语言作为不同模态之间的绑定纽带，因为语言模态已得到充分探索且蕴含丰富语义。具体而言，我们冻结通过VL预训练获得的语言编码器，随后通过对比学习为其他模态训练编码器。最终，所有模态被映射至共享特征空间，实现多模态语义对齐。尽管LanguageBind确保我们能将VL模态扩展至N模态，但仍需一个以语言为中心的对齐数据对构成的高质量数据集。因此，我们提出包含视频、红外、深度、音频及其对应语言的VIDAL-10M数据集（命名）。在我们的VIDAL-10M中，所有视频均来自具有完整语义的短视频平台（而非长视频截取片段），且视频、深度、红外和音频模态均与对应的文本描述对齐。在VIDAL-10M上预训练后，我们在零样本视频-文本检索任务的MSR-VTT数据集上以仅15%的参数规模超越ImageBind 5.8%的R@1指标。此外，我们的LanguageBind在零样本视频、音频、深度和红外理解任务中均取得大幅提升：例如在MSR-VTT上超越InterVideo 1.9%，MSVD上提升8.8%，DiDeMo上提升6.3%，ActivityNet上提升4.4%。在LLVIP和NYU-D数据集上，LanguageBind分别以23.8%和11.1%的top-1准确率超越ImageBind。代码地址：https://github.com/PKU-YuanGroup/LanguageBind.