Cross-modal contrastive distillation has recently been explored for learning effective 3D representations. However, existing methods focus primarily on modality-shared features, neglecting the modality-specific features during the pre-training process, which leads to suboptimal representations. In this paper, we theoretically analyze the limitations of current contrastive methods for 3D representation learning and propose a new framework, namely CMCR (Cross-Modal Comprehensive Representation Learning), to address these shortcomings. Our approach improves upon traditional methods by better integrating both modality-shared and modality-specific features. Specifically, we introduce masked image modeling and occupancy estimation tasks to guide the network in learning more comprehensive modality-specific features. Furthermore, we propose a novel multi-modal unified codebook that learns an embedding space shared across different modalities. Besides, we introduce geometry-enhanced masked image modeling to further boost 3D representation learning. Extensive experiments demonstrate that our method mitigates the challenges faced by traditional approaches and consistently outperforms existing image-to-LiDAR contrastive distillation methods in downstream tasks. Code will be available at https://github.com/Eaphan/CMCR.

翻译：跨模态对比蒸馏近来被研究用于学习有效的3D表示。然而，现有方法主要关注模态共享特征，在预训练过程中忽略了模态特定特征，导致表示效果欠佳。本文从理论上分析了当前对比方法在3D表示学习中的局限性，并提出新框架CMCR（跨模态全面表示学习）以解决这些不足。我们的方法通过更优地整合模态共享特征与模态特定特征，改进了传统方法。具体而言，我们引入遮蔽图像建模和占用估计任务，引导网络学习更全面的模态特定特征。此外，我们提出一种新颖的多模态统一码本，学习跨不同模态共享的嵌入空间。同时，我们引入几何增强的遮蔽图像建模，进一步促进3D表示学习。大量实验表明，我们的方法缓解了传统方法面临的挑战，并在下游任务中持续优于现有图像到激光雷达的对比蒸馏方法。代码将发布于https://github.com/Eaphan/CMCR。