The rising importance of 3D representation learning, pivotal in computer vision, autonomous driving, and robotics, is evident. However, a prevailing trend, which straightforwardly resorted to transferring 2D alignment strategies to the 3D domain, encounters three distinct challenges: (1) Information Degradation: This arises from the alignment of 3D data with mere single-view 2D images and generic texts, neglecting the need for multi-view images and detailed subcategory texts. (2) Insufficient Synergy: These strategies align 3D representations to image and text features individually, hampering the overall optimization for 3D models. (3) Underutilization: The fine-grained information inherent in the learned representations is often not fully exploited, indicating a potential loss in detail. To address these issues, we introduce JM3D, a comprehensive approach integrating point cloud, text, and image. Key contributions include the Structured Multimodal Organizer (SMO), enriching vision-language representation with multiple views and hierarchical text, and the Joint Multi-modal Alignment (JMA), combining language understanding with visual representation. Our advanced model, JM3D-LLM, marries 3D representation with large language models via efficient fine-tuning. Evaluations on ModelNet40 and ScanObjectNN establish JM3D's superiority. The superior performance of JM3D-LLM further underscores the effectiveness of our representation transfer approach. Our code and models are available at https://github.com/Mr-Neko/JM3D.

翻译：三维表征学习在计算机视觉、自动驾驶及机器人领域日益凸显其重要性。然而，当前普遍存在的将二维对齐策略直接迁移至三维领域的趋势，面临三个显著挑战：(1) 信息退化：源于仅将三维数据与单视图二维图像及通用文本对齐，忽略了多视角图像与细粒度子类别文本的必要性。(2) 协同不足：此类策略将三维表征分别与图像和文本特征进行对齐，阻碍了三维模型的整体优化。(3) 利用不充分：所学表征中固有的细粒度信息往往未被充分利用，导致细节信息潜在流失。为解决上述问题，我们提出JM3D——一种融合点云、文本与图像的综合方法。核心贡献包括结构化多模态组织器（SMO），通过多视角图像与层级化文本丰富视觉语言表征；以及联合多模态对齐（JMA），将语言理解与视觉表征相结合。我们的进阶模型JM3D-LLM通过高效微调将三维表征与大语言模型深度融合。在ModelNet40与ScanObjectNN数据集上的评估表明JM3D具有卓越性能，而JM3D-LLM的优异表现进一步验证了表征迁移方法的有效性。代码与模型已开源至https://github.com/Mr-Neko/JM3D。