Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, particularly small models, struggle with processing logical reasoning, question-answering, and handling open scenario categories. On the other hand, generative multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak spatial and local object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we propose the following solutions: Spatial-Enhanced Local Feature Mining for better spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, significantly outperforming existing methods.

翻译：自动驾驶、增强现实、机器人以及具身智能领域的最新进展对三维感知算法提出了迫切需求。然而，当前的三维感知方法，特别是小型模型，在处理逻辑推理、问答以及应对开放场景类别方面存在困难。另一方面，生成式多模态大语言模型虽具备强大的通用能力，但在三维任务中表现欠佳，这归因于其空间与局部物体感知能力较弱、基于文本的几何数值输出质量不佳，以及无法处理相机焦距变化。为应对这些挑战，我们提出以下解决方案：空间增强的局部特征挖掘以提升空间特征提取能力，三维查询令牌驱动的信息解码以实现精确的几何回归，以及基于几何投影的三维推理以处理相机焦距变化。我们采用参数高效微调方法对预训练的多模态大语言模型进行优化，并开发了LLMI3D——一个强大的三维感知多模态大语言模型。此外，我们构建了IG3D数据集，该数据集提供了细粒度的描述与问答标注。大量实验表明，我们的LLMI3D实现了最先进的性能，显著优于现有方法。