Recently, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have shown promise in instruction following and 2D image understanding. While these models are powerful, they have not yet been developed to comprehend the more challenging 3D physical scenes, especially when it comes to the sparse outdoor LiDAR data. In this paper, we introduce LiDAR-LLM, which takes raw LiDAR data as input and harnesses the remarkable reasoning capabilities of LLMs to gain a comprehensive understanding of outdoor 3D scenes. The central insight of our LiDAR-LLM is the reformulation of 3D outdoor scene cognition as a language modeling problem, encompassing tasks such as 3D captioning, 3D grounding, 3D question answering, etc. Specifically, due to the scarcity of 3D LiDAR-text pairing data, we introduce a three-stage training strategy and generate relevant datasets, progressively aligning the 3D modality with the language embedding space of LLM. Furthermore, we design a View-Aware Transformer (VAT) to connect the 3D encoder with the LLM, which effectively bridges the modality gap and enhances the LLM's spatial orientation comprehension of visual features. Our experiments show that LiDAR-LLM possesses favorable capabilities to comprehend various instructions regarding 3D scenes and engage in complex spatial reasoning. LiDAR-LLM attains a 40.9 BLEU-1 on the 3D captioning task and achieves a 63.1\% classification accuracy and a 14.3\% BEV mIoU on the 3D grounding task. Web page: https://sites.google.com/view/lidar-llm

翻译：近期，大语言模型（LLMs）与多模态大语言模型（MLLMs）在指令遵循与二维图像理解方面展现出巨大潜力。尽管这些模型功能强大，但尚未发展出理解更具挑战性的三维物理场景的能力，尤其是在处理稀疏的室外LiDAR数据时。本文提出LiDAR-LLM，该模型以原始LiDAR数据为输入，利用大语言模型卓越的推理能力实现对室外3D场景的全面理解。LiDAR-LLM的核心思想是将3D室外场景认知重构为一个语言建模问题，涵盖3D描述、3D定位、3D问答等任务。具体而言，针对3D LiDAR-文本配对数据稀缺的问题，我们引入三阶段训练策略并生成相关数据集，逐步将3D模态与大语言模型的语言嵌入空间对齐。此外，我们设计了一种视图感知Transformer（VAT）连接3D编码器与大语言模型，有效弥合模态差异并增强大语言模型对视觉特征空间方位感知的理解能力。实验表明，LiDAR-LLM具备理解3D场景相关各类指令并执行复杂空间推理的优异能力。在3D描述任务中，LiDAR-LLM达到40.9的BLEU-1值；在3D定位任务中，分类准确率达63.1%，BEV平均交并比达14.3%。项目主页：https://sites.google.com/view/lidar-llm