Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.

翻译：大语言模型（LLMs）和视觉-语言模型（VLMs）已被证明在常识推理等多类任务中表现出色。尽管这些模型功能强大，但它们并未根植于包含空间关系、可操作性、物理属性、布局等更丰富概念的3D物理世界中。在本工作中，我们提出将3D世界注入大语言模型，并引入全新的3D-LLMs家族。具体而言，3D-LLMs可将3D点云及其特征作为输入，执行涵盖描述、密集描述、3D问答、任务分解、3D定位、3D辅助对话、导航等多样化的3D相关任务。通过设计三类提示机制，我们收集了涵盖这些任务的超过30万条3D-语言数据。为高效训练3D-LLMs，我们首先利用3D特征提取器从渲染的多视图图像中获取3D特征，随后以2D VLMs为骨干网络训练3D-LLMs。通过引入3D定位机制，3D-LLMs能够更好地捕获3D空间信息。在ScanQA上的实验表明，我们的模型以大幅度优势超越现有最优基线（例如，BLEU-1分数超出最优结果9%）。此外，在自建数据集上的3D描述、任务组合及3D辅助对话实验显示，我们的模型优于2D VLMs。定性实例也表明，我们的模型可执行超出现有LLMs和VLMs范畴的更多任务。项目页面：https://vis-www.cs.umass.edu/3dllm/。