Recent research has evidenced the significant potentials of Large Language Models (LLMs) in handling challenging tasks within 3D scenes. However, current models are constrained to addressing object-centric tasks, where each question-answer pair focuses solely on an individual object. In real-world applications, users may pose queries involving multiple objects or expect for answers that precisely reference various objects. We introduce the use of object identifiers to freely reference objects during a conversation. While this solution appears straightforward, it presents two main challenges: 1) How to establish a reliable one-to-one correspondence between each object and its identifier? 2) How to incorporate complex spatial relationships among dozens of objects into the embedding space of the LLM? To address these challenges, we propose a two-stage alignment method, which involves learning an attribute-aware token and a relation-aware token for each object. These tokens capture the object's attributes and spatial relationships with surrounding objects in the 3D scene. Once the alignment is established, we can fine-tune our model on various downstream tasks using instruction tuning. Experiments conducted on traditional datasets like ScanQA, ScanRefer, and Nr3D/Sr3D showcase the effectiveness of our proposed method. Additionally, we create a 3D scene captioning dataset annotated with rich object identifiers, with the assistant of GPT-4. This dataset aims to further explore the capability of object identifiers in effective object referencing and precise scene understanding.

翻译：近期研究证实，大型语言模型在处理三维场景中的复杂任务方面展现出巨大潜力。然而，当前模型仅能应对以物体为中心的任务，即每个问答对仅针对单一物体。在真实应用场景中，用户可能提出涉及多个物体的查询，或期望答案能精准引用不同物体。我们引入对象标识符机制，使对话中可自由引用物体。尽管该方案看似直接，但面临两个主要挑战：1）如何建立每个物体与其标识符间可靠的一一对应关系？2）如何将数十个物体间的复杂空间关系融入语言模型的嵌入空间？为解决这些问题，我们提出两阶段对齐方法：为每个物体学习属性感知标记与关系感知标记。这些标记分别捕获物体属性及其在三维场景中与周围物体的空间关系。建立对齐后，我们可通过指令微调在多种下游任务中对模型进行优化。在ScanQA、ScanRefer及Nr3D/Sr3D等传统数据集上的实验验证了所提方法的有效性。此外，我们借助GPT-4构建了包含丰富对象标识符注释的三维场景描述数据集，旨在进一步探索对象标识符在高效物体引用与精准场景理解中的能力。