Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.2%, and +0.73% on Sr3D, Nr3D, and ScanRefer. Code will be released at https://github.com/ZiyuGuo99/ViewRefer3D.

翻译：从多视角输入理解3D场景已被证明能有效缓解3D视觉定位中的视角差异问题。然而，现有方法通常忽略文本模态中嵌入的视角线索，且未能有效权衡不同视角的相对重要性。本文提出ViewRefer——一种用于3D视觉定位的多视角框架，探索如何同时从文本和3D模态中获取视角知识。在文本分支中，ViewRefer利用大规模语言模型（如GPT）的多样化语言知识，将单一定位文本扩展为多个几何一致的描述。在3D模态中，我们引入带有视角间交叉注意力的Transformer融合模块，以增强跨视角目标之间的交互。在此基础上，进一步提出一组可学习的多视角原型，用于记忆不同视角的场景无关知识，并从两个层面增强框架：视角引导注意力模块以获取更鲁棒的文本特征，以及最终预测阶段的视角引导评分策略。通过该设计范式，ViewRefer在三个基准数据集上取得优越性能，在Sr3D、Nr3D和ScanRefer上分别超越第二名2.8%、1.2%和0.73%。代码将发布在https://github.com/ZiyuGuo99/ViewRefer3D。