Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer. Code is released at https://github.com/Ivan-Tang-3D/ViewRefer3D.

翻译：从多视角输入理解3D场景已被证明能缓解3D视觉定位中的视角差异问题。然而现有方法通常忽略文本模态中蕴含的视角线索，且未能权衡不同视角的相对重要性。本文提出ViewRefer——一种面向3D视觉定位的多视角框架，探索如何从文本与3D双模态中捕获视角知识。在文本分支中，ViewRefer利用大规模语言模型（如GPT）的多样化语言知识，将单一定位文本扩展为多个几何一致性描述。在3D模态中，引入带有跨视角注意力的Transformer融合模块以增强跨视角目标交互。在此基础上，我们进一步提出一组可学习的多视角原型，通过记忆不同视角的场景无关知识，从两方面增强框架：通过视角引导注意力模块生成更鲁棒的文本特征，并在最终预测阶段采用视角引导评分策略。凭借所设计的范式，ViewRefer在三个基准数据集上均取得领先性能，在Sr3D、Nr3D和ScanRefer上分别超越第二名2.8%、1.5%和1.35%。代码已开源至https://github.com/Ivan-Tang-3D/ViewRefer3D。