We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, WildRefer, for this task by fully utilizing the appearance features in images, the location and geometry features in point clouds, and the dynamic features in consecutive input frames to match the semantic features in language. In particular, we propose two novel datasets, STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive comparisons and ablation studies illustrate that our method achieves state-of-the-art performance on two proposed datasets. Code and dataset will be released when the paper is published.

翻译：我们提出了基于自然语言描述和在线捕获的多模态视觉数据（包括2D图像和3D激光雷达点云）在大规模动态场景中进行3D视觉定位的任务。针对该任务，我们提出了一种名为WildRefer的新方法，通过充分利用图像中的外观特征、点云中的位置与几何特征以及连续输入帧中的动态特征，与语言中的语义特征进行匹配。特别地，我们构建了两个新型数据集STRefer和LifeRefer，聚焦于大规模以人为中心的日常生活场景，并提供了丰富的3D目标与自然语言标注。这两个数据集对于野外环境下3D视觉定位的研究具有重要意义，并具有推动自动驾驶与服务机器人发展的巨大潜力。大量对比实验和消融研究表明，我们的方法在两个提出数据集上均达到了最先进性能。代码与数据集将于论文发表后公开。