We introduce the task of 3D visual grounding in large-scale dynamic scenes based on natural linguistic descriptions and online captured multi-modal visual data, including 2D images and 3D LiDAR point clouds. We present a novel method, WildRefer, for this task by fully utilizing the appearance features in images, the location and geometry features in point clouds, and the dynamic features in consecutive input frames to match the semantic features in language. In particular, we propose two novel datasets, STRefer and LifeRefer, which focus on large-scale human-centric daily-life scenarios with abundant 3D object and natural language annotations. Our datasets are significant for the research of 3D visual grounding in the wild and has huge potential to boost the development of autonomous driving and service robots. Extensive comparisons and ablation studies illustrate that our method achieves state-of-the-art performance on two proposed datasets. Code and dataset will be released when the paper is published.

翻译：我们介绍了一种通过自然语言描述和在线采集的多模态视觉数据（包括2D图像和3D LiDAR点云）在大型动态场景中3D视觉定位的任务。我们提出了一种新方法WildRefer，通过完全利用图像中的外观特征，点云中的位置和几何特征以及连续输入帧中的动态特征，来匹配语言中的语义特征。特别地，我们提出了两个新的数据集，STRefer和LifeRefer，这两个数据集关注人类中心的大规模日常生活场景，包括丰富的3D物体和自然语言注释。我们的数据集对于野外3D视觉定位的研究具有重大意义，并有巨大的潜力促进自动驾驶和服务机器人的发展。广泛的比较和消融研究表明，我们的方法在两个提出的数据集上取得了最先进的性能。代码和数据集将在文章发表时公开。