Locating objects referred to in natural language poses a significant challenge for autonomous agents. Existing CLIP-based open-vocabulary methods successfully perform 3D object retrieval with simple (bare) queries but cannot cope with ambiguous descriptions that demand an understanding of object relations. To tackle this problem, we propose a modular approach called BBQ (Beyond Bare Queries), which constructs 3D scene spatial graph representation with metric edges and utilizes a large language model as a human-to-agent interface through our deductive scene reasoning algorithm. BBQ employs robust DINO-powered associations to form 3D objects, an advanced raycasting algorithm to project them to 2D, and a vision-language model to describe them as graph nodes. On Replica and ScanNet datasets, we show that the designed method accurately constructs 3D object-centric maps. We have demonstrated that their quality takes a leading place for open-vocabulary 3D semantic segmentation against other zero-shot methods. Also, we show that leveraging spatial relations is especially effective for scenes containing multiple entities of the same semantic class. On Sr3D and Nr3D benchmarks, our deductive approach demonstrates a significant improvement, enabling retrieving objects by complex queries compared to other state-of-the-art methods. Considering our design solutions, we achieved a processing speed approximately x3 times faster than the closest analog. This promising performance enables our approach for usage in applied intelligent robotics projects. We make the code publicly available at linukc.github.io/bbq/.

翻译：在自然语言中定位所指对象对自主智能体构成重大挑战。现有基于CLIP的开放词汇方法虽能成功处理简单（基础）查询的三维对象检索，但无法应对需要理解对象关系的模糊描述。为解决此问题，我们提出名为BBQ（超越简单查询）的模块化方法，该方法构建具有度量边的三维场景空间图表示，并通过我们提出的演绎场景推理算法，将大语言模型作为人机交互接口。BBQ采用鲁棒的DINO驱动关联构建三维对象，通过先进的光线投射算法将其投影至二维平面，并利用视觉语言模型将其描述为图节点。在Replica和ScanNet数据集上的实验表明，所设计方法能精确构建以对象为中心的三维地图。我们证明其质量在开放词汇三维语义分割任务中，相较于其他零样本方法处于领先地位。同时，研究显示利用空间关系对于包含多个相同语义类别实体的场景尤为有效。在Sr3D和Nr3D基准测试中，我们的演绎方法相较其他先进技术展现出显著提升，能够通过复杂查询检索对象。基于我们的设计方案，处理速度达到最接近同类方法约3倍。这种优异性能使得我们的方法可应用于智能机器人项目。代码已在linukc.github.io/bbq/公开。