We introduce the new setting of open-vocabulary object 6D pose estimation, in which a textual prompt is used to specify the object of interest. In contrast to existing approaches, in our setting (i) the object of interest is specified solely through the textual prompt, (ii) no object model (e.g. CAD or video sequence) is required at inference, (iii) the object is imaged from two different viewpoints of two different scenes, and (iv) the object was not observed during the training phase. To operate in this setting, we introduce a novel approach that leverages a Vision-Language Model to segment the object of interest from two distinct scenes and to estimate its relative 6D pose. The key of our approach is a carefully devised strategy to fuse object-level information provided by the prompt with local image features, resulting in a feature space that can generalize to novel concepts. We validate our approach on a new benchmark based on two popular datasets, REAL275 and Toyota-Light, which collectively encompass 39 object instances appearing in four thousand image pairs. The results demonstrate that our approach outperforms both a well-established hand-crafted method and a recent deep learning-based baseline in estimating the relative 6D pose of objects in different scenes. Project page: https://jcorsetti.github.io/oryon/.

翻译：我们引入了开放词汇物体6D姿态估计这一新设定，其中通过文本提示来指定感兴趣的目标物体。与现有方法不同，在我们的设定中：(i) 仅通过文本提示指定感兴趣的目标物体；(ii) 推理阶段无需任何物体模型（如CAD或视频序列）；(iii) 目标物体在两个不同场景的两个不同视角下成像；(iv) 目标物体在训练阶段未曾出现。为在此设定下运行，我们提出了一种创新方法，该方法利用视觉语言模型从两个不同场景中分割出感兴趣的目标物体，并估计其相对6D姿态。我们方法的核心在于精心设计的策略，将提示提供的物体级信息与局部图像特征融合，从而形成能够泛化到新概念的特征空间。我们在基于两个流行数据集REAL275和Toyota-Light的新基准上验证了方法，这两个数据集共涵盖出现在四千对图像中的39个物体实例。结果表明，在估计不同场景中物体的相对6D姿态时，我们的方法优于成熟的基于手工设计的方法以及近期基于深度学习的基线方法。项目页面：https://jcorsetti.github.io/oryon/。