Domestic service robots offer a solution to the increasing demand for daily care and support. A human-in-the-loop approach that combines automation and operator intervention is considered to be a realistic approach to their use in society. Therefore, we focus on the task of retrieving target objects from open-vocabulary user instructions in a human-in-the-loop setting, which we define as the learning-to-rank physical objects (LTRPO) task. For example, given the instruction "Please go to the dining room which has a round table. Pick up the bottle on it," the model is required to output a ranked list of target objects that the operator/user can select. In this paper, we propose MultiRankIt, which is a novel approach for the LTRPO task. MultiRankIt introduces the Crossmodal Noun Phrase Encoder to model the relationship between phrases that contain referring expressions and the target bounding box, and the Crossmodal Region Feature Encoder to model the relationship between the target object and multiple images of its surrounding contextual environment. Additionally, we built a new dataset for the LTRPO task that consists of instructions with complex referring expressions accompanied by real indoor environmental images that feature various target objects. We validated our model on the dataset and it outperformed the baseline method in terms of the mean reciprocal rank and recall@k. Furthermore, we conducted physical experiments in a setting where a domestic service robot retrieved everyday objects in a standardized domestic environment, based on users' instruction in a human--in--the--loop setting. The experimental results demonstrate that the success rate for object retrieval achieved 80%. Our code is available at https://github.com/keio-smilab23/MultiRankIt.

翻译：家庭服务机器人为日益增长的日常护理与支持需求提供了一种解决方案。结合自动化与操作员干预的人机协同方法，被认为是机器人在社会中实际应用的可行途径。因此，我们聚焦于在人类参与模式下，根据开放词汇的用户指令检索目标物体的任务，并将其定义为物理对象学习排序（LTRPO）任务。例如，给定指令“请前往带圆桌的餐厅，拿起桌上的瓶子”，模型需输出一个排序后的目标物体列表，供操作员或用户选择。本文提出MultiRankIt，一种针对LTRPO任务的新方法。MultiRankIt引入了跨模态名词短语编码器，用于建模包含指代表达的短语与目标边界框之间的关系；并引入跨模态区域特征编码器，用于建模目标物体与其周围多张环境图像之间的关系。此外，我们构建了一个新的LTRPO任务数据集，其中包含带有复杂指代表达的指令，并配以真实室内环境图像，涵盖多种目标物体。我们在该数据集上验证了模型性能，其在平均倒数排名和召回率@k指标上均优于基线方法。进一步，我们开展了实体实验：在标准化家庭环境中，家庭服务机器人根据用户指令（人机协同模式）检索日常物体。实验结果表明，物体检索成功率达到了80%。代码已开源至https://github.com/keio-smilab23/MultiRankIt。