In recent years, a number of models that learn the relations between vision and language from large datasets have been released. These models perform a variety of tasks, such as answering questions about images, retrieving sentences that best correspond to images, and finding regions in images that correspond to phrases. Although there are some examples, the connection between these pre-trained vision-language models and robotics is still weak. If they are directly connected to robot motions, they lose their versatility due to the embodiment of the robot and the difficulty of data collection, and become inapplicable to a wide range of bodies and situations. Therefore, in this study, we categorize and summarize the methods to utilize the pre-trained vision-language models flexibly and easily in a way that the robot can understand, without directly connecting them to robot motions. We discuss how to use these models for robot motion selection and motion planning without re-training the models. We consider five types of methods to extract information understandable for robots, and show the results of state recognition, object recognition, affordance recognition, relation recognition, and anomaly detection based on the combination of these five methods. We expect that this study will add flexibility and ease-of-use, as well as new applications, to the recognition behavior of existing robots.

翻译：近年来，从大规模数据集中学习视觉与语言关系的模型相继发布。这些模型能够执行多种任务，例如回答图像相关问题、检索与图像最匹配的句子、以及定位图像中与短语对应的区域。尽管已有一些应用案例，但预训练视觉-语言模型与机器人领域的联系仍较为薄弱。若将这些模型直接与机器人运动控制相连，由于机器人具身化特性及数据采集难度，其通用性会降低，难以适用于多样化的机体与场景。因此，本研究分类并总结了无需直接连接机器人运动控制，即可灵活且便捷地利用预训练视觉-语言模型的方法，使机器人能够理解这些模型。我们探讨了如何在不重新训练模型的前提下，将这些模型用于机器人运动选择与运动规划。研究提出了五种可提取机器人可理解信息的方法，并展示了基于这五种方法组合实现的状态识别、物体识别、可供性识别、关系识别及异常检测结果。预计本研究将为现有机器人的识别行为增加灵活性与易用性，并拓展新的应用场景。