The advantages of pre-trained large language models (LLMs) are apparent in a variety of language processing tasks. But can a language model's knowledge be further harnessed to effectively disambiguate objects and navigate decision-making challenges within the realm of robotics? Our study reveals the LLM's aptitude for solving complex decision making challenges that are often previously modeled by Partially Observable Markov Decision Processes (POMDPs). A pivotal focus of our research is the object disambiguation capability of LLMs. We detail the integration of an LLM into a tabletop environment disambiguation task, a decision making problem where the robot's task is to discern and retrieve a user's desired object from an arbitrarily large and complex cluster of objects. Despite multiple query attempts with zero-shot prompt engineering (details can be found in the Appendix), the LLM struggled to inquire about features not explicitly provided in the scene description. In response, we have developed a few-shot prompt engineering system to improve the LLM's ability to pose disambiguating queries. The result is a model capable of both using given features when they are available and inferring new relevant features when necessary, to successfully generate and navigate down a precise decision tree to the correct object--even when faced with identical options.

翻译：预训练大语言模型（LLMs）在各类语言处理任务中的优势已显而易见。然而，语言模型所蕴含的知识能否被进一步用于有效消解目标歧义并应对机器人领域的决策挑战？本研究表明，LLM具备解决复杂决策问题的能力——这类问题此前通常借助部分可观察马尔可夫决策过程（POMDPs）进行建模。本研究的核心聚焦点在于LLM的目标消歧能力。我们详细阐述了将LLM集成至桌面环境消歧任务中的方法：机器人需从任意大且复杂的物体集群中辨识并取出用户所需目标对象。尽管采用零样本提示工程进行了多次查询尝试（详见附录），LLM仍难以询问场景描述中未明确提供的特征。为此，我们开发了一套少样本提示工程系统，用于提升LLM提出消歧性查询的能力。最终，该模型既能利用已提供的特征，又能在必要时推断出新的相关特征，从而成功生成并沿着精确决策树导航至正确目标——即便面对完全相同的备选对象亦能胜任。