In this paper, we argue that the next generation of robots can be commanded using only Language Models' prompts. Every prompt interrogates separately a specific Robotic Modality via its Modality Language Model (MLM). A central Task Modality mediates the whole communication to execute the robotic mission via a Large Language Model (LLM). This paper gives this new robotic design pattern the name of: Prompting Robotic Modalities (PRM). Moreover, this paper applies this PRM design pattern in building a new robotic framework named ROSGPT_Vision. ROSGPT_Vision allows the execution of a robotic task using only two prompts: a Visual and an LLM prompt. The Visual Prompt extracts, in natural language, the visual semantic features related to the task under consideration (Visual Robotic Modality). Meanwhile, the LLM Prompt regulates the robotic reaction to the visual description (Task Modality). The framework automates all the mechanisms behind these two prompts. The framework enables the robot to address complex real-world scenarios by processing visual data, making informed decisions, and carrying out actions automatically. The framework comprises one generic vision module and two independent ROS nodes. As a test application, we used ROSGPT_Vision to develop CarMate, which monitors the driver's distraction on the roads and makes real-time vocal notifications to the driver. We showed how ROSGPT_Vision significantly reduced the development cost compared to traditional methods. We demonstrated how to improve the quality of the application by optimizing the prompting strategies, without delving into technical details. ROSGPT_Vision is shared with the community (link: https://github.com/bilel-bj/ROSGPT_Vision) to advance robotic research in this direction and to build more robotic frameworks that implement the PRM design pattern and enables controlling robots using only prompts.

翻译：本文提出，新一代机器人可通过仅使用语言模型的提示进行命令。每个提示通过其模态语言模型（MLM）独立查询特定机器人模态。中央任务模态通过大语言模型（LLM）协调整个通信流程以执行机器人任务。本文将这种新型机器人设计模式命名为“提示机器人模态”（PRM）。此外，本文将该PRM设计模式应用于构建名为ROSGPT_Vision的新型机器人框架。ROSGPT_Vision仅需两个提示即可执行机器人任务：视觉提示和LLM提示。视觉提示以自然语言提取与当前任务相关的视觉语义特征（视觉机器人模态），而LLM提示则规范机器人对视觉描述的响应（任务模态）。该框架自动执行这两个提示背后的所有机制，使机器人能够通过处理视觉数据、做出明智决策并自动执行操作来应对复杂的现实场景。该框架包含一个通用视觉模块和两个独立的ROS节点。作为测试应用，我们利用ROSGPT_Vision开发了CarMate系统，用于监测道路上的驾驶员分心情况并实时向驾驶员发出语音通知。我们展示了ROSGPT_Vision相较于传统方法显著降低了开发成本，并演示了如何通过优化提示策略（无需深入技术细节）提升应用质量。ROSGPT_Vision已向社区开源（链接：https://github.com/bilel-bj/ROSGPT_Vision），旨在推动该方向的机器人研究，并构建更多实现PRM设计模式、支持仅通过提示控制机器人的框架。