Robotics researchers increasingly leverage large language models (LLM) in robotics systems, using them as interfaces to receive task commands, generate task plans, form team coalitions, and allocate tasks among multi-robot and human agents. However, despite their benefits, the growing adoption of LLM in robotics has raised several safety concerns, particularly regarding executing malicious or unsafe natural language prompts. In addition, ensuring that task plans, team formation, and task allocation outputs from LLMs are adequately examined, refined, or rejected is crucial for maintaining system integrity. In this paper, we introduce SafePlan, a multi-component framework that combines formal logic and chain-of-thought reasoners for enhancing the safety of LLM-based robotics systems. Using the components of SafePlan, including Prompt Sanity COT Reasoner and Invariant, Precondition, and Postcondition COT reasoners, we examined the safety of natural language task prompts, task plans, and task allocation outputs generated by LLM-based robotic systems as means of investigating and enhancing system safety profile. Our results show that SafePlan outperforms baseline models by leading to 90.5% reduction in harmful task prompt acceptance while still maintaining reasonable acceptance of safe tasks.

翻译：机器人学研究者日益广泛地将大语言模型（LLM）应用于机器人系统，将其作为接收任务指令、生成任务规划、组建团队联盟以及在多机器人及人类智能体间分配任务的接口。然而，尽管LLM具有诸多优势，其在机器人领域的不断普及也引发了若干安全隐患，特别是在执行恶意或不安全的自然语言指令方面。此外，确保对LLM生成的任务规划、团队组建及任务分配输出进行充分审查、优化或拒绝，对于维护系统完整性至关重要。本文提出SafePlan——一个融合形式逻辑与思维链推理器的多组件框架，旨在增强基于LLM的机器人系统的安全性。通过运用SafePlan的组件（包括指令合理性思维链推理器以及不变性、前置条件与后置条件思维链推理器），我们对基于LLM的机器人系统生成的自然语言任务指令、任务规划及任务分配输出进行了安全性检验，以此作为探究与提升系统安全特性的手段。实验结果表明，SafePlan在保持合理安全任务接受率的同时，将有害任务指令的接受率降低了90.5%，其性能显著优于基线模型。