The emergence of LLM-based agents has garnered considerable attention, yet their trustworthiness remains an under-explored area. As agents can directly interact with the physical environment, their reliability and safety is critical. This paper presents an Agent-Constitution-based agent framework, TrustAgent, an initial investigation into improving the safety dimension of trustworthiness in LLM-based agents. This framework consists of threefold strategies: pre-planning strategy which injects safety knowledge to the model prior to plan generation, in-planning strategy which bolsters safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Through experimental analysis, we demonstrate how these approaches can effectively elevate an LLM agent's safety by identifying and preventing potential dangers. Furthermore, we explore the intricate relationships between safety and helpfulness, and between the model's reasoning ability and its efficacy as a safe agent. This paper underscores the imperative of integrating safety awareness and trustworthiness into the design and deployment of LLM-based agents, not only to enhance their performance but also to ensure their responsible integration into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.

翻译：基于大语言模型的智能体（LLM-based Agents）的兴起引起了广泛关注，但其可信度仍是一个尚待深入探索的领域。由于智能体可直接与物理环境交互，其可靠性与安全性至关重要。本文提出一种基于智能体宪章（Agent Constitution）的框架——TrustAgent，这是提升LLM智能体可信度中安全维度的初步探索。该框架包含三重策略：规划前策略（在生成规划前向模型注入安全知识）、规划中策略（在规划生成过程中增强安全性）以及规划后策略（通过规划后检查确保安全性）。通过实验分析，我们展示了这些方法如何通过识别和预防潜在危险有效提升LLM智能体的安全性。此外，我们探讨了安全性与有用性之间的复杂关系，以及模型推理能力与其作为安全智能体效能之间的关联。本文强调，在设计和部署LLM智能体时，必须将安全意识和可信度融入其中——这不仅是提升性能的关键，更是确保其负责任地融入人本环境的必要条件。数据和代码已开源至https://github.com/agiresearch/TrustAgent。