Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents to perform complex tasks, LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift LLM-based agent safety from ad hoc reliability fixes to proactive guardrails with formal guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.

翻译：基于大语言模型（LLM）的AI智能体通过接入数据源、API、搜索引擎、代码沙箱乃至其他智能体等工具，扩展了LLM的能力。尽管这使智能体能够执行复杂任务，但LLM可能调用非预期的工具交互并引入风险，例如泄露敏感数据或覆盖关键记录，这在企业环境中是不可接受的。当前缓解这些风险的方法（例如基于模型的防护机制）提高了智能体的可靠性，但无法保证系统安全性。信息流控制（IFC）和时间约束等方法旨在提供保证，但通常需要大量人工标注。我们提出一种流程，首先应用系统理论过程分析（STPA）来识别智能体工作流中的危险，推导安全需求，并将其形式化为数据流和工具序列上的可执行规范。为实现此目标，我们引入一种能力增强的模型上下文协议（MCP）框架，该框架要求对能力、机密性和信任级别进行结构化标注。这些贡献共同致力于将基于LLM的智能体安全从临时性的可靠性修补，转向具有形式化保证的主动防护机制，同时减少对用户确认的依赖，并使自主性成为一种审慎的设计选择。