Agentic systems have transformed how Large Language Models (LLMs) can be leveraged to create autonomous systems with goal-directed behaviors, consisting of multi-step planning and the ability to interact with different environments. These systems differ fundamentally from traditional machine learning models, both in architecture and deployment, introducing unique AI safety challenges, including goal misalignment, compounding decision errors, and coordination risks among interacting agents, that necessitate embedding interpretability and explainability by design to ensure traceability and accountability across their autonomous behaviors. Current interpretability techniques, developed primarily for static models, show limitations when applied to agentic systems. The temporal dynamics, compounding decisions, and context-dependent behaviors of agentic systems demand new analytical approaches. This paper assesses the suitability and limitations of existing interpretability methods in the context of agentic systems, identifying gaps in their capacity to provide meaningful insight into agent decision-making. We propose future directions for developing interpretability techniques specifically designed for agentic systems, pinpointing where interpretability is required to embed oversight mechanisms across the agent lifecycle from goal formation, through environmental interaction, to outcome evaluation. These advances are essential to ensure the safe and accountable deployment of agentic AI systems.

翻译：智能体系统彻底改变了大型语言模型（LLM）的运用方式，使其能够构建具有目标导向行为的自主系统，这些系统包含多步规划能力，并能与不同环境进行交互。此类系统在架构和部署方式上均与传统机器学习模型存在本质差异，由此引发独特的人工智能安全挑战，包括目标错位、决策误差累积以及交互智能体间的协调风险。这要求通过设计嵌入可解释性与可说明性，以确保其自主行为全程可追溯、可问责。当前主要为静态模型开发的可解释性技术应用于智能体系统时显现出局限性。智能体系统的时间动态性、决策累积效应和情境依赖行为需要新的分析方法。本文评估了现有可解释性方法在智能体系统语境下的适用性与局限，指出其在提供智能体决策机制有效洞察方面存在不足。我们提出了专门针对智能体系统开发可解释性技术的未来研究方向，明确在智能体全生命周期——从目标形成、环境交互到结果评估——中哪些环节需要嵌入可解释性以建立监督机制。这些进展对于确保智能体人工智能系统的安全可靠部署至关重要。