Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments. However, these same characteristics can create or exacerbate product risks. We studied how industry developers (n=35) perceive, prioritize, and address the risks in their agentic AI products. We found that developers' perceptions of risk were closely tied to the qualities that made the product agentic, such as autonomy, tool use, and usage in a real-world context. Developers prioritized product and business risks before considering downstream societal risks like job displacement and end-user privacy. This prioritization also impacted developers' ability and motivation to mitigate agentic risks. Finally, developers lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity. These findings reveal a capability vs. risk control tension in agentic AI development: developers need to address risks that emerge from agentic capabilities, yet they currently have limited support for doing so without constraining agentic functionality.
翻译:智能体AI系统能够自主行动、使用工具、适应环境并运行于复杂的现实场景中。然而,这些特性也可能催生或加剧产品风险。我们通过研究35位产业界开发者如何感知、权衡与应对其智能体AI产品中的风险发现:开发者对风险的认知与其产品具备的自主性、工具使用、现实场景应用等智能体特性紧密相关;开发者优先考虑产品与商业风险,而后才关注岗位替代、终端用户隐私等下游社会性风险——这种优先级排序也影响了开发者缓解智能体风险的能力与动机。此外,开发者缺乏成熟的智能体风险管控手段,往往通过限制智能体核心特性(如自主性、目标复杂度)的方式来控制风险。这些发现揭示了智能体AI开发中的能力-风险控制张力:开发者亟需应对智能体能力引发的风险,但当前缺乏在不削弱智能体功能的前提下有效管控风险的支持手段。