Developing high-stakes autonomous systems that include Artificial Intelligence (AI) components is complex; the consequences of errors can be catastrophic, yet it is challenging to plan for all operational cases. In stressful scenarios for the human operator, such as short decision-making timescales, the risk of failures is exacerbated. A lack of understanding of AI failure modes obstructs this and so blocks the robust implementation of applications of AI in smart systems. This prevents early risk identification, leading to increased time, risk and cost of projects. A key tenet of Systems Engineering and acquisition engineering is centred around a "left-shift" in test and evaluation activities to earlier in the system lifecycle, to allow for "accelerated delivery of [systems] that work". We argue it is therefore essential that this shift includes the analysis of AI failure cases as part of the design stages of the system life cycle. Our proposed framework enables the early characterisation of risks emerging from human-autonomy teaming (HAT) in operational contexts. The cornerstone of this is a new analysis of AI failure modes, built on the seminal modelling of human-autonomy teams laid out by LaMonica et al., 2022. Using the analysis of the interactions between human and autonomous systems and exploring the failure modes within each aspect, our approach provides a way to systematically identify human-AI interactions risks across the operational domain of the system of interest. The understanding of the emergent behaviour enables increased robustness of the system, for which the analysis should be undertaken over the whole scope of its operational design domain. This approach is illustrated through an example use case for an AI assistant supporting a Command & Control (C2) System.

翻译：开发包含人工智能（AI）组件的高风险自主系统具有高度复杂性；系统错误的后果可能是灾难性的，然而为所有运行场景进行规划却极为困难。在人类操作员面临压力情境（如决策时间紧迫）时，故障风险会进一步加剧。对AI故障模式的理解不足阻碍了智能系统中AI应用的稳健实施，这导致风险早期识别能力缺失，从而增加了项目的时间周期、风险与成本。系统工程与采办工程的核心原则在于将测试评估活动'左移'至系统生命周期更早期，以实现'可运行系统的加速交付'。我们认为，这一左移过程必须将AI故障案例分析纳入系统生命周期的设计阶段。本文提出的框架能够在运行环境中早期识别人机自主协同（HAT）产生的风险特征。其核心是基于LaMonica等人（2022）提出的人机自主团队奠基性模型，构建了全新的AI故障模式分析方法。通过解析人机系统间的交互机制并探究各维度的故障模式，我们的方法为系统化识别目标系统运行域内人机交互风险提供了途径。对涌现行为的深入理解能增强系统稳健性，相关分析应覆盖其运行设计域的完整范畴。本文以支持指挥控制（C2）系统的AI助手为例，具体阐述了该方法的实践应用。