Oversight is rightly recognised as vital within high-stakes public sector AI applications, where decisions can have profound individual and collective impacts. Much current thinking regarding forms of oversight mechanisms for AI within the public sector revolves around the idea of human decision makers being 'in-the-loop' and thus being able to intervene to prevent errors and potential harm. However, in a number of high-stakes public sector contexts, operational oversight of decisions is made by expert teams rather than individuals. The ways in which deployed AI systems can be integrated into these existing operational team oversight processes has yet to attract much attention. We address this gap by exploring the impacts of AI upon pre-existing oversight of clinical decision-making through institutional analysis. We find that existing oversight is nested within professional training requirements and relies heavily upon explanation and questioning to elicit vital information. Professional bodies and liability mechanisms also act as additional levers of oversight. These dimensions of oversight are impacted, and potentially reconfigured, by AI systems. We therefore suggest a broader lens of 'team-in-the-loop' to conceptualise the system-level analysis required for adoption of AI within high-stakes public sector deployment.

翻译：监督被公认为高风险公共部门人工智能应用中至关重要的环节，此类应用的决策可能对个人和集体产生深远影响。当前关于公共部门人工智能监督机制形式的诸多思考，都围绕人类决策者"在环"并据此能够干预以防止错误和潜在伤害的理念展开。然而，在许多高风险公共部门场景中，决策的操作性监督是由专家团队而非个人完成的。已部署的人工智能系统如何融入这些现有的团队监督流程尚未引起足够重视。我们通过制度分析探究人工智能对临床决策现有监督的影响来填补这一空白。研究发现，现有监督嵌套于专业培训要求之中，并高度依赖解释与质询来获取关键信息。专业机构与责任机制也扮演着额外监督杠杆的角色。这些监督维度会受到人工智能系统的影响，并可能被其重构。因此，我们提出更广义的"团队在环"视角，以概念化高风险公共部门部署中采用人工智能所需的系统级分析。