As Artificial Intelligence (AI) systems proliferate, the need for systematic, transparent, and actionable processes for evaluating them is growing. While many resources exist to support AI evaluation, they have several limitations. Few address both the process of designing, developing, and deploying an AI system and the outcomes it produces. Furthermore, few are end-to-end and operational, give actionable guidance, or present evidence of usability or effectiveness in practice. In this paper, we introduce a third-party AI assurance framework that addresses these gaps. We focus on third-party assurance to prevent conflict of interest and ensure credibility and accountability of the process. We begin by distinguishing assurance from audits in several key dimensions. Then, following design principles, we reflect on the shortcomings of existing resources to identify a set of design requirements for AI assurance. We then construct a prototype of an assurance process that consists of (1) a responsibility assignment matrix to determine the different levels of involvement each stakeholder has at each stage of the AI lifecycle, (2) an interview protocol for each stakeholder of an AI system, (3) a maturity matrix to assess AI systems' adherence to best practices, and (4) a template for an assurance report that draws from more mature assurance practices in business accounting. We conduct early validation of our AI assurance framework by applying the framework to two distinct AI use cases -- a business document tagging tool for downstream processing in a large private firm, and a housing resource allocation tool in a public agency -- and conducting expert validation interviews. Our findings show early evidence that our AI assurance framework is sound and comprehensive, usable across different organizational contexts, and effective at identifying bespoke issues with AI systems.

翻译：随着人工智能（AI）系统的普及，对系统性、透明且可操作的评估流程的需求日益增长。尽管已有诸多资源支持AI评估，但它们存在若干局限性。鲜有资源同时涵盖AI系统的设计、开发与部署过程及其产出结果。此外，极少资源能够提供端到端的可操作方案、给出具体实施指导，或在实践中展示其可用性与有效性的证据。本文提出一种第三方AI保障框架以弥补这些不足。我们聚焦于第三方保障，旨在避免利益冲突并确保流程的可信度与问责性。我们首先从多个关键维度区分保障与审计的差异。随后，依据设计原则，通过反思现有资源的缺陷，提出一套AI保障的设计需求。在此基础上，我们构建了一个保障流程原型，包含：（1）用于确定各利益相关方在AI生命周期各阶段参与程度的责任分配矩阵；（2）针对AI系统各利益相关方的访谈规程；（3）用于评估AI系统遵循最佳实践程度的成熟度矩阵；以及（4）借鉴商业会计中更成熟保障实践的保障报告模板。我们通过将框架应用于两个不同的AI用例——大型私营企业的下游处理业务文档标注工具，以及公共机构的住房资源分配工具——并开展专家验证访谈，对AI保障框架进行了早期验证。研究结果表明，我们的AI保障框架具备合理性与全面性，能够跨不同组织情境使用，并能有效识别AI系统的定制化问题。