An increasing number of regulations propose the notion of AI audits as an enforcement mechanism for achieving transparency and accountability for AI systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently have little to no agreed upon practices, procedures, taxonomies, and standards. We propose the criterion audit as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations' ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit, and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which bias audits for hiring algorithms can be performed, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge. Our discussion as informed by experiences in performing these audits in practice highlights the critical role that an audit ecosystem plays in ensuring the effectiveness of such methodology.

翻译：越来越多法规将AI审计视为实现AI系统透明性和问责制的一种执行机制。尽管围绕各类AI审计形式已形成部分共识，但目前针对合规性与保证目的的审计几乎没有公认的实践规范、程序、分类体系和标准。我们提出"标准审计"作为可操作的合规性与保证外部审计框架。该方法借鉴金融审计实践中的要素，并主张AI审计应同样为其利益相关方提供保证，证明AI组织具备以减轻危害、维护人类价值观的方式治理其算法的能力。我们讨论了标准审计的必要条件，并提供了实际开展审计工作的程序性蓝图。通过推导可依据近期生效的《2021年纽约市地方法律第144号》要求对招聘算法进行偏见审计的标准，我们展示了该框架如何适配现行法规。最后，我们批判性地探讨了将更成熟的金融审计行业实践应用于AI审计时的益处、固有局限与实施挑战——当前AI审计领域针对质量保证问题的稳健防护机制尚在萌发阶段。基于实际开展审计的经验，我们的讨论凸显了审计生态系统在确保此类方法论有效性中的关键作用。