Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult. As a result, practitioners make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 390 tools, we map the current ecosystem of available AI audit tools. While there are many tools designed to assist practitioners with setting standards and evaluating AI systems, these tools often fell short of supporting the accountability goals of AI auditing in practice. We thus highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy -- and outline challenges practitioners faced in their efforts to use AI audit tools. We conclude that resources are lacking to adequately support the full scope of needs for many AI audit practitioners and recommend that the field move beyond tools for just evaluation, towards more comprehensive infrastructure for AI accountability.
翻译:审计是识别已部署人工智能(AI)系统风险与局限的关键机制。然而,AI审计的有效执行仍面临巨大困难。为此,实践者们借助多种工具来支撑其工作。基于对35位AI审计从业者的访谈及对390款工具的生态分析,我们绘制了现有AI审计工具的全景图谱。尽管存在大量旨在协助从业者制定标准与评估AI系统的工具,但这些工具在实践中往往难以支撑AI审计的问责目标。因此,我们指出了超越评估范畴的未来工具开发方向——从危害发现到权益倡导——并梳理了从业者在使用AI审计工具时面临的挑战。我们得出结论:当前资源不足以充分满足众多AI审计从业者的全方位需求,并建议该领域应超越仅聚焦评估的工具研发,转向构建更全面的AI问责基础设施。