Large language models (LLMs) represent a major advance in artificial intelligence (AI) research. However, the widespread use of LLMs is also coupled with significant ethical and social challenges. Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust. However, existing auditing procedures fail to address the governance challenges posed by LLMs, which display emergent capabilities and are adaptable to a wide range of downstream tasks. In this article, we address that gap by outlining a novel blueprint for how to audit LLMs. Specifically, we propose a three-layered approach, whereby governance audits (of technology providers that design and disseminate LLMs), model audits (of LLMs after pre-training but prior to their release), and application audits (of applications based on LLMs) complement and inform each other. We show how audits, when conducted in a structured and coordinated manner on all three levels, can be a feasible and effective mechanism for identifying and managing some of the ethical and social risks posed by LLMs. However, it is important to remain realistic about what auditing can reasonably be expected to achieve. Therefore, we discuss the limitations not only of our three-layered approach but also of the prospect of auditing LLMs at all. Ultimately, this article seeks to expand the methodological toolkit available to technology providers and policymakers who wish to analyse and evaluate LLMs from technical, ethical, and legal perspectives.

翻译：大型语言模型是人工智能研究中的重大突破。然而，其广泛应用也伴随着显著的伦理与社会挑战。已有研究指出，审计作为治理机制有望确保AI系统在设计及部署时符合伦理、法律及技术稳健性要求。但现有审计流程无法应对大型语言模型带来的治理难题——这类模型展现出涌现能力并可适配多种下游任务。本文通过提出新型大型语言模型审计蓝图来填补这一空白：具体而言，我们设计了一种三层级方法，使治理审计（针对设计与发布大型语言模型的技术提供方）、模型审计（针对预训练后发布前的大型语言模型）及应用审计（针对基于大型语言模型的应用）能够相互补充与印证。研究表明，当这三个层级的结构化审计协同实施时，可成为识别与管理大型语言模型伦理社会风险的有效机制。但需理性认识审计的预期效能，因此我们不仅讨论了三层级方法的局限性，也探讨了大型语言模型审计本身的可行性边界。最终，本文旨在拓展技术提供方与政策制定者可用的方法论工具箱，使其能从技术、伦理与法律维度综合分析评估大型语言模型。