Large language models are becoming increasingly pervasive and ubiquitous in society via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased and behave irresponsibly, causing harm to people at scale. It is crucial to audit these language models rigorously. Existing auditing tools leverage either or both humans and AI to find failures. In this work, we draw upon literature in human-AI collaboration and sensemaking, and conduct interviews with research experts in safe and fair AI, to build upon the auditing tool: AdaTest (Ribeiro and Lundberg, 2022), which is powered by a generative large language model (LLM). Through the design process we highlight the importance of sensemaking and human-AI communication to leverage complementary strengths of humans and generative models in collaborative auditing. To evaluate the effectiveness of the augmented tool, AdaTest++, we conduct user studies with participants auditing two commercial language models: OpenAI's GPT-3 and Azure's sentiment analysis model. Qualitative analysis shows that AdaTest++ effectively leverages human strengths such as schematization, hypothesis formation and testing. Further, with our tool, participants identified a variety of failures modes, covering 26 different topics over 2 tasks, that have been shown before in formal audits and also those previously under-reported.

翻译：大语言模型通过部署在社会技术系统中日益渗透和普及于社会。然而，这些语言模型无论是用于分类还是生成，已被证明存在偏见且行为不负责任，可能对大规模人群造成伤害。因此，对这些语言模型进行严格审计至关重要。现有的审计工具要么依赖人类，要么依赖AI，或两者结合来发现故障。本文借鉴人机协作和意义建构领域的文献，并与安全公平AI领域的研究专家进行访谈，基于审计工具AdaTest（Ribeiro和Lundberg, 2022）进行扩展。AdaTest由生成式大语言模型驱动。通过设计过程，我们强调意义建构和人机沟通在利用人类与生成模型的互补优势进行协作审计中的重要性。为评估增强型工具AdaTest++的有效性，我们开展了用户研究，参与者对两种商业语言模型——OpenAI的GPT-3和Azure的情感分析模型——进行了审计。定性分析表明，AdaTest++有效利用了人类的优势，如模式化、假设形成与检验。此外，使用我们的工具，参与者在两项任务中识别了涵盖26个不同主题的多种故障模式，这些模式既包括先前在正式审计中已报告的，也包括先前未被充分报道的。