Large language models are becoming increasingly pervasive and ubiquitous in society via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased and behave irresponsibly, causing harm to people at scale. It is crucial to audit these language models rigorously. Existing auditing tools leverage either or both humans and AI to find failures. In this work, we draw upon literature in human-AI collaboration and sensemaking, and conduct interviews with research experts in safe and fair AI, to build upon the auditing tool: AdaTest (Ribeiro and Lundberg, 2022), which is powered by a generative large language model (LLM). Through the design process we highlight the importance of sensemaking and human-AI communication to leverage complementary strengths of humans and generative models in collaborative auditing. To evaluate the effectiveness of the augmented tool, AdaTest++, we conduct user studies with participants auditing two commercial language models: OpenAI's GPT-3 and Azure's sentiment analysis model. Qualitative analysis shows that AdaTest++ effectively leverages human strengths such as schematization, hypothesis formation and testing. Further, with our tool, participants identified a variety of failures modes, covering 26 different topics over 2 tasks, that have been shown before in formal audits and also those previously under-reported.

翻译：大语言模型通过部署在社会技术系统中，正日益渗透并普及至整个社会。然而，无论是用于分类还是生成任务的语言模型，均已被证实存在偏见且行为不负责任，从而对大众造成规模化危害。因此，对这些语言模型进行严格审计至关重要。现有审计工具依赖人类、人工智能或两者协同来发现失效模式。本研究借鉴人机协作与意义建构领域的文献，并对安全与公平AI领域的专家进行访谈，以此改进由生成式大语言模型驱动的审计工具AdaTest（Ribeiro and Lundberg, 2022）。通过设计过程，我们强调意义建构与人机沟通在协作审计中发挥人类与生成模型互补优势的关键作用。为评估增强工具AdaTest++的有效性，我们开展用户研究，让参与者审计两种商业语言模型：OpenAI的GPT-3与Azure的情感分析模型。定性分析显示，AdaTest++能有效利用人类的模式化、假设形成与验证等优势。此外，借助该工具，参与者在两项任务中识别出涵盖26个不同主题的多种失效模式，其中既包括此前正式审计中已报告的案例，也包含以往被低估的问题。