Large language models are becoming increasingly pervasive and ubiquitous in society via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased and behave irresponsibly, causing harm to people at scale. It is crucial to audit these language models rigorously. Existing auditing tools leverage either or both humans and AI to find failures. In this work, we draw upon literature in human-AI collaboration and sensemaking, and conduct interviews with research experts in safe and fair AI, to build upon the auditing tool: AdaTest (Ribeiro and Lundberg, 2022), which is powered by a generative large language model (LLM). Through the design process we highlight the importance of sensemaking and human-AI communication to leverage complementary strengths of humans and generative models in collaborative auditing. To evaluate the effectiveness of the augmented tool, AdaTest++, we conduct user studies with participants auditing two commercial language models: OpenAI's GPT-3 and Azure's sentiment analysis model. Qualitative analysis shows that AdaTest++ effectively leverages human strengths such as schematization, hypothesis formation and testing. Further, with our tool, participants identified a variety of failures modes, covering 26 different topics over 2 tasks, that have been shown before in formal audits and also those previously under-reported.

翻译：大语言模型通过部署在社会技术系统中日益渗透并普及于社会。然而，这些用于分类或生成任务的语言模型已被证明存在偏差且行为不负责任，可能大规模危害人类。因此，严格审计这些语言模型至关重要。现有审计工具或依赖人类、或借助人工智能、或结合两者来发现故障。在本研究中，我们借鉴人机协作与意义建构领域的文献，并对安全与公平AI领域的研究专家进行访谈，以构建基于生成式大语言模型（LLM）的审计工具：AdaTest（Ribeiro和Lundberg，2022）。通过设计过程，我们强调意义建构和人机交流的重要性，以充分发挥人类与生成模型在协作审计中的互补优势。为评估增强工具AdaTest++的有效性，我们开展用户研究，让参与者审计两个商业语言模型：OpenAI的GPT-3和Azure的情感分析模型。定性分析表明，AdaTest++有效利用了人类在图式化、假设形成与检验等方面的优势。此外，借助该工具，参与者在两项任务中识别了涵盖26个不同主题的多种故障模式，这些故障既有此前正式审计中已发现的，也包括先前未充分报告的。