Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.

翻译：标注数据在自然语言处理（NLP）中用于训练模型和评估其性能，发挥着关键作用。随着大型语言模型（LLMs）的最新发展，ChatGPT等模型在许多文本标注任务上展现出零样本能力，其表现可与人类标注者媲美甚至超越。这类LLMs因成本更低且可扩展性更强，可作为人工标注的替代方案。然而，目前鲜有研究将LLMs作为互补型标注者，也未深入探讨如何在人类与LLMs之间最优分配标注工作以实现质量与成本的双重目标。我们提出CoAnnotating——一种面向大规模非结构化文本的人类-LLM协同标注的新型范式。在该框架下，我们利用不确定性来评估LLMs的标注能力。实证研究表明，CoAnnotating能在不同数据集上有效分配工作量，相比随机基线方法性能提升高达21%。代码实现详见：https://github.com/SALT-NLP/CoAnnotating。