Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset's high quality.

翻译：许多自然语言处理（NLP）任务依赖标注数据来训练高性能机器学习模型。然而，数据标注既耗时又昂贵，尤其是当任务涉及大量数据或需要专业领域知识时。近期，GPT-3.5系列模型已在各种NLP任务中展现出显著的少样本和零样本能力。本文首先提出，当提供充分指导和示例时，大型语言模型（如GPT-3.5）可作为优秀的众包标注者。据此，我们提出了AnnoLLM——一种由大型语言模型驱动的标注系统，采用“先解释后标注”的两步方法。具体而言，我们首先提示大型语言模型对给定示例为何被分配特定真实答案/标签进行解释；随后，利用自生成解释构建少样本思维链提示，并用于标注未标注数据。我们在三个任务（用户输入与关键词相关性评估、BoolQ、WiC）上的实验结果表明，AnnoLLM超越或达到了与人类众包标注者相当的性能。此外，我们利用AnnoLLM构建了首个基于对话的信息检索数据集。该数据集旨在促进能够检索对话文本相关文档的检索模型的开发。人工评估验证了该数据集的高质量。