The zero-shot learning capabilities of large language models (LLMs) make them ideal for text classification without annotation or supervised training. Many studies have shown impressive results across multiple tasks. While tasks, data, and results differ widely, their similarities to human annotation can aid us in tackling new tasks with minimal expenses. We evaluate using 5 state-of-the-art LLMs as "annotators" on 5 different tasks (age, gender, topic, sentiment prediction, and hate speech detection), across 4 languages: English, French, German, and Spanish. No single model excels at all tasks, across languages, or across all labels within a task. However, aggregation techniques designed for human annotators perform substantially better than any one individual model. Overall, though, LLMs do not rival even simple supervised models, so they do not (yet) replace the need for human annotation. We also discuss the tradeoffs between speed, accuracy, cost, and bias when it comes to aggregated model labeling versus human annotation.

翻译：大语言模型的零样本学习能力使其无需标注或监督训练即可胜任文本分类任务。众多研究表明，这类模型在多项任务中取得了显著成果。尽管任务、数据及结果差异显著，但其与人类标注的相似性有助于我们以最低成本应对新任务。我们评估了5种最先进的大语言模型（作为"标注器"）在5项任务（年龄、性别、主题、情感预测及仇恨言论检测）中的表现，覆盖英语、法语、德语和西班牙语四种语言。没有任何单一模型能在所有任务、跨语言场景或任务内所有标签维度上表现卓越。然而，针对人类标注者设计的聚合技术显著优于任何单个模型。总体而言，大语言模型仍无法媲美甚至简单的监督模型，因此（目前）无法替代人类标注的需求。此外，我们还探讨了聚合模型标注与人类标注在速度、准确性、成本及偏差之间的权衡关系。