High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.

翻译：雇佣或众包产生的高昂标注成本，使得构建用于训练可靠文本分类器所需的大规模高质量数据集变得复杂。近期研究表明，利用大型语言模型（LLMs）自动化标注过程，可在保持数据质量的同时显著降低成本。LLMs在仇恨言论检测、政治框架分析等下游任务的标注中已展现出良好效果。基于这些领域的成功经验，本研究探讨LLMs是否适用于媒体偏见检测这一复杂任务的标注，以及能否利用此类数据训练下游的媒体偏见分类器。我们构建了首个面向媒体偏见分类的大规模数据集annolexical，其中包含超过48000个经合成标注的样本。基于该数据集微调的分类器，在Matthews相关系数（MCC）上超越所有标注用LLMs模型5-9个百分点，并在两个媒体偏见基准数据集（BABE与BASIL）的评估中，表现接近甚至优于基于人工标注数据训练的模型。本研究证明了该方法能显著降低媒体偏见领域数据集构建乃至分类器开发的成本，同时通过后续的行为压力测试揭示了当前方法存在的若干局限性与权衡。