In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled data consistently exhibit superior or comparable performance compared to their synthetically augmented counterparts. Nevertheless, synthetic augmentation proves beneficial, particularly in improving performance on rare classes within multi-class tasks. Furthermore, we leverage GPT-4 and Llama-2 for zero-shot classification and find that, while they generally display strong performance, they often fall short when compared to specialized classifiers trained on moderately sized training sets.

翻译：在计算社会科学领域，实践者常需处理复杂且资源匮乏的场景，面临数据获取与标注成本高昂、耗时漫长的挑战。本研究旨在建立一套应对此类挑战的指导方针，通过十项复杂度各异的CSS分类任务，对比人类标注数据与GPT-4及Llama-2生成的合成数据在模型训练中的表现差异。同时，我们探究了训练数据规模对模型性能的影响。研究结果表明，基于人类标注数据训练的模型始终表现出优于或等同于合成增强数据模型的性能。然而，合成数据增强技术仍具价值——尤其在提升多类别任务中稀有类别的分类效果方面。此外，我们运用GPT-4与Llama-2进行零样本分类发现：尽管这些模型普遍表现强劲，但在面对中等规模训练集训练的专业分类器时仍显逊色。