This work contributes to the expanding research on the applicability of LLMs in social sciences by examining the performance of GPT-3.5 Turbo, GPT-4, and Flan-T5 models in detecting framing bias in news headlines through zero-shot, few-shot, and explainable prompting methods. A key insight from our evaluation is the notable efficacy of explainable prompting in enhancing the reliability of these models, highlighting the importance of explainable settings for social science research on framing bias. GPT-4, in particular, demonstrated enhanced performance in few-shot scenarios when presented with a range of relevant, in-domain examples. FLAN-T5's poor performance indicates that smaller models may require additional task-specific fine-tuning for identifying framing bias detection. Our study also found that models, particularly GPT-4, often misinterpret emotional language as an indicator of framing bias, underscoring the challenge of distinguishing between reporting genuine emotional expression and intentionally use framing bias in news headlines. We further evaluated the models on two subsets of headlines where the presence or absence of framing bias was either clear-cut or more contested, with the results suggesting that these models' can be useful in flagging potential annotation inaccuracies within existing or new datasets. Finally, the study evaluates the models in real-world conditions ("in the wild"), moving beyond the initial dataset focused on U.S. Gun Violence, assessing the models' performance on framed headlines covering a broad range of topics.

翻译：本研究通过零样本、少样本及可解释提示方法，系统评估了GPT-3.5 Turbo、GPT-4与Flan-T5模型在新闻标题框架偏差检测中的表现，为拓展大语言模型（LLMs）在社会科学领域的应用研究提供了新见解。关键发现是：可解释提示方法能显著提升模型可靠性，凸显了可解释性设置对框架偏差社会科学研究的重要性。GPT-4在提供相关性领域内示例的少样本场景中表现尤为突出；而FLAN-T5的低效表现则表明，小型模型需要针对框架偏差检测任务进行专项微调。研究同时发现，各模型（尤其GPT-4）常将情感化语言误判为框架偏差信号——这揭示了区分新闻标题中"真实情感表达"与"策略性框架偏差"的认知挑战。我们进一步在两组标题子集（分别对应明确/存在争议的框架偏差判定场景）中评估模型，结果表明这些模型能有效标记现有或新增数据集中的潜在标注误差。最后，本研究突破最初聚焦美国枪支暴力的数据集限制，在涵盖多主题框架标题的真实场景（"野外测试"）中评估模型性能。