Several studies have examined the use of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), reporting mixed accuracy. However, questions of reliability remain largely unaddressed. In this study, we go beyond quantitative LLM-human agreement metrics and qualitatively investigate how and why LLMs fail. We also propose actionable recommendations. We analyzed disagreements between LLMs and researchers across six software engineering SRs and over 1,000 primary study papers. For each SR, papers were screened independently by human experts and LLMs in zero-shot mode, resulting in Kappa values ranging from 0.52 to 0.77. Qualitative analysis suggests that human-LLM disagreement results from recurring, identifiable causes, such as boundary ambiguity in key terms, keyword overemphasization, and incorrect topic inference. Based on these findings, we propose recommendations such as validating semantic understanding before deployment, running multiple LLMs, and focusing validation efforts on borderline cases. Future studies are needed to validate the impact of our recommendations, and community efforts are needed to develop normative guidelines on LLM usage in SRs.

翻译：多项研究探讨了在系统评价（SRs）中使用大语言模型（LLMs）进行题目-摘要筛选，报告了参差不齐的准确性。然而，可靠性问题在很大程度上仍未得到解决。在本研究中，我们超越了定量的人机一致性指标，定性研究了大语言模型为何以及如何失败。我们还提出了可操作的建议。我们分析了六个软件工程系统评价中LLMs与研究者之间超过1000篇主要研究论文的分歧。对于每个系统评价，论文由人类专家和LLMs以零样本模式独立筛选，所得Kappa值范围为0.52至0.77。定性分析表明，人机分歧源于反复出现的可识别原因，例如关键术语的边界模糊、关键词过度强调以及主题推断错误。基于这些发现，我们提出了建议，例如在部署前验证语义理解、运行多个LLM，以及将验证工作聚焦于边缘案例。未来需要研究验证我们建议的影响，并需要社区努力制定关于在系统评价中使用LLM的规范性指南。