ArXiv recently prohibited the upload of unpublished review papers to its servers in the Computer Science domain, citing a high prevalence of LLM-generated content in these categories. However, this decision was not accompanied by quantitative evidence. In this work, we investigate this claim by measuring the proportion of LLM-generated content in review vs. non-review research papers in recent years. Using two high-quality detection methods, we find a substantial increase in LLM-generated content across both review and non-review papers, with a higher prevalence in review papers. However, when considering the number of LLM-generated papers published in each category, the estimates of non-review LLM-generated papers are almost six times higher. Furthermore, we find that this policy will affect papers in certain domains far more than others, with the CS subdiscipline Computers & Society potentially facing cuts of 50%. Our analysis provides an evidence-based framework for evaluating such policy decisions, and we release our code to facilitate future investigations at: https://github.com/yanaiela/llm-review-arxiv.

翻译：arXiv近期禁止在其计算机科学领域服务器上传未发表的综述论文，理由是这些类别中LLM生成内容的比例过高。然而，该决策并未提供定量证据支持。在本研究中，我们通过测量近年来综述与非综述研究论文中LLM生成内容的比例来验证这一说法。使用两种高质量的检测方法，我们发现综述与非综述论文中LLM生成内容均显著增加，其中综述论文中的比例更高。但若考虑各类别中发表的LLM生成论文数量，非综述类LLM生成论文的估计值高出近六倍。此外，我们发现该政策对某些领域论文的影响将远大于其他领域，其中计算机科学子学科“计算机与社会”可能面临50%的削减。我们的分析为评估此类政策决策提供了基于证据的框架，并公开代码以促进未来研究：https://github.com/yanaiela/llm-review-arxiv。