The rise of AI has fueled growing concerns about ``hype'' in machine learning papers, yet a reliable way to quantify rhetorical style independently of substantive content has remained elusive. Because bold language can stem from either strong empirical results or mere rhetorical style, it is often difficult to distinguish between the two. To disentangle rhetorical style from substantive content, we introduce a counterfactual, LLM-based framework: multiple LLM rhetorical personas generate counterfactual writings from the same substantive content, an LLM judge compares them through pairwise evaluations, and the outcomes are aggregated using a Bradley--Terry model. Applying this method to 8,485 ICLR submissions sampled from 2017 to 2025, we generate more than 250,000 counterfactual writings and provide a large-scale quantification of rhetorical style in ML papers. We find that visionary framing significantly predicts downstream attention, including citations and media attention, even after controlling for peer-review evaluations. We also observe a sharp rise in rhetorical strength after 2023, and provide empirical evidence showing that this increase is largely driven by the adoption of LLM-based writing assistance. The reliability of our framework is validated by its robustness to the choice of personas and the high correlation between LLM judgments and human annotations. Our work demonstrates that LLMs can serve as instruments to measure and improve scientific evaluation.

翻译：人工智能的兴起加剧了人们对机器学习论文中"炒作"现象的担忧，然而，如何独立于实质内容来量化修辞风格，一直缺乏可靠的方法。由于夸张的语言既可能源于扎实的实证结果，也可能仅是修辞风格使然，二者往往难以区分。为分离修辞风格与实质内容，我们提出一个基于反事实的LLM框架：多个LLM修辞角色从相同实质内容生成反事实文本，由LLM评审员通过成对比较进行评估，并采用Bradley--Terry模型对结果进行聚合。将该方法应用于2017年至2025年采样的8,485篇ICLR投稿，我们生成了超过25万篇反事实文本，实现了对机器学习论文修辞风格的大规模量化。研究发现，即使在控制同行评审评价后，愿景式表述仍能显著预测下游关注度，包括引用量和媒体关注度。我们还观察到2023年后修辞强度急剧上升，并通过实证证据表明这种增长主要源于LLM写作辅助工具的普及。该框架的可靠性通过其对角色选择的鲁棒性以及LLM判断与人工标注之间的高度相关性得到验证。我们的工作表明，LLM可作为衡量和改进科学评估的工具。