LLM-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences. We have to assume that not only reviewers are using LLM-assistance, but also that authors use LLMs to revise their papers before submitting. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review (ARR) to evaluate LLM reviews from both the author and the reviewer perspective. First, we identify a limited alignment of LLM reviews with human ones. In the best-case scenario, the alignment is reasonable. However, we also find that LLM-human alignment varies substantially across prompts and models. Finally, we investigate the scenario in which the author uses an iterative draft-revise workflow to improve the submission according to the LLM review. We find that this "gaming" of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35\% of papers. We publish our code: https://github.com/uhh-hcds/reviewarcade.

翻译：大语言模型生成的学术论文评审正获得广泛关注，甚至被主流会议正式试点采用。我们必须假设，不仅审稿人使用LLM辅助评审，作者也可能在投稿前使用LLM修改论文。本研究基于2025年ACL滚动评审(ARR)的论文开展实证实验，从作者和审稿人双重角度评估LLM评审质量。首先，我们发现LLM评审与人工评审存在有限对齐——在最佳情况下对齐程度尚可，但LLM-人工对齐程度会因提示词和模型差异而显著波动。最后，我们探讨了作者采用"草稿-修改"迭代流程、根据LLM评审结果改进投稿的可行性。研究发现，这种对LLM评审的"博弈"策略在特定场景下确实有效，能使最多35%论文的总体评分获得统计显著性提升。相关代码已开源：https://github.com/uhh-hcds/reviewarcade。