We study the ability of LLMs to generate feedback for scientific papers and develop MARG, a feedback generation approach using multiple LLM instances that engage in internal discussion. By distributing paper text across agents, MARG can consume the full text of papers beyond the input length limitations of the base LLM, and by specializing agents and incorporating sub-tasks tailored to different comment types (experiments, clarity, impact) it improves the helpfulness and specificity of feedback. In a user study, baseline methods using GPT-4 were rated as producing generic or very generic comments more than half the time, and only 1.7 comments per paper were rated as good overall in the best baseline. Our system substantially improves the ability of GPT-4 to generate specific and helpful feedback, reducing the rate of generic comments from 60% to 29% and generating 3.7 good comments per paper (a 2.2x improvement).

翻译：我们研究了大型语言模型生成科学论文反馈的能力，并开发了MARG——一种利用多个大语言模型实例进行内部讨论的反馈生成方法。通过将论文文本分发给不同智能体，MARG能够处理超出基础大语言模型输入长度限制的完整论文内容；同时，通过专业化智能体设计并结合面向不同评论类型（实验、清晰度、影响力）的定制化子任务，该方法提升了反馈的有用性和针对性。用户研究表明，使用GPT-4的基线方法超过半数时间会生成泛泛或极其泛泛的评论，最佳基线方法中每篇论文仅产生1.7条整体质量良好的评论。我们的系统显著提升了GPT-4生成具体且有帮助反馈的能力，将泛泛评论的比例从60%降至29%，每篇论文生成3.7条优质评论（提升2.2倍）。