Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews using ReviewFlow and a baseline environment with minimal guidance. Participants produced more comprehensive reviews using ReviewFlow than the baseline, calling out more pros and cons, but they still struggled to provide actionable suggestions to address the weaknesses. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold peer review process on scientific work and beyond.

翻译：摘要：同行评审是科学的基石。研究社区通过同行评审评估贡献并提升科学工作的整体质量。每年，新社区成员首次被招募为同行评审员。技术如何帮助新手遵守其社区的同行评审实践与标准？为更深入理解同行评审实践与挑战，我们开展了一项包含10名新手和10名专家的形成性研究。研究发现，许多专家采用一种工作流程：标注、记录笔记，并将笔记综合成符合社区标准的合理评审意见。而新手缺乏关于如何阅读和评估投稿、以及如何构建论文评审结构的及时指导。为支持同行评审过程，我们开发了ReviewFlow——一种AI驱动的工作流程，通过上下文反思辅助新手批判性评论和标注投稿、提供即时知识支持以评估新颖性，并通过笔记到大纲的合成帮助对齐同行评审与社区期望。在一项受试者内实验中，16名无经验评审员使用ReviewFlow和仅提供最低指导的基线环境撰写评审。使用ReviewFlow时，参与者撰写的评审比基线更全面，指出了更多优缺点，但仍难以提供解决弱点的可操作建议。尽管参与者赞赏ReviewFlow提供的精简流程支持，但也对在科学评审过程中使用AI表示担忧。我们探讨了使用AI在科学工作及其他领域搭建同行评审流程的启示。