Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond.

翻译：摘要：同行评审是科学的基石。研究社区通过同行评审评估研究贡献并提升科学工作的整体质量。每年，新成员首次被招募为同行评审员。技术如何帮助新手遵循其社区的同行评审实践与标准？为深入理解同行评审的实践与挑战，我们进行了一项形成性研究，涉及10名新手和10名专家。研究发现，许多专家采用标注、笔记记录及综合笔记形成符合社区标准的合理评审工作流。新手在如何阅读和评估投稿、如何构建论文评审方面缺乏及时指导。为支持同行评审过程，我们开发了ReviewFlow——一种由AI驱动的工作流，通过情境化反思帮助新手批判性评估并标注投稿、通过现场知识支持评估创新性、并通过笔记到大纲的整合帮助评审与社区期望对齐。在一项被试内实验中，16名无经验评审员在两种条件下撰写评审：使用ReviewFlow与使用仅提供最小指导的基线环境。使用ReviewFlow时，参与者撰写了更全面的评审，识别出更多优缺点。尽管参与者赞赏ReviewFlow提供的简化流程支持，他们也表达了在科学评审过程中使用AI的担忧。我们讨论了使用AI构建同行评审过程对科学工作及其他领域的影响。