ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
CoRR(2024)
摘要
Peer review is a cornerstone of science. Research communities conduct peer
reviews to assess contributions and to improve the overall quality of science
work. Every year, new community members are recruited as peer reviewers for the
first time. How could technology help novices adhere to their community's
practices and standards for peer reviewing? To better understand peer review
practices and challenges, we conducted a formative study with 10 novices and 10
experts. We found that many experts adopt a workflow of annotating,
note-taking, and synthesizing notes into well-justified reviews that align with
community standards. Novices lack timely guidance on how to read and assess
submissions and how to structure paper reviews. To support the peer review
process, we developed ReviewFlow – an AI-driven workflow that scaffolds
novices with contextual reflections to critique and annotate submissions,
in-situ knowledge support to assess novelty, and notes-to-outline synthesis to
help align peer reviews with community expectations. In a within-subjects
experiment, 16 inexperienced reviewers wrote reviews using ReviewFlow and a
baseline environment with minimal guidance. Participants produced more
comprehensive reviews using ReviewFlow than the baseline, calling out more pros
and cons, but they still struggled to provide actionable suggestions to address
the weaknesses. While participants appreciated the streamlined process support
from ReviewFlow, they also expressed concerns about using AI as part of the
scientific review process. We discuss the implications of using AI to scaffold
peer review process on scientific work and beyond.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要