ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

As an important step for ensuring the scientific quality of work, peer review has been adopted by most journals and conferences (Price and Flach, 2017). In a typical conference review process, each reviewer needs to evaluate the paper’s quality, make an acceptance decision and provide reviews for their assigned papers (Shah et al., 2018; Shah, 2022). The reviewers are usually experts in the area who have fruitful experience and knowledge to assess or evaluate the quality and contribution of the paper. After reviewers submit the review, a discussion takes place between reviewers and a meta-reviewer, who will carry out the final decision on the acceptance of the paper.

The number of papers in research communities has increased exponentially in recent years (Wang et al., 2020). While this may be positively viewed as an acceleration of scientific progress, the disparity between growth rates of the submission and reviewer pools also creates more burden for reviewers (McCook, 2006; Stelmakh et al., 2019; Stelmakh et al., 2021a; Shah, 2022). To avoid overloading reviewers, conferences need to find new sources of reviewers as there are not enough experienced reviewers to review all papers (Stelmakh et al., 2021a). These novice and junior reviewers constitute a large fraction of the reviewer pool in computer science conferences (Stelmakh et al., 2021b). For example, in 2023, the ACM CHI Conference on Human Factors in Computing Systems reported that more than 50% of ACs who reviewed papers are first-time ACs (chi, 2015). Given this large fraction, conferences need to ensure that newly added junior reviewers do not compromise the quality of the process, that is, are able to write reviews of quality comparable to the experienced reviewers.

A previous study explored and compared the reviews written by experienced reviewers versus junior reviewers and the study showed that junior reviewers were slightly harsher in scoring the clarity of the submissions (Shah et al., 2018; Stelmakh et al., 2021a). In the meantime, other works provide empirical evidence that junior reviewers are more critical than their senior counterparts and reveal that graduate students’ review comments are not very useful (Mogul, 2013; Patat et al., 2019). Faced with these doubts and challenges, helping novice junior reviewers to write a high-quality review becomes crucial.

Reviewing is a time-consuming and mentally demanding task (Wang et al., 2020; Shah, 2022). A constructive and comprehensive review can improve the quality of the paper, while a bad, random, dismissive, or biased review brings frustration and anger to authors (Wang et al., 2020). To provide a high-quality review, reviewers need to go through a multi-step workflow, including understanding the contribution of the paper, accessing the merit of scientific contribution and providing an evaluation together with a comprehensive written review (Smith, 2006; Jefferson et al., 2002; Xiong and Litman, 2011; Langford and Guzdial, 2015; Hua et al., 2019; Cheng et al., 2020; Gao et al., 2019; Wang et al., 2020; Yuan et al., 2021). To make a fair judgment, reviewers need to have enough background knowledge to grasp the main idea of the paper and evaluate its contribution. More importantly, reviewers need to equip critical thinking skills to think deeply about the author’s judgments, like whether the claims are reasonable and why the approaches are chosen (Facione, 1990; Moore, 2013). A typical peer review not only contains the paper summary and its contribution but also raises weaknesses from different aspects together with constructive feedback or thought-provoking questions. During this process, it requires readers to actively analyze, synthesize, and evaluate the paper content (Moore, 2013).

Researchers are typically trained extensively in conducting the research itself, but they often lack formal instructions in the peer review process. Hence, it becomes even more challenging for inexperienced junior reviewers to gain expertise quickly (Shah, 2022). Existing research explored instructional methods to “teach” or “train” junior reviewers. A previous study provided a training video and found that it increased the inter-reviewer agreement, alignment with the scoring rubric, and the amount of time reading the review criteria (Sattler et al., 2015). Another study offers novice reviewers a more guided introduction to the different stages of the reviewing process, such as how to lead a discussion among reviewers, to help novices write better reviews. The results showed that with this guidance on the reviewing stages, novice reviewers could deliver more “above expectation” reviews (Stelmakh et al., 2021a). However, their guidance is only limited to introducing the different parts of the reviewing process, such as rebuttal and discussion, and providing novices opportunities to ask expert questions on the general process (Stelmakh et al., 2021a). Outside of the general review process, academic peer review is a complex activity that involves both understanding a paper submission’s stated contributions and evaluating whether the paper crosses an acceptable threshold for the research community. To explore how we might support the peer review process, we start with a formative study to understand how experts approach this task and what novices see as key challenges.

Previous research has used computational methods to provide support to streamline several parts of the peer review process, such as matching submissions with appropriate reviewers or assessing review quality (Bartoli et al., 2016; Wang et al., 2020; Hua et al., 2019; Yuan et al., 2021; Arous et al., 2021; Shah et al., 2018). However, fewer empirical studies that attempt to scaffold the entire workflow for reviewing academic papers, which includes reading, note-taking, evaluating, decision making, and synthesizing this into a written review.

To help novices improve problem-solving skills in complex cognitive tasks, cognitive apprenticeship introduces several strategies, including modeling, coaching, scaffolding, and reflection (Vygotsky and Cole, 1978; Collins, 2006). Scaffolding is instructional support provided by experts to promote learning, especially when concepts and skills are being first introduced to novice students (Reiser, 2004; Collins, 2006). These supports include advanced organizers, modeling, worked examples, concept maps, explanations, handouts, and prompts (Oakes et al., 2018; Nesbit and Adesope, 2013; Atkinson et al., 2000; Charney and Carlson, 1995; Bui and McDaniel, 2015; Collins, 2006). Previous research shows that effective scaffolding can help novices perform work nearly as well as experts (Yuan et al., 2016; Hui and Sprouse, 2023; Hui et al., 2018). When scaffolding is mediated by technology, including AI-based methods, it creates more opportunities for instructors and learners but also brings more challenges in making the scaffolding contextualized, adaptive, and effective (Hannafin et al., 1999).

Researchers used prompts and guided questions to scaffold learners in the paper reading process (Peng et al., 2022; Yuan et al., 2023; Chen et al., 2022). To facilitate critical paper reading, researchers developed CReBot which interactively asks section-level critical thinking questions for routine paper readers and novice readers. Results showed that the interactive question prompts CReBot provided might not be better than static guidelines for beginners to conduct critical thinking. Furthermore, researchers developed CriTrainer which can adaptively provide questions in the reading process together with hints and feedback to help readers critically think and comprehend the paper content. Interestingly, on the opposite of CReBot, their result showed that CriTrainer can improve learners’ ability to raise understandable, relevant, and critical questions after the training sessions. Their results highlighted the benefits of its text-specific critical thinking questions provided by the system. However, guided questions used in CReBot and CriTrainer are both template-based and did not fully use the user-selected content on the paper.

Existing research also used existing examples to scaffold complex writing processes (Yuan et al., 2016; Hui and Sprouse, 2023; Hui et al., 2018). Scholars found that scaffolding can help students learn about form and organization by analyzing the examples and templates (Doan, 2021; Collins, 2006). In the context of writing introductory help requests, providing high-quality examples and expert-informed templates can increase learning and writing quality (Hui et al., 2018). Another writing support system used “scaffolded annotation” that broke down examples into each component to help professional writing students improve their early-stage drafts (Hui and Sprouse, 2023). In traditional instruction scenarios, experts take a large amount of time to curate examples, create guidance, or author rubrics (Reiser, 2004). However, experts who created examples still faced the challenges of effectively adapting and contextualizing into the current learning step.

Advances in AI and LLMs provide opportunities to provide efficient and context-specific scaffolding in the learning process (Ippolito et al., 2022; Chung et al., 2022; Clark et al., 2018; Gero and Chilton, 2019b, a; Rahman et al., 2020; Lee et al., 2022; Gero et al., 2021; Ippolito et al., 2022; Mirowski et al., 2022). For example, TaleBrush allowed users to create a story with AI through sketching to aid the planning of writing. Then it enabled writers to generate diverse storylines and interactively refine them (Chung et al., 2022). Another system Wordcraft explored how to support users collaborating with generative language models to co-write a story (Yuan et al., 2022). Spark used a language model to generate prompts related to a scientific concept to facilitate scientific writing (Gero et al., 2021). However, as far as we know, none of the studies that used scaffolding strategies focused on the context of conference peer review writing. Our research explores how we might guide a complex, multi-faceted workflow like peer reviewing and the potential role of AI in creating contextually adaptive scaffolds.

Writing is a complex, iterative process (Flower and Hayes, 1981). The Hayes model describes the cognitive processes an individual writer engages in during the process of writing (Flower and Hayes, 1981; Hayes, 2012; Gero et al., 2022). In the cognitive process of writing, there are three major components: planning, translating, and reviewing, as shown in Figure 5. Several systems are developed to facilitate different stages of writing (Bhat et al., 2023; Zhang et al., 2023). For example, VISAR is an AI-enabled writing assistant that helps writers brainstorm and revise hierarchical goals and organize argument structures in the planning stage. To facilitate the iterative planning and revising process in writing, intelligent systems further use the chain of thoughts (COT) prompting method to break down the large problem into step-by-step prompts (Wei et al., 2022). Specifically, the Re3 framework and DOC framework used the COT approach to decompose the writing tasks where they first generate an outline and then automatically turn the outline into the story generation (Yang et al., 2022b, a). Their evaluation demonstrated that the decomposition approach can improve the coherence of long story generation and is highly controllable where humans can control the story generation by modifying the outlines.

Control and agency are extremely important in the conference peer review writing process, as human reviewers should play the role of driving the writing process. While the development of LLMs can provide scaffolding opportunities for this writing process, it is important to address the concerns and limitations of LLMs in this context (Fok and Weld, 2023). The prior work that aligns most closely with the concept of applying AI techniques to reviewing academic papers that a machine model that automatically generates feedback using LLMs. Results showed that LLM feedback could benefit researchers in earlier stages of manuscript preparation while researchers struggle with an in-depth critique of study methods (Liang et al., 2023). Instead of using an automatic method, humans should be in the loop to drive and control the writing process (Fok and Weld, 2023). Drawing on these insights, we designed the ReviewFlow system not to automate any parts of the process, but rather to scaffold key considerations and to give novices agency over how to apply machine-generated language suggestions.

From the interview study with novices and the observational study with experts, we identified a range of challenges perceived by novices as well as insights on expert practices (see Figure 2). Novices lack confidence in identifying the novelty of research based on prior research and in knowing how to structure a peer review that meets community standards. Experienced reviewers tend to avoid biasing the decision-making and preserve agency in reviewing. The juxtaposition of these novice challenges and the expert practices suggests four core design goals for intelligent scaffolding:

DG1: Offer lightweight and adaptive scaffolds that facilitate reflection throughout the paper Novices highlighted their need for guidance in critically evaluating papers from various perspectives, expressing uncertainty about the key points to focus on (C1). In educational settings, instructors typically serve as scaffolds when introducing new concepts or knowledge to novice learners (Collins, 2006). Prior studies in scientific paper reading developed methods to offer contextual hints or guided questions that encouraged reflection and critical thinking (August et al., 2022; Rachatasumrit et al., 2022). For example, Paper Plain provided a collection of key questions that guide readers to answering passages and plain language summaries of those passages (August et al., 2022). Similarly, in the context of our study, the objective is to provide guidance comparable to that of expert peer reviewers. The emphasis is on delivering locally relevant questions that help users think critically about each section of the paper.

DG2: Enable in-situ knowledge support for assessing novelty compared to prior work Novices struggled with insufficient background knowledge for evaluating papers (C2). Providing knowledge support in situ can externalize the user’s working memory, aid in sense-making, and facilitate a swift reviewing and resumption of task contexts (Kuznetsov et al., 2022). Prior studies in scientific literature review highlight the importance of in-situ knowledge support (Kang et al., 2022; Chang et al., 2023). To help novice reviewers who lack background knowledge to evaluate the novelty while reading the paper, our goal is to enable in-situ knowledge support by extracting the abstract of the cited paper.

DG3: Model the expert workflow of reading, evaluating, and synthesizing practiced by experienced reviewers Novices expressed their need for more specific guidance from experts in terms of their process and structure (C3). Prior work showed that surfaced expert practices can better structure and scaffold the process for novices (Kim et al., 2015). Motify illustrates that storytelling patterns extracted from expert stories can be used to effectively scaffold novices to create video stories (Kim et al., 2015). In the context of writing introductory help requests, providing high-quality examples and expert-informed templates can increase learning and writing quality (Hui et al., 2018). Our approach involves the modeling of the expert workflow encompassing reading, evaluating, synthesizing, and revising. This model is then employed to structure the peer review process for novice reviewers.

DG4: Guide the reviewing process such that the work aligns with community standards Novices reported that they lacked effective models for tone and expectation (C4). More guidance on the expectation of the review can help reviewers to reflect on their tone and structure. In addition, experts raised valid concerns about the potential for intelligent tools to introduce biases or influence the thinking process. Existing studies have highlighted apprehensions regarding AI exhibiting biases in human-AI collaboration tasks (Kawakami et al., 2022; Poursabzi-Sangdeh et al., 2021). Notably, research has shown that providing information, including explanations, generated by AI has the potential to mislead users in decision-making (Lakkaraju and Bastani, 2020; Goyal et al., 2023; Buçinca et al., 2021). Therefore, a key design objective is to steer clear of biasing the decision-making process and focus solely on encouraging justifications. This approach aims to assist novice reviewers in producing high-quality and original reviews without compromising the integrity of the evaluation process.

ReviewFlow’s front-end is a React web application built on top of an existing PDF highlighting library ²²2https://github.com/agentcooper/react-pdf-highlighter. The front end is responsible for displaying the PDF, managing the annotation data, and displaying the text input used to write the review draft. The back-end uses a Flask server which handles the GPT-4 ³³3https://openai.com/research/gpt-4 API endpoints, MongoDB endpoints for storing data, and GROBID (gro, 2008) for PDF data extraction. This data includes all of the parsed text of the PDF, its PDF coordinates, and citations linked to their references. Detailed prompts are provided in the Appendix.

We conducted a within-subjects experiment with 16 novice participants where each participant experienced both the ReviewFlow condition and the Baseline condition in two sessions separately. We counterbalanced the order of two conditions and papers using a Latin Square design to minimize the potential order effects. To reduce knowledge transfer and minimize fatigue, we scheduled the two study sessions of each participant at least 24 hours apart. The ReviewFlow condition includes all scaffolding features, while the Baseline condition includes the minimal guideline. In the baseline interface, users can still highlight, take notes, and tag notes. Community guideline is also provided.

To compare the quality of written review in both conditions, we performed ANCOVA tests to examine the effect of the two conditions on each quality measure, accounting for the review length, participants’ knowledge of the topic, the order of the experiment condition as co-variants. As shown in Figure 7, we found that the reviews in the ReviewFlow condition are significantly more comprehensive than the Baseline condition (p = 0.04$*$, F = 3.55). We found no significant interaction effect between the order of the experiment conditions and no statistically significant interaction effect between the knowledge of the two papers.

Furthermore, we evaluated several proxies of review quality including the number of strengths and weaknesses and participants’ self-rated satisfaction with the written review. As shown in Table 1, participants wrote longer reviews and called out more strengths and weaknesses in the paper in the ReviewFlow condition. However, there is no significant improvement in the constructiveness of the review. This indicates that the ReviewFlow system can scaffold participants to capture more pros and cons but still cannot help participants write constructive solutions for each weakness. Also, participants’ self-rated satisfaction with reviews written using the ReviewFlow tends to be higher than using the Baseline. ANCOVA test with proxies did not a show significant effect. Participants reflected on the reason that “I captured more pros and cons since I took more notes this time, so I feel satisfied with the review as it covered more aspects” [P3].

After participants had experienced the two systems, we asked them to compare two conditions. As shown in Figure 8, participants highly preferred ReviewFlow over the Baseline. 93.7% of participants perceived the interface as enjoyable to use. Participants mentioned that the scaffolding features make the review process more engaging and less boring (N=5). All participants think that the system is useful and they like the guidance provided in reviewing. For instance, P11 described the guidance they received as similar to experts – “I feel like some experts, such as my advisor, were sitting next to me to prompt me in particular ways and show me how to write the review”[P11].

We further measured participants’ perceptions of the tasks in each condition. In the post-survey, we asked participants to rate cognitive workload, such as distraction and engagement, the feeling of control and collaboration with the interface, and perceived learning gains. As shown in Figure 9, 75% of the participants reported that they are engaged in the process, but the add-on scaffolding features bring in some distraction for 68.8% of the participants. However, participants highlighted that it is not a bad distraction, but served as a staging process that motivated them to think. For instance, P8 explained that “Since you don’t want to just keep going through the paper without getting anything from it. The distraction is like a pop-up that keeps prompting you about review criteria from different places”[P8].

All participants believe that the ReviewFlow helped them learn the peer review process. 68.8% of participants have the feeling of collaborating with the interface. Participants identified the collaboration mainly happened during the process of answering contextual cues in each section and using the note-to-outline synthesis feature. The two-step process where participants can first summarize notes into a high-level outline and then expand it provided them the feeling of “iterating with an assistant”[P9]. Even though the current system uses AI-generated outlines, all participants agreed that they still have the control over writing process. One participant mentioned the reason is that “It just synthesized what I have already written in the notes and I can still write it by myself”[P9]. 37.5% of the participants described that the notes synthesis did not bias them in the current session, but they still worried that “busy reviewers might randomly create notes and use the outline to make a decision”[P14].

AI has come a long way since the early and annoying attempts at supporting work (e.g. Clippy). With the development of LLMs, AI can automate or augment many aspects of knowledge work, including information discovery, sensemaking, and writing (Palani et al., 2023; Liu et al., 2023a; Zhang et al., 2023). While LLMs have become incredibly valuable, the risk now might the risk now might be the tendency, especially among tech-focused innovators, of taking automation too far and dehumanizing work. Prior research has explored whether models can predict a paper decision or even draft a review (Liang et al., 2023). Our ReviewFlow system was designed with the intention of balancing the use of technology with human values. Our system strikes this balance by taking cues from the learning sciences research on scaffolding (Saye and Brush, 2002; Holton and Clarke, 2006). The goal of ReviewFlow is not to produce reviews, per se, but to convey an understanding of how to think while reading a submission and writing a review. Our values place more emphasis on training and preparing novices (perhaps for their next peer review), not just on getting to a final peer review.

Prior work indicates that well-design scaffolding can help novices operate more like experts (Yuan et al., 2016). The concept of Zone of Proximal Development (ZPD) represents the space between what a learner is capable of doing unsupported and what the learner cannot do even with support (Harland, 2003; Wass et al., 2011). Scaffolding works most effectively when it meets novices when they need support and tapering off when they have internalized the best practices. The scaffolding in ReviewFlow is intelligent and contextual. For instance, users get reflection cues that adapt specifically to the current section.

ReviewFlow was explicitly designed to support novices, but it does not adapt to someone’s existing knowledge or experience with the task. A longitudinal deployment would provide insight on whether novices continue to prefer ReviewFlow, or whether it is useful for early experiences with peer reviewing. As novices become more proficient, they may eventually prefer to work with less structure (e.g. traditional text editor), although even old timers likely find value in ReviewFlow’s pragmatic support for capturing and synthesizing notes. Future research could extend the current system to be more context-aware and more adaptive to users and then study its use with both novices and experienced reviewers over time to gain design insights for interactive scaffolding. Furthermore, future work could explore other properties of scaffolding, such as the timing of when it’s provided, the communication modality, the format or representations used, or the strategies leveraged, such as comparing generated examples vs. guiding questions.

ReviewFlow incorporates three types of scaffolding to support cognitive activities: contextual scaffolding generates reflection questions to aid in paper reading; knowledge scaffolding focuses on citations to facilitate paper evaluation; and structural scaffolding synthesizes notes into structured outlines to assist in review writing. The current study provided insights into user perceptions of different scaffolding by observing their usage of all the features and asking them to rate each feature’s usefulness and accuracy. More rigorous and well-controlled ablation studies could help us understand the underlying factors impacting the ReviewFlow experience. For instance, a study with three experimental conditions, each providing only one type of scaffolding—contextual scaffolding, knowledge scaffolding, and structural scaffolding—would allow for a more comprehensive evaluation of the quality of review writing, task efficiency, and the learning impacts for the user.

Our study of ReviewFlow indicates novice reviewers can benefit from intelligent scaffolding: it helps them evaluate the submission and write more comprehensive reviews. Participants mentioned how they gradually learned about the research community’s practices and became more confident in their review writing. Beyond just writing reviews, the strategies for intelligent scaffolding built into ReviewFlow have the potential to provide value to other aspects of the peer review ecosystem (Shah et al., 2018). For example, similar intelligent scaffolding can be used to support novice reviewers to revise their papers and write a persuasive rebuttal. The process of meta-reviewing, or summarizing independent reviews, could also benefit from scaffolding since there are new people jumping into that role each cycle (Sun et al., 2024).

Future deployments of this type of intelligent scaffolding would require careful consideration and round-table discussions within the research community. Previous research revealed that writing with an opinionated language model can affect participants’ attitudes towards social topics in writing (Jakesch et al., 2023). In our study, even though our scaffolding was carefully designed to allow the user to drive the process and to avoid biasing the decision, we still heard participants who were concerned about other people using such a system. Interestingly, very few participants were concerned that their own decision or writing was being biased by the AI, instead, they worried about how others would appropriate the intelligent support and how it might erode a community’s trust in the peer review system. To deploy the system in the future and build trust with people in the community, we need to make sure that the system is robust and trustworthy (Wang et al., 2023). A real-world deployment would need to have a large-scale consent process and commitment from key stakeholders.

Beyond its application in peer reviewing, intelligent scaffolding could support a range of of complex knowledge work in science, such as paper reading, literature reviewing, sense-making, and paper writing (Palani et al., 2023; Liu et al., 2023a; Zhang et al., 2023; Lin et al., 2024). Notable projects like the semantic reader project have developed interactive and dynamic reading interfaces to aid paper reading and citation discovery (Lo et al., 2023; Kang et al., 2022; August et al., 2022; Fok et al., 2023). These studies suggest broader possibilities for incorporating intelligent, process-driven scaffolding into science work.

Generative AI and LLMs introduce numerous ethical considerations in the design of human-AI collaboration systems. In the context of academic publishing and peer review, a primary concern involves the potential violation of academic integrity and harm to authorship when directly incorporating automatically generated content into writing artifacts. We try to mitigate this in ReviewFlow by constraining the use of LLMs to primarily sensemaking activities, like guided note-taking. In the final step, ReviewFlow synthesizes an outline, not paragraphs, even though LLMs are quite capable of doing so. To limit data sharing, ReviewFlow only sends the LLM the reviewer’s notes and highlighted segments, not the full paper. While individual reviewers may still choose to (mis)use LLMs in these ways, ReviewFlow subtly prioritizes learning and thinking over getting the job done fast.

Another concern revolves around the tendency of LLMs to create inaccurate information or to mislead people(Huang et al., 2019; Jakesch et al., 2023). In line with our scaffolding strategy, ReviewFlow includes a checklist pop-up that allows users to engage in self-reflection, with reminders to proofread and fact-check their review. To enhance transparency in the writing process, future systems might explore the development of intelligent highlighting. For example, highlights could directly color-code the portions that were user-generated and those that were automatically generated, providing a clear visual indication and promoting a more transparent and accountable writing process.

Given the uncertainties, future LLM-powered tool designers should consider strategies for inducing users into a “reflective skepticism” around all data, especially those produced by machine models. As exemplified in prior work  (Lai, 2011), this involves fostering a mindset of critical evaluation and thoughtful questioning to counteract the potential limitations, hallucinations, biases, or misinformation that may arise from these language models.