Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review

Robert F. Bornstein

doi:10.1037/arc0000076

Bornstein, Robert F.

2021-07-01 00:00:00

Archives of Scientific Psychology 2021, 9, 1–11 © 2021 American Psychological Association DOI: https://doi.org/10.1037/arc0000076 2169-3269 Archives of Scientific Psychology www.apa.org/pubs/journals/arc Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review Robert F. Bornstein Derner School of Psychology, Adelphi University AB S T R AC T Recent findings indicate that many adults in the United States are skeptical regarding scientific evidence. Some of this skepticism is rooted in political partisanship, but distrust of research findings exists on both ends of the political spec- trum. One way to begin to restore faith in the validity and utility of scientific research is to take a closer look at the effec- tiveness of the manuscript-evaluation process. To that end, this article examines current findings regarding the strengths and limitations of manuscript review and identifies ways that the process can be improved. To move toward a more evi- dence-based framework, we must take into account two psychological dynamics that shape the review process. First, manuscript review is what is known in psychology as a “signal-detection task”: Reviewers and editors attempt to distin- guish signals (publishable manuscripts) from noise (manuscripts that do not warrant publication). Second, manuscript review involves a complex social interaction among authors, reviewers, and editors, each of whom brings particular goals to the endeavor and each of whom must communicate with multiple audiences simultaneously to achieve those goals. The practical implications of the findings in this area are discussed, and preliminary steps for moving toward a more evi- dence-based approach to manuscript review are outlined. S CIE N T IF IC A BS T RA CT Given the critical importance of manuscript review, the development of a more evidence-based approach is needed. Manuscript review is the final step in the research process, and limitations in this aspect of the process may have a nega- tive impact on the quality of published research in psychology and in other disciplines as well. This article examines cur- rent findings regarding the strengths and limitations of manuscript review and identifies ways that the process can be improved. To move toward a more evidence-based framework, the current conceptualization of the review process as a method for evaluating manuscripts must be complemented by an appreciation of two dynamics that help shape this pro- cess. First, manuscript review is a signal-detection task wherein reviewers and editors attempt to distinguish signals (pub- lishable manuscripts) from noise (manuscripts that do not warrant publication). Second, manuscript review involves a complex social interaction among authors, reviewers, and editors, each of whom brings particular goals to the endeavor and each of whom must communicate with multiple audiences simultaneously to achieve those goals. The practical impli- cations of research in this area are discussed, and preliminary steps for moving toward a more evidence-based approach to manuscript review are outlined. Keywords: manuscript review, peer review, evidence-based practice, signal detection, strategic self-presentation Peer review as a method for assessing the quality of scientiﬁc work the new scientific societies and academies of the 17th century were crucial prior to publication began more than 300 years ago. As Zuckerman for the social invention of the scientific journal.. . . From the earlier prac- tice of simply putting manuscripts into print, without competent and Merton (1971, p. 68) noted: This article was published July 1, 2021. Robert F. Bornstein https://orcid.org/0000-0001-6203-225X Correspondence concerning this article should be addressed to Robert F. Bornstein, Derner School of Psychology, Adelphi University, 212 Blodgett Hall, Garden City, NY 11530, United States. Email: bornstein@adelphi.edu This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. BORNSTEIN evaluation of their content by anyone but the author himself, there slowly attempt to assess the quality of submitted manuscripts and identify developed the practice of having the substance of manuscripts legitimated those that merit publication in a particular journal. Beyond manuscript .. . through evaluation by institutionally assigned and ostensibly compe- quality, other evaluative criteria are sometimes used, such as research tent reviewers. importance, novelty, and potential impact. These evaluative criteria are often speciﬁed by a particular journal (e.g., when reviewers are Although the initial impetus behind the development of scientiﬁc asked to complete a series of manuscript ratings as part of their journals was efﬁciency—articles presented new information more assessment). In other instances, individual reviewers simply choose to rapidly and succinctly than did books—over time, the function of employ additional criteria (e.g., potential impact) when formulating journals evolved into a mechanism by which readers could judge the their assessments of manuscript quality. worth of scientiﬁc contributions: Better studies got published in more In addition to assessing manuscript quality, many reviewers and selective journals. Peer review remains the accepted procedure for editors strive to enhance the quality of submitted manuscripts by pro- vetting the scientiﬁc rigor and importance of manuscripts submitted viding constructive feedback to authors. This goal of “quality for publication, and it is also the process by which scientists them- improvement” is applied more commonly in the social sciences than selves are evaluated. Career success (e.g., promotion) and access to in the physical and biomedical sciences, where there is less emphasis resources (e.g., grant funding) depend largely on one’s ability to navi- on providing feedback to enhance the quality of submitted manu- gate the manuscript-review process and publish in the most selective scripts. As a result, reviews in the physical and biomedical sciences and prestigious journals. tend to be briefer and focus primarily (sometimes exclusively) on enu- As Tumin and Tobias (2019) and others observed, given increasing merating the strengths and limitations of the research reported in the pressure to publish coupled with a relative scarcity of journal pages, manuscript (see Park, 2009). manuscript review has become more complex and challenging. In With respect to process, manuscript review has traditionally been recent years, there have been numerous reports of problematic behav- conceptualized as an assessment procedure, akin to that in which ior on the part of authors (Lawrence, 2003), reviewers (Wicherts et clinicians engage when rendering a diagnosis (Bornstein, 1991) and al., 2012), and editors (McCarty, 2002; and at least a few notable that in which organizations engage when selecting among job appli- instances wherein manuscripts that eventually became highly inﬂuen- cants (Evans & Woolridge, 1987). As in these other tasks—which, tial received negative initial reviews (see, e.g., Garcia, 1981; Horro- like manuscript reviews, involve rendering a dichotomous yes/no bin, 1990). Those who have scrutinized the manuscript-review decision—reviewers are presumed to base their assessments on system in psychology and other disciplines are divided with regard to evidence, to try to minimize the impact of potentially biasing in- their overall assessment of the current situation. At the more positive formation (e.g., the gender of a patient, job applicant, or author), end, while acknowledging the challenges inherent in manuscript and to be as dispassionate as possible when rendering a decision. review, some have concluded that overall, the system works reason- ably well (see Ramam, 2016). Rennie (2016, p. 31) offered a less op- Manuscript Review as a Form of timistic perspective, noting: Evidence-Based Practice Peer review is touted as a demonstration of the self-critical nature of sci- The concept of evidence-based practice (EBP) originated in clinical ence. But it is a human system. Everybody involved brings prejudices, settings, ﬁrst medicine, then psychology and other allied health disci- misunderstandings and gaps in knowledge, so no one should be surprised plines; as a result, most deﬁnitions of EBP emphasize its clinical fea- that peer review is often biased and inefficient. It is occasionally corrupt, tures. The American Psychological Association (APA) deﬁnes EBP sometimes a charade, an open temptation to plagiarists. Even with the in psychology as “integration of the best available research with clini- best of intentions, how and whether peer review identifies high-quality science is unknown. It is, in short, unscientific. cal expertise in the context of patient characteristics, culture, and pref- erences” (APA, 2006, p. 273). In clinical contexts, EBP is most Given the critical importance of manuscript review, the develop- commonly applied to treatment (Gaudiano & Miller, 2013), but in ment of a more evidence-based framework is needed. This article dis- recent years, it has also been used to enhance psychological assess- cusses the strengths and limitations of manuscript review and ment and diagnosis (Keeley et al., 2016). EBP has also been utilized identiﬁes ways that the process can be improved. To move toward a to inform work in organizational and forensic settings, as well as in more evidence-based approach, the current conceptualization of the management, education, nursing, and other disciplines (see Bennett & review process as a method for evaluating manuscripts must be com- Bennett, 2000). plemented by an appreciation of two dynamics that shape this pro- It is time to use the principles of EBP to move toward a more evi- cess. First, manuscript review is a signal-detection task wherein dence-based framework for manuscript review. Based on extant deﬁ- reviewers and editors attempt to distinguish signals (publishable nitions of EBP within and outside psychology, it is possible to offer a manuscripts) from noise (manuscripts that do not warrant publica- working deﬁnition of evidence-based manuscript review (EBMR), as tion). Second, manuscript review involves a complex social interac- follows: tion among authors, reviewers, and editors, each of whom brings particular goals to the endeavor and each of whom must communicate EBMR involves integration of the best available research on manuscript with multiple audiences simultaneously to achieve those goals. The review with the reviewer’s scientific expertise, taking into account the practical implications of research in this area are discussed, and initial intra- and interpersonal dynamics of the review process, and the context steps toward developing a more evidence-based framework for manu- within which a review is carried out (e.g., the focus, readership, and selec- script review are offered. tivity of a particular publication outlet). Manuscript Review: Goals and Process Manuscript review is a narrower term than peer review, the latter also used to describe assessment of grant proposals, performance within an organization, The goals of manuscript review vary somewhat across disciplines. and other forms of peer evaluation. Despite these differences, the terms are A universal feature of manuscript review is that reviewers and editors often used interchangeably in studies of the manuscript-review process. This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. EVIDENCE-BASED MANUSCRIPT REVIEW A key assumption of this deﬁnition is that manuscript review is a Absence of Compelling Validity Evidence form of assessment and should therefore fulﬁll the criteria for evi- This criticism actually involves two different (albeit related) issues. dence-based assessment (Bornstein, 2017; Rennie & Flanagin, 2018). First, it is difﬁcult to operationally deﬁne the concept of validity in These include (a) construct validity (evidence should conﬁrm that the context of manuscript reviews: What, exactly, should reviewer reviewer ratings and editorial judgments actually predict manuscript and editorial ratings predict? The most common criterion employed quality) and (b) scientiﬁc utility (reviewer and editorial comments by those who study the manuscript-review process is research quality should have a tangible positive impact on the quality of articles pub- (see Alam et al., 2011; Ernst & Resch, 1994), although some have lished). Scientiﬁc utility in EBMR is analogous to clinical utility in argued that the validity of manuscript reviews is best assessed by the EBP literature. including indices of research impact as well (Albers et al., 2011; Elson et al., 2020). The outcome criteria employed by researchers in Best Available Research: Criticisms of the Review examining the validity of manuscript review include citation metrics Process, and Evidence Bearing on These Criticisms for published papers (Siler et al., 2015); postpublication evaluations of the quality and impact of articles by others working in the ﬁeld Moving toward EBMR requires awareness of the limitations of the (including F1000Prime peer ratings; Bornmann, 2015b); and most current manuscript-review process so that these limitations may be recently, alternative metrics (“altmetrics”) of publication impact, corrected or their negative effects minimized. Research in this area including Mendeley citation rates and frequency of mention on social has increased substantially in recent years, yielding a burgeoning lit- media (e.g., Twitter; see Bornmann & Haunschild, 2018). erature that includes studies from within and outside psychology, as The vast majority of studies to date have used indirect in vivo prox- well as conferences (Kaplan, 2005), special issues of journals in psy- ies that capture research impact but not manuscript quality (e.g., cita- chology and the biomedical sciences (Suls & Martin, 2009), and tion metrics). Relatively few investigations have assessed manuscript articles on the review process in informatics and informetrics journals quality directly (e.g., via peer ratings of published papers), in part (Bornmann, 2015a). Numerous criticisms of the review process have because these latter investigations are more time- and labor-intensive. emerged as a result of this increased scrutiny (see Bornmann, 2010; As Tahamtan and Bornmann (2018) noted, the degree to which cita- Heesen & Bright, 2020). These can be grouped into ﬁve categories, tion frequency is a viable proxy for research quality is open to ques- discussed in the following sections. tion: Their review of factors that predict citations of published works showed that a number of article-related features unrelated to research Inadequate Interreviewer Reliability quality predict citation patterns, including—among other things—the paper’s title, topic, keywords, methodology (reviews and meta-analy- Interrater reliability in manuscript review is generally modest, with ses are cited more frequently than original studies), and references reliability indices in the range of .3 to .4 when quantiﬁed using intra- (articles with a large number of references are cited more frequently class correlation coefﬁcients (ICCs; see Bornmann et al. [2010] for a than articles with fewer references). Moreover, as Ye and Bornmann meta-analysis of research bearing on this issue). Interreviewer agree- (2018) noted, some papers “catch on” quickly, whereas others take ment tends to be better in the biomedical and physical sciences than longer to be noticed. in the social sciences, and evidence suggests that providing additional Beyond these methodological challenges, two other issues arise in structure to reviewers (e.g., numerical rating scales to assess speciﬁc studies of the validity of manuscript review. First, the relationship qualities of manuscripts and the studies described therein) may between journal prestige and various indices of publication quality enhance interreviewer consensus (Armstrong, 1997). Although there and impact is complex: As Traag (2019) showed, higher-quality articles tend to appear in more prestigious journals, but it is also true is disagreement regarding how best to interpret interreviewer reliabil- that articles appearing in more prestigious journals are perceived as ity ﬁndings (see Cicchetti, 1999), in general, critics regard these data being of higher quality, and are cited more frequently, by virtue of as a limitation of the current review system and as evidence that inde- where they are published. Second, there is no way to evaluate the va- pendent assessors cannot agree regarding the quality and importance lidity of reviewer and editorial ratings for articles that were never of manuscripts submitted for publication. published; studies in this area are, by deﬁnition, limited to the subset Taken at face value, results in this area suggest that interreviewer of manuscripts that eventually appeared in the scientiﬁc literature. reliability in manuscript assessment is inadequate. This conclusion These obstacles notwithstanding—and with an appreciation of might not be a fair depiction of the state of manuscript review, how- methodological constraints inherent in extant investigations—studies ever, in that—as Heid and Zbiek (2009) and Robson et al. (2015) in this area have generally produced supportive results, with correla- noted—editors often select reviewers with contrasting perspectives tions (rs) between reviewer ratings and various indices of publication regarding a particular topic or research method to obtain constructive quality and impact ranging from about .20 to .40 (Bornmann et al., dissensus among evaluators. Similarly, editors sometimes solicit 2010; Simonton, 2016). These patterns suggest that reviewer and edi- reviewers with complementary areas of expertise (e.g., theory vs. torial assessments do predict research quality and impact, although methodology) to obtain feedback regarding different aspects of a critics disagree regarding the implications of obtained reviewer rat- manuscript. Contextualized in this way, evidence bearing on interre- ing–publication quality effect sizes (compare, e.g., the conclusions of viewer reliability must be interpreted in a more nuanced manner. In Simonton [2016] with those of Del Guidice [2020]). Interpretive situations where like-minded reviewers assess a particular manu- script, interreviewer reliability should be relatively high. In situations As these outcome criteria suggest, research on manuscript review has where reviewers with contrasting perspectives assess a manuscript, tended to emphasize convergent validity (i.e., associations between reviewer reliability should be lower. No studies have tested this hypothesis judgments and indices of research quality and impact). Although few researchers discuss discriminant validity in manuscript assessments (i.e., the directly, however. Although researchers have examined the impact of absence of a relationship between reviewer ratings and variables theoretically reviewer characteristics on aspects of reviewer bias (discussed in an unrelated to manuscript quality), some discriminant validity concerns are ensuing section), they have not examined the impact of these charac- actually addressed in studies of reviewer bias (e.g., the impact of author and teristics on interreviewer reliability. institutional prestige on manuscript ratings). This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. BORNSTEIN disagreements notwithstanding, these validity coefﬁcients are gener- bogus manuscript reporting results that either supported or failed to ally in line with those obtained in other areas of psychological assess- support the effectiveness of transcutaneous electrical nerve stimula- ment where ratings provided by peers or professionals are used to tion from 33 medical researchers whose previous publications predict subsequent outcomes (e.g., peer ratings of social behavior, cli- reﬂected acceptance versus skepticism regarding this intervention; in nician ratings of patients’ psychopathology; Bornstein, 2011; Meyer this investigation, reviewers’ reasons for rejecting manuscripts that et al., 2001). reported results inconsistent with their preexisting views focused on aspects of the paper’s methodology and implications. It seems that Bias on the Part of Reviewers and Editors when reviewers encounter manuscripts that contradict their beliefs, they may “misattribute” their negative reaction to other qualities of A broad array of potential sources of bias have been discussed in the submitted manuscript. this context, including biases against unpopular topics, controversial Some writers have suggested that requiring reviewers to reveal theories, counterinituitive ﬁndings, nonsigniﬁcant results, replications their identities to authors would increase reviewer accountability of previously published data, less established authors, and less pres- and decrease the potential for conﬁrmatory bias and other extrane- tigious institutions. Although there are countless anecdotal examples ous factors to taint manuscript reviews. Although survey data show of reviewer and editorial bias, there have been surprisingly few con- that the majority of reviewers prefer anonymous to open reviews trolled studies examining its occurrence (see Rennie [2010] for a dis- (Bennett et al., 2018), data also indicate that the majority of cussion of impediments to research on bias in manuscript review). reviewers would continue to evaluate manuscripts if they were Peters and Ceci’s (1982) classic investigation of previously pub- required to reveal their identity to authors (Snell & Spencer, 2005). lished manuscripts submitted again (and rejected 89% of the time by Reviewers’ preferences notwithstanding, research examining the the journals that had already published them) does appear to represent effect of reviewer anonymity versus signed reviews has generally evidence of “prestige bias” in manuscript evaluations, in that the reported no signiﬁcant effect of this procedural change on review names and institutions of the authors on these manuscripts were quality (see Godlee, 2002). replaced by ﬁctitious author names (which are, by deﬁnition, com- Similar results emerge when researchers examine the impact of dis- pletely unknown) and obscure institutions (e.g., the Tri-Valley Center guising (or “masking”) the identities of authors: Although some for Growth and Understanding). Additional evidence regarding the investigations have found modest differences in review quality for impact of author and institutional characteristics on reviewer ratings “masked” versus “unmasked” manuscripts, most studies ﬁnd that this suggests that author prestige (Bakanic et al., 1990), institutional pres- procedural change has no impact on manuscript ratings (Jefferson et tige (Bornmann & Daniel, 2009), and author gender (Heesen & Bright, 2020) may affect reviewer ratings of submitted manuscripts. It is clear al., 2002; Justice et al., 1998). Research also suggests that masking is that papers reporting nonsigniﬁcant ﬁndings are rarely published in often ineffective, with between 40% and 60% of reviewers able to peer-reviewed journals (Mell & Zietman, 2014), and analogue studies determine authorship in masked manuscripts based on information comparing reviewer evaluations of identical manuscripts that vary with contained in the document (e.g., use of particular measures or meth- respect to the statistical signiﬁcance of reported ﬁndings conﬁrm that ods, works cited in the manuscript; see Bennett et al., 2018; New- reviewers tend to evaluate manuscripts that report statistically signiﬁ- combe & Bouton, 2009). One might argue that because masking is cant results more positively than those that do not (Emerson et al., effective in about half of submitted manuscripts, it may be worth 2010). There has been little empirical evidence addressing whether retaining as a bias-reducing strategy, but in this context, it is impor- unpopular topics, controversial theories, and counterintuitive results tant to note that masking is likely to be most effective in concealing affect reviewer and editorial assessments of manuscript quality and the identity of less established authors, who are not strongly associ- publishability. ated with particular research topics and whose papers involve fewer Beyond the potential sources of bias already discussed, critics have self-citations than those of more well-known authors. Thus, a argued that the concordance of ideas and ﬁndings in a submitted reviewer’s inability to infer author identity in a masked manuscript manuscript with the reviewer’s own beliefs (i.e., conﬁrmation bias) may lead the reviewer to conclude, correctly or incorrectly, that the may inﬂuence assessments of research quality (see Cooper, 2009). paper was not written by an established ﬁgure in the ﬁeld—a subtle Extant ﬁndings indicate that conﬁrmation bias does inﬂuence the out- (but potentially signiﬁcant) source of bias in manuscript evaluation. come of manuscript assessments. In an early analogue investigation, Goodstein and Brazis (1970) found that reviewers’ preexisting beliefs Subjective Criticism and Arbitrary Criteria for Rejection regarding astrology predicted ratings of bogus manuscripts whose ﬁndings appeared to support (or refute) the value of astrological signs Although widely discussed, subjectivity and arbitrariness are difﬁ- in predicting human behavior. A later analogue study showed that cult to deﬁne and operationalize with regard to manuscript assess- reviewers’ political orientations (conservative vs. liberal) predicted ment; no controlled empirical studies examining these issues have their evaluations of bogus manuscripts whose ﬁndings suggested that been conducted. A number of critics have argued that reviewers and left-leaning protesters were (or were not) psychologically healthy editors sometimes fail to distinguish important ﬂaws from trivial (Abramowitz et al., 1975). problems and justify manuscript rejection based on issues that could Several investigations of conﬁrmatory bias have gone beyond potentially be corrected through revision (Ranade & Kumar, 2007; global publishability judgments, examining reviewers’ ratings of spe- Rennie, 2010). As Schwartz and Zamboanga (2009, p. 57) noted: ciﬁc manuscript qualities. When Mahoney (1977) asked 75 Journal of Applied Behavior Analysis reviewers to evaluate bogus manu- reviewers and editors [should] judge the value of a manuscript in terms of scripts that reported results conﬁrming or disconﬁrming reviewers’ a the overall contribution to the field and the absence of fatal flaws that priori beliefs, manuscripts with results that supported reviewers’ would seriously undermine the ability of the manuscript to make the con- beliefs received signiﬁcantly higher ratings on indices of methodolog- tributions that the authors are claiming. It is important that papers not be ical rigor and overall scientiﬁc contribution. Similar results were rejected simply because of a multitude of small problems that are “added obtained by Ernst and Resch (1994), who obtained evaluations of a up” and used to justify rejection. This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. EVIDENCE-BASED MANUSCRIPT REVIEW regarding a particular manuscript reﬂects that person’s subjective Unconstructive Criticism and Unnecessarily threshold (called c in SDT), which attempts simultaneously to mini- Harsh Feedback mize the number of “false positives” (poor manuscripts that should Most experienced authors can describe instances wherein reviewers not have been accepted) and “false negatives” (publishable manu- and/or editors have been less than collegial in providing feedback, scripts that were erroneously rejected). and there have been numerous examples of reviewer and editorial Beyond providing a context for understanding false-positive and communications that were unconstructive and unnecessarily harsh false-negative decisions, one advantage of using a signal-detection (see Kotchoubey et al., 2015). These behaviors appear to be relatively approach to conceptualizing manuscript review is that SDT provides rare, however, and there have also been many examples of situations a framework for separating a reviewer’s ability to distinguish good wherein reviewers and editors offered supportive, helpful suggestions from poor-quality research (sensitivity) from that reviewer’s thresh- and provided informal mentorship to authors through their written old regarding what constitutes a publishable manuscript, which may comments (Robinson & Agne, 2010). Unconstructive criticism and vary across context. For example, a reviewer’s knowledge regarding harsh feedback sometimes occur during manuscript review, and the selectivity of a particular journal might cause that reviewer to although, on occasion, this may reﬂect a situation wherein an author’s adjust their threshold upward (for more selective journals) or down- ideas or ﬁndings contradict a reviewer’s personal beliefs (or seem to ward (for journals that are less selective). Similar adjustments undermine that reviewer’s own ﬁndings), it is also due in part to the would occur based on the reviewer’s understanding of a particular fact that some reviewers choose to respond bluntly rather than kindly journal’s mission and goals and the degree to which a manuscript is when providing their critique. As Cooper (2009) noted, sometimes consistent with those goals. As a result, the same manuscript (sig- reviewers offer legitimate feedback, but they do it in such a way that nal) might be deemed publishable in one context and unpublishable the potential value of that feedback is overshadowed by the dismis- in another. sive tone of the commentary. SDT also provides a framework to understand how a reviewer’s (or editor’s) sensitivity and response threshold are inﬂuenced by The Intra- and Interpersonal Dynamics of Manuscript factors unrelated to manuscript quality and journal characteristics. Review: Quality Assessment, Signal Detection, and For example, when reviewers render decisions under suboptimal conditions (e.g., high cognitive load, a looming deadline), sensitiv- Strategic Self-Presentation ity suffers. Variables such as the prestige of an author or the con- The ﬁrst component of EBMR, discussed in the previous sections, cordance of a study’s ﬁndings with the reviewer’s beliefs may cause involves utilizing the best available research to enhance the review that reviewer to shift their criterion level in a conservative or liberal process. The second component of EBMR involves understanding the direction. intra- and interpersonal dynamics of manuscript review and the way Although no studies have used SDT to examine the dynamics of these dynamics are affected by the context within which a review is manuscript review directly, Peters and Ceci (1982) noted that their carried out (i.e., the focus, selectivity, and readership of a particular ﬁndings are generally consistent with a signal-detection interpreta- journal). Evidence from other areas of psychology helps illuminate tion, insofar as altering features of manuscripts ostensibly unrelated these dynamics, suggesting modiﬁcations that may improve the to research quality (e.g., an author’s institutional afﬁliation) review process. Two domains in particular provide helpful perspec- appeared to cause some reviewers to adjust their criterion level tive regarding this issue. upward to minimize the possibility of a false-positive review. Such results have often been interpreted as evidence of reviewer bias— Manuscript Review as Signal Detection and in some instances, they may well be—but as Miller (2006,p. 425) pointed out, information such as an author’s reputation or insti- A fundamental premise of signal-detection theory (SDT) is that tutional afﬁliation “provide clues, albeit imperfect ones, as to the humans and other organisms are often required to classify ambiguous competency of a manuscript’s author(s), and these clues can be used states of the world into one of two discrete categories, with these di- .. . as unconscious or conscious shortcuts around any uncertainty chotomous decisions reﬂecting the perceiver’s judgment that some about the value of a submission.” sort of signal is either present or absent (Green & Swets, 1966). As Bornmann (2015a) reached a similar conclusion following a Wixted (2020) noted, signal-detection judgments arise in a broad review of the various ways that reviewers use heuristics, often without array of contexts, including perception, memory, decision making, di- being aware that they are doing so, to aid in identifying publishable agnosis of medical and psychological disorders, personnel selection, manuscripts. Similar conclusions regarding the utility of such “quick threat assessment, forensics, and other areas. SDT conceptualizes the but shallow” decision-making processes have emerged in signal- perceiver’s decision to identify a stimulus as being present or absent detection investigations of perception, memory, and other phenom- as the product of two processes: sensitivity and response bias. Sensi- ena, where researchers found that perceivers’ initial reactions and gut tivity is primarily affected by stimulus features and the perceiver’s responses often lead to decisions that are as accurate as—or more ability to distinguish stimulus from noise, whereas response bias is accurate than—decisions that involve more extensive deliberation inﬂuenced by factors such as a perceiver’s tendency to be conserva- regarding stimulus properties (see, e.g., Rosenstreich & Ruderman, tive or liberal in identifying a stimulus as being present, as well as 2016). As Lipworth (2009) pointed out, subjectivity is inherent in contextual factors, such as the perceived costs of different types of peer review (as in all aspects of science and practice); she went on to errors. High sensitivity coupled with low bias leads to optimal per- suggest that formance in signal-detection tasks (see Lynn & Barrett, 2014). Viewed through this lens, identifying publishable research is a type even if it were possible to set aside this mode of reasoning, to do so would of signal-detection task wherein reviewers and editors attempt to dis- mean that we would lose a mode of judgment that is extremely useful. tinguish those manuscripts (signals) whose perceived quality and con- Intuition is known to be useful in recognizing patterns and extracting tribution (i.e., whose intensity, d ) is great enough that they can be probabilistic contingencies, and in leading to conclusions that can be cor- reliably distinguished from the pool of lower-quality manuscripts rect even if those doing the reasoning cannot be explicit about the princi- (i.e., noise) in the system. The response of a reviewer or editor ples underlying their impressions. (p. 333) This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. BORNSTEIN treatment. Bradley’s(1981) survey of 411 members of the Psycho- Manuscript Review as Complex Social Interaction nomic Society, American Psychological Association, and American Although SDT provides a useful framework for understanding the Statistical Association revealed that at one time or another, 76% of intrapersonal dynamics of manuscript review, these intrapersonal dy- authors felt pressure to conform to subjective reviewer judgments, and namics do not occur in a vacuum. As Olbrecht and Bornmann (2010) 8% made revisions they knew to be incorrect. and Suls and Martin (2009) noted, greater attention to the social psy- As Table 1 shows, a somewhat similar dynamic characterizes the chology of manuscript review is needed. Table 1 summarizes the behavior of reviewers, whose goals not only include those tradition- goals of authors, reviewers, and editors throughout the review pro- ally associated with the role of reviewer (i.e., assessing manuscript cess. To achieve these goals, each actor must communicate with mul- quality and offering constructive feedback) but may also include tiple audiences simultaneously, with communications tailored to impression-management strategies directed toward the editor. It is present particular kinds of information to each recipient. For example, common for reviewers to be selected in part because they have pub- an effective manuscript review provides helpful feedback to the lished their own work in a particular journal, and by impressing the author while simultaneously presenting an assessment of the manu- editor with a timely and incisive review, they may facilitate the publi- script’s publishability to the journal editor (McDermott & Turk, cation of future manuscripts in that journal. In a widely cited study, 2017). When Lipworth et al. (2011) and Lipworth and Kerridge Amabile (1983) found that when participants were asked to form (2011) conducted open-ended interviews with 35 journal editors and impressions of stimulus persons who had provided positive or nega- reviewers in the United States, the United Kingdom, and Australia, tive assessments of a book, negative reviewers were perceived to be they found that reviewers and editors were cognizant of myriad inter- more competent and intelligent than positive reviewers. As Suls and personal inﬂuences on manuscript reviews, including complex rela- Martin (2009) noted, to the degree that reviewers are aware (either ex- tions of power and authority that interact to shape communications plicitly or implicitly) that providing rigorous manuscript assessments throughout the process. Authors are aware of these complex relations causes one to be perceived in these ways, this would increase their as well (Dutta, 2006). motivation to include as many cogent criticisms as possible in their Not surprisingly, among the author’s most important goals are to review. do good work and present that work in the most compelling way pos- In many respects, the editor is in the most challenging position of sible. Other goals reﬂect authors’ efforts to present themselves in par- all. They must identify the best research for their journal while taking ticular ways to others involved in the process. For example, authors into account reviewers’ recommendations (which often conﬂict) and employ various self-presentation strategies via their communications synthesizing these reviewers’ comments, coupled with their own with reviewers and editors (e.g., by describing revisions made in assessment of the manuscript, to provide constructive feedback to the response to reviewer and editorial suggestions in a way that empha- author. Interpersonal challenges faced by the editor include being sizes the author’s competence, cooperativeness, and ﬂexibility; see respectful to the author while rendering a ﬁrm decision and construct- Taylor, 2016). Moreover, experienced authors not only work toward getting their current manuscript accepted for publication, but by ing a decision letter that communicates to each reviewer that they val- emphasizing their cooperativeness and ﬂexibility, they simultaneously ued that person’s input (even if they did not). Studies show that attempt to strengthen their relationship with the journal editor to create ﬁnding competent, conscientious reviewers is not an easy task: Mus- a context wherein future manuscripts will receive fair and favorable taine and Tewksbury’s (2016) survey of 117 social science journal Table 1 Contrasting Goals of Author, Reviewer, and Editor in the Manuscript-Review Process Author 1. Do good work. 2. Prioritize research that will be influential and widely cited. 3. Promote one’s work to reviewers and the editor by emphasizing its value and empirical rigor. 4. Appear competent and flexible in communications with the editor and reviewers to increase the likelihood of manuscript acceptance. 5. Cultivate a positive long-term relationship with the editor to maximize the likelihood that future submissions will receive fair and favorable treatment. Reviewer 1. Assess the quality of submitted manuscripts. 2. Enhance the quality of manuscripts via constructive feedback. 3. Work efficiently to moderate the amount of time and effort devoted to each manuscript. 4. Impress editor (and other reviewers) through rigorous manuscript assessments and by including incisive comments and suggestions in review. 5. Cultivate a positive long-term relationship with the editor to maximize the likelihood that future submissions will receive fair and favorable treatment. Editor 1. Assess the quality of submitted manuscripts based on own reading and reviewers’ comments. 2. Identify the best research for that particular journal and readership. 3. Enhance the quality of manuscripts via constructive feedback. 4. Maximize the journal impact factor and other metrics that enhance the journal’s reputation. 5. Implement various strategies to manage editorial workload (e.g., desk rejections, reliance on skilled and reliable associate editors). 6. Be respectful to authors of submitted manuscripts while remaining firm in decision making (especially challenging when the author is well established in a particular area). 7. Be respectful to reviewers and diplomatic in constructing the editorial decision letter (especially challenging when reviewers offer sharply contrasting perspectives). 8. Cultivate positive long-term relationships with reviewers to maximize the likelihood that they will continue to evaluate manuscripts submitted to this journal and other journals for which the editor may solicit reviews. Note. These goals and interpersonal dynamics are based in part on writings by Bornstein (2002), Cooper (2009), Dutta (2006), Heesen and Bright (2020), Lawrence (2003), Lipworth et al. (2011), Suls and Martin (2009), Taylor (2016), and Tumin and Tobias (2019). This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. EVIDENCE-BASED MANUSCRIPT REVIEW editors in the United States found that 41% of requests for reviewers “the power of collective intelligence”). Consistent with these results, are turned down, and 57% of submitted manuscripts had at least one Herron (2012) found that the predictive validity of medical journal ar- potential reviewer say no (also see Derraik [2015] for a discussion of ticle evaluations provided by nonexpert readers actually exceeded that challenges editors face in obtaining reviewers with appropriate scien- of expert readers when a large number of nonexpert ratings were tiﬁc expertise who also meet their deadlines). obtained. West and Bergstrom (2011) reached similar conclusions in In this context, it is not surprising that editors often select their synthesis of the literature on political decision making. In a study reviewers with whom they are personally acquainted, in part because of manuscript assessments by biomedical researchers, Evans et al. they are familiar with that person’s expertise and in part because these (1993) found that younger, less experienced reviewers provided potential reviewers are more likely to accept the editor’s request. As higher-quality reviews than did more experienced reviewers (in part Ramam (2016, p. 600) noted, “Section editors use their professional because they devoted more time to evaluating the manuscript). Rec- networks and their knowledge of workers in the ﬁeld to pick referees. ognizing that it is difﬁcult to recruit reviewers even under the current Understandably, referees are more likely to respond to an invitation system, continued research examining the potential beneﬁts of from someone they know.” When editors recruit reviewers from their “crowdsourcing” in manuscript assessments may be fruitful. own professional networks, their motivation to treat that reviewer Although implementing a crowdsourcing approach to manuscript courteously increases: Not only might they need that person’s input assessment may be challenging, one possible strategy would be to ask in future manuscript assessments, but they will likely interact with a pool of the journal’s consulting editors to screen submitted manu- that person in professional contexts as well (e.g., as collaborator, as scripts, each providing a single publishability rating (but not detailed mentor or mentee, as member of a group of colleagues who share par- feedback) using Bornmann’s (2015a) “quick but shallow” assessment ticular interests). approach. Those manuscripts that receive the most positive ratings would be sent to a small number of expert reviewers for more detailed Signal Detection, Self-Presentation, and Manuscript feedback. Other crowdsourcing approaches to manuscript assessment Review: Practical Implications could employ panels similar to those typically used to evaluate grant proposals. Dickersin et al.’s (2007) overview of the initial screening Conceptualization of manuscript review as a signal-detection task process undertaken collectively by the editorial board of the Journal involving complex social interactions not only illuminates the intra- of the American Medical Association also provides helpful context for and interpersonal dynamics of manuscript review but also suggests considering procedural changes in this area. some initial action steps that may help improve the review process. An alternative crowdsourcing strategy would be to implement post- Five stand out. publication review as a method for obtaining feedback from a broad array of researchers with interest and expertise in a particular topic. Implement Debiasing Strategies to Enhance Reviewer and Heesen and Bright (2020) articulated one possible framework for this Editorial Decision Making approach, wherein completed manuscripts would be made publicly accessible prior to peer review, using online repositories (e.g., Open There is a vast literature on biases and distortions in human infor- Science Framework, openICPSR). Other researchers would have the mation processing (see Fernandez, 2013; Kahneman, 2003). There is opportunity to offer feedback and suggestions via online reviews, a more modest literature on strategies for attenuating these biases and which would also be publicly available. Authors would then have the distortions, commonly called debiasing strategies. Some debiasing option of integrating into their manuscripts those suggestions they strategies are designed to minimize negative situational inﬂuences, found most helpful. Beyond the transparency that would result from whereas others moderate the impact of human processing limitations public availability of peer reviews alongside manuscripts, an addi- and expectancy effects (Croskerry et al., 2013a, 2013b). Debiasing tional advantage of this approach is that papers could continue to be strategies have been applied in an array of contexts, including eco- improved over time as helpful comments and critiques accrue. nomics, forensics, risk management, and medical and psychological decision making (Sellier et al., 2019). Table 2 summarizes ﬁve debiasing strategies that may enhance Examine the Impact of Reviewer Selection on Manuscript reviewer and editorial decision making and reduce the potential for Evaluations unintended bias to taint manuscript assessments. These debiasing Editors’ strategy of selecting reviewers based in part on personal strategies integrate evidence regarding human judgment under uncer- ties helps the process run more efﬁciently, but this strategy can also tainty with ﬁndings regarding the intra- and interpersonal dynamics be problematic in certain respects. As described by Crane (1972) and of manuscript review discussed in the previous sections. Implementa- others, invisible colleges are informal networks of researchers who tion of these and other debiasing strategies may help attenuate the interact frequently and share common interests and values. Invisible negative impact of extraneous factors (e.g., conﬁrmatory bias, institu- colleges often reﬂect common intellectual heritage (e.g., shared men- tional prestige bias) on reviewer and editorial judgments (see Cros- tors), and their members typically collaborate in various ways (e.g., kerry [2013b] for a detailed discussion of procedures for implementing coauthoring manuscripts, contributing chapters to volumes edited by these debiasing strategies in vivo). others in the group). With this in mind, when an editor recruits a Conduct Studies to Determine the Optimal Number of Reviewers per Manuscript If this procedure were adopted, it would be advantageous for journals with relatively few consulting editors to increase the size of their editorial boards, so Signal-detection research suggests that researchers should explore subsets of consulting editors could evaluate each manuscript during the initial screening process, reducing reviewer burden. the possible beneﬁts of using a larger number of reviewers to assess The ﬁnal component of Heesen and Bright’s (2020) proposed framework is submitted manuscripts. Cummings and Quimby (2018) found that that the strongest papers that were initially made available online could be combining multiple novice raters’ judgments in a standard signal- published in traditional journals, with journal editors acting as curators, detection task yielded more accurate classiﬁcation decisions than selecting for inclusion those “prepublished” articles that are of greatest interest were yielded by individual expert raters (a phenomenon they termed to the readership of their journal. This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. BORNSTEIN Table 2 Debiasing Strategies for Manuscript Reviewers and Journal Editors Become familiar with research on signal detection to understand the separable influences of sensitivity and response bias on manuscript evaluations. Remain current regarding research on specific forms of bias that influence manuscript assessments (e.g., confirmatory bias, author-prestige bias). Receive ongoing training in theories of reasoning and distortion in human decision making (e.g., heuristics, stereotypes, affect biases) and effective debiasing techniques. Self-monitor for fatigue, distraction, and cognitive overload when evaluating manuscripts and composing formal feedback. Be mindful of the possible impact of self-presentation goals on manuscript assessments (especially important when reviewer and editor have a preexisting/ ongoing relationship). Note. Detailed discussions of various debiasing techniques are provided by Croskerry et al. (2013a, 2013b) and Norman (2009). member of their invisible college to evaluate a manuscript, the editor Funding can also be used to increase the number of studies examin- may, consciously or unconsciously, attribute greater weight to this ing scientiﬁc utility—studies that help determine the degree to which reviewer’s assessment than to that of other reviewers. As Blashﬁeld reviews actually facilitate editorial decision making and enhance the and Reynolds (2012) have shown, it is possible to map researchers’ quality of published research. The ﬁrst issue can be addressed in part invisible colleges, delineating subtle professional links among mem- via research examining editorial ratings of review quality and helpful- bers of the group. ness (although it is possible that editors assign higher ratings to Although no investigations have examined the impact of invisible- reviews that conﬁrm their preexisting beliefs, one potential conse- college bias on manuscript assessments, in a widely cited study of quence of invisible-college bias). The second issue might be best nepotism in peer review, Wenneras and Wold (1997) found that rat- addressed using a pre/post design. If independent raters were asked to ings of postdoctoral fellowship applications were signiﬁcantly more compare the quality of initial submissions and reviewer-enhanced (re- positive for applicants with formal or informal relationships with one vised) manuscripts, unaware of manuscript status, manuscripts in the or more committee members who were involved in making ratings (a latter category should receive more positive ratings, which would pro- phenomenon they termed the “friendship bonus”). Studies contrasting vide compelling evidence of scientiﬁc utility in manuscript review. the relative impact of evaluations made by invisible-college members Funding could also be used to facilitate communication across dis- ciplines. In recent years, a number of special issues of journals have versus other reviewers on editorial decisions would help determine the degree to which invisible-college bias may affect the outcome of been devoted to peer review, but by and large, these discussions have manuscript assessments. If evidence suggests that editors differen- been discipline speciﬁc. As Rennie (2002), Siler et al. (2015), and tially weight feedback from close associates versus less familiar others noted, there has been only modest exchange among researchers reviewers, one possible mechanism for attenuating potential con- in the social sciences, biomedicine, and information science in inves- tigations of manuscript review, despite the fact that many of the same founds in this area would be to have the journal’s editorial assistant remove the identities of reviewers before reviews are forwarded to challenges emerge in different disciplines. Funding for cross-discipli- nary conferences (and edited volumes resulting from these conferen- the editor. ces) would allow researchers from different areas to share knowledge, insights, and potential solutions. Prioritize Funding for Research on Manuscript Review Given the critical role of manuscript review in identifying high- Include Peer Review as a Core Competency in Professional quality research, funds should be set aside for the study of manuscript Psychology review. If .01% (i.e., one ten-thousandth) of existing National Insti- tutes of Health and National Science Foundation budgets, which are Because there is no formal mechanism for training in manuscript currently $43 billion and $8 billion, respectively, were devoted to reviewing or journal editing, novice reviewers and new editors are of- research on peer review, this would represent about $5 million per ten mentored informally by more experienced reviewers and journal year. Support for research on manuscript review could also come editors (Lovejoy et al., 2011). Such informal mentorship is an impor- from the Agency for Health Care Research and Quality and various tant component of professional development, but it may also inadver- international funding bodies. Increased funding would facilitate pro- tently propagate the status quo in reviewer and editorial decision gress in several ways. For example, there are currently very few stud- making. ies that employ experimental methods to examine the impact of Peer review should be a core competency in professional psychol- extraneous factors (e.g., conﬁrmatory bias, prestige bias) on manu- ogy and included in graduate and postgraduate training. Like other script review (cf., Emerson et al., 2010; Ernst & Resch, 1994), in part competencies, psychologists’ progress in this area can be assessed in because such studies are more costly and labor-intensive than in vivo terms of their evolving knowledge, skills, and attitudes. Although a investigations of archival data sets. With additional ﬁnancial support, central element of competency in peer review involves scientiﬁc ex- the impact of reviewers’ preexisting beliefs on interreviewer reliabil- pertise, various other skills are required to be an effective manuscript ity could simultaneously be investigated naturalistically (by treating reviewer. These include (a) providing manuscript assessments that are reviewer characteristics as moderators of interreviewer agreement) informative, articulate, and based on logic and evidence; (b) articulat- and via analogue investigation where reviewers with similar versus ing recommendations to authors constructively, respectfully, and contrasting perspectives were asked to review a manuscript or set of empathically; and (c) providing feedback to editors that takes into manuscripts. If parallel results were obtained using these contrasting account journal-speciﬁc factors (e.g., the selectivity and readership of methods, this would provide strong evidence regarding the impact of a particular publication outlet). Competency in peer review also conﬁrmatory bias on manuscript assessment. Similar strategies could requires that the reviewer engage in ongoing self-reﬂection to mini- be used to examine the impact of author gender, institutional prestige, mize the impact of bias in manuscript assessment and attenuate the and other variables on reviewers’ judgments of manuscript quality. confounding effects of role-related impression management. This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. EVIDENCE-BASED MANUSCRIPT REVIEW Given the importance of manuscript review in research and health- Bornmann, L. (2015b). Interrater reliability and convergent validity of F1000Prime peer review. Journal of the Association for Information Science service psychology, incorporation of training into existing doctoral and Technology, 66(12), 2415–2426. 10.1002/asi.23334 curricula may be warranted (see Scrimgeour & Pruss, 2016). Delinea- Bornmann, L., & Daniel, H. D. (2009). Reviewer and editor biases in journal tion of a formal competency in peer review would facilitate this pro- peer review: An investigation of manuscript refereeing at Angewandte cess. To enhance professional development and ensure that reviewers Chemie International Edition. Research Evaluation, 18(4), 262–272. and editors remain up to date with best practices in EBMR, comple- Bornmann, L., & Haunschild, R. (2018). Do altmetrics correlate with the qual- tion of ongoing continuing education in manuscript assessment ity of papers? A large-scale empirical study based on F1000Prime data. should be required for journal editors and reviewers. PLoS ONE, 13(5), Article e097133. 10.1371/journal.pone.0197133 Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization Conclusion study of journal peer reviews: A multi-level meta-analysis of inter-rater reli- ability and its determinants. PLoS ONE, 5(12), Article e14331. 10.1371/ Moving toward a more evidence-based framework for manuscript journal.pone.0014331 review will not be simple; this article represents a modest ﬁrst step to- Bornstein, R. F. (1991). Manuscript review in psychology: Psychometrics, ward that goal. Looking ahead, psychology is in an ideal position to demand characteristics and an alternative model. Journal of Mind and take the lead in implementing EBMR and improving the review pro- Behavior, 12(4), 429–468. cess. Psychologists have been at the forefront of research on manu- Bornstein, R. F. (2002). Peer review in neuropsychology: Can we increase effectiveness without sacrificing rigor? Cortex, 38(3), 403–405. 10.1016/ script review for several decades, and psychologists of all specialties S0010-9452(08)70668-3 have training and experience in assessment, a skill central to every Bornstein, R. F. (2011). Toward a process-focused model of test score validity: area of scientiﬁc and applied psychology (Bornstein, 2011). As a Improving psychological assessment in science and practice. Psychological result, psychologists are ideally suited to examine the intra- and inter- Assessment, 23(2), 532–544. 10.1037/a0022402 personal dynamics of peer review and implement changes to optimize Bornstein, R. F. (2017). Evidence based psychological assessment. Journal of these dynamics. Personality Assessment, 99(4), 435–445. 10.1080/00223891.2016.1236343 Bradley, J. V. (1981). Pernicious publication practices. Bulletin of the Psy- chonomic Society, 18(1), 31–34. 10.3758/BF03333562 References Cicchetti, D. V. (1999). Peer review in the physical sciences: Implications for neuropsychology. Journal of Clinical and Experimental Neuropsychology, Abramowitz, S. I., Gomes, B., & Abramowitz, C. V. (1975). Publish or 21(3), 412–421. 10.1076/jcen.21.3.412.920 politic: Referee bias in peer review. Journal of Applied Social Psychol- Cooper, M. L. (2009). Problems, pitfalls, and promise in the peer-review pro- ogy, 5(3), 187–200. 10.1111/j.1559-1816.1975.tb00675.x cess. Perspectives on Psychological Science, 4(1), 84–90. 10.1111/j.1745- Alam, M., Kim, N. A., Havey, J., Rademaker, A.,Ratner, D.,Tregre, B.,West, 6924.2009.01109.x D. P., & Coleman, W. P. (2011). Blinded vs. unblinded peer review of Crane, D. (1972). Invisible colleges: Diffusion of knowledge in scientific com- manuscripts submitted to a dermatology journal: A randomized multi-rater munities. University of Chicago Press. study. British Journal of Dermatology, 165(3), 563–567. Croskerry, P., Singhal, G., & Mamede, S. (2013a). Cognitive debiasing 1: Ori- Albers, C. A., Floyd, R. G., Fuhrmann, M. J., & Martinez, R. S. (2011). Publi- gins of bias and theory of debiasing. BMJ Quality & Safety, 22(Suppl. 2), cation criteria and recommended areas of improvement within school psy- ii58–ii64. 10.1136/bmjqs-2012-001712 chology journals as reported by editors, journal board members, and Croskerry, P., Singhal, G., & Mamede, S. (2013b). Cognitive debiasing 2: manuscript authors. Journal of School Psychology, 49(6), 669–689. Impediments to and strategies for change. BMJ Quality & Safety, 22(Suppl. 10.1016/j.jsp.2011.10.002 2), ii65–ii72. 10.1136/bmjqs-2012-001713 Amabile, T. M. (1983). Brilliant but cruel: Perceptions of negative evaluators. Cummings, M. L., & Quimby, P. W. (2018). The power of collective intelli- Journal of Experimental Social Psychology, 19(2), 146–156. 10.1016/0022- gence in a signal detection task. Theoretical Issues in Ergonomics Science, 1031(83)90034-3 19(3), 375–388. 10.1080/1463922X.2017.1381776 American Psychological Association. (2006). Evidence-based practice in Del Guidice, M. (2020). How well do bibliometric indicators correlate with scien- psychology. American Psychologist, 61(4), 271–285. 10.1037/0003- tific eminence? A comment on Simonton (2016). Perspectives on Psychological 066X.61.4.271 Science, 15(1), 202–203. 10.1177/1745691619872763 Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, Derraik, J. G. B. (2015). The principles of fair allocation of peer review: How fairness, and innovation. Science and Engineering Ethics, 3(1), 63–84. much should a researcher be expected to contribute? Science and Engineering 10.1007/s11948-997-0017-3 Ethics, 21(4), 825–828. 10.1007/s11948-014-9584-2 Bakanic, V., McPhail, C., & Simon, R. J. (1990). If at first you don’t succeed: Dickersin, K., Ssemanda, E., Mansell, C., & Rennie, D. (2007). What do Review procedures for revised and resubmitted manuscripts. The American Soci- JAMA editors say when they discuss manuscripts that they are considering ologist, 21(4), 373–391. 10.1007/BF02691847 for publication? Developing a schema for classifying the content of editorial Bennett, K. E., Jagsi, R., & Zietman, A. (2018). Radiation oncology authors discussion. BMC Medical Research Methodology, 7(1), Article 44. 10.1186/ and reviewers prefer double-blind peer review. Proceedings of the National 1471-2288-7-44 Academy of Sciences of the United States of America, 115(9), e1940. Dutta, M. J. (2006). The ten commandments of reviewing: The promise of a 10.1073/pnas.1721225115 kinder, gentler discipline. Health Communication, 20(2), 197–200. 10.1207/ Bennett, S., & Bennett, J. W. (2000). The process of evidence-based practice in s15327027hc2002_11 occupational therapy: Informing clinical decisions. Occupational Therapy Jour- Elson, M., Huff, M., & Utz, S. (2020). Metascience on peer review: Testing nal, 47(4), 171–180. 10.1046/j.1440-1630.2000.00237.x the effects of a study’s originality and statistical significance in a field Blashfield, R. K., & Reynolds, S. M. (2012). An invisible college view of the DSM–5 personality disorder classification. Journal of Personality Disor- experiment. Advances in Methods and Practices in Psychological Science, ders, 26(6), 821–829. 10.1521/pedi.2012.26.6.821 3,53–65. Bornmann, L. (2010). Does the journal peer review select the “best” from the Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in works submitted? The state of empirical research. IETE Technical Review, 27(2), 93–96. 10.4103/0256-4602.60162 peer review. Archives of Internal Medicine, 170(21), 1934–1939. 10.1001/ Bornmann, L. (2015a). Complex tasks and simple solutions: The use of heuris- archinternmed.2010.406 tics in the evaluation of research. Journal of the Association for Information Ernst, E., & Resch, K. L. (1994). Reviewer bias: A blinded experimental Science and Technology, 66(8), 1738–1739. 10.1002/asi.23318 study. The Journal of Laboratory and Clinical Medicine, 124(2), 178–182. This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. BORNSTEIN Evans, A. T., McNutt, R. A., Fletcher, S. W., & Fletcher, R. H. (1993). The Lovejoy, T. I., Revenson, T. A., & France, C. R. (2011). Reviewing manu- characteristics of peer reviewers who produce good-quality reviews. Jour- scripts for peer-review journal: A primer for novice and seasoned reviewers. nal of General Internal Medicine, 8(8), 422–428. 10.1007/BF02599618 Annals of Behavioral Medicine, 42(1), 1–13. 10.1007/s12160-011-9269-x Evans, K., & Woolridge, B. (1987). Journal peer review: A comparison with Lynn, S. K., & Barrett, L. F. (2014). “Utilizing” signal detection theory. Psycholog- employee peer performance appraisal. Journal of Social Behavior and Per- ical Science, 25(9), 1663–1673. 10.1177/0956797614541991 sonality, 2(4), 385–396. Mahoney, M. J. (1977). Publication prejudices: An experimental study of con- Fernandez, J. (2013). Self-deception and self-knowledge. Philosophical Stud- firmatory bias in the peer review system. Cognitive Therapy and Research, ies, 162(2), 379–400. 10.1007/s11098-011-9771-9 1(2), 161–175. 10.1007/BF01173636 Garcia, J. (1981). Tilting at the papermills of academia. American Psycholo- McCarty, R. (2002). Science, politics, and peer review: An editor’s dilemma. gist, 36(2), 149–158. 10.1037/0003-066X.36.2.149 American Psychologist, 57(3), 198–201. 10.1037/0003-066X.57.3.198 Gaudiano, B. A., & Miller, I. W. (2013). The evidence-based practice of psy- McDermott, S., & Turk, M. A. (2017). The art of manuscript review—And chotherapy: Facing the challenges that lie ahead. Clinical Psychology manuscript development. Disability and Health Journal, 10(1), 1–2. Review, 33(7), 813–824. 10.1016/j.cpr.2013.04.004 10.1016/j.dhjo.2016.10.009 Godlee, F. (2002). Making reviewers visible: Openness, accountability and Mell, L. K., & Zietman, A. L. (2014). Introducing prospective manuscript credit. Journal of the American Medical Association, 287(21), 2762–2765. review to reduce publication bias. International Journal of Radiation Oncol- 10.1001/jama.287.21.2762 ogy, 90(4), 729–732. 10.1016/j.ijrobp.2014.07.052 Goodstein, L. D., & Brazis, K. L. (1970). Credibility of psychologists: An em- Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, pirical study. Psychological Reports, 27(3), 835–838. 10.2466/ R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001). Psychological pr0.1970.27.3.835 testing and assessment: A review of evidence and issues. American Psychol- Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophy- ogist, 56(2), 128–165. 10.1037/0003-066X.56.2.128 sics. Wiley. Miller, C. C. (2006). Peer review in the organizational and management scien- Heesen, R., & Bright, L. K. (2020). Is peer review a good idea? The British ces: Prevalence and effects of reviewer hostility, bias, and dissensus. Acad- Journal for the Philosophy of Science. Advance online publication. emy of Management Journal, 49(3), 425–431. 10.5465/amj.2006.21794661 10.1093/bjps/axz029 Mustaine, E. E., & Tewksbury, R. (2016). A survey of social science journal Heid, M. K., & Zbiek, R. M. (2009). Manuscript review as scholarly work. editors for behind-the-scenes data on the publication process. Journal of Journal for Research in Mathematics Education, 40(5), 474–476. 10.5951/ Scholarly Publishing, 47(3), 231–249. 10.3138/jsp.47.3.231 jresematheduc.40.5.0474 Newcombe, N. S., & Bouton, M. E. (2009). Masked reviews are not fairer Herron, D. M. (2012). Is peer review obsolete? A model suggests that post- reviews. Perspectives on Psychological Science, 4(1), 62–64. 10.1111/ publication reader review may exceed the accuracy of traditional peer j.1745-6924.2009.01102.x review. Surgical Endoscopy, 26(8), 2275–2280. 10.1007/s00464-012-2171- Norman, G. (2009). Dual processing and diagnostic errors. Advances in Health Sciences Education: Theory and Practice, 14(Suppl. 1), 37–49. 10.1007/ Horrobin, D. F. (1990). The philosophical basis of peer review and the sup- s10459-009-9179-x pression of innovation. Journal of the American Medical Association, Olbrecht, M., & Bornmann, L. (2010). Panel peer review of grant applications: 263(10), 1438–1441. 10.1001/jama.1990.03440100162024 What do we know from research on social psychology on judgment and de- Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002). Effects of edito- cision-making in groups? Research Evaluation, 19(4), 293–304. 10.3152/ rial peer review—A systematic review. Journal of the American Medical 095820210X12809191250762 Association, 287(21), 2784–2786. 10.1001/jama.287.21.2784 Park, D. C. (2009). Publishing in the psychological sciences: Enhancing jour- Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & the nal impact while decreasing author fatigue. Perspectives on Psychological PEER Investigators. (1998). Does masking author identity improve peer Science, 4(1), 36–37. 10.1111/j.1745-6924.2009.01103.x review quality? A randomized trial. Journal of the American Medical Asso- Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological jour- ciation, 280(3), 240–242. 10.1001/jama.280.3.240 nals: The fate of published articles, submitted again. Behavioral and Brain Kahneman, D. (2003). A perspective on judgment and choice: Mapping Sciences, 5(2), 187–255. 10.1017/S0140525 X00011183 bounded rationality. American Psychologist, 58(9), 697–720. 10.1037/ Ramam, M. (2016). The manuscript review process: What do editors do? In- 0003-066X.58.9.697 dian Journal of Dermatology, Venereology and Leprology, 82(6), 599–602. Kaplan, D. (2005). How to fix peer review: Separating its two components– 10.4103/0378-6323.191541 improving manuscripts and judging scientific merit–would help. Journal of Ranade, S. A., & Kumar, N. (2007). Improved system of anonymous peer Child and Family Studies, 14(3), 321–323. 10.1007/s10826-005-6845-3 review of manuscripts. Current Science, 93(12), 1659–1660. Keeley, J. W., Reed, G. M., Roberts, M. C., Evans, S. C., Medina-Mora, Rennie, D. (2002). Fourth international congress on peer review in biomedical M. E., Robles, R., Rebello, T., Sharan, P., Gureje, O., First, M. B., publication. Journal of the American Medical Association, 287(21), Andrews, H. F., Ayuso-Mateos, J. L., Gaebel, W., Zielasek, J., & Saxena, 2759–2760. S. (2016). Developing a science of clinical utility in diagnostic classification Rennie, D. (2010). Integrity in scientific publishing. Health Services Research, systems: Field study strategies for ICD–11 mental and behavioral disorders. 45(3), 885–896. 10.1111/j.1475-6773.2010.01088.x American Psychologist, 71(1), 3–16. 10.1037/a0039972 Rennie, D. (2016). Make peer review scientific. Nature, 535(7610), 31–33. Kotchoubey, B., Butof, S., & Ranganatha, S. (2015). Flagrant misconduct of 10.1038/535031a reviewers and editors: A case study. Science and Engineering Ethics, 21(4), Rennie, D., & Flanagin, A. (2018). Three decades of peer review congresses. 829–835. 10.1007/s11948-014-9583-3 Journal of the American Medical Association, 319(4), 350–353. 10.1001/ Lawrence, P. A. (2003). The politics of publication. Nature, 422(6929), jama.2017.20606 259–261. 10.1038/422259a Robinson, J. D., & Agne, R. R. (2010). Kindness, gentility, and rejection: An Lipworth, W. (2009). Beyond the consulting room: Intuition and intersubjec- analysis of 99 manuscript reviews. Health Communication, 95(6–7), tivity in peer review. Australasian Psychiatry, 17(4), 331–334. 10.1080/ 504–511. 10.1080/10410236.2010.496530 Robson, K., Pitt, L., & West, D. C. (2015). Navigating the review process: Lipworth, W., & Kerridge, I. (2011). Shifting power relations and the ethics of Reviewers’ suggestions for a manuscript. Journal of Advertising Research, journal peer review. Social Epistemology, 25(1), 97–121. 10.1080/ 55(1), 9–15. 10.2501/JAR-55-1-009-017 02691728.2010.534567 Lipworth, W. L., Kerridge, I. H., Carter, S. M., & Little, M. (2011). Journal Rosenstreich, E., & Ruderman, L. (2016). Not sensitive, yet less biased: A sig- peer review in context: A qualitative study of the social and subjective nal detection theory perspective on mindfulness, attention, and recognition dimensions of manuscript review in biomedical publishing. Social Science memory. Consciousness and Cognition, 43,48–56. 10.1016/j. & Medicine, 72(7), 1056–1063. 10.1016/j.socscimed.2011.02.002 concog.2016.05.007 This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association. EVIDENCE-BASED MANUSCRIPT REVIEW Schwartz, S. J., & Zamboanga, B. L. (2009). The peer review and editorial Tumin, D., & Tobias, J. D. (2019). The peer review process. Saudi Journal of Anaes- system: Ways to fix something that might be broken. Perspectives on Psy- thesia, 13(5), 52–58. 10.4103/sja.SJA_544_18 chological Science, 4(1), 54–61. 10.1111/j.1745-6924.2009.01106.x Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer review. Nature, Scrimgeour, G. J., & Pruss, S. D. (2016). Writing highly effective reviews of a 387(6631), 341–343. 10.1038/387341a0 scientific manuscript. Freshwater Science, 35(4), 1076–1081. 10.1086/ West, J. D., & Bergstrom, C. T. (2011). Can ignorance promote democracy? Science, 334(6062), 1503–1504. 10.1126/science.1216124 Sellier, A. L., Scopelliti, I., & Morewedge, C. K. (2019). Debiasing training Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting improves decision making in the field. Psychological Science, 30(9), the daylight in: Reviewing the reviewers and other ways to maximize trans- 1371–1379. 10.1177/0956797619861429 parency in science. Frontiers in Computational Neuroscience, 6,20–29. Siler, K., Lee, K., & Bero, L. (2015). Measuring the effectiveness of scientific 10.3389/fncom.2012.00020 gatekeeping. Proceedings of the National Academy of Sciences of the Wixted, J. T. (2020). The forgotten history of signal detection theory. Journal United States of America, 112(2), 360–365. 10.1073/pnas.1418218112 of Experimental Psychology: Learning, Memory, and Cognition, 46(2), Simonton, D. K. (2016). Giving credit where credit’s due: Why it’s so hard to 201–233. 10.1037/xlm0000732 do in psychological science. Perspectives on Psychological Science, 11(6), Ye, F., & Bornmann, L. (2018). “Smart girls” versus “sleeping beauties” in the 888–892. 10.1177/1745691616660155 sciences: The identification of instant versus delayed recognition by using Snell, L., & Spencer, J. (2005). Reviewers’ perceptions of the peer review pro- the citation angle. Journal of the Association for Information Science and cess for a medical education journal. Medical Education, 39(1), 90–97. Technology, 69(3), 359–367. 10.1002/asi.23846 10.1111/j.1365-2929.2004.02026.x Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: Suls, J., & Martin, R. (2009). The air we breathe: A critical look at practices Institutionalization, structure and functions of the referee system. Minerva, and alternatives in the peer-review process. Perspectives on Psychological 9(1), 66–100. 10.1007/BF01553188 Science, 4(1), 40–50. 10.1111/j.1745-6924.2009.01105.x Tahamtan, I., & Bornmann, L. (2018). Core elements in the process of citing publications: Conceptual overview of the literature. Journal of Informetrics, 12(1), 203–216. 10.1016/j.joi.2018.01.002 Taylor, B. W. (2016). Writing an effective response to a manuscript review. Received August 18, 2020 Freshwater Science, 35(4), 1082–1087. 10.1086/688886 Revision received January 21, 2021 Traag, V. A. (2019). Inferring the causal effect of journals on citations. PsyArXiv. https://arxiv.org/abs/1912.0864 Accepted January 22, 2021 n This document is copyrighted by the American Psychological Association or one of its allied publishers. Content may be shared at no cost, but any requests to reuse this content in part or whole must go through the American Psychological Association.

http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png

Archives of Scientific Psychology American Psychological Association

http://www.deepdyve.com/lp/american-psychological-association/improving-what-is-published-toward-an-evidence-based-framework-for-0Nv8k9RJI1

Loading next page...

Publisher: American Psychological Association
eISSN: 2169-3269
DOI: 10.1037/arc0000076
Publisher site: See Article on Publisher Site

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review

Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review

Improving What Is Published: Toward an Evidence-Based Framework for Manuscript Review

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies