This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

trusted source

proofread

Score, then rank: Researchers propose an integrated approach to grant review assessments

Score, then rank: Researchers propose an integrated approach to grant review assessments
Exploratory plots for AIBS Panel 1 (2020). Top: Boxplots of scores by proposal. Bottom: Stacked barcharts of ranking places assigned across judges by proposal. Credit: Research Integrity and Peer Review (2023). DOI: 10.1186/s41073-023-00131-7

The public funding of science is responsible for many of the biomedical and other scientific breakthroughs on which our lives depend. However, the process through which funding decisions are made, the peer review of grant proposals, has been historically understudied, and current approaches can lead to undesirable outcomes.

Writing in Research Integrity and Peer Review, Stephen A. Gallo, then affiliated with the American Institute of Biological Sciences, and Michael Pearce, Carole J. Lee, and Elena A. Erosheva from the University of Washington highlight problems related to proposal rating in grant review, and propose a protocol to ameliorate them.

In many programs, say the authors, proposals are funded primarily on the basis of reviewer ratings and summary statistics derived from them that represent reviewers' assessment of proposal quality. Relying solely on ratings, however, can pose problems for establishing priorities, "especially when distinguishing between similarly rated proposals, especially for projects of high quality, where only a few minor weaknesses can make the difference between a funded project and a rejection."

Furthermore, say Gallo and colleagues, the use of ratings rubrics "often forces reviewers to penalize a project based on a series of identified but minor weaknesses, despite the fact that they may find the application overall to be potentially significant."

To address these concerns, the authors suggest adding rankings to the traditional approach and describe an innovative methodology called the Mallows-Binomial, which can then produce integrated proposal scores, as well as an induced preference ordering. In so doing, the approach allows funders to consider reviewers' scores alongside their opinions about how well the proposals stack up against one another. Pearce states that the inclusion of ranking will "more accurately distill reviewer opinion into a useful output to make an informed funding decision."

For research funders, the authors suggest clear benefits of the Mallows-Binomial protocol in comparison with ratings-only approaches. These include "providing a ranked priority list with confidence metrics, a higher degree of discrimination between similarly rated proposals, and a robustness to outliers and reviewer inconsistencies," features that will be important in distilling reviewer opinion "into a useful output to make the best, most informed decision" for funding crucial research.

More information: Stephen A. Gallo et al, A new approach to grant review assessments: score, then rank, Research Integrity and Peer Review (2023). DOI: 10.1186/s41073-023-00131-7

Citation: Score, then rank: Researchers propose an integrated approach to grant review assessments (2023, August 1) retrieved 7 March 2024 from https://phys.org/news/2023-08-score-approach-grant.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Double-anonymous peer review reduces reviewer bias, finds a three-year trial

1 shares

Feedback to editors