A grant review committee is a group of people assembled to review grant proposals and make decisions on awards for a grant program. Grant review committees can vary in size, expertise, and time demands depending on the grantmaker and the grant program. The grant review committee’s role is to objectively evaluate all grant applications, and collaboratively make smart recommendations regarding funding to maximize impact.
Running a grant review committee: Best practices (from Submittable)
- Clarifying the process – This should include defining the objectives, outlining the review process, and determining the timeline. This will help to clearly communicate expectations with prospective grant review committee members, and set the stage for a smooth process.
- Defining the objectives – Objectives will vary based on the organization and program, and defining the objectives of the grant will help provide clarity around who to recruit to the review team. Factors to consider can include background and areas of expertise.
- The review workflow – This will will depend on the program, but typical workflows may involve the following steps:
- Screening applicants for your eligibility criteria to save your review team time.
- A first pass review with a simple thumbs up or thumbs down rating.
- A second round of review with a detailed review rubric.
- Timeline – To determine a timeline, it helps to begin with the end in mind. Ask yourself: When do you want to award the grantees? Then you can work backward, with answers to these questions:
- How long do you need to allot for each round of review?
- How long will you accept applications to your grant program?
- How much time will it take to recruit your grant review team?
- Assembling a diverse review committee – This is a key step in the review process where having a diverse committee will provide broad levels of knowledge and experience to thoroughly assess the proposals.
- Invitations to potential reviewers are based on their area(s) of expertise, experience in grant writing, peer review process, and in consultation with the committee Chairs, and grants staff.
- The committee membership should consider the need to cover the full range of relevant research areas, methodologies and experience; the need for diversity and an inclusive team. Factors to consider can include ethnicity, gender, age, ability, sexual orientation, background, education, socioeconomic status, professional and personal experience, and geography.
- Patient/Stakeholder/Lay reviewers are members of the community who are oftentimes invited to sit on review committees to offer valuable perspectives for the review. This can include personal experiences as a patient or as an advocate or provide guidance from a stakeholder’s viewpoint.
- Communicating Expectations – Clearly communicate the work and time demands required by your committee members and any compensation that is being offered upfront or early on during the review process
- Roles and Responsibilities – Peer Review Committees are responsible for evaluating individual applications; scoring/rating each application; discussing applications at committee meeting; and, recommending a budget and term to support the proposed research if the application is approved.
Reviewer compensation
Organizations have the option to compensate their reviewers for their efforts towards reviewing and scoring grant proposals, and this topic may be approached as a professional consulting project with rates by which they will be appropriately compensated. Compensation may be in various non-monetary forms such as discounts to annual conferences.While this varies between organizations, budgets may allow for compensation as a gesture of appreciation towards the review committee, but reviewers may waive their compensation.
- $250 – $1000 per review has been reported and some organizations have offered bonuses (extra $100) for expedited reviews (2-week turnaround times) during shorter review periods. This may vary between length of the proposal (LOI versus a full proposal).
Examples
-
- $250 for a 2-page LOI and $500 for a full proposal as a primary reviewer
- $1000 per review
- Annual compensation packages are a single fee paid to reviewers for their time and expertise. This can vary from $1200 – $6000 depending on the workload.
- Patient/Stakeholder reviewers have been compensated
- Example: $500 stipend per reviewer who is given 5 – 6 applications or approximately $60 per hour
- Gift-in-kind for tax purposes
- 50% of the rate to pay a full reviewer
- If review panels require in-person attendance, travel will be reimbursed.
- The Fair Market Calculator has also been suggested to calculate compensation.
Other Considerations
- Levels of compensation – Some levels of compensation may vary based on whether they are just a member or a committee chair.
- $1700 for a committee chair
- $1200 for a member
- Negative points to consider:
- This can make the relationship between members of the scientific community and the organization transactional, and it can take money away from activities related to the It can question whether compensation may be a motivating factor for reviewers to invest more time into their reviews and impact how thorough and thoughtful they are during the review process.
- Benchmarking and tracking this may be of interest to HRA and to member organizations
- “Over the past 10-15 years it feels like there’s been a transition from a system where reviewing for free or for a very low honorarium was considered to be a normal part of “good scientific citizenship” to a world where paying for reviewing is considered more fair. The review-for-free system seemed to work OK when reviewers were usually tenured senior professors who had sufficient time to conduct reviews on the side without negatively impacting their careers. However I am now recruiting more diverse review committees in terms of seniority of the researcher, size of lab, and level of resources, and we are actively trying to increase the number of researchers from groups currently underrepresented in research. In this context it seems more fair to pay them.”
Principals of Peer Review
Confidentiality – Confidentiality is information about a person that will not be disclosed directly or indirectly to anyone else without that person’s prior expressed consent. The information provided by applicants in their applications is made available to grants staff and reviewers for reviewing purposes only. Committee members are not to approach or communicate in any way with applicants, or anyone outside of the committee, regarding any information relating to the review of a specific application, or offer opinions on the applicant’s chances of success. In turn, applicants are not to contact committee members regarding the status of their applications.
Conflict of Interest (COI) – A conflict of interest (COI) is a circumstance by which a reviewer is involved in or affiliated with financial or personal interests or considerations that may influence or compete with the outcome of the review process. A conflict of interest may be deemed to exist, or be perceived as such, when committee members and observers:
-
- Are a relative or close friend, or have a personal relationship with the applicants;
- Are in a position to gain or lose financially/materially from the funding of the application;
- Have had long-standing scientific or personal differences with the applicants;
- Are currently affiliated with the applicants’ institutions, organizations or companies*;
- Are closely professionally affiliated with the applicants, as a result of having in the last six years:
- Frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
- Been a supervisor or a trainee of the applicants;
- Collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; or,
- Been employed by the institution, when an institution is the applicant.
- Feel for any reason unable to provide an impartial review of the application.
Examples of COI Policies from HRA Members
Fairness – Reviewers must be willing to:
-
- Exercise rigorous judgement
- Be impartial and reasonable
- Understand and take into account, in a balanced way the particular context of each application
- Provide a constructive, quality review which helps the applicant by pointing out strengths and weaknesses that contributed to the application rating
Transparency – Transparency can be ensured through:
-
- Using the application requirements and review criteria established for each grant program, as seen in the funding opportunity details
- Publishing Peer Review Committee membership lists
- Publishing the lists of all successful applications of the grant program
Reviewer Guidelines
The guidelines are generally set by which the organization determines to be significant to their interests to advance the fields, the assessment of the investigator’s training, technical expertise, productivity, and the novelty of their ideas and creativity. Other criteria for reviewers to consider are the proposal’s ability to be innovative and its approach to addressing the issue(s) at hand. The environment that the investigator, especially if they have not achieved full faculty, is important to determine if the facilities, equipment, and staff are sufficient to support the proposed the work. Finally, the overall impact and the contribution to science should be a considering in scoring a proposal.
Hearing Health Foundation Guidelines
Scoring/Grading Process
Type of Peer Review
- Single Blind – Applicant doesn’t know the identity of the reviewer
- Double Blind – Reviewer doesn’t know the identity of the applicant, and vice-versa
Levels of Peer Review
Single-stage: Proposal reviews are completed in a single review step.
Multi-stage:
- Stage One: Reviewers independently review and score grant applications based on the evaluation criteria for the grant mechanism.
- Stage Two: Reviewers gather either in-person or virtually to discuss applicants, the strengths and weakness of their proposals, and make funding recommendations for board/leadership approval.
Scale
Most grantmakers use a set of criteria and scoring rubric when evaluating grant applications. Rubrics are detailed outlines for how each application will be read and scored. A comprehensive rubric helps reviewers stay consistent, minimizes personal bias, and provides a useful resource to answer questions.
NIH scoring system and procedure: Many of the scoring scales followed by organizations are built around the NIH’s scoring system. This scale is defined by a descriptive scale to describe impact as well as a numerical breakdown (High (1-3), medium (4-6), and low (7-9)).
(Scoring scale from HHF)
Examples from HRA members:
AHA Scoring Scale and Guidelines
HRA Summary of Analyzing Reviewers Scores
Review Meetings
Reviews can be conducted both in-person or remotely. The review committee can determine the meeting format and the frequency of meetings needed to complete the review process. During these sessions, the committee will discuss summary statements and provide feedback to applicants. Due to the pandemic, meetings have shifted to virtual sessions. HRA members hosted a webinar on Running a Successful Virtual Peer Review in April 2020. HRA has also compiled resources for Transforming to Virtual Scientific Conferences.
Contract Peer Review
If additional support is needed to manage and conduct peer review, multiple members have recommended the American Institute of Biological Sciences.
Bias in Peer Review
Bias in peer review can present in the form of both implicit and explicit biases. Currently, there are efforts within each organization to reduce these biases by addressing diversity, equity, and inclusion. HRA hosted webinar to address bias reduction.
HRA is also focused on reducing bias in grantmaking through the Inclusive Grantmaking Initiative (IGI).
- Transparency, Recognition and Innovation in Peer Review in the Life Sciences [ASAPBio Meeting Summary, February 2018]
- Validity of Peer Review Examined by AIBS
- AIBS Risk evaluation in peer review of grant applications
- Understanding Unconscious Bias by The Royal Society (YouTube video)