NHMRC recognises that all researchers and support staff have been affected, to varying degrees, by the COVID-19 pandemic. These effects include interruptions to research activities, increased workloads for front-line health care workers and public health researchers, and increased caring responsibilities. As a result, the capacity of many researchers to meet the original application deadlines and to contribute to peer review during 2020 is likely to be reduced.
In response, NHMRC made changes to several NHMRC schemes this year, including extending the application period for Ideas Grants by five weeks (to 10 June 2020). These were difficult decisions, made with the aim of providing clarity and a degree of certainty for researchers and institutions at a challenging time.
In announcing the extension to the application deadline for the Ideas Grant scheme, we flagged our intention to use a streamlined peer review process for this round. The purpose of this message is to provide further information on that streamlined peer review process for the Ideas Grant scheme in 2020.
Why use a streamlined peer review process in 2020?
The purpose of the streamlined process is to reduce the burden on peer reviewers and to maximise the chance that outcomes can be released before the end of the year despite the delayed application deadline. NHMRC recognises the importance of Ideas Grants for the employment of many researchers from January 2021. A significant delay in the release of outcomes would therefore add to uncertainty for researchers and their employing institutions.
The following sections outline the planned changes to peer review for the 2020 Ideas Grant round and summarise the background information that has guided these changes.
How will Ideas Grant applications be assessed in 2020?
Briefly, Ideas Grants will be awarded based on the assessments provided by four independent assessors assigned to each application (with a minimum of three assessments accepted if four cannot be obtained). Grant Review Panels (GRPs) will not be used. A similar process was used for the 2019 and 2020 rounds of the Investigator Grant scheme.
As in 2019, peer reviewers will be matched to applications based on their conflict of interest and suitability declarations. This year, applicants are required to provide additional information to help maximise the matching of suitable assessors to each application. NHMRC will also appoint experienced reviewers over first-time reviewers where possible.
Budgets will continue to be scrutinised as part of the peer review process and applicants must ensure that all budget items are fully justified. As usual, peer reviewers will be asked to make recommendations to NHMRC about any budget items that are not sufficiently justified and may need to be reduced or removed.
The changes to the process for 2020 have been made following consultation with NHMRC’s Research Committee and examination of data from the 2019 rounds of the Ideas Grant and Investigator Grant schemes, as outlined below.
Background to the changes to the 2020 Ideas Grant assessment process
Following NHMRC’s public consultation on peer review processes for the new grant program in 2017, NHMRC has continued to consider ways both to streamline and to improve the quality of NHMRC’s peer review processes. As discussed extensively in 2017, we are seeking a balance between the competing goals of improved rigour and reduced burden on peer reviewers. These discussions were continuing before the COVID-19 outbreak – including the possible removal of GRPs.
There are arguments in favour and against the use of GRPs for grant assessment.
In their favour, many peer reviewers value the experience of coming together to discuss grants and reach an agreed view on their relative merits. GRP meetings also give early career researchers the opportunity to observe or participate in the peer review process.
Arguments against the use of GRPs include the following:
- Preparation for and participation in GRP meetings place a significant additional burden on peer reviewers.
- Some potential reviewers are unable to participate in GRPs over several days because of caring or professional responsibilities.
- GRP meetings add many weeks to peer review timelines, especially for large schemes such as the Ideas Grant scheme (6 weeks of meetings in addition to time for peer reviewer preparation and administration in 2019).
- Grouping applications and peer reviewers into GRPs constrains NHMRC’s ability to assign applications to the most suitable peer reviewers.
Data from the 2019 Ideas Grant round showed that, as a result of GRP discussion:
- The majority of applications (75%) decreased their score (by an average of 0.4 points) compared with their total pre-GRP score determined by the four spokespersons (SPs) alone;
- The remaining applications (25%) increased their score (by an average of 0.2 points);
- Most SPs (74%) did not change their scores; where a change did occur, it was most frequently for a single criterion, moving up or down by a single point.
These findings are consistent with some published studies of peer review (cited in Pier et al, Res Eval. 2017 26(1): 1–14), showing that panel discussion does not significantly alter outcomes compared to independent evaluations. Some other funding agencies also rely on independent assessments without GRPs (for example New Zealand’s Endeavour Fund).
NHMRC has considered using GRPs for ‘discussion by exception’ in which only applications nominated by assessors are discussed. This process was used for Investigator Grants in 2019 (by video/teleconference) but was discontinued for the 2020 round for the following reasons:
- 2019 assessors expressed concern about unfairness and inconsistency because only some applications were discussed.
- The process added about 7 weeks to the peer review process.
- NHMRC data showed that only about 5% of applications were nominated for discussion, SPs rarely changed their scores and the discussions led to a change in outcome for less than 1% of applications.
For the reasons outlined above and in light of this year’s special circumstances, NHMRC will not run GRPs or discussion by exception in the 2020 Ideas Grant round.
Plans for assessment of Ideas Grant applications in future rounds
As noted above, NHMRC had been considering the possible streamlining of peer review processes for future rounds of Ideas Grants. Shortening of the peer review process would reduce peer reviewer burden, reduce the time from application to the release of outcomes, allow a longer gap between release of Investigator Grant outcomes and the Ideas Grant application deadline, and open up the possibility of running two rounds of Ideas Grants per year. However, we had envisaged that any substantial change, such as the removal of GRPs, would only be introduced in consultation with the sector and would be accompanied by other steps to improve the quality and rigour of peer review.
The COVID-19 outbreak requires us to make significant changes to peer review in 2020 without the benefit of wide consultation and other improvements to the process. While the circumstances are far from ideal, NHMRC will monitor the outcomes of the streamlined process this year to inform discussions about any future changes. As always, we will welcome feedback from applicants and peer reviewers once the round is complete.
No comments:
Post a Comment