The peer review process for awarding funds to international science research consortia: a qualitative developmental evaluation

Document reviews

Among the 73 eligible applications (26 in round one, 47 in round two) submitted to the award-making institution, 34 were shortlisted for full review (15 in round one, 19 in round two): 6/6 (round one/round two) in renewable energy, 4/8 in soil-related science, and 5/5 in water and sanitation. Each eligible application was scored by three panel members who were assigned applications on the basis of their relevant expertise. Applications that met the assessment criteria and had highest scores were shortlisted. Each shortlisted application was reviewed by at least five external specialists and their comments taken into consideration during the final selection meeting.

Guidance notes for panel members on the conduct of the face to face meeting and the assessment criteria, and summary details of the shortlisted applications for both rounds, were provided to us by the grant-awarding institution. We were not involved in the shortlisting process. We used a pre-designed matrix to extract information from the panel guidance notes about the assessment criteria against which applications were to be judged, the role of the panel members, the role of the panel chair and the panel code of conduct.

Observation of peer review selection panel meetings

Overt observation of the panel meetings involved the direct observation (Lamont, 2009; McNaughton et al., 2013) of the panel in their natural setting; as observer-researchers we did not participate in the process. The observations of each panel review meeting were conducted by three (round one) or two (round two) researchers. The researchers gave a brief presentation at the beginning of each meeting outlining the role of CRU within the programme and the purpose of their observations during the panel meetings. Panel members were given the opportunity to ask questions and the researchers gave an undertaking to maintain confidentiality and anonymity.

An iterative approach was used for the observation of the round one panel meeting to allow flexibility in the research approach. This was important as there was no published relevant research in this area and there were no appropriate data collection tools or findings that could be used or adapted. We used the assessment criteria in the panel guidance document to develop tools that enabled data to be collected against each criterion during observation of the selection panel meeting (see box 1 and box 3). In addition, the data collection for the observation of the panel meeting for the first round of shortlisted applications was partially influenced by a Swedish Research Council publication that broadly explored the modus operandi of its evaluation panels (Ahlquist et al., 2013). This document primarily influenced the development and content of observation matrices used for note-taking during the overt observation.

For the round one panel meeting, two separate matrices were used to capture data on a) the contents of each application and b) panel members’ contributions and interactions (Supplementary file 1). The matrices ensured data were captured for assessment criteria areas of scientific excellence, research capacity strengthening and additional criteria (e.g. gender, career stage). It also enabled information to be captured on the selection process and panel members’ contributions (e.g. the process and format used to discuss each application, time spent on various aspects of each application including applicants’ demographic and professional backgrounds, details of how and which assessment criteria were used and discussed and other communication among the panel members).

For the round two panel meeting, the observation matrices were revised based on lessons learned from observation of the round one meeting. The main changes were to combine the two matrices into one with sub-divisions for key assessment themes (scientific strength, research capacity strengthening, other assessment criteria, themes not covered in assessment criteria). This matrix was completed for each shortlisted application as it was discussed by the panel members (Supplementary file 2). The observation of the round two panel meeting used the same researcher’s observation guide as in round one to ensure methodological consistency. As in round one, the round two observation focussed on processes and contents, verbal and non-verbal communication of panel members and any other relevant observations, with particular emphasis on which and how assessment criteria were used. This was because the award-making institution had made adjustments to the assessment criteria - to improve clarity and to make it compulsory to include strengthening of laboratories - based on our recommendations after the first panel meeting.

Data management and analysis of panel meeting observations. All the researchers’ hand-written observation notes were transferred onto an Excel spreadsheet. These data were coded and analysed under the a priori categories of ‘assessment criteria’ and ‘other observations’. Items with the same code that emerged from the data were amalgamated into themes during debriefing discussions among the researchers after each panel meeting observation. This process facilitated a balanced interpretation of the data and helped reduce the subjectivity of the researchers’ interpretations. The frequency with which key items were discussed at the meeting was also calculated. The datasets concerning the observations of the panel meetings have not been made available because they contain sensitive information and it was not possible to anonymise them without losing the meaning of the data.

Interviews with panel members

Semi-structured interviews (Supplementary file 3) with nine (six in round one, three in round two) purposively selected panel members were conducted after each of the two meetings to explore their views and perceptions of the award-making process. The choice of panel members to interview was designed to maximise diversity in nationality, expertise profile and gender. Interviews were conducted by two researchers by phone/Skype within three months of the panel meeting.

The assessment criteria were also used to inform the guides used to interview panel members. The interview guide topics included panel members’ experiences and perceptions of the award selection process, their role in the panel and their understanding of scientific excellence, research capacity strengthening and research partnerships. Since these interviews revealed some differences between interviewees’ responses, the interview guide was revised after round one so that these differences could be explored in more depth during the round two interviews. Interviews lasted 30–60 minutes. They were audio-recorded (with permission) and the interviewers took notes summarising the main issues discussed under each topic (Dataset 1). Interviewees were assured that the information they gave would be treated confidentially and anonymously to avoid the possibility that panel members’ quotes could be attributable. Since the number of interviewees was small, demographic details for interviewees’ quotes have not been provided in order to maintain anonymity.

Data management and analysis of interviews. All hand-written notes were transcribed electronically and checked against the recordings for accuracy. The data in the notes were then coded using codes which were developed iteratively based on themes that emerged from the transcripts and agreed among the research team. Once coding had been completed, links within and between codes were explored and data were grouped into higher level themes to allow for data interpretation and explanation. The panel guidance notes were also used in the analysis to determine how they were understood and followed by panel members during the meetings.

Ethical considerations

The study was approved by the Liverpool School of Tropical Medicine Research Ethics Committee (ref Research Protocol 13.14RS)

Round one

Pre-award process. The award-making institution received 26 eligible applications for round one of the programme grant. During the initial assessment stage, each panel member was allocated a selection of the eligible applications based on their area of expertise. Each application was assessed independently by three panel members using scores between one (poor) and seven (outstanding) against the assessment criteria. Fifteen applications were shortlisted for external review, six in water and sanitation, six in renewable energy and three in soil-related science. At least five external specialists assessed each application and their reports were sent to the panel members prior to the panel meeting. Their scores were collated and applications were ranked according to their average score. Each application, with its external specialists’ reports, was allocated to three of the panel members to lead on the discussion during the award-making meeting.

Award-making panel meeting. The round one panel meeting started with a general introduction of all participants, followed by a presentation by the award making institution giving relevant background information about the programme and a brief outline of the selection criteria to be used when discussing the applications during the meeting (box 1).

These assessment criteria reflected those in the panel guidance notes. However, an indication of the differential weighting between ‘primary’ and ‘additional considerations’ was provided in the introductory presentation at the panel meeting but was not included in the written guidance the panel members received in advance of the meeting.

Process: The overall format of discussion of each application was consistent throughout the meeting and the applications were not discussed in any particular order (e.g. alphabetical) which may have introduced bias. One panel member led on each proposal and presented their opinion of the application before the other two panel members contributed their comments. After the discussion of the application by the three panel members assigned to that application, all panel members were invited to express their views about the proposed project.

Applications were assigned to each panel member according to their area of expertise while aiming to also provide diversity in terms of Francophone or Anglophone and UK or SSA-based. The observation revealed that the three panel members assigned to discuss a selected application demonstrated a thorough understanding of what was written in the proposals and took into account the applicants’ expertise and backgrounds. Panel members consistently made use of the external reviewers’ comments to inform discussions about the applications. A summary report of the discussions was provided for successful applicants.

In total panel members spent 189 minutes (excluding breaks and general discussions) discussing the 15 applications (average 13 minutes per application), spending an average of 15 minutes on applications they recommended for a grant and 11 minutes on proposals that were not successful. In one instance, panel members spent 20 minutes on a ‘maybe’ application that straddled the boundary between success and rejection. It was decided that ‘maybe’ applications would be discussed further after the panel members had gone through all proposals. The rationale was that this allowed for more informed decisions of ‘maybe’ applications as panel members could compare these to all other applications.

Content: The observations revealed substantial variations in how each application was presented and therefore how it was discussed by the panel members. Panel members were not asked to use a specific format for presenting applications and, although most spoke to several of the assessment criteria, not all criteria were discussed for each application (see Table 1 for frequency of themes discussed for each round). Scientific excellence and innovation were given the greatest attention by panel members as well as the applicants’ credentials (e.g. number of publications, journal impact factors, frequency of publication) and success in obtaining grants, including prior collaborative projects.

Research capacity strengthening was discussed for almost all applications but for a shorter time and in less detail than scientific excellence. It was often used as an additional selection criterion to differentiate between two applications of similar quality. The observation revealed that panel members commonly used the term ‘capacity strengthening’ in relation to research training for individuals, including PhD candidates. There were limited discussions surrounding broader aspects of capacity strengthening such as PhD supervision plans and the institutional research infrastructure. Other aspects of some, but not all, applications were discussed by the panel, such as presentation of the application, budgets and applicant’s career age (i.e. research-active years since their PhD) especially in relation to their potential (in)experience of managing large awards. Unsuccessful applicants received feedback based on a summary of the discussions at the panel meeting and pre-meeting written reviews.

Post award interviews with panel members

Interviews with panel members after the selection meeting showed that they were positive about the process and emphasised the high levels of rigour, collaboration and balanced expertise of the panel members. Their assessment was primarily based on their previous experiences of serving on other review panels. Responses to questions about how the panel worked as a team showed that all those interviewed agreed that the committee was cooperative and respectful and that panel members were given enough time to speak and listened to each other. Interviewees highlighted that for each application consensus was reached based on thorough and constructive discussions. A few interviewees attributed the success of the panel to effective chairing of the meeting.

Specific expertise in one or more of the three priority research areas of the programme was quoted as the main reason why panel members believed they were approached to participate in the review process. One panel member also thought she had been selected because she was female and an African panel member felt he was invited because he could provide an African perspective and contextual expertise. He noted that during the review process he ‘scrutinised the applications with regard to development relevance for Africa’.

Most interviewees felt that they were provided with clear guidelines about how to review the applications. However, interview data mirrored the researchers’ observations that for some aspects panel members had different understandings about the interpretation or weighting of the assessment criteria, particularly relating to the balance between scientific excellence and research capacity strengthening, as the following two quotes exemplify:

Interview data showed that panel members had a common understanding of scientific excellence, focussing primarily on innovative research projects and the applicants’ publication and grant records. However, responses to interview questions about what they thought were key characteristics of research capacity strengthening showed a range of different responses. Characteristics mentioned by the various panel members included: sustainable physical and human infrastructure of institutions, multi-disciplinary and international collaborations, training of researchers at different levels (including students, postdoctoral researchers and supervisors), research uptake strategies and sustainable monitoring and evaluation processes. There did not appear to be a common understanding of the concept of ‘research capacity strengthening’ among the panel members.

Learning generated by developmental evaluation of round one selection process and used to influence round two. Recommendations for improvements to the selection panel process for round two were derived from the findings from research into round one selection processes (box 2). These recommendations included improving the clarity and specificity of assessment criteria guidelines, a more standardised structure for presenting each application and increasing the diversity of the panel.

Round two

Pre-award process. The shortlisting process for the 47 eligible applications was the same as for round one except for the use of revised guidance on assessment criteria (see box 3).

Award-making panel meeting. The overall format of the panel meeting was similar to round one with presentation of the applications by the relevant lead panel members, followed by open discussion by the whole panel.

Process. Compared to round one, there was less variation in how the applications were discussed and more consensus in the decision-making. In total panel members spent 178 minutes (excluding breaks and general discussions) discussing the shortlisted 19 applications (average 9 minutes/application versus 13 minutes/application in round one). The relative time spent on successful, unsuccessful and ‘maybe’ applications was similar to that in round one. Panel members spent on average 11 minutes on the 5 applications they recommended for a grant (15 minutes in round one), 9 minutes on the 14 unsuccessful proposals (11 minutes in round one) and 14 minutes on the ‘maybe’ applications (20 minutes in round one).

Content. Compared to round one there was less variation in the focal areas of the discussions concerning scientific strength. These areas covered reviewers’ scoring, the novelty of the research, research methods and how well individual research projects were integrated within a consortium (see Table 1 for frequency of themes discussed for each round). The panel also discussed areas not covered in round one and not included in the revised assessment guidance, such as dissemination of research findings, transferability of results and risk assessments. Compared to round one there was also more discussion on development relevance for Africa and greater consensus among the panel regarding applications recommended and not recommended for funding which was reflected in the shorter time needed to discuss each application in round two compared to round one.

There were no major differences between round one and round two in how panel members discussed the credentials of scientists and their complementarity within consortia. Discussions focused on applicants’ complementary research backgrounds, especially their publication and grant records, frequently using the term ‘well-published’.

Research capacity strengthening components of applications were discussed in 13 out of the 19 applications; these were not necessarily those in which scientific strength was considered to be possibly insufficient. The panel considered many of the points under research capacity strengthening that were listed in the revised guidance notes (e.g. relevance of training, equipment and statistics training, cross partner training). However, the training of laboratory technicians was only discussed once and there was limited consideration of how proposals would impact on the wider research infrastructure. Although sustainable research networks were mentioned as a goal of the scheme this topic was not explicitly discussed by the panel members.

Post award interviews with panel members

All three interviewees assessed the award-making process positively, commenting particularly on how well the panel worked as a group, the chairing of the meeting and how the panel made informed decisions about funding. They highlighted a high level of mutual respect among the panel members and an increased level of familiarity with the process based on experiences from round one. Despite using the revised assessment guidance notes in round two, the interviews revealed that there was still some ambiguity in the interpretation of the weighting of the assessment criteria. Whereas two panel members highlighted the importance of scientific strength, noting ‘we have to select science, capacity strengthening was always the second tier’, another panel member (Interviewee 7) explained:

One panel member felt that capacity strengthening was given less attention during this panel meeting compared to the previous one, but in contrast another interviewee (Interviewee 8) noted that there was more consensus around the focus on capacity strengthening issues:

In line with the panel guidance notes, interviewees noted that additional points, such as gender and the career level (i.e. early- or mid-career) of African scientists and non-Anglophone applications, were discussed in less detail than scientific strength and capacity strengthening.

Study strengths and limitations

The overt observation method was chosen because it would yield a rich amount of data when exploring the award-making process, how panel members used the guidance notes and how the two rounds of award-making differed from each other. The presence of more than one researcher taking independent notes at each meeting and discussing findings among researchers immediately after the meeting promoted an unbiased interpretation of the data and enabled the researchers to document processes and interactions that the panel members themselves might consider self-evident or not worth mentioning. However, overt observation does not allow exploration of the cause of observed phenomena and it is not possible to be confident that observations represent normal behaviours since subjects may act differently when they know they are being watched (McNaughton et al., 2013). To mitigate this possibility the observation data were compared with information from the semi-structured interviews with panel members. The interviews provided information about experiences of the panel members which could not be captured through observations. It is possible that interviewees might have given responses that did not truly reflect their feelings as they were aware that information might be shared (anonymously) with the award making institution. Therefore, before each interview the interviewees were reassured that their information would be treated confidentially and anonymously.

Generalisable lessons learnt about evaluating complex multi-national research consortia applications

1.    The constitution of the panel should be diverse enough to reflect the scientific topics, LMIC context, gender and language of the applications and should be maintained across different rounds of awards to ensure consistency.

2.    The shortlisting and selection process, with involvement of external reviewers, pooled scores and discussions during the panel meeting, is in line with funders’ benchmarks and can be applied to these complex consortia applications to provide a rigorous and equitable selection mechanism.

3.    Assessment criteria need to be defined in terms of content and their weighting. The number of assessment criteria should be limited to those most revelvant to the programme’s aims to enhance consistency during the discussions.

4.    The guidelines should specify whether criteria such as gender and career stage must be considered in all cases, or whether they should only be used as ‘tie breakers’ and how much weight they should carry.

5.    For programmes which have dual (but not exclusive) aims of generating high quality research and strengthening research capacity: