Methods and Uses of Peer Review -Challenges and Lessons Learned From a

Canadian Perspective

Jennifer Birta, National Research Council CanadaNew Frontiers in (R&D) Evaluation Conference

Vienna, Austria25 April 2006

2

Presentation Overview

• Overview of the National Research Council

of Canada (NRC)

• Terminology

• Case Study– Overview of initiative– Peer reviews conducted– Challenges and opportunities

• Conclusions

3

National Research Council Canada (NRC)

2004-2005

# of papers published: 1,286

# of patents issued: 86

# of licenses issued: 105

• Agency of the Canadian federal government

• Principal scientific research body conducting intramural research

• Range of research from fundamental to applied in areas such as:– Molecular sciences– Astronomy– Biotechnology– Environmental technologies

• Provision of innovation support– Industrial Research Assistance Program– Technology cluster strategy

4

National Research Council Canada (NRC)

• 20 institutes and 10 other research centres

• 4,000+ employees

• 1,300+ students, visiting/guest workers

• Laboratories and facilities across the country

• Total expenditures 2004-05: $712M

• Income 2004-05: $103.7M

5

Terminology

PURPOSES

Journal manuscript review

Proposal review

Evaluation review

Review of research results for competence, significance and originality prior to publication

Assessment of proposals for future funding decisions

Performance assessment of level of excellence and impact of past work

METHODS

Paper-based

Panel

Standing Committee

Provide written opinion only, no interaction

Come together to conduct a single review

Conduct many reviews over a term

6

Case Study: Overview of NRC’s Genomics and

Health Initiative

• Horizontal initiative across 10 NRC Institutes• Research focus fundamental and applied (e.g., diagnosis of

disease, crop enhancement)• Annual expenditures ~ $25M (CDN)• Started in 1999, divided into phases of 3 year lengths, internal

competitive process used to select programs• Responsibility of the VP Life Sciences, managed by a coordination

office and a standing committee

7

Case Study: Peer Reviews of NRC-GHI

• Research program selection every 3 years– Competitive approach for funding– Scientific and market-driven assessment criteria– Two-tiered approach used

• Periodic retrospective evaluation– Assessment of relevance, success, alternatives and

effectiveness of design/delivery– Seven methodologies used

• Peer review achievement of objectives and progress made

– Two-tiered approach used

8

Case Study: Challenges

Subjective Bias

• Reliance on human judgement

• Trust scientists’ desire not to diminish their reputations

Confidentiality

• Possibility of inappropriate use of knowledge gained during review

• Precautionary measure to sign a confidentiality agreement

• Best defence not to disclose sensitive or proprietary information

9

Case Study: Challenges

Conflict of Interest• Ability of reviewers to participate without conflict with their own

interests• Difficult to find qualified individuals with no prior knowledge of

program under review• Sources of potential reviewers• Disclosure of conflict of interest

Timing• Short timeframe between two peer reviews• Most qualified individuals had already participated• Result low return rate

10

Case Study: Challenges

Reviewer Fatigue

• More S&T organizations relying on peer review

• Difficulty finding specialists, willing and able to participate

Level of Commitment

• Paper-based reviews lack sense of community that exists with panels/committees

• Dependent on profile/reputation of initiative under review

• Might be increased with provision of honoraria

11

Case Study: Challenges

Supporting Materials

• Existing documentation has limited applicability

• Customized materials required

Costs

• Travel and hospitality expenses, honoraria (if applicable)

• Loss of productivity for reviewers and those being reviewed

12

Case Study: Opportunities

Standing Committees

• Benefit from familiarity and sense of obligation

Integration

• Combine retrospective and prospective portions of reviews

Honoraria

• Revisiting NRC’s policy

13

Conclusions

Way forward…

• Has the support of most researchers/scientists

• Alternatives will do little to reduce pressures on the system

• Remains the most appropriate means to assess scientific meritof past, present and future work

14