1 csr’s mission and function and what’s new in peer review martha m. faraday, ph.d. scientific...
Post on 16-Jan-2016
220 Views
Preview:
TRANSCRIPT
1
CSR’s Mission and Function
and
What’s New in Peer Review
Martha M. Faraday, Ph.D.
Scientific Review OfficerDivision of AIDS, Behavioral & Population Sciences
Risk Prevention & Health Behavior IRGPsychosocial Risk & Disease Prevention Study Section
Date : April 22, 2009
National Institutes of HealthU.S. Department of Health and Human Services
National Institutes of HealthNational Institutes of Health
National Instituteon Alcohol Abuseand Alcoholism
National Instituteon Alcohol Abuseand Alcoholism
National Instituteof Arthritis andMusculoskeletal
and Skin Diseases
National Instituteof Arthritis andMusculoskeletal
and Skin Diseases
National CancerInstitute
National CancerInstitute
National Instituteon Drug Abuse
National Instituteon Drug Abuse
National Instituteof Environmental Health Sciences
National Instituteof Environmental Health Sciences
National Instituteon Aging
National Instituteon Aging
National Instituteof Child Health
and HumanDevelopment
National Instituteof Child Health
and HumanDevelopment
National Institute onDeafness and Other
CommunicationDisorders
National Institute onDeafness and Other
CommunicationDisorders
National EyeInstitute
National EyeInstitute
National HumanGenome Research
Institute
National HumanGenome Research
Institute
National Instituteof Mental Health
National Instituteof Mental Health
National Instituteof NeurologicalDisorders and
Stroke
National Instituteof NeurologicalDisorders and
Stroke
National Instituteof General
Medical Sciences
National Instituteof General
Medical Sciences
National Instituteof Nursing Research
National Instituteof Nursing Research
National Libraryof Medicine
National Libraryof Medicine
Center for Center for Scientific ReviewScientific Review
National Centerfor Complementary
and AlternativeMedicine
National Centerfor Complementary
and AlternativeMedicine
National Instituteof Allergy and
Infectious Diseases
National Instituteof Allergy and
Infectious Diseases
FogartyInternational
Center
FogartyInternational
Center
National Centerfor ResearchResources
National Centerfor ResearchResources
Clinical Center
Clinical Center
National Center on Minority Health andHealth Disparities
National Center on Minority Health andHealth Disparities
National Institute of Biomedical Imagingand Bioengineering
National Institute of Biomedical Imagingand Bioengineering
Office of the DirectorOffice of the Director
Center for InformationTechnology
Center for InformationTechnology
National Heart,Lung, and Blood
Institute
National Heart,Lung, and Blood
Institute
National Instituteof Dental andCraniofacial
Research
National Instituteof Dental andCraniofacial
Research
National Instituteof Diabetes andDigestive and
Kidney Diseases
National Instituteof Diabetes andDigestive and
Kidney Diseases
CSR Mission StatementCSR Mission Statement
To see that NIH grant applications receive
fair, independent, expert, and timely reviews -
free from inappropriate influences - so the
Institutes and Centers within the NIH can fund
the most promising research.
CSR Peer Review: 2008 StatisticsCSR Peer Review: 2008 Statistics
• 77,000 applications received77,000 applications received
• 56,000 applications reviewed 56,000 applications reviewed
• 16,000 reviewers16,000 reviewers
• 240 Scientific Review Officers240 Scientific Review Officers
• 1,600 review meetings1,600 review meetings
Scientific Review ProcessScientific Review Process
Dual Review System for Grant ApplicationsDual Review System for Grant Applications
Second Level of ReviewSecond Level of Review NIH Institute/Center CouncilNIH Institute/Center Council
First Level of Review : First Level of Review : CSR/Institute ReviewCSR/Institute Review
Scientific Review Group (SRG)Scientific Review Group (SRG)
(Study Section)(Study Section)
Translational and Translational and
Clinical SciencesClinical Sciences
Cardiovascular and Cardiovascular and
Respiratory Sciences Respiratory Sciences
Surgical Sciences, Surgical Sciences,
Biomedical Biomedical
Imaging and Imaging and
Bioengineering Bioengineering
Musculoskeletal, Oral Musculoskeletal, Oral
And Skin Sciences And Skin Sciences
Oncology 2 – Oncology 2 –
Translational Clinical Translational Clinical
Vascular and Vascular and
HematologyHematology
Physiological and Physiological and
Pathological Sciences Pathological Sciences
Endocrinology, Endocrinology,
Metabolism, Metabolism,
Nutrition &Nutrition &
Reproductive SciencesReproductive Sciences
ImmunologyImmunology
Infectious DiseasesInfectious Diseases
& Microbiology& Microbiology
Digestive, Kidney &Digestive, Kidney &
Urological Systems Urological Systems
Neuroscience, Development Neuroscience, Development
and Agingand Aging
Brain Disorders &Brain Disorders &
Clinical NeuroscienceClinical Neuroscience
Molecular, Cellular &Molecular, Cellular &Developmental NeurosciencDevelopmental Neuroscience
Integrative, Functional & Integrative, Functional &
Cognitive NeuroscienceCognitive Neuroscience
Emerging Technologies &Emerging Technologies &
Training in NeuroscienceTraining in Neuroscience
Biology of Development Biology of Development
& Aging& Aging
Biobehavioral &Biobehavioral &
Behavioral ProcessesBehavioral Processes
Risk, Prevention& Risk, Prevention&
Health Behaviors Health Behaviors
Population Sciences & Population Sciences &
EpidemiologyEpidemiology
Healthcare Delivery Healthcare Delivery
& Methodologies& Methodologies
AIDS &AIDS &
Related ResearchRelated Research
AIDS, Behavioral AIDS, Behavioral
and Population Sciencesand Population SciencesBasic and Integrative Basic and Integrative
Biological SciencesBiological Sciences
Biological Chemistry & Biological Chemistry &
Macromolecular Macromolecular
Biophysics Biophysics
Bioengineering SciencesBioengineering Sciences
& Technologies& Technologies
Genes, Genomes Genes, Genomes
& Genetics & Genetics
Oncology 1 – Basic Oncology 1 – Basic
TranslationalTranslational
Cell BiologyCell Biology
Interdisciplinary Interdisciplinary
Molecular Molecular
& Training& Training
CSR Review DivisionsCSR Review Divisions
• Standing Study Sections when the subject matter of the application matches the referral guidelines for the study section
• Special Emphasis Panels (SEPs) when the subject matter does not fit into any study section, or when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAs)
Within an IRG, applications are assignedfor review to
Assignment to CSR Review Groups
• Read the instructions
• Never assume that reviewers “will know what you mean”
• Refer to the literature thoroughly
• State rationale of proposed investigation
• Include well-designed tables and figures
• Present an organized, lucid write-up
• Remember to address human subjects, vertebrate animals, potential biohazards; these could affect your score
• Obtain pre-review from faculty at your institution
NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm
When Preparing Your ApplicationWhen Preparing Your Application
Directing Your Application to a Specific Study Section
• Peruse CSR Study Section Guidelines* to Identify a Possible Home for Your Application
http://csr.nih.gov/
* Recently revised; alternative study sections listed in approximate order of degree of overlap
• Submit a Cover Letter
• About CSRAbout CSR
• News and ReportsNews and Reports
• Peer Review Meetings
• Resources for Applicants Resources for Applicants
CSR Web Site: http://www.csr.nih.gov
Role of Scientific Review Officer (SRO)Role of Scientific Review Officer (SRO)
• Performs administrative and technical review of applications to ensure completeness and accuracy
• Selects reviewers based on broad input
• Manages study section meetings
• Prepares summary statements
• Provides any requested information about study section recommendations to Institutes/Centers and National Advisory Councils/Boards
Designated Federal official with overall responsibility for the review process
WHOM DO I CONTACT?WHOM DO I CONTACT?
Before review, contact the Scientific Review Officer in CSR
After review, contact your Program Officer in the NIH funding institute or center
Pre-Meeting Review ProcessPre-Meeting Review Process
• Appropriate reviewers recruited by SRO; minimum of 3 “interactive” reviewers per application
• Conflicts of interest identified
• Applications made available to reviewers ~6 weeks prior to meeting
• Critiques and preliminary scores posted by assigned reviewers on NIH web site at least 2-3 days prior to meeting
• Critiques and preliminary scores (excluding conflicts) available to review group prior to meeting
Where Do We Find Reviewers?Where Do We Find Reviewers?
• National Registry for Society-Recommended Reviewers
• Successful applicants
• Word of mouthRecommendations from study section membersRecommendation from NIH IC staff
• CRISP (crisp.cit.nih.gov)
• PubMed
• Scientific Conferences
Traditional* Review Meeting Process
• Upper half applications discussed:Reviewers are guided by specific review criteria Protections for Humans, Vertebrate Animals, Environment (Biohazard) may affect final scoreAssigned reviewers recommend scores for each application in upper half; all members not in conflict vote their conscience (outlier score policy pertains)Other considerations not affecting final score are discussed (e.g., budget, foreign applicants, resource sharing plans)
• Lower half applications not discussed, not assigned an overall score
* Aspects of this process will change in May, 2009
http://enhancing-peer-review.nih.gov
Post Meeting Review Process
• Scores are provided to investigators within 3 working days
• Summary Statements for discussed and scored applications include Resume & Summary of Discussion, (largely unedited) critiques, and other recommendations (e.g., Budget)
• Summary Statements for lower half (Not Discussed) applications receive (largely unedited) critiques and review criteria scores but no overall impact scores
• All Summary Statements are made available within 30 days of meeting (10 days for new investigators’ R01s)
What’s New in Peer Review?What’s New in Peer Review?
Enhancing Peer ReviewEnhancing Peer Review
““Fund the best science, by the best scientists, Fund the best science, by the best scientists, with the least administrative burden…” with the least administrative burden…”
Elias Zerhouni, MD, Former Director, NIHElias Zerhouni, MD, Former Director, NIH
2008: The Year of Peer Review2008: The Year of Peer Review
RecommendationsRecommendations
Amended Applications:Amended Applications:
To speed the funding of meritorious science and To speed the funding of meritorious science and minimize reviewer burden:minimize reviewer burden:
•As of January 25, 2009, all original new applications (i.e., never submitted) and competing renewal applications will be permitted only a single amendment (A1)single amendment (A1).
What’s New in Peer ReviewWhat’s New in Peer Review
• New Investigators/Early Stage InvestigatorsNew Investigators/Early Stage Investigators
• Enhanced Review CriteriaEnhanced Review Criteria
• Template-Based CritiquesTemplate-Based Critiques
• Scoring Scale (9 point scale)Scoring Scale (9 point scale)
Criterion ScoringCriterion Scoring
Overall Impact ScoreOverall Impact Score
21
New Investigators/Early Stage InvestigatorsNew Investigators/Early Stage Investigators
• New Investigator (NI): New Investigator (NI):
PD/PI who has not yet competed successfully for a substantial NIH research grant
o For multiple PD/PIs-all PD/PIs must meet requirements for NI status
• Early Stage Investigator (ESI):Early Stage Investigator (ESI):
PD/PI who qualifies as a New Investigator AND is within 10 years of completing the terminal research degree or is within 10 years of completing medical residency (or equivalent)
• Applies only to R01 applicationsApplies only to R01 applications• New Investigators/Early Stage Investigators will be clustered New Investigators/Early Stage Investigators will be clustered
together for reviewtogether for review
22
Enhanced Review CriteriaEnhanced Review Criteria
• Overall Impact: Overall Impact: Assessment of the likelihood for the project to exert a sustained, powerful influence on the exert a sustained, powerful influence on the research field(s) involvedresearch field(s) involved
• New Core Criteria Order:New Core Criteria Order:Significance
Investigator(s)
Innovation
Approach
Environmento Review criteria enhanced and expandedReview criteria enhanced and expanded
23
CritiquesCritiques
Goal: To improve the quality of the critiques and to Goal: To improve the quality of the critiques and to focus reviewer attention on the review criteria:focus reviewer attention on the review criteria:
•Provide clear, concise, and explicit information•Aid in identifying the strengths and weaknesses of each criterion
Template-Based CritiquesTemplate-Based Critiques
Template-Based CritiquesTemplate-Based Critiques
• Critique template contains a total of 18 boxesCritique template contains a total of 18 boxesReviewers should provide text for only those criteria that Reviewers should provide text for only those criteria that are applicable.are applicable.
25
1. Significance1. Significance 6. Resubmission 13. Overall Impact13. Overall Impact
2. Investigator(s)2. Investigator(s) 7. Renewal 14. Budget and Period of Support
3. Innovation3. Innovation 8. Revision 15. Select Agents
4. Approach4. Approach 9. Protection of Human Subjects
16. Applications from Foreign Organization
5. Environment5. Environment 10. Inclusion of Women, Minorities, and Children
17. Resource Sharing Plan
11 Vertebrate Animals 18. Additional 18. Additional Comments to Comments to ApplicantApplicant
12. Biohazards
Template-Based CritiquesTemplate-Based Critiques
• Goal: is to write evaluative statements and to Goal: is to write evaluative statements and to discourage summarizing the applicationdiscourage summarizing the application
Comments should be in the form of bullet points or if necessary short narratives
Do not record scores Do not record scores on the critique template
The entire template is uploaded to IAR to become part of the summary statement.
26
1. SignificanceSignificance Please limit text to ¼ pagePlease limit text to ¼ page
Strengths Weaknesses
Goal: To improve the transparency of the scoring Goal: To improve the transparency of the scoring process:process:
• Score applications on five review criteria using five review criteria using a scale of 1-9a scale of 1-9.
• Preliminary overall impact score using 1-9 overall impact score using 1-9 scalescale.
Should not be the average of the criterion scores.
Not DiscussedNot Discussed applications will receive initial criterion scores from the three assigned reviewers
Scoring – 9 Point ScaleScoring – 9 Point Scale
Scoring DescriptionsScoring Descriptions
Impact Score Descriptor Strengths/Weaknesses
High Impact
1 Exceptional
Weaknesses
2 Outstanding
3 Excellent
Moderate Impact
4 Very Good
5 Good
6 Satisfactory
Low Impact
7 Fair
8 Marginal
9 Poor
29
ClusteringClustering
• NI/ESI R01 NI/ESI R01 applications will be clustered together in review.
ESI applications will not be separately ESI applications will not be separately clustered within the NI\ESI group.clustered within the NI\ESI group.o NI/ESI applications will be identified for
reviewers so there can be appropriate review in context of career stage.
o Expectations of preliminary data and publication track record less than for established investigators.
Order of ReviewOrder of Review
Goal: Discuss applications in order of average Goal: Discuss applications in order of average preliminary score.preliminary score.
Why:Why:•Concern - variation of scores during different times of the meeting.
One recommendation was to recalibrate scores at One recommendation was to recalibrate scores at the end of the meeting .the end of the meeting .
Solution:Solution:•Recalibrate “dynamically” throughout meeting.
Order of ReviewOrder of Review
• For calibration purposes: For calibration purposes: Begin meeting by discussing the best scored application (any activity code)
o NI/ESI R01s clustered beginning of meetingNI/ESI R01s clustered beginning of meetingo All other activity codes clustered if feasible (if at
least 10 discussed (may include R03, R15, and R21s as a group that can be clustered)
Order of ReviewOrder of Review
SummarySummaryDiscussion order is based on the average of the impact scores from assigned reviewers
Final scores of discussed applications may differ from preliminary scores as re-calibration happens dynamically
Discuss ~ 50-60% of applications
SRO will then ask if there are any other applications that panel wishes to discuss
The remaining applications will not be discussed(applications receive criterion scores only)
o Same after review of ~60% of SBIR applications
Not DiscussedNot Discussed
Final ScoresFinal Scores
• Discussed applications will receive an overall score from each eligible (i.e., without conflicts of interest) panel member and these scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.
• Percentiles will be reported in whole numbers.
•Summary statement will be shortershorter and more focused.
•DiscussedDiscussed applications will also have a summarysummary of of the panel’s discussiondiscussion at the meeting.
•ALL applications will be scoredALL applications will be scored.
Not discussed Not discussed applications will receive criterion scorescriterion scores only.
Summary StatementsSummary Statements
Recruiting the Best ReviewersRecruiting the Best Reviewers
Move a meeting a year to the West Coast
Additional review platforms
Develop a national registry of volunteer reviewers Searchable database with 4,000 reviewers
Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). 1574 chartered members used flexible deadlines during the last 6 months
Provide flexible time for reviewers Choice of 3 times/year for 4 years or2 times/year for 6 years
THANK YOU!THANK YOU!
This is CSRThis is CSR
top related