The Grant Writing and Review Process at NIH

Joshua SmythProfessor of Biobehavioral Health and Medicine; Associate Director,

SSRI

Rhonda BeLueAssociate Professor of Health Policy and Administration

Jenae NeiderhiserProfessor of Psychology

Danielle DownsAssociate Professor of Kinesiology and Obstetrics & Gynecology

Thanks to:

Lori Francis, Associate Professor of Biobehavioral Health

Brittany Frost, Social Science Research Institute

NIH Organization NIH Funding Mechanisms The Grant Writing Process

Focus on the R01 The NIH Review Process

Overview of Review Meeting The Scoring Process A Penn State example

Workshop Evaluation

Workshop Outline

I. The NIH Department of Health and Human

Services National Institutes of Health

25 Awarding Institutes/Centers aka ICs e.g., NICHD, NIMH, NIDA, NIA, etc.

Center for Scientific Review Office of the Director

The NIH Extramural Team

A. Program

B. Grants Management

C. Review

A. The Program/Institute Staff

Program Administrator

Maintains knowledge of scientific area Attends study section meetings Makes funding recommendations Monitors scientific progress Identifies scientific area of importance Reports to senior staff Development of programs and initiatives

B. Grants Management

Interprets Federal regulations and policies

Assures compliance with Federal regulations and policies

Monitors financial aspects of projects Interprets regulations and policy

C. Review:Scientific Review Group (1st

Level)

Center for Scientific Review (CSR) or NIH Institute & Center (IC)

Scientific Review Group (SRG) Non-federal scientists with relevant

expertise Led by a Scientific Review Officer (SRO)

http://www.csr.nih.gov/Roster_proto/sectionI.asp

C. Review:Advisory Council or Board (2nd

Level)

The potential awarding IC performs the second level of review

NIH program staff examine applications for impact (formerly “priority”) scores, percentile rankings, & summary statements against the IC’s needs

Program staff provide grant funding plan to Advisory Council or Board

Advisory Council or Board advises the IC director Director makes final decision

NIH Grant Application Cycle

Allocates Funds

Investigator NIHInstitution

Submits Application

Peer Review

Council Review

Funding Decision

Initiates Research

Conducts Research

Grant Application: It’s a process, not an event

1. Communicate with Program Officer Introducing ideas, getting feedback, pre-review

2. Get your proposal to the right review committee

Review the rosters and talk to colleagues Effectively wording the abstract Make a written request

3. Seek feedback from colleagues and consultants on drafts of the grant (prepare ahead!)

4. Consider who is likely to review your grant (review the rosters) and make sure to know and cite their work when relevant

5. Recognize that funding on first submission is rare

Ks: NIH Career Development Awards (K01, K02, K05, K07, K08, K22 [K99/R00])

P01: Research Program Project Grant P30: Center Core Grants R01: NIH Research Project Grant Program R03: NIH Small Grant Program R13: NIH Support for Conferences and

Scientific Meetings (R13, U13) R15: NIH Academic Research Enhancement

Award (AREA)

II. NIH Grant Mechanisms

R21: NIH Exploratory/Developmental Research Grant Award

R34: NIH Clinical Trial Planning Grant T series: NRSA Training Grants (T32, T34, T35,

T90, etc.) U series: Research Project Cooperative

Agreement Diversity Supplements: Research

Supplements to Promote Diversity in Health-related Research

Roadmap: NIH Roadmap Initiatives (Director’s Pioneer Award; Director’s New Innovative Program)

NIH Grant Mechanisms (continued)

R01 (primary focus today)

Used to support a discrete, specified, circumscribed research project

NIH’s most commonly used grant program

Body of the grant (Research Strategy) is 12 pages (mostly single spaced)

Generally awarded for 3 to 5 years Up to $500,000/year without exceptions

needed Advance permission required for $500,000 or

more (direct costs) in any year

New and Early Stage Investigators:A Competitive Edge

New Investigator has not previously served as a PI for an R01; may have been an investigator or received other smaller, developmental or research training awards

Early Stage Investigator (ESI) is within 10 years of completing her/his terminal research degree, or completing medical residency

Grant writing is: A skill like any other… But not the same skill as article writing

Instead, more of a problem-based writing activity (theory and practice problem)

III. The Grant Writing Process

Start early, make a timeline and STICK TO IT Should allow time for serious pre-submission

review and subsequent revision Develop a relationship with project

officers It is not possible to overdo clarity Let your passion come through in your

proposal Take advantage of early stage and new

investigator opportunities

A few preliminary tips

Know what has been done Know what has been funded

NIH website RePORTER (formerly CRISP) Decide on the problem

Important enough to get funded but simple enough to explain as clean design

in 12 pp for the R01; less for some other mechanisms

Assemble team Complementary skills;

Seniority/competence/other by association; People you can count on

Getting ready to write

Communicate with program officer Establish a relationship and trust

(funding decision) Acquire information on mechanism

and priorities Obtain input on aims/proposal

Getting ready to write

Face Page Table of Contents Performance Sites Other information

Project Summary (Description) Public Health Relevance Statement Facilities & Resources

Main Sections of the NIH Application(see Francis et al. for some examples)

Key Personnel Biosketches -- with personal

statements Budgets (for each study year)

Budget Justification Other sections (not discussed

today); for example: Clinical Trial and Human Embryonic

Stem Cell (HESC) List of Research Plan Attachments

More Sections

[Introduction – revisions only] Specific Aims: The basis for the proposal’s

organization Research Strategy

Significance and Innovation Approach

Preliminary studies Design Sample/recruitment/power analyses Procedures & measures Analyses

Main Sections of the R01: Specific Research Plan

Page Limit Guide: Plan your proposal with these limits in mind

Section of Application Page Limits

Introduction (for resubmission application only) 1

Specific Aims 1

Research Approach: R03, R13/U13, R21, R36, R41, R43, Fellowships (F), SC2, SC3

6

Research Approach: R01, single project U01, R10, R15, R18, U18, R33, R24, R34, U34, R42, R44, DP3, G08, G11, G13, UH2, UH3, SC1

12

Biographical Sketch 4

Page limits may vary for other funding mechanisms.

Check Funding Opportunity Announcement: http://grants.nih.gov/grants/guide/search_results.htm?scope=pa&year=active

Protection of Human Subjects Women and Minorities Planned Enrollment Table Children

References Cited Letters of Support Resource Sharing Plan Checklist SO START EARLY!

And More Sections

Clarity is key – each point follows naturally Tell a (scientific) story

Everything that needs to be there, is; nothing extra

Communicate your excitement Get feedback early on

And often

General themes of success

Impact and Significance Practice (2-3 sentences)

Prevalence of problem in population Important social concern/public health

problem Theory (model)

Building, testing, using Innovation

New directions, value added Compelling Preliminary Research

General themes to R01 Success

Specific Aims may be only part read by some reviewers No more than one page Tell the entire story

Functions as an abstract would in a manuscript

End with (actual) specific aims Hypotheses and aims must align

Keys to Success – Specific Aims

Background and significance Used to be an explicit section; no

longer This is where you ‘hook’ the reader

on your story Functions as the ‘introduction’ does in

a manuscript Likely no more than 2-3 pages

Keys to Success – Background

Research team collaboration no longer required to be documented Still very helpful

Preliminary study ‘types’ – document: Relevant previous research Research that demonstrates

competence in requisite domains Capacity to recruit in specific

populations and/or contexts Pilot data specific to proposal

Preliminary Studies

Methods are very important Overall -- clarity and detail May include a table that traces aims to

hypotheses to constructs to measures (table/s)

Is the design feasible? Are there gaps in the methods (e.g., fidelity

for interventions) Statistics are essential – product must match

aims Consider a methodologist team member

Include a detailed timeline

Keys to Success - Methods

Preference for representative samples Students only if relevant to

age/situation (e.g., college drinking) Generalizability from a single

entity (university, clinic, state) Justified exclusions

Unit of assignment is unit of analysis

Sample size/power analysis For each outcome/planned test

And More Methods: The Sample

Direct Costs Senior Personnel (PI, co-Is, project director)

(PSU fringe at 36.5%) Other Personnel (full time staff, RAs, part time

wages) (PSU fringe 36.5% for full time staff, 13.2% for Grad Assts AY; 7.9% for part time wages and summer)

Equipment Travel Participant/Trainee Support Costs

Budgets

Other Materials and supplies Publication costs Consultant services

Subawards/Consortium/Contractual Fees Other

Indirect Costs (~50% at PSU, but does not include all expenses)

Budget justification

Budgets, cont.

Receiving the Summary Statements: The Hardest Part!

1. Reviews critical, even harsh2. Reviewers usually find grant’s

weaknesses, while recognizing strengths

3. Summary statements spend much more time on critique than praise

4. Many investigators experience a mixture of rage and depression when they read their summary statements and easily lose perspective

5. Take a day or two (or more!) and then read again with a cooler head

Receiving the Summary Statements: Bouncing Back!

1. Ask experienced colleagues to read reviews

2. Don’t interpret criticism as hopeless3. Program Officer may be helpful in

clarifying critique4. If “discussed” (rather than triaged), you

have a chance of funding in next round5. The lower the initial score, the fewer

problems and more likely to be successful after revision

1. Persistence pays off in the grant process!!

2. Second submission must respond to the critiques through revision or clearly defending reasoning

3. Same reviewers may or may not review resubmission, but will see critique

Resubmission:Resilience and Flexibility!

Most Common Reasons for a Poor Score (in priority order)

Lack of impact or significance Lack of new or original ideas Hypotheses ill-defined, superficial, lacking, unfocused, or

unsupported by preliminary data Methods unsuitable, not feasible, not rigorous or not likely to

yield results; methods don’t clearly link to aims Design not logical, inappropriate instrumentation, poor

timing or conditions; doesn’t link well to aims Data management and analysis vague, not rigorous;

analyses don’t clearly link to aims Inadequate expertise or knowledge of field for PI; too little

time to devote to the work Poor resources or facilities; limited access to appropriate

population

When to Revise Basic idea was significant and

innovative or these can be bolstered

Design/measurement/analysis problems can be clarified (more information) or fixed

Need preliminary data Problem is poor writing

A. The Review Meeting B. Review Discussion C. The Scoring Process D. A Penn State Example

IV. The NIH Review Process

A. The Review Meeting: The SRO’s Role Prior to Meeting Point of contact until review group meets

(then project officer) Analyze submissions for completeness and

conflicts Recruit ad hoc reviewers as needed Schedule 1-2 day meeting Assign applications to reviewers (at least 3)

Primary, secondary, discussant Create review order based on preliminary

impact scores from best to worst within categories

Reviewers’ Role Prior to Meeting Familiarize self with criteria, mechanisms, and

scoring Review assigned applications

Assign scores to each criteria and other areas Write bulleted strengths and weaknesses for each

criteria Reviews are advice to institutes for funding

decisions, not advice to PI Post scores and comments on NIH Commons Read other reviews of assigned applications Prepare presentation of reviews Skim/read non-assigned applications

Format of the Review Meeting

SRO opening remarks Chair orientation New investigator R01 grants Other R01 grants Other grant types (R03, R15, R21,

R34) Applications discussed in order of

Impact Score; bottom 50% are not discussed

Confidentiality Review order Proposals below median within

each category may not be discussed

SRO Opening Remarks

Chair Orientation Start with reviewer impact scores

May differ from posted scores Goal of discussion is to clarify not

reach agreement If scores are similar, shorter discussion If scores are dissimilar, longer

discussion Recommended time

Primary – 5 minutes Secondary – 3 minutes Discussant – 2 minutes

B. Review Discussion Identify proposal Members in conflict leave Reviewers provide preliminary impact scores Reviews

Impact, Significance, Investigators, Innovation, Approach, Environment

Stress main points, do not repeat previous points Non-reviewers typically ask questions to clarify

Human Subjects issues affecting scoring Open discussion to entire committee

Review Discussion (continued)

Ask for reviewers impact scores again Identify the reviewers’ recommended

range Ask if anyone wants to score outside the

range Entire committee records impact score Discuss budget and other issues

C. The Scoring Process

1. Overall Impact Score: likelihood project will “exert a sustained, powerful influence on the research field(s) involved (1-9 scale)

2. A separate 1-9 score for each of 5 core criteria (Significance, Investigators, Innovation, Approach, Environment)

3. Additional review criteria help determine scientific and technical merit BUT are not scored separately

4. Additional review considerations are addressed by reviewers, but are not scored & are discussed after group scores

Score Criteria

Overall Impact: will project exert a sustained, powerful influence on the research field(s) as indexed by 5 core review criteria1. Significance: important problem addressed; how will this improve scientific knowledge, technical capability, and/or clinical practice

Impact Addresses: Probability of whether the research will exert a

sustained, powerful influence on the research field

Significance Addresses: Does the project address an important

problem or a critical barrier to progress in the field?

If the aims are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?

What is the Difference Between Impact and Significance ?

Score Criteria (continued) 2. Investigators: PI & other researchers well suited to the project; appropriate experience & training; ongoing record of accomplishments; complementary & integrated experience; leadership approach, governance, and organizational structure appropriate for project

3. Innovation: the work challenges and seeks to shift current research or practice paradigms; utilizing novel theory, approaches or methods, instrumentation, or interventions; the work is novel

(Be innovative, but maybe not too innovative…)

Score Criteria (continued)4. Approach: strategy, methodology, analyses are well-reasoned and appropriate; potential problems & alternative strategies thought through; benchmarks set; risk is managed

Most common reviewer complaint is lack of detail here

Typically the longest section

5. Environment: the environment will contribute to the project’s success; institutional support, equipment, & other resources sufficient; unique features of the environment, subject population, collaborative arrangements

Additional Review Criteria (not scored)

Human Subjects:1. Protection of human subjects2. Data safety monitoring plan (clinical trials only)3. Inclusion of women, minorities, children4. Vertebrate animals5. Biohazards

Additional Review Considerations

1. Budget and period of support2. Select agent research (infectious

agents)3. Applications from foreign organizations4. Resource sharing plans

CRITERIA SCORING SYSTEMHIGH1. Exceptional: Exceptionally strong with essentially no

weaknesses2. Outstanding: Extremely strong with negligible

weaknesses3. Excellent: Very strong with only some minor weaknessesMEDIUM4. Very Good: Strong but with numerous minor weaknesses5. Good: Strong but with at least one moderate weakness6. Satisfactory: Some strengths but also some moderate

weaknessesLow7. Fair: Some strengths but with at least one major

weakness8. Marginal: A few strengths and a few major weaknesses9. Poor: Very few strengths and numerous major

weaknesses

CRITERIA SCORING SYSTEM (continued)

Minor Weakness: An easily addressable weakness that does not substantially lessen impact

Moderate Weakness: A weakness that lessens impact

Major Weakness: A weakness that severely limits impact

CRITERIA SCORING SYSTEM (continued)

Final Overall Impact Score:Mean of all reviewers’ final impact scores X 10Range = 10 (high impact) -- 90 (low impact)

NOTE: Scoring likely to produce applications with identical scores (“ties”). Thus, other factors (e.g., mission relevance, portfolio balance) will be considered when all other things are essentially equal

Research Plan Components Specific Aims Research Strategy

Includes Background & Significance; Preliminary Studies/Progress Report; Research Design & Methods

Facilities and Equipment Reflects the Environment criterion

For ESIs should describe the institutional investment in the success of the investigator

Biographical Sketch [NEW! as of Jan 25th, 2015] Personal statement – why well-suited for project, 4 pubs Contribution to science - you describe up to 5 of your

most significant scientific contributions (<=1/2 page each); up to 4 pubs or other products for each contribution area

Key Sections in R01 Proposal Format

D. A Penn State Example

Mock NIH Study SectionChair: Joshua Smyth, Associate Director, SSRIReviewer #1: Rhonda BeLue, Associate Professor of Health Policy and AdministrationReviewer #2: Jenae Neiderhiser, Professor of PsychologyReviewer #3: Danielle Downs, Associate Professor of Kinesiology and Obstetrics & Gynecology Review group members: Workshop attendees

Enhancing Peer Review Criteria: http://grants.nih.gov/grants/guide/notice-files/not-od-09-025.html

Page Limits: http://grants.nih.gov/grants/forms_page_limits.htm

Human Subjects: http://grants.nih.gov/grants/policy/hs/index.htm

SF424 guidelines for submission: http://grants.nih.gov/grants/funding/424/index.htm

Glossary: http://grants.nih.gov/grants/glossary.htm

Links of Interest

NIH RePorter (formerly CRISP)https://libraries.ucsd.edu/info/resources/nih-reporter-formerly-crispRevised Applicationshttp://grants.nih.gov/grants/policy/amendedapps.htmNIH Grant Writing Tip Sheets http://grants.nih.gov/grants/grant_tips.htm Getting an RO1http://sciencecareers.sciencemag.org/career_development/previous_issues/articles/1190/getting_an_nih_r01

General Resources

NSF Proposal Writinghttp://www.cs.cmu.edu/~sfinger/advice/advice.html

Other Proposal Writing Guides http://www.learnerassociates.net/proposal/

Reasons Proposals Failhttp://chronicle.com/article/How-to-Fail-in-Grant-Writing/125620/--let

More general resources

New and Early Stage Investigatorshttp://grants.nih.gov/grants/new_investigators/

New and Early Stage Investigator Resources

http://public.csr.nih.gov/Pages/default.aspx

http://grants.nih.gov/grants/oer.htm (forms, grant search, etc.)

NIH Websites

SSRI Listserv

New subscribers can join the SSRI listserv by sending mail to:

L-SSRI-subscribe-request@lists.psu.edu