Noni Byrnes

René Etcheberrigaray

Richard Nakamura

Review of CSR’s study sections by areas of science

SRG: Scientific Review Group

Study Section – managed by Scientific Review Officer (SRO)

IRG: Integrated Review Group

Organizational Unit – group of 6-10 scientifically-related study sections, each managed by an SRO, supervised by IRG Chief

Definitions

Genes, Genomes and Genetics(GGG) IRG

Genomics, Computational Biology and Technology (GCAT)

Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE)

Molecular Genetics A (MGA) Molecular Genetics B (MGB)

Prokaryotic Cell and Molecular Biology (PCMB)

Therapeutic Approaches to Genetic Diseases Study Section (TAG)

Bioengineering Sciences and Technologies (BST) IRG

Biodata Management and Analysis (BDMA)

Biomaterials and Biointerfaces (BMBI)Gene and Drug Delivery Systems (GDD)

Instrumentation Systems and

Development (ISD)Modeling and Analysis of Biological

Systems (MABS) Nanotechnology (NANO)

High Throughput Screening Assays for Probe Discovery (ZRG1-BST-U (55)

Example: study sections have scientific overlap other study sections both within and outside their IRGs

“As recommended by the PSBR Report and the CSR Advisory Committee, CSR will examine the functioning of the new IRGs and study sections…...”

Panel on Scientific Boundaries of Review (PSBR)1999-2003

• By IRG (examination of scientific guidelines of SRGs, rosters, summary statements, metrics, etc.) – Presentation by Chief, Q/A and internal review by CSR management

• By IRG (examination of scientific guidelines of study sections, rosters, summary statements, metrics, etc.) - Presentations by Chief and SROs, Q/A and internal review by CSR management

• By IRG – email input from selected reviewers from all SRGs, teleconference with study section Chairs, white paper by Chief with summary of external input from reviewers/Chair teleconference, presentations by Chief and SROs, Q/A and internal review by CSR management

• By IRG - external scientific working group in-person meeting, provided them bibliometric/scientific overlap, roster, metrics, study section guidelines, asked questions about quality, review process, white paper by Chief, presentation by Chief to external working group

Evolution of the IRG Review Process at CSR

• By IRG – external working group commented on operations in IRG, rosters, balance between regular/adhoc members, diversity, etc.

• Issues discussed includes – use of surveys, exit interviews, ranking, H-indices, bibliometrics, should Chairs be used to recruit new members, NIH A2 policy, new investigators, percent “not discussed”, IRG function (via discussion with SROs)

• The main scientific changes made to study sections were endorsement of proposals made by the IRG Chief in the white paper and presentation

Latest IRG Evaluation Results – process vs. science

1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs)

2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG)

3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group

4) Ask 1-2 questions designed to focus discussion on science, not process

Proposed Plan

1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs)

2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG)

3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group

4) Ask 1-2 questions designed to focus discussion on science, not process and provide specific guidance

Proposed Plan

Boyack KW, Chen MC, Chacko G (2014) Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health. PLoS ONE 9(8): e104244. doi:10.1371/journal.pone.0104244

Figure 3. Consensus study section network.

• CSR committee (senior management, plus IRG Chiefs representing each scientific division at CSR)

• Consensus network of study sections• Scientific guidelines for study sections, list of overlapping study

sections listed on the web• Data on review of study section member applications in related study

sections• Close examination of scientific content of applications by committee

members – abstracts/specific aims for content • Input from Senior Staff, CSR IRG Chiefs – adjust scientific clusters

1. Form Scientific Clusters

1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs)

2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG) within a cluster

3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group

4) Ask 1-2 questions designed to focus discussion on science, not process

Proposed Plan

1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs)

2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG)

3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group

4) Ask 1-2 questions designed to focus discussion on science, not process, and provide specific guidance

Proposed Plan

• Current scientific guidelines for each study section in cluster

• Titles, abstracts, specific aims for applications randomly selected from 3 rounds

• Workload trends for up to 9 rounds

3. Information Provided to Working Group

1) Review by scientific clusters, not IRGs – each cluster with about 12-20 study sections (SRGs)

2) Assemble working group of scientifically broad, senior scientists, preferably with interest in more than one study section (SRG)

3) Provide enough information, time and guidance for meaningful evaluation and recommendation by working group

4) Ask 1-2 questions designed to focus discussion on science, not process, and provide specific guidance

Proposed Plan

How well does the scientific scope of the study sections align with the current state of the science? Recommend modifications if:

– a study section is currently over- or under-subscribed

– the scope of the science reviewed by a study section is not adequately representative of the scope of the field and emerging fields

4. Questions for the Working Group

• Read the study section guidelines for all 16 panels, look at the workload (including workload trends over the last 9 rounds), and then look over the sample abstracts/aims for each panel to get a better idea of what type of science is reviewed there.

• Does this study section:–Appear to be in the optimal size range (approximately 60-80 applications, +/- 10 ok) to encourage

scientific breadth/competition – if not, recommend a change–Appear to map onto the current state of the scientific field well – if not, recommend a change–Appear to have cohesion – i.e. the scientific areas are broad but interrelated, not mismatched, if not,

recommend a change

 • “Change” can be: change in guidelines, merge study sections, create new study sections, move an area of

science from one study section to another or several others, add areas of science that appear to be missing

• The only caveats in recommending a change – –there should be some scientific overlap between study sections – enough to ensure competition and a

choice of review venue for investigators, but not so much that the science/expertise is scattered in too many places

–any new study section should have enough applications in a broad, cohesive field to be sustainable, i.e. no small, boutique, insular panels.

4. (cont’d) Specific Guidance for Working Group

Non-hypothesis driven areas:

• Biomaterials

• Basic biomedical imaging

• Instrumentation/tools design

• Computational biology, modeling, informatics

Proof of Concept: Cluster #1 (“bioengineering”)

Healthcare Delivery and Methodologies IRG (HDM) Biomedical Computing and Healthcare Informatics

(BCHI) Biostatistical Methods and Research Design (BMRD)

Community Influences on Health Behavior (CIHB) Community-Level Health Promotion (CLHP)

Dissemination and Implementation Research in Health (DIRH)

Health Disparities and Equity Promotion (HDEP) Understanding and Promoting Health Literacy SEP

(HDM-X) Health Services Organization and Delivery (HSOD)

Nursing and Related Clinical Sciences (NRCS)

Surgical Sciences, Biomedical Imaging & Bioengineering IRG (SBIB)

Biomedical Imaging Technology A (BMIT-A) Biomedical Imaging Technology B (BMIT-B)

Bioengineering, Technology, and Surgical Sciences (BTSS)

Clinical Molecular Imaging and Probe Development (CMIP)

Clinical and Translational Imaging Applications (DTCS 81)

Medical Imaging (MEDI) Surgery, Anesthesiology, and Trauma (SAT)

Bioinformatics in Surgical Sciences, Biomedical Imaging and Bioengineering (ZRG1-SBIB-Q (80)

Genes, Genomes and Genetics IRG (GGG) Genomics, Computational Biology and Technology

(GCAT) Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE)

Molecular Genetics A (MGA) Molecular Genetics B (MGB)

Prokaryotic Cell and Molecular Biology (PCMB) Therapeutic Approaches to Genetic Diseases Study

Section (TAG)

Bioengineering Sciences and Technologies IRG (BST)

Biodata Management and Analysis (BDMA) Biomaterials and Biointerfaces (BMBI)

Gene and Drug Delivery Systems (GDD) Instrumentation Systems and Development (ISD)

Modeling and Analysis of Biological Systems (MABS) Nanotechnology (NANO)

High Throughput Screening Assays for Probe Discovery (ZRG1-BST-U (55)

Musculoskeletal, Oral and Skin Sciences IRG (MOSS)

Arthritis, Connective Tissue and Skin (ACTS)Musculoskeletal Rehabilitation Sciences (MRS)

Musculoskeletal Tissue Engineering (MTE) Oral, Dental and Craniofacial Sciences (ODCS)

Skeletal Biology Development and Disease (SBDD) Skeletal Biology Structure and Regeneration (SBSR) Skeletal Muscle and Exercise Physiology (SMEP)

Interdisciplinary Molecular Sciences and Training IRG (IMST)

Enabling Bioanalytical and Imaging Technologies (EBIT)

Biological Chemistry and Macromolecular Biophysics IRG (BCMB)

Biochemistry and Biophysics of Membranes (BBM) Macromolecular Structure and Function A (MSFA) Macromolecular Structure and Function B (MSFB) Macromolecular Structure and Function C (MSFC) Macromolecular Structure and Function D (MSFD)

Synthetic and Biological Chemistry A (SBCA) Synthetic and Biological Chemistry B (SBCB)

Emerging Technologies and Training in Neurosciences IRG (ETTN)

Bioengineering of Neuroscience, Vision and Low Vision Technologies (BNVT)

Molecular Neurogenetics Study Section (MNG) Neuroscience and Ophthalmic Imaging Technologies

(NOIT)

Healthcare Delivery and Methodologies IRG (HDM) Biomedical Computing and Healthcare Informatics

(BCHI) Biostatistical Methods and Research Design (BMRD)

Community Influences on Health Behavior (CIHB) Community-Level Health Promotion (CLHP)

Dissemination and Implementation Research in Health (DIRH)

Health Disparities and Equity Promotion (HDEP) Understanding and Promoting Health Literacy SEP

(HDM-X) Health Services Organization and Delivery (HSOD)

Nursing and Related Clinical Sciences (NRCS)

Surgical Sciences, Biomedical Imaging & Bioengineering IRG (SBIB)

Biomedical Imaging Technology A (BMIT-A) Biomedical Imaging Technology B (BMIT-B)

Bioengineering, Technology, and Surgical Sciences (BTSS)

Clinical Molecular Imaging and Probe Development (CMIP)

Medical Imaging (MEDI) Surgery, Anesthesiology, and Trauma (SAT)

Bioinformatics in Surgical Sciences, Biomedical Imaging and Bioengineering ZRG1-SBIB-Q (80)

Genes, Genomes and Genetics IRG (GGG) Genomics, Computational Biology and Technology

(GCAT) Genetics of Health and Disease (GHD) Genetic Variation and Evolution (GVE)

Molecular Genetics A (MGA) Molecular Genetics B (MGB)

Prokaryotic Cell and Molecular Biology (PCMB) Therapeutic Approaches to Genetic Diseases Study

Section (TAG)

Bioengineering Sciences and Technologies IRG (BST)

Biodata Management and Analysis (BDMA) Biomaterials and Biointerfaces (BMBI)Gene and Drug Delivery Systems (GDD)

Instrumentation Systems and Development (ISD)Modeling and Analysis of Biological Systems (MABS)

Nanotechnology (NANO) High Throughput Screening Assays for Probe Discovery

(ZRG1-BST-U (55)

Musculoskeletal, Oral and Skin Sciences IRG (MOSS)

Arthritis, Connective Tissue and Skin (ACTS)Musculoskeletal Rehabilitation Sciences (MRS) Musculoskeletal Tissue Engineering (MTE)

Oral, Dental and Craniofacial Sciences (ODCS) Skeletal Biology Development and Disease (SBDD) Skeletal Biology Structure and Regeneration (SBSR) Skeletal Muscle and Exercise Physiology (SMEP)

Interdisciplinary Molecular Sciences and Training IRG (IMST)

Enabling Bioanalytical and Imaging Technologies (EBIT)

Biological Chemistry and Macromolecular Biophysics IRG (BCMB)

Biochemistry and Biophysics of Membranes (BBM) Macromolecular Structure and Function A (MSFA) Macromolecular Structure and Function B (MSFB) Macromolecular Structure and Function C (MSFC)

Macromolecular Structure and Function D (MSFD) Synthetic and Biological Chemistry A (SBCA) Synthetic and Biological Chemistry B (SBCB)

Emerging Technologies and Training in Neurosciences IRG (ETTN)

Bioengineering of Neuroscience, Vision and Low Vision Technologies (BNVT)

Molecular Neurogenetics Study Section (MNG) Neuroscience and Ophthalmic Imaging Technologies

(NOIT)

• Creation of new study sections in emerging areas (e.g. Microphysiological Systems, Clinical Informatics)

• Dissolution/redistribution of study sections that are undersubscribed or need to merge/redesign into new panels

• Modification of study section scope and guidelines to better map onto the scientific field (e.g. imaging committees)

• Evaluate with another cluster (e.g. MSFD)

• No change (e.g. MABS, ISD)

Recommendations from Working Group Meeting (April 29, 2015)

• Discussion with relevant CSR and IC staff about practical considerations

• Develop an implementation plan (when appropriate/feasible)

• Council advice – continue SRG evaluations with this type of cross-IRG process?

Next Steps

Committee:Advisor: Richard Nakamura, Director, CSRMembers:John Bowers, Acting Director, Division of Clinical & Translational Sciences (DTCS)Noni Byrnes, Director, Division of Basic & Integrative Biological Sciences (DBIB)Valerie Durrant, Acting Director, Division of Neuroscience, Development & Aging (DNDA)Rene Etcheberrigaray, Deputy DirectorJonathan Ivins, Chief, Digestive, Kidney and Urological Sciences (DKUS) IRGRichard Panniers, Chief, Genes, Genomes & Genetics (GGG) IRG

Syed Quadri, Chief, Oncology 2 Translational Clinical (OTC) IRG

Ray Jacobson, SRO, Bioengineering Science & Technologies (BST) IRG

CSR Staff

QUACKENBUSH, JOHN, PHDPROFESSOR OF COMPUTATIONAL BIOLOGY ANDBIOINFORMATICSDEPARTMENT OF BIOSTATISTICSHARVARD SCHOOL OF PUBLIC HEALTH, DANA-FARBER CANCER INSTITUTE

SNYDER, MICHAEL P, PHDPROFESSOR AND CHAIR OF GENETICSDEPARTMENT OF GENETICSSTANFORD UNIVERSITY

TOGA, ARTHUR W, PHDPROVOST PROFESSORLABORATORY OF NEURO IMAGINGTHE INSTITUTE FOR NEUROIMAGING AND INFORMATICSKECK SCHOOL OF MEDICINE OF USCUNIVERSITY OF SOUTHERN CALIFORNIA

VUNJAK-NOVAKOVIC, GORDANA, PHDMIKATI FOUNDATION PROFESSOR OF BIOMEDICAL ENGINEERINGDEPARTMENT OF BIOMEDICAL ENGINEERING AND MEDICAL SCIENCECOLUMBIA UNIVERSITY

YASZEMSKI, MICHAEL J, MD, PHDPROFESSOR OF ORTHOPEDICSDEPARTMENTS OF ORTHOPEDIC SURGERY AND BIOMEDICAL ENGINEERINGMAYO CLINIC COLLEGE OF MEDICINE, ROCHESTER

CSR AC ADVISORS: BRINTON, ROBERTA DIAZ, PHDR. PETE VANDERVEEN CHAIR IN THERAPEUTIC DISCOVERY AND DEVELOPMENTBIOMEDICAL ENGINEERING AND NEUROLOGYNORRIS FOUNDATION LABORATORY FOR NEUROSCIENCE RESEARCHUNIVERSITY OF SOUTHERN CALIFORNIA

MAYO, STEPHEN L, PHDBREN PROFESSOR OF BIOLOGY AND CHEMISTRYWILLIAM K. BOWES JR. LEADERSHIP CHAIRDIVISION OF BIOLOGY AND BIOLOGICAL ENGINEERINGCALIFORNIA INSTITUTE OF TECHNOLOGY

CHAIR:

GRAINGER, DAVID W, PHD

UNIVERSITY DISTINGUISHED PROFESSOR

DEPARTMENT OF PHARMACEUTICS AND PHARMACEUTICAL CHEMISTRY

UNIVERSITY OF UTAH

MEMBERS:

ABECASIS, GONCALO, DPHIL

FELIX E. MOORE COLLEGIATE PROFESSOR OF BIOSTATISTICS

DEPARTMENT OF BIOSTATISTICS

UNIVERSITY OF MICHIGAN

ALLBRITTON, NANCY L, MD, PHD

DEBRECZENY DISTINGUISHED PROFESSOR

DEPARTMENT OF BIOMEDICAL ENGINEERING

UNIVERSITY OF NORTH CAROLINA

CHUTE, CHRISTOPHER G, MD, DRPH

BLOOMBERG DISTINGUISHED PROFESSOR OF HEALTH INFORMATICS

DIVISION OF GENERAL INTERNAL MEDICINE

JOHNS HOPKINS UNIVERSITY

GORE, JOHN C PHD, PHD

HERTHA RAMSEY CRESS UNIVERSITY PROFESSOR

VANDERBILT UNIVERSITY MEDICAL CENTER INSTITUTE OF IMAGING SCIENCE

GRILL, WARREN M. PHD, PHD

ADDY PROFESSOR OF BIOMEDICAL ENGINEERING

DEPARTMENT OF BIOMEDICAL ENGINEERING

DUKE UNIVERSITY

HYLTON-WATSON, NOLA M., PHD

PROFESSOR OF RADIOLOGY

DEPARTMENT OF RADIOLOGYAND BIOMEDICAL IMAGING

UNIVERSITY OF CALIFORNIA, SAN FRANCISCO

LATHROP, RICHARD H, PHD

PROFESSOR

COMPUTER SCIENCE DEPARTMENT

DONALD BREN SCHOOL OF INFORMATION ANDCOMPUTER SCIENCE

UNIVERSITY OF CALIFORNIA, IRVINE

External Working Group