proposal review process mock review webinar

68
Proposal Review Process Mock Review Webinar Louis Everett, Scott Grissom & Don Millard Division of Undergraduate Education

Upload: samira

Post on 25-Feb-2016

42 views

Category:

Documents


1 download

DESCRIPTION

Proposal Review Process Mock Review Webinar. Louis Everett, Scott Grissom & Don Millard. Division of Undergraduate Education National Science Foundation. Caution. Most of the information presented in this session represents the presenter’s opinion and is not an official NSF position. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Proposal Review Process Mock Review Webinar

Proposal Review Process Mock Review Webinar

Louis Everett, Scott Grissom & Don Millard

Division of Undergraduate EducationNational Science Foundation

Page 2: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 2

Most of the information presented in this session represents the presenter’s opinion and is not an official NSF position

Caution

2

Page 3: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 3

Introduction (5min) Overview of the review process (5min) Instructions on producing a panel review (5min) Organization of participants into teams and designation of panel chairs (5min)

Local teams discuss strengths and weaknesses (20min) Teams locally report/discuss results (10min) Reporting to virtual group with ratings – note: facilitators will be asked to select an individual to report (10min)

BREAK (15min)

Individuals consider ways to improve the proposal (5min) Local teams discuss suggestions for improvement (10min) Teams locally report/discuss results (5min) Report back to virtual group - note: facilitator picks an individual to report (5min) PD commentary on responses (10min) Think – What have I learned today that I will use in preparing my next proposal? (5min) Share your thoughts with local participants (5min) Facilitator reports results back to virtual group (5min) Wrap-up Q&A (10min)

Webinar Agenda

Page 4: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 4

Help participants to: Become more familiar with the proposal review process

Better understand the TUES-specific criteria

Better understand the use of intellectual merit/broader impact criteria in reviewing proposals

Develop more competitive proposals that effectively meet the expectations of the TUES program

Please note: A number of the following slides are provided for informational purposes – we will not be going through all of them

Webinar Goals - Expected Outcomes

Page 5: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 5

TUES Program

Page 6: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 6

Title changed to emphasize the special interest in projects that have the potential to transform undergraduate STEM education

Review criteria was modified to emphasize the desire for projects that: Propose materials, processes, or models that have the potential to

Enhance student learning Be adapted easily by other sites

Involve a significant effort to facilitate adaptation at other sites Institutionalize the approach at the investigator's college or university as

appropriate (e.g., for the Type) Have the potential to contribute to a cultural shift in undergraduate STEM

education

TUES vs. CCLI

Page 7: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 7

TUES Program Vision: Excellent STEM education for all undergraduate

students

Reflects national concerns about producing: ◦Skilled STEM professionals (including K-12 teachers) ◦Citizens knowledgeable about STEM and how it relates to

their lives

Seeks to build a community of faculty committed to improving undergraduate STEM education

Encourages projects with potential to advance and transform undergraduate STEM education

Page 8: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 8

Creating Learning Materials and Strategies Guided by research on teaching and learning Incorporate and be inspired by advances within the discipline

Implementing New Instructional Strategies Contribute to understanding on how existing strategies:

Can be widely adopted Are transferred to diverse settings Impact student learning in diverse settings

Developing Faculty Expertise Enable faculty to acquire new knowledge and skills in order to revise their curricula

and teaching practices Involve a diverse group of faculty

TUES Project Components

Page 9: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 9

Assessing and Evaluating Student Achievement: Develop and disseminate valid and reliable tests of STEM knowledge Collect, synthesize, and interpret information about student understanding,

reasoning, practical skills, interests, attitudes or other valued outcomes

Conducting Research on Undergraduate STEM Education: Explore how:

Effective teaching strategies and curricula enhance learning and attitudes Widespread practices have diffused through the community Faculty and programs implement changes in their curriculum

TUES Project Components (cont)

Page 10: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 10

Projects developing instructional materials and methods should:

◦ Be based on how students learn

◦ Consider transferability and dissemination throughout the project's lifetime

◦ Involve efforts to facilitate adaptation at other sites in more advanced projects

Instructional Materials and Methods Projects

Page 11: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 11

Expect to award approximately 10% Total budget: up to $200,000 for 2 to 3 years

$250,000 when 4-year and 2-year schools collaborate Typically involve a single institution & one program

component – but there are exceptions Contribute to the understanding of undergraduate STEM education Informative evaluation effort based on the project's specific

expected outcomes Institutionalized at the participating colleges and universities Deadlines:

May 26, 2011 (A-M) May 27, 2011 (N-W)

Type 1 Projects

Page 12: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 12

Type 2 Projects 20 to 25 awards expected Total budget: up to $600K for 2 to 4 years

Type 3 Projects 3 to 5 awards expected Budget negotiable, but not to exceed $5M over 5 years

Tues Central Resource Projects 1 to 3 awards expected Budget negotiable, depending on the scope and scale of the activity

Small focused workshop projects -- 1 to 2 years & up to $100K Large scale projects -- 3 to 5 years & $300K to $3M

Deadline: January 13, 2012

Type 2, 3, and CRP Projects

Page 13: Proposal Review Process Mock Review Webinar

Review Process

Page 14: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 14

NSF Peer Review Process Reviewers are solicited by program directors◦ For example - A Typical TUES Type 1 Panel Review:

4 panels/program director (60-70 proposals/PD) 6-8 reviewers/panel 17 proposals/panel – not all read by every panelist Approximately 130 engineering reviewers

Reviewers assign individual ratings and prepare written comments on Intellectual Merit & Broader Impacts prior to coming to the panel meeting

Panel meeting is held in DC area – over a 1.5 day period

Page 15: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 15

Panel Review Meeting Panel Chair establishes order of proposal review

process

Proposals are discussed individually

A “scribe” is designated to capture all of the points brought up in discussion and produce a summary review – called the “Panel Summary”

All reviewers return on day 2 to approve all the Panel Summaries

Page 16: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 16

NSF program directors

◦ Informs recommendations relative to funding

◦ Guides pre-award negotiations

Applicants

◦ If proposal is funded: Provides suggestions for improving project

◦ If proposal is not funded: Provides information to guide a revision of the proposal

Audience for Reviews

Page 17: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 17

NSF Review Criteria All proposals are evaluated using the NSB-approved

review criterion ◦Intellectual merit ◦Broader impacts

The TUES Solicitation provides two sets of suggested questions to help define these criteria◦Standard NSF set ◦TUES specific set

Page 18: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 18

Suggested questions are only a guide for considering intellectual merit and broader impacts

Suggested questions are NOT:

◦ A complete list of “requirements”

◦ Applicable to every proposal

◦ An official checklist

Caution Regarding Suggested Questions

Page 19: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 19

Will the project: Include activities important in advancing knowledge? Involve qualified proposer(s)? Contain creative and original concepts? Have a well conceived and organized plan? Include sufficient access to resources?

NSF Suggested Questions for Intellectual Merit

Page 20: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 20

Will the project: Advance discovery - promote teaching & learning? Broaden participation of underrepresented groups? Enhance the infrastructure? Include broad dissemination? Benefit society?

NSF Suggested Questions for Broader Impacts

Page 21: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 21

Will the project: Produce one or more of the following:

Exemplary materials, processes, or models that enhance student learning and can be adopted by other sites

Important findings related to student learning? Build on existing knowledge about STEM education? Have explicit and appropriate expected measurable outcomes integrated into an

evaluation plan? Include an evaluation effort that is likely to produce useful information? Institutionalize the approach at the investigator's college or university (as

appropriate for the Type)

TUES Suggested Questions for Intellectual Merit

Page 22: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 22

Will the project: Involve a significant effort to facilitate adaptation at other

sites? Contribute to the understanding of STEM education? Help build and diversify the STEM education community? Have a broad impact on STEM education in an area of

recognized need or opportunity? Have the potential to contribute to a significant advancement

and cultural shift in undergraduate STEM education?

TUES Suggested Questions for Broader Impacts

Page 23: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 23

Writing the Review

Page 24: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 24

The Entire Proposal is Used to Inform Reviewers◦ Project Summary◦ Project Description◦ Biographical Sketches◦ Budget◦ Supplementary Documentation

Review Material

24

Page 25: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 25

Overview A review should indicate an opinion on the merit of the project

The rating should indicate an overall evaluation of the proposal’s merit◦ Excellent, Very Good, Good, Fair, and Poor

Describe positive aspects◦ Not just list them -- Provide details

Identify concerns (or weaknesses)◦ Not just list them -- Provide details

Offer suggestions for improvement

Rating and text should be consistent

Page 26: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 26

Uses appropriate style Contains adequate details Contains understandable, specific, and complete statements Relates strengths and weakness to review criteria Indicates why an item is a strength or weakness Justifies the proposal rating in the written critique

A reader should be able to guess the rating from the written text

Characteristics of Informative Reviews and Panel Summaries

Page 27: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 27

Specific and Complete Comments Identify a strength or weaknesses

◦ “The evaluation plan is a strength.”

Identify a strength or weaknesses and indicate why it is one◦ “The evaluation plan includes a competent, independent evaluator...”

◦ “The background discussion is well referenced, shows a good understanding of the prior work, supports the proposed work...”

Page 28: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 28

Specific and Complete Comments (cont.) Identify a strength or weaknesses and indicate why it is one and

why it is important◦ “The evaluation plan includes a competent, independent evaluator, … and it

will guide the investigators as the project evolves and provide a measure of its effectiveness at the end.”

Identify a strength, indicate why it is one, why it is important, and how it could be improved◦ “The evaluation plan is a strength because it includes a competent,

independent evaluator, … and this will guide the investigators as the project evolves and provide a measure of its effectiveness at the end. It could be improved by adding …”

Page 29: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 29

Try to be constructive in your written comments◦ Provide suggestions to help applicants improve their proposals

Do not be overly critical in your ratings◦Most fundable proposals have some weaknesses◦ Some are correctable through negotiations

Other Important Ideas

Page 30: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 30

Strengths & Weakness

Page 31: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 31

Local teams discuss the strengths and weaknesses identified in the individual reviews Team “scribe” takes notes during discussion

Teams locally report/discuss results with facilitator’s guidance Facilitators guide discussion and selection of an individual to report to

the full virtual group Participating organizations will be called upon by webinar

moderator – PLEASE BE READY TO REPORT BACK Program Directors will offer comments on reports

Team Activity Process

Page 32: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 32

Redundancy is OK – indicative of common perception Responses will be solicited as follows:

Intellectual Merit Strengths Weaknesses

Broader Impacts Strengths Weaknesses

Overall Perception Is this a proposal worthy of funding?

Reporting

Page 33: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 33

ONE MINUTEUntil Reporting Begins

Page 34: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 34

Redundancy is OK – indicative of common perception Responses will be solicited as follows:

Intellectual Merit Strengths Weaknesses

Broader Impacts Strengths Weaknesses

Overall Perception Is this a proposal worthy of funding?

Reporting

Page 35: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 35

Questions?

Page 36: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 36

BREAK(15 min)

Page 37: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 37

ONE MINUTEUntil Next Session

Page 38: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 38

Suggestions for Improvement

Page 39: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 39

Individuals consider ways that the proposal could be improved – create a list

Local teams discuss the suggestions for improvement Team “scribe” takes notes during discussion

Teams locally report/discuss results with facilitator’s guidance Facilitators guide discussion and selection of an individual to report to

the full virtual group Participating organizations will be called upon by webinar

moderator – PLEASE BE READY TO REPORT BACK Program Directors will offer comments on reports

Improvements Activity Process

Page 40: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 40

Try to minimize redundant responses Identify approaches for:

Building on strengths Overcoming weaknesses

Responses should include potential improvements to: Idea Project infrastructure Project implementation plan Evaluation plan Dissemination plan Proposal participation/involvement

Reporting

Page 41: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 41

ONE MINUTEUntil Reporting Begins

Page 42: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 42

Try to minimize redundant responses Identify approaches for:

Building on strengths Overcoming weaknesses

Responses should include potential improvements to: Idea Project infrastructure Project implementation plan Evaluation plan Dissemination plan Proposal participation/involvement

Reporting - Reminder

Page 43: Proposal Review Process Mock Review Webinar

Sample Proposal Review

Comments Regarding the Proposal’s Intellectual Merit, Broader Impacts and Weaknesses

Page 44: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 44

The following comments reflect a combination of the proposal’s panel review, individual reviewers and inputs provided by NSF program officers

Sample Proposal Review Comments

Page 45: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 45

The proposal is the evolution of the successful studio pedagogy to a mobile studio pedagogy.

The proposal builds on a instructional methodology that is accepted in the STEM literature.

The proposal is student-focused and grounded in the STEM knowledge base.

A primary innovation is the development of a low cost virtual instrument board that, when coupled with a computer/laptop, provides students with a portable experimental platform

Intellectual Merit (1)

Page 46: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 46

This experimental platform serves as a low cost replacement for the laboratory equipment found in the original studio classrooms and provides the students with an "anywhere, anytime" experimental platform. The PIs are well qualified to carry out the work.

The proposal was responsive to the cyclic model of knowledge creation contained in the TUES/CCLI request for proposals.

The panel felt the proposal would be strengthened by a more detailed evaluation plan with clearly stated, measurable outcomes.

Intellectual Merit (2)

Page 47: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 47

The inclusion of an historically black college or university and a community college should magnify the impact of the proposed pedagogy.

The broader impacts of the proposed work include the potential to significantly impact a large number of under-served students by providing a low cost, portable experimental platform.

The partnership was made concrete through an instructor exchange program between the institutions.

Broader Impacts

Page 48: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 48

While the evaluation team appears to have worked together with the PIs for several years, some basic information about the qualifications of those individuals should have been included in the proposal.

The proposal could be strengthened with more definition of the role of each partner.

It is not clear that it will positively impact student learning; just because a student has the means to perform an experiment 24/7, does not mean that they will.

Weaknesses (1)

Page 49: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 49

The proposal could benefit from clarifying or identifying the role of laboratory experiences where actual industrial-quality laboratory equipment is utilized. Too much reliance on the proposed pedagogy might also leave the student with limited experience in the use of the real industrial grade and scale laboratory equipment.

Moving from the original studio to the mobile studio essentially means that the new work is now based on an untested model. The proposal could be strengthened by addressing this issue.

Weaknesses (2)

Page 50: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 50

It has not been demonstrated that the mobile studio is as effective as the original studio with its significant faculty involvement and structure.

This proposal would have been much stronger if some preliminary data on efficacy was included.

The proposal mentions a pilot test that was done at Howard University but no details are provided.

The proposal fails to explicitly address how it will focus on components of the cyclic model for innovation in STEM education.

Weaknesses (3)

Page 51: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 51

Practical Aspects of Review Process

Page 52: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 52

Reviewers have: Many proposals

Ten or more from several areas Limited time for your proposal

20 minutes for first read Different experiences in review process

Veterans to novices Different levels of knowledge in proposal area

Experts to outsiders Discussions of proposals’ merits at panel meeting

Share expertise and experience

Practical Aspects of Review Process

52

Page 53: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 53

Write down a list of suggestions (guidelines) that you would suggest that a colleague should follow - to deal with practical aspects of the Review Process

2 minutes

Practical Aspects of Review Process

53

Page 54: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 54

ONE MINUTE

Page 55: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 55

The following comments reflect perspectives provided by NSF program officers

Considering the Practical Aspects of the Review Process

Page 56: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 56

Use good style (clarity, organization, etc.) Be concise, but complete Write simply but professionally Avoid jargon and acronyms Check grammar and spelling Use sections, headings, short paragraphs & bullets (Avoid dense, compact

text)

Reinforce your ideas Summarize; Highlight (bolding, italics)

Give examples

Proposal Writing Guidelines

56

Page 57: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 57

Use tables, figures – where it makes sense

Reinforce your ideas Summarize; Highlight (bolding, italics)

Give examples

Proposal Writing Guidelines

57

Page 58: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 58

Provide appropriate level of detail

Pay special attention to Project Summary

Summarize goals, rationale, methods, and evaluation and dissemination plans

Address intellectual merit and broader impacts Explicitly and independently

Three paragraphs with headings: “Summary” “Intellectual Merit” “Broader Impacts”

Proposal Writing Guidelines

58

Page 59: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 59

Follow the solicitation and GPG Adhere to page, font size, and margin limitations

Use allotted space but don’t pad the proposal

Follow suggested (or implied) organization

Use appendices sparingly (check solicitation to see if allowed)

Include letters showing commitments from others “Support letters” are not allowed Avoid form letters

Proposal Writing Guidelines

59

Page 60: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 60

Prepare credible budget Consistent with the scope of project Clearly explain and justify each item

Address prior funding when appropriate Emphasize results

Sell your ideas but don’t over promote

Proofread the proposal

“Tell a story” and turn a good idea into a competitive proposal

Proposal Writing Guidelines

60

Page 61: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 61

Good idea + need Right people + infrastructure Assessment of outcomes that measure effect on

student learning (with goals/objectives linked to evaluation)

Active dissemination plan Efforts to broaden participation of underrepresented

groups

A Successful Proposal

Page 62: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 62

Reflection

Page 63: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 63

Individuals consider: What have I learned today that I will be able to used in preparing my next proposal?

Create a list (5min) Share your list with local participants (5min) Facilitators report back to virtual group (5min)

Participating organizations will be called upon by webinar moderators – PLEASE BE READY TO REPORT BACK

Reflection Activity

Page 64: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 64

Closing Comments

Page 65: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 65

Model good practices that increase/improve learning Facilitate direct experience with the methods and processes of

inquiry/experimentation Empower the student Enhance ability to produce and evaluate innovative results

Products Solutions to problems Metrics on outcomes

Establish a community that will help inform, disseminate and sustain engineering education efforts

Toward Transformation…

Page 66: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 66

Good idea + need Right people + infrastructure Can be readily adopted at other sites Assessment of outcomes that measure effect on student

learning (with goals/outcomes linked to evaluation) Active dissemination plan Shows promise for institutionalization Efforts to broaden participation of underrepresented groups

A Successful TUES Proposal…

Page 67: Proposal Review Process Mock Review Webinar

Mock Review Webinar - 67

Questions?

Page 68: Proposal Review Process Mock Review Webinar

Thanks for participating!Please respond to the survey located at:

http://www.step.eng.lsu.edu/nsf/participants

Louis Everett – [email protected] Grissom – [email protected]

Don Millard – [email protected]