investing in innovation (i3) reviewer orientation

June 2010

Scale-Up Grant Competition

Investing in Innovation (i3) Reviewer Orientation

Note: These slides are intended as guidance only. Pleaserefer to the official documents published in the Federal Register.

Schedule for TodayTopic Time

• Overview of the Role of a Peer Reviewer and the Review Process

5 minutes

• Overview of i3 Program 15 minutes

Q&A Period #1 15 minutes

• Selection Criteria 20 minutes


• Detailed i3 Review Process 10 minutes

• Scoring Applications, Writing Comments and Using e-Reader

10 minutes


2

i3 Peer Review Process

3

For all three grant types…For all three

grant types…

• The Department is using independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest

• Two peer reviewers who are evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation. Three peer reviewers who are subject matter experts will score selection criteria A, C, E, F and G.

• Applications were assigned to panels by one of the four absolute priorities where possible.

• Subject matter reviewers will determine whether any competitive preference priority points should be added

Scale-Up Only…

Scale-Up Only…

• Scale-Up finalists will be invited to an in-person interview in which they may clarify elements of their application to a panel of reviewers – finalists will not be able to present new information.

What is expected of a Peer Reviewer?

4

ProcessProcess

BehaviorBehavior

• Review the entire i3 application package and FAQs

• Review and become thoroughly familiar with the selection criteria, factors, and notes

• Familiarize yourself with the e-Reader system – used for inputting scores and comments

• Participate in all scheduled conference calls

• Provide appropriate comments that justify the score awarded and are helpful to the applicant

• Revise comments as suggested by your panel monitor

• Return all forms as required to ensure payment and completion of review process

• Be available the entire review process

• Draw upon your expertise

• Maintain confidentiality throughout the review process

PreparationPreparation

What must a Peer Reviewer really do? Receive applications and panel assignments from Synergy Identify any Conflicts of Interest, and notify Synergy or panel monitor

immediately Participate in Orientation webinar Register in and become acquainted with e-Reader Read applications (It may be helpful to you to take notes!) Write draft comments, assign scores, and submit drafts in e-Reader

before panel discussion (We recommend cutting and pasting from Word)

Receive feedback on comments from Panel Monitor Revise draft comments based on feedback as appropriate When comments/scores are acceptable to Panel Monitor, submit

final comments/scores Sign and return Technical Review Signature Forms to Synergy Remember to submit All Necessary Forms to Synergy

5

Prior to Panel

Discussion

Prior to Panel

Discussion

After Panel DiscussionAfter Panel Discussion



5 minutes







10 minutes


6

i3 Is One Part of UnprecedentedDirect Federal Investment in Education

7

• $4.35B - Race to the Top Fund, including $350MM for development of assessments

• $3.5B* - School Improvement Grants

• $650MM - Investing in Innovation Fund

• $650MM - Education Technology

• $300MM* - Teacher Incentive Fund

• $250MM - Statewide Longitudinal Data Systems

* Includes regular FY 09 appropriations

SFSF$48.6B

FormulaGrants$26B

Race To The Top and Other Grants($9.7B in FY2009 Funding)

AR

RA

K-1

2 In

vest

men

tA

ligne

d w

ith F

our

Ass

uran

ces

Investing in Innovation (i3) Fund Summary

8

PurposePurpose

FundingFunding

ApplicantsApplicants

To provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on:•Improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates

$650 million to be obligated by September 30, 2010

Eligible applicants are: (1)Local educational agencies (LEAs) (2)Nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools

Types of Awards Available Under i3

9

i3i3

EstimatedFunding Available

Up to $5MM/award Up to $30MM/award Up to $50MM/award

Evidence Required

Reasonable – research findings or hypotheses, including related research or theories in education and other sectors

Moderate – either high internal validity and medium external validity, or vice versa

Strong – both high internal validity and high external validity

Scaling Required

Able to further develop and scale

Able to be scaled to the regional or state level

Able to be scaled to the national, regional, or state level

Applicants

10

Eligible Applicants can be:

1)A local educational agency (LEA)

2)A nonprofit organization in partnership with (a) one or more LEAs or (b) a consortium of schools

Key Definition: Partners

11

Official partner means any of the entities required to be part of a partnership under section 14007(a)(1)(B) of the ARRA (i.e., a non-profit organization, an LEA, or a consortium of schools).

Official partner means any of the entities required to be part of a partnership under section 14007(a)(1)(B) of the ARRA (i.e., a non-profit organization, an LEA, or a consortium of schools).

Other partner means any entity, other than the applicant and any official partner, that may be involved in a proposed project.

Other partner means any entity, other than the applicant and any official partner, that may be involved in a proposed project.

In the case of an eligible applicant that is a partnership between a nonprofit organization and(1) one or more LEAs or (2) a consortium of schools, the partner that was the applicant, and became the grantee when the partnership was selected to receive an award, may make subgrants to one or more of the official partners

Why It Is Important

All Eligible Applicants Must Implement Practices, Strategies, or Programs for

High-Need Students

12

MUST

MUST

High-need student means a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient.

High-need student means a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient.

13

Improve Achievementfor High-Need

Students


Students

Teacher and Principal Effectiveness


Improved Use of Data SystemsImproved Use

of Data Systems

College- and Career-ready Standards and

High Quality Assessments



Improving Achievement in Persistently Low-

performing Schools


performing Schools

Early Learning(0 or 1 point)


College Access and Success

(0 or 1 point)


(0 or 1 point)

Serving Students with Disabilities and Limited

English Proficient Students(0 or 1 point)



Serving Students in Rural LEAs

(0, 1, or 2 points)


(0, 1, or 2 points)

i3 Absolute Priorities

Required forall applications

Must select one(Absolute Priority)

May select one or more(Competitive Preference)

14

Notes on Absolute Priority 1Innovations that Support Effective Teachers and Principals

• “…increase the number or percentages of teachers or principals who are highly effective teachers or principals or reduce the number or percentages of teachers or principals who are ineffective, especially for teachers of high-need students…”

• “…by identifying, recruiting, developing, placing, rewarding, and retaining highly effective teachers or principals (or removing ineffective teachers or principals).”

• “…teacher or principal effectiveness should be determined through an evaluation system that is rigorous, transparent, and fair; performance should be differentiated using multiple rating categories of effectiveness; multiple measures of effectiveness should be taken into account, with data on student growth as a significant factor, and the measures should be designed and developed with teacher and principal involvement.”

Two Possible Routes

Two Possible Routes

Multiple Measures of Effectiveness

Multiple Measures of Effectiveness

Multiple MethodsMultiple Methods

15

Notes on Absolute Priority 2 Innovations that Improve the Use of Data

• “…(a) encourage and facilitate the evaluation, analysis, and use of student achievement or student growth data by educators, families, and other stakeholders in order to inform decision-making and improve student achievement, student growth, or teacher, principal, school, or LEA performance and productivity; or (b) enable data aggregation, analysis, and research”

• “…data must be disaggregated using the student subgroups described in section 1111(b)(3)(C)(xiii) of the ESEA…”

Two Possible Areas of FocusTwo Possible

Areas of Focus

Data Disaggregation

Data Disaggregation

16

Notes on Absolute Priority 3 Innovations that Complement the Implementation of High

Standards and High-Quality Assessments• “…standards and assessments that measure students’

progress toward college and career-readiness…”

• “…may include, but are not limited to, … (a)increase the success of underrepresented student

populations in academically rigorous courses and programs…;

(b)increase the development and use of formative assessments or interim assessments, or other performance-based tools and “metrics” that are aligned with high student content and academic achievement standards; or

(c)translate the standards and information from assessments into classroom practices that meet the needs of all students, including high-need students.

• “…eligible applicant must propose a project that is based on standards that are at least as rigorous as its State’s standards…”

Focus on College & Career Readiness

Focus on College & Career Readiness

Range of Allowable Projects

Range of Allowable Projects

RigorousStandardsRigorousStandards

17

Notes on Absolute Priority 4 Innovations that Turn Around

Persistently Low-Performing Schools

• “Whole-school reform, including, but not limited to, comprehensive interventions to assist, augment, or replace Investing in Innovation Fund Absolute Priority 4 schools, including the school turnaround, restart, closure, and transformation models of intervention … or …”

• “Targeted approaches to reform, including, but not limited to:

(1) Providing more time for students to learn core academic content by expanding or augmenting the school day, school week, or school year, or by increasing instructional time for core academic subjects

(2) integrating ‘‘student supports’’ into the school model to address non-academic barriers to student achievement

(3) creating multiple pathways for students to earn regular high school diplomas”

Projects May Choose Either

Approach

Projects May Choose Either

Approach

18

i3 Priority 4 Schools

Under Absolute Priority 4, the Department provides funding to support strategies, practices, or programs that are designed to turn around schools that are in any of the following categories:

(a)persistently lowest-achieving schools (as defined in the final requirements for the School Improvement Grants program, see http://www2.ed.gov/programs/sif/faq.html);

(b)Title I schools that are in corrective action or restructuring under section 1116 of the Elementary and Secondary Education Act (ESEA); or

(c)secondary schools (both middle and high schools) eligible for but not receiving Title I funds that, if receiving Title I funds, would be in corrective action or restructuring under section 1116 of the ESEA.

19


Students


Students



Improved Use of Data SystemsImproved Use

of Data Systems






performing Schools


performing Schools




(0 or 1 point)


(0 or 1 point)






(0, 1, or 2 points)


(0, 1, or 2 points)

i3 Competitive Preference Priorities

Required forall applications

Must select one(Absolute Priority)

May select one or more(Competitive Preference)

20

Notes on Competitive Preference Priority 5 Innovations for Improving Early Learning Outcomes

• “…improve educational outcomes for high-need students who are young children (birth through 3rd grade) by enhancing the quality of early learning programs”

• “…(a) improving young children’s school readiness (including social, emotional, and cognitive readiness) so that children are prepared for success in core academic subjects (as defined in section 9101(11) of the ESEA); (b) improving developmental milestones and standards and aligning them with appropriate outcome measures; and (c) improving alignment, collaboration, and transitions between early learning programs that serve children from birth to age three, in preschools, and in kindergarten through third grade.”

Focus on High-need Children

Focus on High-need Children

Projects Must Address All 3Projects Must Address All 3

21

Notes on Competitive Preference Priority 6 Innovations that Support College Access and Success

• “… enable kindergarten through grade 12 (K–12) students, particularly high school students, to successfully prepare for, enter, and graduate from a two- or four-year college…”

• “…(a) address students’ preparedness and expectations related to college; (b) help students understand issues of college affordability and the financial aid and college application processes; and (c) provide support to students from peers and knowledgeable adults.”

Focus on College

Graduation

Focus on College

Graduation

Projects Must Address All 3Projects Must Address All 3

22

Notes on Competitive Preference Priority 7 Innovations to Address the Unique Learning Needs of Students

with Disabilities and Limited English Proficient Students

• “…address the unique learning needs of students with disabilities, including those who are assessed based on alternate academic achievement standards, or the linguistic and academic needs of limited English proficient students.”

• “…must provide for the implementation of

particular practices, strategies, or programs that are designed to improve academic outcomes, close achievement gaps, and increase college- and career-readiness, including increasing high school graduation rates (as defined in this notice), for students with disabilities or limited English proficient students.”

Focus on Either Student

Population

Focus on Either Student

Population

Projects That Improve Specific

Outcomes

Projects That Improve Specific

Outcomes

23

Notes on Competitive Preference Priority 8 Innovations that Serve Schools in Rural LEAs

• “…focus on the unique challenges of high-need students in schools within a rural LEA…”

• “…must include practices, strategies, or programs that are designed to improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or improve teacher and principal effectiveness in one or more rural LEAs.”

Focus on Specific

Locations

Focus on Specific

Locations

Projects May Address Range

of Outcomes

Projects May Address Range

of Outcomes



5 minutes







10 minutes


24



5 minutes







10 minutes


25

i3 Selection Criteria and Points

26

* Development grants will be judged in two tiers: all eligible applications will be scored on Criteria A, C, E, F, and G and the competitive preference priorities; then high-scoring applications will be scored on Criteria B and D by a different panel of reviewers.

Selection Criteria Development Validation Scale-Up

A. Need for the Project and Quality of the Project Design 25 20 15

B. Strength of Research, Significance of Effect and Magnitude of Effect

10* 15 20

C. Experience of the Eligible Applicant

25 20 15

D. Quality of the Project Evaluation 15* 15 15

E. Strategy and Capacity to Bring to Scale or to Further Develop and Bring to Scale

5 10 15

F. Sustainability 10 10 10

G. Quality of the Management Plan and Personnel 10 10 10

Total Points 100 100 100

Subject Matter Reviewers will Score Criteria A, C, E, F, and G (not including the Evidence and Evaluation Criteria)

Subject Matter Reviewers

Scale-Up GrantsA. Need for the Project and Quality of the Project

Design (up to 15 points)

28

The Secretary considers the need for the project and quality of the design of the proposed project. In determining the need for the project and quality of the design of the proposed project, the Secretary considers the following factors:

1)The extent to which the proposed project represents an exceptional approach to the priorities the eligible applicant is seeking to meet (i.e., addresses a largely unmet need, particularly for high-need students, and is a practice, strategy, or program that has not already been widely adopted).

2)The extent to which the proposed project has a clear set of goals and an explicit strategy, with actions that are

(a) aligned with the priorities the eligible applicant is seeking to meet, and

(b) expected to result in achieving the goals, objectives, and outcomes of the proposed project.

Scale-Up Grants C. Experience of the Eligible Applicant (up to 15

points)

29

The Secretary considers the experience of the eligible applicant in implementing the proposed project. In determining the experience of the eligible applicant, the Secretary considers the following factors:

1)The past performance of the eligible applicant in implementing large, complex, and rapidly growing projects.

2)The extent to which an eligible applicant provides information and data demonstrating that—

(a) In the case of an eligible applicant that is an LEA, the LEA has— (i) Significantly closed the achievement gaps between groups of students described in section 1111(b)(2) of the ESEA, or significantly increased student achievement for all groups of students described in such section; and (ii) Made significant improvements in other areas, such as graduation rates or increased recruitment and placement of high-quality teachers and principals, as demonstrated with meaningful data; or

(b) In the case of an eligible applicant that includes a nonprofit organization, the nonprofit organization has significantly improved student achievement, attainment, or retention through its record of work with an LEA or schools.

Scale-Up GrantsE. Strategy and Capacity to Bring to Scale (up to

15 points)

30

The Secretary considers the quality of the eligible applicant’s strategy and capacity to bring the proposed project to scale on a national, regional, or State level. In determining the quality of the strategy and capacity to bring the proposed project to scale, the Secretary considers:

1)The number of students proposed to be reached by the proposed project and the capacity of the eligible applicant and any other partners to reach the proposed number of students during the course of the grant period.

2)The eligible applicant’s capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to bring the proposed project to scale on a national, regional, or State level working directly, or through partners, either during or following the end of the grant period.

3)The feasibility of the proposed project to be replicated successfully, if positive results are obtained, in a variety of settings and with a variety of student populations. Evidence of this ability includes the proposed project’s demonstrated success in multiple settings and with different types of students, the availability of resources and expertise required for implementing the project with fidelity, and the proposed project’s evidence of relative ease of use or user satisfaction.

4)The eligible applicant’s estimate of the cost of the proposed project, which includes the start-up and operating costs per student per year (including indirect costs) for reaching the total number of students proposed to be served by the project. The eligible applicant must include an estimate of the costs for the eligible applicant or others (including other partners) to reach 100,000, 500,000, and 1,000,000 students.

5)The mechanisms the eligible applicant will use to broadly disseminate information on its project so as to support replication.

Scale-Up Grants F. Sustainability (up to 10 points)

31

The Secretary considers the adequacy of resources to continue the proposed project after the grant period ends. In determining the adequacy of resources for the proposed project, the Secretary considers the following factors:

1)The extent to which the eligible applicant demonstrates that it has the resources to operate the project beyond the length of the Scale-up grant, including a multi-year financial and operating model and accompanying plan; the demonstrated commitment of any other partners; and evidence of broad support from stakeholders (e.g., State educational agencies, teachers’ unions) critical to the project’s long-term success.

2)The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the eligible applicant and any other partners at the end of the Scale-Up grant.

Scale-Up Grants G. Quality of the Management Plan and

Personnel (up to 10 points)

32

The Secretary considers the quality of the management plan and personnel for the proposed project. In determining the quality of the management plan and personnel for the proposed project, the Secretary considers:

1)The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks, as well as tasks related to the sustainability and scalability of the proposed project.

2)The qualifications, including relevant training and experience, of the project director and key project personnel, especially in managing large, complex, and rapidly growing projects.

3)The qualifications, including relevant expertise and experience, of the project director and key personnel of the independent evaluator, especially in designing and conducting large-scale experimental and quasi-experimental studies of educational initiatives.

Evidence and Evaluation Reviewers will Score Criteria B and D only

Evidence and Evaluation Reviewers

Scale-Up Grants B. Strength of Research, Significance of Effect,

and Magnitude of Effect (up to 20 points)

34

The Secretary considers the strength of the existing research evidence, including the internal validity (strength of causal conclusions) and external validity (generalizability) of the effects reported in prior research, on whether the proposed project will improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates. Eligible applicants may also demonstrate success through an intermediate variable that is strongly correlated with improving these outcomes, such as teacher or principal effectiveness. In determining the strength of the existing research evidence, the Secretary considers the following factors: 1)The extent to which the eligible applicant demonstrates that there is strong evidence (as defined in this notice) that its implementation of the proposed practice, strategy, or program will have a statistically significant, substantial, and important effect on improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates. 2)The importance and magnitude of the effect expected to be obtained by the proposed project, including the extent to which the project will substantially and measurably improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates. The evidence in support of the importance and magnitude of the effect would be the research-based evidence provided by the eligible applicant to support the proposed project.

Strong Evidence: Scale-up

35

Internal Validity and External Validity

Evidence from previous studies whose designs can support causal conclusions (i.e., studies with high internal validity), and studies that in total include enough of the range of participants and settings to support scaling up to the State, regional, or national level (i.e., studies with high external validity)

Practice, Strategy, or Program in Prior Research

The same as that proposed for support under the Scale-up grant

Participants and Settings in Prior Research

Included the kinds of participants and settings proposed to receive the treatment under the Scale-up grant

Significance of Effect

Effect in prior research was statistically significant, and would be likely to be statistically significant in a sample of the size proposed for the Scale-up grant

Magnitude of Effect Based on prior research, substantial and important for the target population for the Scale-up project

Strong Evidence: Scale-up (cont’d)

36

Scale-Up Grants D. Quality of the Project Evaluation (up to 15

points)

37

The Secretary considers the quality of the evaluation to be conducted of the proposed project. In determining the quality of the evaluation, the Secretary considers the following factors:

1)The extent to which the methods of evaluation will include a well-designed experimental study or, if a well-designed experimental study of the project is not possible, the extent to which the methods of evaluation will include a well-designed quasi-experimental study.2)The extent to which, for either an experimental study or a quasi-experimental study, the study will be conducted of the practice, strategy, or program as implemented at scale.3)The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes.4)The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings.5)The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively.6)The extent to which the proposed evaluation is rigorous, independent, and neither the program developer nor the project implementer will evaluate the impact of the project.



5 minutes







10 minutes


38



5 minutes







10 minutes


39

i3 Peer Review Process

40

For all three grant types…For all three

grant types…

• The Department is using independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest

• Two peer reviewers who are evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation. Two – three peer reviewers who are subject matter experts will score selection criteria A, C, E, F and G.

• Applications were assigned to panels by one of the four absolute priorities where possible.

• Peer reviewers will determine whether any competitive preference priority points should be added

Scale-Up Only…

Scale-Up Only…

• Scale-Up finalists will be invited to an in-person interview in which they may clarify elements of their application to a panel of reviewers – finalists will not be able to present new information.

What is expected of a Peer Reviewer?

41

ProcessProcess

BehaviorBehavior

• Review the entire i3 application package and FAQs

• Review and become thoroughly familiar with the selection criteria, factors, and notes

• Familiarize yourself with the e-Reader system – used for inputting scores and comments

• Participate in all scheduled conference calls

• Provide appropriate comments that justify the score awarded and are helpful to the applicant

• Revise comments as suggested by your panel monitor

• Return all forms as required to ensure payment and completion of review process

• Be available the entire review process

• Draw upon your expertise

• Maintain confidentiality throughout the review process

PreparationPreparation

What must a Peer Reviewer really do? Receive applications and panel assignments from Synergy Identify any Conflicts of Interest, and notify Synergy or panel

monitor immediately Participate in Orientation webinar Register in and become acquainted with e-Reader Read applications (It may be helpful to you to take notes!) Write draft comments, assign scores, and submit drafts in e-Reader

before panel discussion (We recommend cutting and pasting from Word)

Receive feedback on comments from Panel Monitor Revise draft comments based on feedback as appropriate When comments/scores are acceptable to Panel Monitor, submit

final comments/scores Sign and return Technical Review Signature Forms to Synergy Remember to submit All Necessary Forms to Synergy

42

Prior to Panel

Discussion

Prior to Panel

Discussion

After Panel DiscussionAfter Panel Discussion

Conflict of Interest: Direct vs. Indirect

43

Direct Conflict of

Interest

Direct Conflict of

Interest

Indirect Conflict of

Interest

Indirect Conflict of

Interest

A peer reviewer has a DIRECT conflict of interest if: The individual’s financial interests were affected by the outcome of the Investing in

Innovation competition; An individual helped prepare an Investing in Innovation application, even if he or she

has no financial interest in the outcome of that application; An individual has agreed to serve as an employee or consultant, or otherwise provide

assistance or advice, on any project for which funding is being sought in any Investing in Innovation application, or has been offered the opportunity to do so, and has not yet accepted or declined.

A peer reviewer has an INDIRECT conflict of interest if any of the following individuals or

organizations has a personal financial interest in the outcome of the competition: The reviewer’s spouse, his or her child, a member of his or her household, or any

relative with whom he or she has a close relationship; Any employer the reviewer has served within the last 12 months; a business partner;

an organization the reviewer has served as an officer, director, or trustee within the last 12 months; or an organization that he or she serves as an active volunteer;

Any person or organization with whom the reviewer is negotiating for, or has an arrangement concerning, future employment; or

Any professional associate – including any colleague, scientific mentor, or student – with whom the reviewer is currently conducting research or other professional activities, or with whom the reviewer has conducted such activities within the last 12 months.

Conflict of Interest: What must a Peer Reviewer do?

44

Before Receiving

Applications

Before Receiving

Applications

After Receiving

Applications

After Receiving

Applications

• Complete the conflict of interest questions that were provided by Synergy.

• Review the applications and contact your panel monitor immediately if, while reading the applications, you identify an area in which you may have a direct or indirect conflict of interest.

• Specifically, a peer reviewer should review the applicant, official and other partners, and, if applicable, sources for the private-sector match to ensure that he or she does not have a conflict of interest with any of the entities associated with each application.

Other than the peer reviewer, what are the key roles within the peer review process?

45

Competition Manager

Competition Manager

• Manages relationship with Contractor (SEI/Synergy)• Coordinates across all panel monitors• Supports all panel monitors

• Contact reviewers to schedule panel meetings• Assist reviewers with any questions• Identify any conflicts of interest• Make sure reviewers enter comments/scores in e-Reader

promptly• Keep panel discussions on track• Promptly send feedback on written comments/scores to

reviewers and notify reviewers when scores and be submitted in final form

• Assign peer reviewers to panels, and re-assign as required• Mail applications to peer reviewers• Processes paperwork and compensates peer reviewers• Provide peer reviewers resources on their website

Panel MonitorPanel Monitor

Contractor (SEI/ Synergy)

Contractor (SEI/ Synergy)



5 minutes







10 minutes


46

Scoring Applications Numeric scores indicate how well the applicant responded to the

selection criteria Use the entire range of points for each criterion. For each of the competitive preference priorities, applicants do not earn a

point simply for addressing the priority. Reviewers must award points based on how well applicants address the requirements of the priority.

Make sure that your comments are consistent with your numeric scores If full points are awarded to a criterion, all comments should be in

“strengths” section and “no weaknesses found” should be entered in the “weaknesses” section.

If partial points are awarded to a criterion, there should be points and comments under both the “strengths” and “weaknesses” sections.

47

Writing Comment - “Do’s & Don’ts”

48

“Do’s” “Don’ts”

Read and score each application against the selection criteria

Provide comments that support the numeric score and are based on the strengths and weaknesses of an application’s responses to the selection criteria.

Write clear, concise, and objective comments for each of the Selection Criteria, including all factors

Write in complete sentences Check the spelling and grammar of

all comments before you submit Include application page numbers

where helpful in your comments

• Compare applications to one another

• Summarize, paraphrase or quote information presented in the application without adding comments that explain your judgment about that information

• Write brief one- or two-word notations to justify your score on a particular selection criterion

• Use inflammatory or derogatory language when writing your comments

• Ask questions of the applicant in your comments

Sample Comment – #1

“The application proposes to create a professional development program for teachers that will instruct them how to use performance data from class assignments and to use it to improve their teaching. It’s an excellent idea.”

Problems: The comment simply paraphrases the application, and it does not explain why the reviewer likes the idea.

Revision: “One of the strengths of the proposed professional development program is that it gives teachers direct practice in using the newly created online repository of student assignments that can be used “in the moment” when classroom assessments identify a learning deficiency. As the application demonstrates on p. 17, this technology-based approach aligns with research showing that learning deficits can be remedied if addressed immediately.”

Why is it Better?: It explains more clearly why the reviewer believes this is a quality program, and it includes specific references to the proposal.

49

Sample Comments – #2“This management plan seems thorough and thoughtful, with appropriate LEA representation on the advisory council. The allocation of leadership and responsibilities seems appropriate to the tasks, and the schedule laid out in the chart seems reasonable.”

Problems: At first glance, these comments seem fine, but they are not as helpful as they could be. More detail is necessary to explain what about the management plan is “thorough and thoughtful.” What characteristics of the schedule or the allocation of responsibilities is the reviewer referencing?

Partial Revision: “…It makes sense in this case to split administrative leadership of the project from the coordination of the each of the school-based implementation sites and to assign these duties to different individuals…”

Why is it Better?: The revision makes more clear what specific aspects of the leadership plan are “appropriate” and “reasonable.”

50

Sample Comments – #3

(Weakness) “The administrative employee is on contract and is not a full-time employee. Is that advisable?”

Problem: It may appear that points have been deducted based on an issue of uncertainty, and the applicant cannot respond. It’s generally best to avoid questions. No weakness has been specifically identified.

Revision: Because the administrative employee is contracted, rather than serving as a full-time employee, it is unclear whether the applicant would be able to absorb this individual’s critical functions after the grant period is over.

Why is it Better?: The revision refrains from asking a question and more clearly explains why there might be a weakness in the way this employee has been budgeted.

51

e-Reader 101 Go to http://e-grants.ed.gov and click the “Continue” button at the

bottom of the page and then “Register” button at the left on the next page

Indicate that you are NOT an ED Employee and select the e-Reader module

Complete the User Registration page and Submit, making sure you used the correct email address. A system generated password will be e-mailed to you

Use the username and password to login*

e-Reader is viewed best in Internet Explorer (IE) 5. However, it currently supports Netscape 6.2, Firefox 2.2 and older versions of IE, Netscape, and Firefox.

e-Reader is MAC compatible and does support Safari. Please make sure that you have Cookies and JavaScript enabled

in your browser.

Tuesday, Friday and Saturday: Available 24 hours Monday and Thursday: 6am – midnight; Sunday: midnight – 8pm

and Wednesday: midnight – 7pm

System Requirements

System Requirements

Help Desk Hours of

Operation

Help Desk Hours of

Operation

RegistrationRegistration

*Existing e-Reader users should use their existing username and password. If it has been a while since you last logged in, your profile may need to be updated.

Other Important Resources

53

Investing in Innovation Fund Website: (http://www.ed.gov/programs/innovation/index.html)

Notice of Final Priorities, Requirements, Definitions, and Selection Criteria

Application Package (includes the Notice Inviting Applications)

Frequently Asked Questions Evidence Summary Table Selection Criteria Summary Table

i3 Overview (PowerPoint) i3 At-A-Glance (Quick Reference) Archived recordings of the i3 Webinars i3 Glossary

Please submit all questions to your panel monitor.

Please submit all questions to your panel monitor.



5 minutes







10 minutes


54

APPENDIX

• Key Definitions• Other Important Resources• Disabling Smart Quotes in MS Word

56

Absolute Priority 1: Key Definitions

Highly effective teacher means a teacher whose students achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth. Eligible applicants may include multiple measures…

Highly effective teacher means a teacher whose students achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth. Eligible applicants may include multiple measures…

Highly effective principal means a principal whose students, overall and for each subgroup as described in section 1111(b)(3)(C)(xiii) of the ESEA, achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth. Eligible applicants may include multiple measures...

Highly effective principal means a principal whose students, overall and for each subgroup as described in section 1111(b)(3)(C)(xiii) of the ESEA, achieve high rates (e.g., one and one-half grade levels in an academic year) of student growth. Eligible applicants may include multiple measures...

57


Student growth means the change in student achievement data for an individual student between two or more points in time. Growth may be measured by a variety of approaches…

Student growth means the change in student achievement data for an individual student between two or more points in time. Growth may be measured by a variety of approaches…

Student achievement means(a)For tested grades and subjects:

(1) A student’s score on the State’s assessments under section 1111(b)(3) of the ESEA; and, as appropriate,

(2) other measures of student learning, such as those described in paragraph (b) of this definition, provided they are rigorous and comparable across classrooms; and (b) For non-tested grades and subjects: alternative measures of student learning and performance such as student scores on pre-tests and end-of-course tests; student performance on English language proficiency assessments; and other measures of student achievement that are rigorous and comparable across classrooms.

Student achievement means(a)For tested grades and subjects:

(1) A student’s score on the State’s assessments under section 1111(b)(3) of the ESEA; and, as appropriate,

(2) other measures of student learning, such as those described in paragraph (b) of this definition, provided they are rigorous and comparable across classrooms; and (b) For non-tested grades and subjects: alternative measures of student learning and performance such as student scores on pre-tests and end-of-course tests; student performance on English language proficiency assessments; and other measures of student achievement that are rigorous and comparable across classrooms.

58


Core academic subjects means “English, reading or language arts, mathematics, science, foreign languages, civics and government, economics, arts, history, and geography.”

The Department interprets the core academic subject of “science” to include STEM education (science, technology, engineering, and mathematics).

Core academic subjects means “English, reading or language arts, mathematics, science, foreign languages, civics and government, economics, arts, history, and geography.”

The Department interprets the core academic subject of “science” to include STEM education (science, technology, engineering, and mathematics).

59


Persistently lowest-achieving schools means: (1) any Title I school in improvement, corrective action, or restructuring that:(i)is among the lowest-achieving five percent of Title I schools in improvement, corrective action, or restructuring or the lowest-achieving five Title I schools in improvement, corrective action, or restructuring in the State, whichever number of schools is greater; or (ii) is a high school that has had a graduation rate as defined in 34 CFR 200.19(b) that is less than 60 percent over a number of years;(2) any secondary school that is eligible for, but does not receive, Title I funds that (i) is among the lowest-achieving five percent of secondary schools or the lowest-achieving five secondary schools in the State that are eligible for, but do not receive, Title I funds, whichever number of schools is greater; or (ii) is a high school that has had a graduation rate as defined in 34 CFR 200.19(b) that is less than 60 percent over a number of years.

Persistently lowest-achieving schools means: (1) any Title I school in improvement, corrective action, or restructuring that:(i)is among the lowest-achieving five percent of Title I schools in improvement, corrective action, or restructuring or the lowest-achieving five Title I schools in improvement, corrective action, or restructuring in the State, whichever number of schools is greater; or (ii) is a high school that has had a graduation rate as defined in 34 CFR 200.19(b) that is less than 60 percent over a number of years;(2) any secondary school that is eligible for, but does not receive, Title I funds that (i) is among the lowest-achieving five percent of secondary schools or the lowest-achieving five secondary schools in the State that are eligible for, but do not receive, Title I funds, whichever number of schools is greater; or (ii) is a high school that has had a graduation rate as defined in 34 CFR 200.19(b) that is less than 60 percent over a number of years.

60

Competitive Preference Priority 7: Key Definitions

High school graduation rate means a four-year adjusted cohort graduation rate consistent with 34 CFR 200.19(b)(1) and may also include an extended-year adjusted cohort graduation rate consistent with 34 CFR 200.19(b)(1)(v) if the State in which the proposed project is implemented has been approved by the Secretary to use such a rate under Title I of the ESEA.

High school graduation rate means a four-year adjusted cohort graduation rate consistent with 34 CFR 200.19(b)(1) and may also include an extended-year adjusted cohort graduation rate consistent with 34 CFR 200.19(b)(1)(v) if the State in which the proposed project is implemented has been approved by the Secretary to use such a rate under Title I of the ESEA.

61

Competitive Preference Priority 8: Key Definitions

Rural LEA means an LEA that is eligible under the Small Rural School Achievement program or the Rural and Low-Income School program authorized under Title VI, Part B of the ESEA.

Rural LEA means an LEA that is eligible under the Small Rural School Achievement program or the Rural and Low-Income School program authorized under Title VI, Part B of the ESEA.

Disable “Smart Quotes” in MS Word

Peer Reviewers are encouraged to write their comments is MS Word, then copy and past them into e-Reader. Before you start typing in MS Word , you should disable Smart Quotes to avoid formatting problems in e-Reader. The instructions are below:

o In MS Word on the Tools menu, click AutoCorrect. o Go to the AutoFormat As you Type tab. o Uncheck the option “Straight quotes” with “smart quotes.” o Uncheck the option Symbol characters (- -) with symbols (--).o Go to the AutoFormat tab. o Uncheck the option “Straight quotes” with “smart quotes.” o Uncheck the option Symbol characters (- -) with symbols (--). o Click the OK button.

62

investing in innovation (i3) reviewer orientation

Documents

panel of reviewers finalists

review processpreparationwhat

minutesi3 peer review

panel assignments

panel monitorreturn

independent peer reviewers

ereader10 minutesqa

writing comments