Analysis of the Quality of Professional Development Programs for Mathematics and Science Teachers:

Findings from a Cross-State Study

Rolf K. Blank

Nina de las Alas

Carlise Smith

February 2007

(With errata February 2008)

Report prepared under a grant to the Council of Chief State School Officers from National Science Foundation, Grant # REC 0438359

Council of Chief State School Officers, Washington, DC. PI: Rolf K. Blank rolfb@ccsso.org: 202 336-7044

Improving Evaluation of Professional Development

in Mathematics and Science Education

Table of Contents

Executive Summary of Study Findings ........................................................................................... 1 Study Purpose .................................................................................................................................. 3

State Leader Needs .................................................................................................................... 3 A Common Rubric for Assessing Quality of Programs ............................................................ 4 Study Design ............................................................................................................................. 5 Table 1: Teacher Professional Development Programs in CCSSO Study ................................ 6

Phase I: Analysis of Quality of Professional Development Activities from

Review of Program Designs ............................................................................................... 8 Content Focus............................................................................................................................ 8 Active Learning....................................................................................................................... 10 Coherence of Professional Development ................................................................................ 12 Collective Participation of Teachers ....................................................................................... 14 Sufficient Time—Duration and Frequency of Professional Development Activities ............. 15 Program Quality Summary...................................................................................................... 16

Phase II: Program Evaluation Designs and Initial Findings

from Analysis of Evaluation Results ................................................................................ 17 Evaluation Designs.................................................................................................................. 17 Plans for Evaluation Reporting and Dissemination................................................................. 27 Summary of Findings on Program Quality Review ................................................................ 28 Results from Analysis of Evaluation – Initial Findings .......................................................... 29

Appendix

Table 2: Funding Sources of Teacher Professional Development Programs Table 3: Evaluation Designs of Programs Table 4: Program Evaluation Report Results, as of December 2006

Evaluation Study Reports Reviewed References

1

Analysis of the Quality of Professional Development Programs for Mathematics and Science Teachers: Findings from a Cross-State Study

In 2005 the Council of Chief State School Officers (CCSSO) began a study of teacher professional development programs in mathematics and science through a grant from the National Science Foundation. States nominated professional development programs for the study, and to conduct the study, the CCSSO team has worked with state coordinators and local program directors and evaluators. The study is designed to assist education leaders in all states by providing a cross-state analysis of the quality of professional development programs and evaluations using a common rubric developed from recent research on program effectiveness. Executive Summary of Study Findings The CCSSO analysis is based on a study sample of 25 professional development initiatives across 14 states that are representative of the current leading efforts to improve the teaching of math and science in public schools. The findings from the CCSSO-led study provide a picture of the status and prospects for math and science teacher professional development, and particularly initiatives supported through federal and state funding. Content Focus: The CCSSO analysis found that current leading professional development is providing content knowledge development for teachers in math and science, especially for elementary and middle grades teachers. In 22 of 25 programs reviewed, the activities were rated as significantly focused on content knowledge in math or science. Additionally, a majority of programs reviewed were rated positively for providing important pedagogical content knowledge in math or science for teachers. Active Learning: The professional development activities are using active methods of learning for teachers in a large majority of programs. In comparison to the findings from research on professional development in the mid-1990s, the sample of programs in this study were surprising in the prevalence of active roles by teachers, including developing and presenting sample lessons, use of coaching and mentoring, developing new lessons or assessments, and interaction among teachers about ways to improve their practice. Collective participation: While most of the 25 reviewed programs did organize teachers and activities by common subject area and grade level, only a minority of the sample programs focused on delivery of professional development to teachers through a school-based strategy where teachers were learning with their school colleagues. The predominant organizing pattern was to plan teacher development for a treatment group drawn a large number of schools and districts, with only a small number of teachers from each school. Coherence: In almost all programs rated, the reviewers found a description of how the program was designed to be aligned to state content standards. Additionally, a majority of program materials described how the development was consistent with local curriculum or with curriculum materials teachers were intended to use. Sufficient Time: The average time for professional development activities including follow-up work in schools was found to be significantly greater than the typical math and science professional development documented and described in the mid-1990s. Evaluation: The reviewed programs included evaluation designs with a number of evaluation objectives and tools in most programs. The CCSSO review covered four specific evaluation targets, and a majority of programs included at least one measure in each area: (a) quality of implementation of development

2

activities, (b) gains in teacher knowledge, (c) change in classroom practices, and (d) increase in student achievement. Thus in general the programs were very ambitious in emphasizing methods of evaluation. The second phase of the study is now analyzing the evaluation results being reported from the 25 programs, and then comparing planned evaluations with what is actually completed and reported to educators and decision-makers. Issues for State Education Leaders: Through the cross-state and cross-program review, CCSSO has been able to identify several issues for state leaders. First, professional development programs for teachers are increasing in total time for teachers but most designs are intended to provide about one year of teacher involvement in a specific set of activities. Then, many programs are designed to begin with a new cohort of teachers. A key question is whether teachers have sufficient time to fully implement what may have been learned, and whether follow-up activities with teachers are sufficient to have an impact on improving teaching. Second, we do not have good evidence of the degree to which programs are designed and built from research concerning prior program designs that has positive outcomes, or that current programs have specific timelines and evaluation reporting plans that will allow leaders to improve the next wave of professional development based on what is learned. With the recent new attention to scientific designs to allow measurement of impact of professional development on learning, we would like to see greater focus on how results from evaluations will be provided to decision-makers at specific points of time in the course of a project, and not long after the program activities have concluded. A third issue is how evaluations of professional development are designed in relation to the purposes for the evaluations. A majority of programs planned to use student achievement data to evaluate effects of professional development. About half the programs that reported results did include multiple years of achievement data that could be analyzed to determine change. However, in many cases the link could not be made between teachers that received professional development and the students that they subsequently taught (i.e., pre-post measurement). About one-fourth of programs evaluations did include comparison of a treatment group with a control group of teachers. Our analysis suggests that many professional development programs could use a scientific design without great extra costs. State decision-makers should consider evaluation designs that can be completed by linking data from state student assessments or local assessments with data on professional development for teachers. Integrating data systems will remove the need for buying or creating new evaluation tools for each program to be evaluated. Finally, some evaluations emphasize methods that require extensive evaluator staff time, such as classroom observations. These methods provide more in-depth qualitative data and they may be useful on a small scale at initial stages of implementation of a program, but they become very expensive as programs are expanded.

3

Study Purpose: Compare Current Professional Development to Evidence from Research In the present education policy environment, a high priority has been placed on improving teacher and teaching quality in U.S. schools. Standards-based educational improvement requires teachers to have deep knowledge of their subject and the pedagogy that is most effective for teaching the subject. States and school districts are charged with establishing and leading professional development programs, some with federal funding support, which will address major needs for improved preparation of teachers. Central to efforts to improve the quality of professional development is research-based evidence of effective programs and analysis of the characteristics of programs that make them effective.

Current policies at national and state levels are attempting to address the need for improving the preparation of teachers through professional development programs that have been demonstrated through research to be effective. Research over the past decade has provided a base of knowledge about the characteristics of effective programs of teacher professional development in mathematics and science. Recent federal programs directed toward teacher professional development, such as No Child Left Behind (NCLB) Title II and NSF teacher enhancement programs, cite recent findings from research indicating the characteristics of programs that are effective in improving teaching and learning (see Banilower, Boyd, Pasley, & Weiss, 2006; Birman & Porter, 2002, Corcoran & Foley, 2003; Cohen & Ball, 1999; Cohen & Hill, 2001; Frechtling, 2001; Garet, Porter, Desimone, Birman, & Yoon, 1999; Hiebert, 1999; Loucks-Horsley, Hewson, Love & Stiles, 1998; Kennedy, 1998; Weiss, Banilower, McMahon, & Smith, 2001) and state and local program grantees are asked to based their designs on the research. However, current evidence from large-scale national studies shows that most professional development provided to teachers does not meet these quality characteristics (Garet et al., 2001; Desimone, et al, 2002; Corcoran, et al., 2003). As educators, we know what kinds of programs should be developed and implemented based on research findings, but the field of practice is not there yet.

This CCSSO project is working to address the central problem of many professional development programs and initiatives being initiated and supported by state, local and federal funds, but a large portion of these programs are not designed based on existing research-based evidence of what constitutes effective development of teacher knowledge and skills that results in improved teaching and increased student learning. The study is designed to provide feedback to state education leaders about the degree to which a set of nominated professional development programs from a range of states meet criteria of high quality professional development established from research.

State Leader Needs State Education Agencies (SEAs) and state leaders have several roles in providing leadership with professional development for teachers. Each of these roles can benefit from research and evaluation findings concerning the characteristics of programs that have positive impact on improving teacher knowledge and skills and improving the quality of teaching and learning in classrooms:

• States write and set policies requiring teacher professional development, and in most states that have professional development requirements for re-certification the policies have a major effect on the types of professional development that are offered;

• State education agencies manage federal and state funds that are designated for teacher professional development and staff often have responsibility for determining the size, scope, focus and types of professional development teachers will receive, and many programs have specific evaluation requirements. Under the NCLB Title IIB funding for math and science states are responsible for awarding MSP local grants and each grantee must provide evaluations and reports on the effects of the MSP funding. As this program has grown since 2002, many states are taking a larger role in providing a statewide program evaluation design for grantees;

• State education specialists who manage professional development or a curriculum subject area often provide recommendations to local districts and others concerning their professional

4

development programs and strategies—including programs during scheduled in-service days, evening/weekend or summer programs, and teacher course credit options;

• State education agencies manage other areas that have significant and important implications for teacher professional development, including: development and implementation of state content standards for student learning, review and recommend curriculum materials and textbooks, and standards for teacher licensure and certification, and school improvement strategies and designs for low-performing schools and districts.

As a result of the important roles of state leaders in shaping and leading professional development in math and science education, CCSSO has undertaken the current cross-state study of the characteristics of high-quality, effective professional development. Two key expected uses of the study results for state leaders are

• Guidance on how to use findings from research on program quality—in designing, selecting, or leading professional development programs supported by state agencies or in advising local agencies. State leaders will receive specific examples of the characteristics of professional development that will match what we know from research, and we can use evidence and examples from the current programs that are reviewed; and

• Assistance with evaluation designs, tools, and use of evaluations—to improve the role of states in establishing designs and principles for evaluation and expectations for how the evaluation results will be used. All of the programs nominated by states for the CCSSO study have methods of evaluating effects of programs and activities, but the quality and appropriateness of methods used vary widely. The study will provide a review of current methods and recommendations on how states can strengthen their evaluations and improve the usefulness of the evidence.

A Common Rubric for Assessing Quality of Programs In May 2005, CCSSO convened a study planning meeting with state education specialists from the 15 participating SEAs and the study advisory panel, comprised of experts in teacher professional development research and evaluation. The meeting produced two types of recommendations for the study: first, the kinds of evidence and measures of effects that should be used to evaluate the quality of professional development programs across the states, and, second, the needs of state program managers for information on program effects that will help to guide them in designing and selecting quality programs. The input from state partners and research experts provided a sound basis for a study design that will provide comparison of current practices to accumulated research. In addition, we expect that the study findings will address the practical needs of state education leaders for improving decisions regarding new professional development programs. Based on a recent body of research on characteristics of effective professional development, the study team conducted an analysis of the program designs and intended curriculum for teacher professional development in a sample of 27 programs across 15 states that were nominated for the study. The study includes two phases. The first phase addressed two main questions:

• What is the quality of professional development across the nominated sample of programs? What is the extent of variation in quality?

• What are the main program characteristics contributing to high ratings for quality that can be identified and replicated in future program design and development?

The present report provides cross-program findings from the first phase of the study.

5

The second phase of the study is an analysis of outcomes and findings from the program evaluation reports. The analysis is based on data included in program evaluation reports being submitted to the CCSSO study team. This study phase is addressing several questions:

• What evaluation results and evidence were reported? What were the major outcomes? • To what extent were the evaluation methods described in proposals carried out? • When were evaluation findings reported to stakeholders? How were they disseminated?

The present report provides an initial analysis of what evaluation results were reported as of end of 2006. Study Design The first study phase was carried out through the development and application of the "Rubric for Professional Development Program Review” (CCSSO, 2005). The Rubric was developed in the first year of the study with assistance and recommendations from the advisory panel plus input and comment from other invited experts in program evaluation and math and science teacher development. CCSSO devised the rubric as a guide and rating instrument to conduct comparable reviews across programs using common indicators. The reviews were based on written documentation of the programs submitted by program directors, including proposals, curricula, annual reports, and program data. Each indicator of professional development program quality (e.g., focus on content knowledge), was computed from aggregate scores across 3 to 5 measures using a numeric rating (e.g., 1 to 5). A critical element of the review and rating process are definitions of expected program elements, and these statements were carefully crafted from the research literature and prior studies and then reviewed by staff and project advisers. The rubric and definitions are available on the CCSSO website— http://www.ccsso.org/projects/improving_evaluation_of_professional_development.

The program review and rating of professional development quality using the CCSSO rubric was conducted by review panels comprised of experts in math and science curriculum, school administration and program evaluation. A total of 27 professional development programs were nominated as "high quality programs" by state education leaders from 15 states, with states using their own criteria for the nominations. Review and analysis of the professional development programs using the CCSSO review rubric requires a minimum number of documents and depth of written information. Cooperation was requested from local program directors to provide the needed materials. With follow-up steps by CCSSO staff, the requested program description information was provided for 25 of the 27 programs. Each of the 25 programs were reviewed and rated by a panel of 3 to 4 experts. The key elements of program information are outlined in the following table. Across the 25 programs, a majority of those nominated by state leaders were programs supported by state MSP grants (from federal Title IIB funds). This is not surprising since many of the state representatives from 15 volunteer states are also responsible for managing MSP at the state level. Additionally, several of the programs nominated are supported by NSF MSP grants. The set of programs considered in this study are not a representative sample of all teacher professional development in math and science across the participating states. The programs tend to be well designed and have a focus on building teacher knowledge in math or science (a requirement for both MSP funding programs). We know that states voluntarily nominated their high quality programs for the review, and we know that almost all are programs were selected in their states through a process of proposal, review, and competitive selection. Table 1 provides a listing of key information for the 25 professional development programs included in the CCSSO study. Table 2 in the Appendix identifies funding sources for the reviewed programs.

6

Table 1 - Teacher Professional Development Programs in CCSSO Study

Program Number, Subject, Grade Duration, Time

Northeast Front Range (CO) 70 - 150 Gr. 6-8 teachers in math and science/year

1 year per teacher cohort, at least 120 hours of PD

Rocky Mountain Middle School (CO)

120 Gr. 5-8 teachers in math and science/(600 – 5 yrs.)

15 months, 20 graduate credit hours toward MA

Teacher Quality Enhancement/Secondary Teacher

Enhancement Project - TQE/STEP (CO)

100 Gr. 6-12 teachers in English, social studies, math and science/year

2 years, 45 graduate credit hours toward licensure

Science Lead Teacher (DE) 100 Gr. K-10 lead science teachers/year 1 year,16 hours

Science Coalition Courses (DE) 1,200 Gr. K-10 science teachers/year 1 year, Gr. K-5: 27-54 hours, Gr. 6-10: 30-45 graduate credit hours

LAUNCH II (FL) 150 Gr. 3-8 teachers in math and science/year 1 year, 60 hours

Developing Mathematical Thinking - DMT (ID) 56 Gr. K-6 math teachers/3 years 3 years, 135 hours

iCATS: Beyond the Textbook (IN) 50 Gr. 3-5 teachers/year 1 year, 45 hours

Making Math More Meaningful - M4 (IN) 50-60 Gr. K-9 math teachers/year 1 year, 100 hours

High School Mathematics Alliance Grant (KY) 12 high school teachers/3 years 3 years, 42 days

Math/Science Teachers Engaged as Mentors - MS*TEAMS (KY)

27 Gr. 6-8 teachers in math and science/3 years 3 years, 119 hours

Mathematics Access and Teaching in High Schools - MATHS (ME) 128 Gr. 6-12 math teachers/year 3 years, 100 hours

EduTron (MA) 40-50 Gr. 4-8 math teachers/year 1 year, 45 hours

Coalition for Higher Standards Math Partnership Program - Lesley/MassInsight (MA)

35 Gr. 4-8 teachers in math and science/year

6 sequential courses, 270 class hours

Teachers as Leaders and Learners - TaLL (NJ) 100-200 Gr. K-8 math teachers/year 3 years, 37.5 graduate credit

hours

Consortium for New Explorations in Coherent Teacher Education -

CONNECT-ED (NJ)

27 Gr. K-12 teachers in math and science/year

6 months for design team/6 hours for workshop attendee, for a total of 20 hours for design team, 6 hours per Big Idea module

Pursuing Excellence in Middle School Math & Science - Nash-

Rocky Mount (NC)

175-250 Gr. 6-8 teachers of math and science/year 1 year, 209 hours

BreakThrough Mathematics (OH) 100 Gr. K-12 math teachers/year 3 six-week sessions, 18 hours of PD.

Comprehensive, Data-Based Professional Development for Middle School Mathematics

Teachers (OH)

35 Gr. 4-8 math teachers/year 1 year, 49 hours

7

Program Number, Subject, Grade Duration, Time

High Desert (OR) 25 Gr. K-8 math teachers/5 years 1 year, 304 hours

Willamette Watershed Project (OR) 15 Gr. 5-8 math teachers/year 1 year, 9 graduate credit hours

Oregon Mathematics Leadership Institute - OMLI (OR)

565 elementary, middle, high teachers in math/5 years

5 years, 27 graduate course credit hours or 360 hours.

Mathematics and Science Coaching Initiative - South Carolina Coaching

Initiative (SC)

30-45 Gr. K-5 teachers in math and science/year 1 year, 170 hours

Northwest Wisconsin Partnership for Mathematics and Science Education -

NW Wisconsin (WI)

29 Gr. 6-8 teachers in math and science/year 1 year, 100-120 hours

Wisconsin Academy Staff Development Initiative Retention and

Renewal - R & R (WI)

40-80 Gr. K-12 teachers in math and science/year

2 years, 66 hours of PD for novice teachers, 154 hours of PD for experienced teachers

8

Phase I: Analysis of Quality of Professional Development Activities from Review of Program Designs

The analysis of program quality using the CCSSO rubric focused on five criteria or dimensions of program effectiveness that research has shown to be indicative of high-quality professional development programs, i.e. the criteria differentiate programs that have positive effects in changing instructional practices (Corcoran, et al., 2003; Desimone, et al, 2002; Garet et al., 2001; Hiebert, 1999; Kennedy, 1998; Loucks-Horsley, et al., 1998):

• content focus • active learning by teachers • coherence with curriculum/standards • collective participation • sufficient time—frequency, duration, follow-up

The aggregate analysis also focuses on three elements of evaluation of program quality and outcomes-- evaluation design, reported outcomes and dissemination. Content Focus Among the key professional development features confirmed by research to positively affect teaching practice in math or science is the extent to which the professional development program or activity focuses on the subject content of mathematics or science (or specific concepts in mathematics or science) (Garet, et al., 2001). The expert review panels reviewed and rated programs on content focus as they address two interconnected types of teacher development: (a) increased subject matter knowledge of teachers in math or science, and (b) increased pedagogical content knowledge of math education or science education. Programs were scored on a 5-point scale that describes the content emphasis and specificity of the teachers’ opportunities to develop their knowledge and their capacity to teach math or science topics and concepts. Figure 1 provides a graphic summary of the programs ratings for content focus across the programs that were reviewed.

(a) Subject content in math or science. Overall, the analysis of content focus showed that an overwhelming majority—22 of 25 programs—were focused on providing improved teacher preparation in mathematics or science content knowledge. Thirteen of the 25 programs received high ratings of 4 or 5 for focus on subject content, while eight of 25 programs were rated as 3 on subject content knowledge. Three examples of high content focus are

• Northeast Front Range (CO): A partnership of five districts and five institutes of higher

education provide participant teachers with 120 hours of in-depth study of data analysis & probability, geometry, earth/space science, forces & motion, and life science;

• Developing Mathematical Thinking - DMT (ID): A program funded by the Micron Foundation grant and state MSP grant, DMT is designed to increase elementary teachers’ capacity to teach for understanding by addressing teachers’ conceptual understandings and misunderstandings behind arithmetic calculations (addition, multiplication, subtraction, division).

• Teachers as Leaders and Learners TaLL (NJ): A program offers math graduate courses for middle grades teachers that cover arithmetic, algebra, geometry, and measurement as well as grade-specific science topics and provides integrated content and pedagogical instruction.

9

(b) Pedagogy in content area. The second measure of content focus is emphasis on pedagogical content knowledge—understanding how to teach subject content to students in the classroom. The program reviews by our panels showed that almost half of the programs are high scoring on pedagogical content knowledge (12 of 25), and 9 of 25 were rated as medium with a 3. Examples of highly rated programs (scoring 5) are

• DMT (ID): The program utilized a cognitively guided instruction (CGI) framework

throughout its delivery and integrates use of multiple student assessments to collect evidence of student skills and understandings;

• EduTron (MA): Participant teachers created knowledge maps that demonstrate the connection among math concepts and procedures, toolkits that document common areas for student misunderstanding of math concepts, and application problems and solutions related to the project topics;

• High Desert (OR): The program focused on teachers planning on how to teach the content they learned to the target student populations using a range of active learning strategies.

(c) Addressed needs of target group. A third measure of content focus is the degree to which

professional development is specifically designed to address the identified content weaknesses or needs of teachers in the target population. The scores across programs were consistently lower on this measure. Only seven of the 25 programs were rated highly, and two examples are

• Rocky Mountain Middle School (CO): The program utilizes a teacher content inventory to

determine the needs of individual teachers and intermittent surveys distributed among participant teachers, content development team members, and higher education instructors to identify needs as and a series of proscribed structured follow-up sessions to discern the retention and application of learned materials.

• EduTron: The program utilizes the participant teachers’ pre-test as a needs assessment. The coursework takes into account those identified needs. The program provides on-demand tutorials and runs institutes that can break out into parallel tracks based on the teachers’ comprehension of the subject matter.

Other programs in the study used pre-treatment assessments for the participant (and in some cases, control group) teachers but only as a means to gauge where the teachers’ baseline abilities and knowledge were, and not specifically to design the professional development program. Our overall assessment is that a weakness of many professional development programs is the lack of targeting of activities addressing teachers’ content knowledge and skills on the specific needs of teachers identified for the program.

10

Figure 1 - Programs' Ratings for Content Focus (N = 25 Programs)

Active Learning Research has shown that professional development (PD) programs that provide teachers opportunities to “talk, think, try out and hone new practices” facilitate increases in knowledge and skills, and changes in teaching practice (Desimone, et al., 2002; Garet et al., 2001; Lieberman, 1996). Results from our program reviews show that all programs do engage the participant teachers in active learning. Our rubric for program review included six measures of the intended PD curriculum. The measures of active learning engagements are

• Participant teachers demonstrate, model or lead instruction; present or lead discussion; • Teachers plan lessons to meet identified student needs; • Coaching or mentoring of teachers in the classroom; • Develop assessments or review/score assessments of student work; • Teachers observe other teachers; and • Teachers engage in a learning network or regular discussion with teachers, either face-to-face or

online.

Figure 2 provides a graphic summary of how the programs were rated on methods of providing active learning in professional development. Eleven of the 25 programs include all six or five of the types of active learning engagements, and another 12 of the programs include three or four types of active learning engagements. The methods most often found across programs are: having teachers demonstrate, model, or lead instruction; engaging teachers in a learning network with other teachers; having teachers plan lessons to meet student needs; and providing coaching or mentoring in the classroom.

0

2

4

6

8

10

12

14

Provides study of content area

Provides study of pedagogical content area

Addresses identified teacher needs

High Scores (4s or 5s)Medium Scores (3s)Low Scores (1s or 2s)

# of

Pro

gram

s

11

• Northeast Front Range: During the school year, teachers follow a lesson study process using materials from what they learned during their summer institute. The process involves other participant teachers viewing and critiquing a lesson taught by a colleague. Teachers see the lesson implemented with different groups of students, and use the key concepts of planning, implementing, reflecting, communicating and differentiating spread of information, and gauging understanding from students.

• Mathematics Access in High School - MATHS (ME): Teachers conduct lesson studies, create concept maps and examine student thinking and misconceptions using diagnostic probes, participate in monthly study groups to discuss instructional strategies in the classroom and follow-up on tried approaches, and teacher leaders mentor and observe other participant teachers.

• Math/Science Teachers Engaged as Mentors - MS*TEAMS (KY): Teachers and administrators are required to have common planning time, vertical and horizontal team meetings, classroom visits and visits to other schools to establish and support professional networks and study groups within school and within the program.

• Willamette Watershed Project (OR): Three released-time days are provided for each treatment teacher to observe teacher colleagues when they implement new teaching content and methods, and to analyze, discuss, and create model lessons. Treatment teachers design a lesson for a two week unit in the classroom based on content presented and learned during the summer workshop.

• Oregon Math Leadership Institute - OMLI (OR): The institute recruits and trains two teachers and one administrator per school to work as a team of intellectual leaders, master teachers and change agents for their school and district.

• Coalition for Higher Standards Math Partnership Program - Lesley/MassInsight (MA): The partnership provides education of math teachers as coaches and resources for the district. Supported by a network of higher education advisors and a listserv system, each team leader/mentor regularly meets and works with three peer teachers in their school to establish a learning community at their schools.

Figure 2 - Ratings on Active Learning in Professional Development across Programs (N = 25 Programs)

0

5

10

15

20

25

Plan lessons Coaching or mentoring in classroom

Develop, review or

score assessments

Observe other

teachers

Engage in learning network

# of

Pro

gram

s

YesNo

Model or lead instruction,

present or lead discussion

12

Coherence of Professional Development Coherence of the professional development program speaks to how the program and its implementation are aligned to the curriculum being implemented in classrooms, the standards of learning that define expectations for students, and the needs of teachers. It is important for a professional development program to be relevant to the day-to-day operations of schools and teachers, and that the content and continuing education provided to teachers is consistent with the instructional practices and knowledge needed for their specific assignments. Moreover, a professional development program that is aligned with overall policies and practices in the teacher’s school system creates a supportive environment that encourages improvement in teaching practices and facilitates long-term sustainability of the changed practices (Grant, Peterson, & Shojgreen-Downer, 1996). The specific activities for teacher professional development may be provided predominantly offsite (i.e., away from the school) or they may be predominantly school-based (e.g., coaching, mentoring, teacher learning network), but in either approach the degree of coherence of the program are key measures of quality. Five indicators of coherence were included in the program review rubric:

• The PD program is consistent with school curriculum or learning goals for students; • The PD program is aligned with state or district standards for student learning or student

performance; • The PD program is designed to meet state teacher certification or licensure standards; • The PD program is consistent with state rules for “highly qualified” teachers under NCLB; and • PD activities are aligned with the overall goals of the program.

A large majority of the programs that were reviewed had evidence of coherence across these indicators. Five programs had characteristics of all five of the indicators for coherence, and seven programs had evidence of meeting four of the five indicators. As shown in Figures 3, the three most commonly cited indicators of coherence are alignment with state or district standards (22 of 25), alignment with school curriculum or learning goals for students (20 of 25), and alignment with the broader goals for professional development (20 of 25). In addition, many of the programs cited that they particularly address the need for “highly qualified” teachers in the classrooms as defined by NCLB, and are therefore consistent with state rules on that matter (16 of 25).

(a) Consistent with local curriculum or learning goals. The following are examples of how several programs provided coherence with school curriculum and learning goals: • Consortium for New Explorations in Coherent Teacher Education - CONNECT-ED (NJ):

Teacher development is based on six Big Idea Modules (BIMs) that are linked across the grade levels. The Modules are designed by district teams that consist of one teacher from each grade level, a district administrator and either a higher education scientist or education indicator scientist;

• OMLI: The teacher content preparation includes an explicit emphasis on student discourse. Teachers learn to model pedagogical techniques used in K-12 classrooms that are the focus of the Collegial Leadership workshops conducted during the summer institutes.

(b) Aligned with standards. The following are examples of how several programs provided

coherence between the designed professional development and state or district standards:

• MATHS: In focusing on a learning goal from the state standards, teachers are taught to use “Curriculum Topic Study”—a methodological study process developed by the provider (MMSA)—and teachers identify a hierarchy of K-12 mathematics knowledge and skills in order to clarify specific math concepts, ideas, and skills;

13

• OMLI: The program uses the recommendations from the Conference Board of the Mathematical Sciences for the preparation of teachers. Furthermore, the program’s six mathematics content courses are consistent with the standards of the National Council of Teachers of Mathematics.

Note: This rating does not measure how well the teacher learning addressed teacher needs for their classrooms or school. A high proportion of rated PD programs received a positive rating for the program design (on paper) being consistent with the stated local curriculum goals, but the rating does not take into account whether teachers actually used the learning in their classrooms or whether it matched their specific school curriculum.

(c) Meets Needs for highly qualified teachers. Several examples of how programs were designed to increase the qualifications of teachers in the target population are • Lesley/MassInsight: Program is designed to meet the state’s grades 5-8 teacher professional

licensure requirement. Teachers who complete the program receive an Advanced Professional Certificate, and they satisfy state HOUSSE plan for middle school mathematics;

• TaLL: Teachers who complete the program earn from 12-15 graduate credits in mathematics and receive a mathematics specialization recognition certificate. They satisfy the New Jersey state standards for highly qualified teachers.

Figure 3 - Programs' Ratings for Coherence (N = 25 Programs)

0

5

10

15

20

25

Consistent with school

curriculum or learning goals

Aligned with state or district standards

Meets teacher certification or

licensure

Consistent with state rules for HQ

teachers

# of

Pro

gram

s

YesNo

14

Collective Participation of Teachers A “structural feature” that is key to high-quality professional development is collective participation of teachers. Research has demonstrated that a key factor in teachers implementing what they gained from professional development in their classrooms is to carry-out the program with teachers that will support and encourage the movement from teacher learning to more effective practices. As shown in Figure 4, the program review included three measures of coherence of professional development:

• Involve teachers in the same department or content area. Twenty-three of 25 programs reviewed met this indicator;

• Involve teachers in the same grade or grade span. Twenty-one of 25 programs reviewed met this indicator.

Many programs in the CCSSO program review (supported by Title II B or NSF MSP) were targeted for teachers across multiple districts, a region, or statewide, and the indicators above are consistent with these designs. The third indicator focuses on school-based designs.

• Professional development targeted teachers from the same school. About half—13 of 25 programs—met this indicator.

In some of the school-based programs, the professional development is designed to involve teachers across the grade spans to work together to understand the connections of content and skills needed in prior and subsequent grades to their assignment. Among the programs reviewed, all had an in-school component with follow-up activities, coaching or mentoring, but another pattern of professional development is to spend a majority of time in development activities with teachers in the same school learning and working together to increase their knowledge and skills. Examples of how the school-focused programs operated to provide collective participation include:

• Northeast Front Range (CO): The program was designed for teachers to work with teachers of adjoining grades in middle schools, so that 5th grade teachers know what is expected of their future 6th grade students by working with 6th grade teachers;

• LAUNCH II (FL): The program focuses on teachers in elementary and middle schools in the same feeder pattern to work together. Moreover, the program requires participating schools to make a commitment to building and sustaining a leadership cadre for mathematics and science within their building sites;

• Science Coalition Courses (DE): Most of the professional development activities take place in the schools of participating elementary teachers and the teachers are developing ways to work and learn with their peers to improve science teaching;

• iCATs: Beyond the Textbook (IN): The program targeted schools that had lower than expected ISTEP+ scores in sixth grade—teachers of fourth and fifth graders in the selected schools as well as special education teachers were asked to participate in the professional development activities.

15

Figure 4 - Program Ratings for Collective Participation (N = 25 Programs)

Sufficient Time—Duration and Frequency of professional development activities An abundance of research has presenting findings to document the merits of professional development activities that are sustained over time. Programs that provide participant teachers with more contact hours in professional development courses or activities and support the teachers through follow-up activities or assistance with implementation over a prolonged period are more likely to promote deeper understanding in the content area and extend practice of newly-learned skills with students (Garet et al, 2001). Findings from the CCSSO review of 25 professional development programs demonstrates a clear departure from one-time, day-long workshops to deeper and more sustained professional development activities. As experienced by participant teachers, 20 of the 25 programs provide at least a year’s span of teacher participation, and 10 programs continued teacher involvement for fifteen months to five years. The longer span of teacher participation generally is carried out with multiple workshops or follow-up events during the course of the academic year. A sequence found in 12 programs is first a concentrated period of content-focused development activities, generally a summer institute or extended workshop (often on a college campus) which focuses on one or two content areas, and then from 2 to 6 follow-up workshops during the course of the next academic year. As a result, participant teachers who complete the course of activities for these programs log significant professional development hours. In particular, participant teachers for the following course are expected to accumulate more than 100 hours up of contact and work-related time upon completion of the program:

0

5

10

15

20

25

Participating with teachers from the

same school

Participating with teachers in same dept. or

content area

Participating with teachers from same

grade span

# of

Pro

gram

s

YesNo

16

Program Planned Time in PD

Northeast Front Range 120 hours Rocky Mountain Middle School 120 hours DMT 135 hours M4 100 hours MS*TEAMS 119 hours MATHS 100 hours Lesley/Mass Insight 270 hours Nash-Rocky Mount 209 hours High Desert 304 hours South Carolina Coaching Initiative 170 hours NW Wisconsin 100-120 hours R & R 154 hours for mentor teachers

Program Quality Summary Across the five research-based criteria applied in the CCSSO review, many of the programs were consistently rated positively on a majority of the indicators and criteria, and we can identify five programs which were rated positively on all five criteria—Northeast Front Range, DMT, MS*TEAMS, TaLL, and OMLI. We would cite these programs as exhibiting many of the qualities of high-quality professional development programs. Further descriptive information on specific features of these programs can be accessed on the CCSSO project website— http://www.ccsso.org/projects/improving_evaluation_of_professional_development/.

17

Phase II: Program Evaluation Designs and Initial Findings from Analysis of Evaluation Results

The second objective of the study is to report on the effects of the identified programs on improving teaching and learning in math and science. This report will discuss the program evaluation designs used across the reviewed programs, and identify the types of educational outcomes being assessed in their evaluations, and how the evaluations plan to report on results and the audiences to whom the evaluation results are shared. Program evaluation information was collected during the program review from 23 of the initial 27 programs. Evaluation Designs The evaluation designs of the 23 PD programs were organized into four types of outcomes:

(a) quality of implementation of professional development activities (b) gain in content knowledge and pedagogy skills of teachers (c) change in instructional practices or curriculum that is taught (d) improvement in student achievement

Almost all of the programs—21 of 23—have evaluation designs that are designed to assess professional development across at least three of the four types of outcomes. Of the 21, seven have evaluation designs that cover all four dimensions and other outcomes. These programs utilize multiple measures to tell report to their stakeholders on how the program is being implemented and the degree to which the program activities are having effects on teaching and learning. In several cases, the programs also track for other outcomes specific to the program. Figure 5 - Summary of Program Evaluation Design (N = 23 Programs)

0

5

10

15

20

25

Quality of PD Activities

Teacher Content

Knowledge

Instructional Practices or Curriculum

Student Achievement

Other Outcomes

# of

Pro

gram

s

YesNo

18

A complete reporting of the program evaluation designs is shown in Table 3 in the Appendix.

(a) Implementation of PD Activities. The predominant method of determining the quality of implementation, found in a majority of the review programs, is the use of post-treatment surveys of teacher participants and leaders. These surveys are used to determine the characteristics of the participant teachers, to assess teacher views of what worked for them and what did not get addressed, and the level of satisfaction with the professional development experienced by teachers. A minority of programs utilize more complex measures of quality of implementation including pre-post treatment surveys (e.g., DMT, Lesley/MassInsight, Nash-Rocky Mount, Willamette Watershed, South Carolina Coaching Initiative), focus groups with PD staff (e.g., Northeast Front Range) , and observations of activities conducted by evaluators (e.g., DMT, LAUNCH II, CONNECT-ED, NW Wisconsin, R & R). In most cases, the post-treatment survey tools were independently developed by project evaluators. Survey tools used in several programs were National Survey of Science and Mathematics Education (2000 redesigned), Reformed Teacher Observation Protocol (RTOP), and Professional Development Observation Protocol (PDOP). Figure 6 illustrates OMLI’s approach to capturing teachers’ impressions and experiences with the PD activities:

Figure 6 - Examples of Post-Treatment Survey Items, OMLI Teacher Survey

19

(b) Gains in Teacher Content knowledge. Across the reviewed programs two types of assessment tools were found: a) post-treatment assessments and b) pre-post treatment assessments. Pre-post assessments measure change in teacher content knowledge over the specific content or concept area with the pre-test providing a control for level of prior knowledge. Noteworthy designs for teacher content knowledge gains were used by DMT, LAUNCH II and EduTron. Edutron administers a post-treatment assessment six months after the PD activities and LAUNCH II administers a post-treatment assessment in the spring of the following school year to determine if and how much of the new information was retained by the teachers.

Three specific teacher knowledge assessment instruments were identified in use across the sample of programs:

• One set of teacher assessments is the Diagnostic Teacher Assessments in Mathematics and

Science (DTAMS) developed by researchers at the University of Louisville. Programs such as Northeast Front Range, Rocky Mountain Middle School, High School Mathematics Alliance, and Nash-Rocky Mount utilize the Louisville DTAMs to take inventory of teachers’ knowledge in areas such as algebraic thinking and geometry. The instrument takes on average an hour per subject area for teachers to complete and it includes 10 multiple-choice and 10 open-response questions. The results are reported to evaluators in about 2 weeks. The following items in Figure 7 illustrate the types of test items participant teachers in these programs encounter:

20

Figure 7: Examples of Test Items on Teacher Knowledge, Diagnostic Teacher Assessment in Mathematics and Science

• A second set of teacher assessments of knowledge instrument found across several program is the Content Knowledge for Teaching Mathematics Measures - CKT-M measures. Developed by Deborah Ball, Heather Hill and colleagues at University of Michigan, this tool is used by BreakThrough Math and R&R programs. The instrument is distinguished by in their ability to “capture whether teachers can not only answer the mathematics problems they assign students, but also how teachers solve the special mathematical tasks that arise in teaching, including evaluating unusual solution methods, using mathematical definitions, representing mathematical content to students, and identifying adequate mathematical explanations” (need a citation here). The following is an example of a test item from that instrument:

Figure 8 - Example of Teacher Assessment Item for Pedagogical Knowledge, Content Knowledge for Teaching Mathematics Measures

21

• A third method of teacher assessment is Praxis. Two programs—TQE/STEP and High Desert MSP—rely on Praxis to capture changes in teacher content and pedagogical content knowledge. The Praxis is produced by Educational Testing Service (ETS) and one or more of the three components (basic skills, content, pedagogy) are used in many states for certification and licensure of new teachers. The following are examples of test items that can be found on the assessment:

Figure 9 - Sample Test Items, Praxis

Lastly, in several cases such as with DMT, EduTron, MS*TEAMS, Lesley/MassInsight, TaLL, CONNECT-ED, and Willamette Watershed programs utilize program-specific, instructor

22

designed assessments. The following is an example of an open-response test item Willamette Watershed instructors use to assess their participant teachers: Figure 10: Example of Test Item on Teacher Content Knowledge, Willamette Watershed

(c) Change in Instructional Practices. To assess any changes in instructional practices, [programs chiefly rely on two instruments, post-treatment surveys asking for teacher self-report information on how instruction changed or classroom observations with trainer observers using a protocol. Three instruments supported by research on validity and reliability were identified in use across the sample of programs were

• Surveys of Enacted Curriculum (SEC), a teacher self-report instrument developed by

CCSSO SEC Collaborative and WCER, it is currently in used in many states; used to measure change in instruction practices and instructional content, and alignment to standards. Below is an example of survey items on the K-12 Science Survey that focuses on instructional practices:

23

Figure 11 - Example of Survey Items, Surveys of Enacted Curriculum

• Reformed Teaching Observation Protocol (RTOP), developed by Arizona State University-Arizona Collaborative for Excellence in the Preparation of Teachers, gathers information on the context, lesson design and implementation, lesson content and classroom culture for indicators of reformed teaching. The following observation items, demonstrate areas that the instrument attempts to reveal:

Figure 12 - Examples of Observation Items, Reformed Teacher Observation Protocol

• Classroom Observation Protocol (COP), developed by Horizon Research, Inc. for LSC

study, it is a three-part protocol that collects background information about the class including class culture and information about the lesson including the purpose, content and design of the lesson plan and its implementation. Figure 13 provides an example of one area

24

the instrument attempts to capture, namely possible impact of the instruction on student achievement:

Figure 13 - Sample Observation Items, Classroom Observation Protocol

These instruments address classroom practices used by teachers and other measures for analysis of change in practices related to professional development, including school context, implementation, lesson design and implementation, content, and classroom culture. During the course of the study, CCSSO identified 15 different teacher survey instruments that are available, including many that were used by the programs (For complete list of instruments, see http://www.ccsso.org/projects/improving_evaluation_of_professional_development/Cross_State_Study/). Programs that had evaluations that included interviews and classroom observations obtained greater depth of information about teacher response and implementation process, but fewer numbers of teachers. For example, participant teachers in the LAUNCH II program reported increased frequency in the use of inquiry-oriented strategies in the classroom and in-class activities that support reformed, standards-based teaching and prevent backsliding to “traditional” teaching methods. In addition, participant teachers reported being more aware of their own misconceptions about science and learning about alternative ways to teach subject area matters through the Conceptual Change model. In many instances, the teachers commented on their intent to try the new strategies in the classroom.

(d) Measuring improvement in student achievement. All but 5 of the 23 programs included a plan

for measuring change in scores on standardized student assessments, based on review of proposals and first-year reports. None of the programs had evaluation designs of achievement gains using randomized control trials.

25

However, quasi-experimental designs were included in six programs to provide scientific evidence that student achievement change can be attributed to the professional development treatment. These programs utilized data from statewide assessments, or an assessment specifically developed for this purpose, to compare test results of students of the participant teachers with students of non-participant teachers in schools with similar demographical backgrounds and proficiency levels prior to the program’s start (i.e., treatment vs. comparison group design). Our analysis identified unique comparison designs. One program, the Colorado’s Rocky Mountain Middle School, utilizes regression discontinuity methodology. In this design, participant teachers are targeted because their students’ low test scores suggest that they need professional development. Assuming that the Rocky Mountain professional development was effective, the students’ test scores of these post–treatment participant teachers should show a shift to a level equal to those of teachers who did not receive the professional development. Another program, DMT, is conducting its program evaluation using three treatment schools and two comparison schools. This design allows for comparison of students and teachers who are comparable according to the students’ socioeconomic backgrounds and the students’ initial test results prior to the administration of the professional development and filters out possible self-selection of teachers into the program. The study takes into account the school cultures and contexts of the treatment and comparison schools alike. However not all programs rely on standardized student assessments. The OMLI uses a student discourse observation protocol that was developed from Horizon Research Inc.’s research in the Local Systemic Initiative. Trained observers using the protocol note the kinds of exchanges among students and between students and teachers that take place in the classroom and how in the hierarchy of higher-order thinking skills those discourses are. Figure 14 illustrates the codes used to document the modes, type and tools for discourse found in the classroom during a lesson:

26

Figure 14 - Examples of Illustration of Student Learning through Observation, Student Discourse Observation Protocol

(e) Other outcomes. The most common measure beyond the four categories is the number of teachers receiving “highly-qualified” status. Northeast Front Range, Rocky Mountain Middle School , TQE-STEP, MATHS, and BreakThrough Math are programs that record and track changes to their participant teachers’ “highly-qualified” status.

Using a focus group protocol on K-12 mathematics teachers and administrators that compose the instructional team, OMLI determines the degree to which a professional learning community is taking form and taking root at the local level. Independently, OMLI and LAUNCH II also examine effect on the staffs of partner institutes of higher education, such as changes in the faculty’s own instructional practices and changes in course offering in schools of education.

27

Lastly, in the case of R & R, the placement and retention of newly-inducted teachers in the teaching workforce and decreases in turnover rates for all teachers are critical to the program’s existence.

Figure 15 - Programs’ Ratings on Other Outcomes (N = 23 Programs)

Plans for Evaluation Reporting and Dissemination Although the plans for professional development program evaluation across the sites are all designed to touch on at least one of the four outcomes, stated plans for reporting on the results of those evaluations are mostly unclear. Whereas 21 of 23 programs have evaluation designs that look at the programs across at least 3 of the four outcomes, only twelve of the 23 programs include descriptions of how they plan to report on evaluation results. Four programs describe reporting plans and schedules of reports for four outcomes: Northeast Front Range, LAUNCH II, High Desert, OMLI, and R & R. Most programs planned to report on gains in teacher content knowledge (15 of 23) and implementation of program activities, gains in teacher pedagogical content knowledge, and improvements in student achievement (each at 12 of 23). These plans are consistent with the intended program activities which consistently emphasize teacher content knowledge followed by teacher pedagogical content knowledge.

0

1

2

3

4

5

6

7

No. of HQ Teachers Prof. LearningCommunity

Effects on IHEPartners

Teacher Placement &Retention

Hi-Level StudentCourse Enrollment

Dev. Of LeadershipCadre

# of

Pro

gram

s

28

Figure 16 - Programs’ Ratings on Reporting of Evaluation Results (N = 23 Programs)

Only a few programs include information on their plans for dissemination of findings. Ten of 23 programs indicated they would share the evaluation results with major stakeholders, while 15 described sharing results with district or state officials, and 13 indicated sharing results with grant offices. However, only six programs described a method of sharing results with the public.. Programs that have the widest plans for dissemination include LAUNCH II, TaLL, High Desert MSP. Half the program documents (12 of 23) described methods of sharing results with their participant teachers and 11 indicated plans for reporting to school administrators. In general, written plans for reporting and dissemination of evaluation findings are missing in many programs. This is an area for concern especially knowing the ratings of quality our review found in many of the math-science professional development programs. As we analyze the evaluation reports from these programs in the next phase of the CCSSO study, the question of how programs report and disseminate results will be important. Summary of Findings on Program Quality Review The review of professional development programs based on common criteria from research helps educators and decision-makers gain insight into the current status of programs being designed and implemented for math and science teachers. We suggest that the sample of 25 programs that were reviewed and rated are strongly representative of leading efforts to improve the teaching of math and science in public schools through focused efforts with teacher professional development. The findings of the CCSSO-led review provide a picture of the current status of math and science professional development and particularly efforts supported through federal and state funding. The review findings from our study of 25 programs in 14 states show that professional development is providing more content knowledge development for teachers in math and science, especially for elementary and middle grades teachers. The voluntarily selected sample of programs was rated on average as heavily focused on content knowledge in math or science. Also, increasing knowledge of

0

2

4

6

8

10

12

14

16

Participant Teachers School Administrators District or State Officials Grant Donors (IncludingFederal grant donors)

General public

# of

Pro

gram

s

YesNo

29

how to teach important content (pedagogy) is a key feature of a majority of the reviewed programs. A second major finding is that methods of professional development include active methods of learning by teachers. Compared to the research literature of the mid-1990s, this set of programs was surprising in the prevalence of active roles by teachers including developing and presenting sample lessons, use of coaching and mentoring, and interaction among teachers about ways to improve their practice. We also found that a majority of programs included several professional development activities, including coaching or mentoring, networking, presenting or leading discussion with colleagues, and developing new lessons or assessments. Third, existing research from existing studies show that teachers benefit from learning with their colleagues. However, while most of the 25 reviewed programs did organize teachers and activities by common subject area and grade level, only a small minority of the sample programs focused on delivery of professional development to teachers with their school colleagues. The predominant organizing pattern was to plan teacher development for a treatment group teachers that may be drawn from a large number of schools and districts, with only a small number from each school. Exceptions to this pattern were found, however. School-based professional development was the strategy for several programs (e.g., LAUNCH II, Science Coalition, Northeast Front Range, iCATs, DMT). Another organizing pattern was to provide training for teacher leaders (e.g., Lesley/MassInsight, CONNECT-ED) with the expectation that those trained teacher leaders would proceed to pass on key learning to teachers in their schools. In almost all programs rated, the reviewers found a description of how the program was designed to be aligned to state content standards. Additionally a majority of programs did describe how the development was consistent with local curriculum in their school or with curriculum materials they would use. Thus, a majority of programs were providing knowledge development for teachers in the content area and grade they were teaching, however we do not know from written program review to what extent the development specific to what teachers would be teaching in their classes. This question is for evaluation of effects of the training on classroom practices. The duration and total time planned for teacher professional development was consistently found across programs as much more extensive than the math and science professional development documented and described in the mid-1990s (e.g., Garet et al, 2001). We would grant the programs nominated by state leaders for this study are likely to be those that exceed the average (across all professional development for teachers) in duration and time. However, as an indicator of the direction of professional development supported by federal and state grants, we can state that the amount of time per teacher for professional development in specific content-focused programs is increasing. The final area of review was evaluation design, and the results were very positive on the number of evaluation tools and objectives that were built into the 25 reviewed programs. Our review covered four specific evaluation targets. A majority of programs included at least one measure of quality of PD activities, gains in teacher knowledge, change in classroom practices, and increase in student achievement. That is, the program descriptions were very ambitious in emphasizing methods of evaluation. The MSP Title IIB program under NCLB does emphasize evaluation requirements and the U.S. Department of Education’s monitoring system on grants asks for each program to report on evaluations. However, the review identified a wide range of well-tested evaluation tools that were being employed in these studies, and two-thirds of the programs described how effects of teacher development on student achievement would be measured. Results from Analysis of Evaluations – Initial Findings In Fall 2006 CCSSO requested evaluation reports from all of the sample programs that included specific evaluation designs for professional development. The study team staff reviewed the evaluation reports received in order to document which of the evaluation results were completed and released and to determine what findings were available. Since most professional development activities in the sample

30

programs began in the 2004-05 school year, the evaluation results reported and analyzed were mainly for activities evaluated during the second program year (i.e., 2005-06). Table 4 in the Appendix summarizes the status of evaluation reports across the reviewed programs.

1) What evaluation reports were received and analyzed across programs? Of the 25 programs that were reviewed by CCSSO, 20 project directors submitted evaluation reports to CCSSO and 18 of these reports contained specific findings or data in relation to the four outcomes that were analyzed in the review. Thus, several programs either did not have evaluation reports prepared or did not have reports that could be shared externally, and several included descriptive information about the programs such as workshop agenda but not data to evaluate specific objectives. The program evaluation reports include a range of outcomes data and descriptive analyses, and some of the evaluation evidence pertains to program outcomes other than for professional development, including pre-service, curriculum materials, and partnership development. The CCSSO study team identified the outcomes from professional development included in the evaluation reports submitted.

2) What outcomes from professional developed were reported by end of December 2006?

• Quality of Implementation. Evaluation data were reported from 13 programs on the quality

of how professional development activities were implemented (i.e., extent of teacher participation, fidelity of implementation of PD design, and response of teachers to the professional development). Many of the programs reviewed are still in operation, often in year two of operation. The evaluation data in this category are typically used as formative evaluations to assist decision-makers and leaders to assess how the program is operating and how improvements can be made.

• Gains in teacher knowledge. Twelve of the 20 evaluation reports included data and findings

regarding gains in teacher knowledge related to the professional development activities being evaluated. Further analysis of these data across sites will allow us to determine whether there are common factors in producing gains in knowledge and how the several knowledge tests worked to measure gains.

• Change in classroom instructional practices in relation to the professional development.

Twelve of the 20 reports also reported on the extent to which professional development actually produced change in instructional practices. A variety of measures and tools were found to be in use.

• Evaluation data reported on effects of professional development on student

achievement. Of the 20 evaluation reports analyzed by CCSSO, twelve of the reports included data on student achievement gains for students of teachers receiving professional development, but some were aggregate data at the district or school level. Six of the evaluations compared student scores of teachers in treatment group, those receiving specific professional development being analyzed as compared to a control group of students of other teachers in the target school districts or schools. All of the 12 evaluations reported student achievement trends from one year (2005) to the next year (2006). A small number tracked achievement grains for two years (2004–06).

31

This study phase is addressing three questions:

• What evaluation results and evidence were reported? What were the major outcomes? • To what extent were the evaluation methods described in proposals carried out? • When were evaluation findings reported to stakeholders? How were they disseminated?

The next CCSSO project report in Fall 2007 will focus on the second phase of the study—i.e., analyzing outcomes and findings from the program evaluation reports. The analysis so far has focused on the first question –what has been evaluated and reported. The next report will look more closely at the cross-program findings and the common patterns in results found across program models.

Table 2

Program Name Funding SourceNortheast Front Range (CO) Title II-B (State MSP)

Rocky Mountain Middle School (CO) Title II-B (State MSP) Teacher Quality Enhancement/Secondary Teacher

Enhancement Project - TQE/STEP (CO)NSF MSP; Title II-B (State MSP); Other Private Foundations & Organizations

Science Lead Teacher (DE) Title II-A (Teacher Quality); State; NSF-Local Systemic Initiative; Other Private Foundations and Institutions

Science Coalition Courses (DE) Title II-A (Teacher Quality); State; NSF-Local Systemic Initiative; Other Private Foundations and Institutions

Launch II (FL) Title II-A (Teacher Quality); Title II-B (State MSP)Developing Mathematical Thinking - DMT (ID) Title II-B (State MSP); Other Private Foundations & Organizations

iCATS: Beyond the Textbook (IN) Other Federal Government Grants -- Title II-D (Ed Tech)Making Math More Meaningful – M4 (IN) Title II-B (State MSP)

High School Mathematics Alliance Grant (KY) StateMath/Science Teachers Engaged as Mentors - MS*TEAMS

(KY) Title II-B (State MSP)

Mathematics Access and Teaching in High Schools - MATHS (ME) Title II-A (Teacher Quality); Title II-B (State MSP)

EduTron (MA) Title II-B (State MSP)Coalition for Higher Standards Math Partnership Program -

Lesley/MassInsight (MA) Title II-B (State MSP); State; Other Private Foundations & Organizations

Teachers as Leaders and Learners - TaLL (NJ) StateConsortium for New Explorations in Coherent Teacher

Education - CONNECT-ED (NJ) Other Federal Government Grants -- ED (FIPSE grant); Other Private Foundations & Organizations

Pursuing Excellence in Middle School Math & Science – Nash-Rocky Mount (NC) Title II-B (State MSP)

BreakThrough Mathematics (OH) Title II-A (Teacher Quality); Title II-B (State MSP)Comprehensive; Data-Based Professional Development for

Middle School Mathematics Teachers (OH) Title II-B (State MSP)

High Desert (OR) Title II-B (State MSP) Willamette Watershed Project (OR) Title II-B (State MSP)

Oregon Mathematics Leadership Institute - OMLI (OR) NSF MSP; Title II-B (State MSP) Mathematics and Science Coaching Initiative – South

Carolina Coaching Initiative (SC) Title II-B (State MSP); State

Northwest Wisconsin Partnership for Mathematics and Science Education – NW Wisconsin (WI) Title II-B (State MSP)

Wisconsin Academy Staff Development Initiative Retention and Renewal – R & R (WI)

NSF-Teacher Enhancement Program

# Supported through NSF MSP grants 2# Supported through Other NSF grants 3

# Supported through Title IIA grants 5# Supported through Title IIB grants 18

# Supported through other Federal Gov't. grants 2# Supported through State funds 6

# Supported through other Private Foundations & Organizations 9

Funding Sources of Teacher Professional Development Programs

Created by CCSSO, 2007Grant No. REC0438359

Table 3

Program Outcomes to Be Measured Instruments Post-treatment survey of participant teachersObservation (without protocol) of professional development activitiesYear-end, post-structured follow-up surveys of instructional team, development team, participant teachersFocus group of 15 participant teachers (non-random sampling)Post-treatment surveys of partner organizationsPre-/post-treatment teacher assessments in earth science & physicsPost-treatment surveysU. of Louisville (DTAMS) pre-/post-treatment teacher assessment (pilot year)Teacher assessment in earth science & physicsPost-treatment survey of participant teachersYear-end, post-treatment surveys of participant teachersFocus group of 15 participant teachers (non-random sampling)

Student achievement gains CSAP (statewide student assessment) - math section onlyPre-/post-treatment interviews of participant teachers (random sampling)Post- treatment surveys of instructional team and participant teachers

Post-structured-follow-up surveys of development team and participant teachers

Observation protocol for participant teachers (RTOP)Pre-/post- treatment interviews of participant teachers (random sampling)Post- treatment surveys of instructional team and participant teachers

Post-structured-follow-up surveys of development team and participant teachers

Observation protocol for participant teachers (RTOP)Pre-/post-treatment teacher assessments in physics (Horizon), chemistry, earth science, algebra (U. of Louisville (DTAMS)), biology (U. of Louisville (DTAMS)), statistics & probability (U. of Louisville (DTAMS))

Change in instructional practices Post-structured-follow-up surveys of development team and participant teachers

Student achievement gains CSAP (statewide student assessment) in math section only.Post-treatment surveys for probationary teachers Interviews of participant teachersFocus groups of participant teachers (with protocol)Program for Licensing Assessments for Colorado Educators (PLACE) or Praxis (national standardized teaching licensure exams)Student portfolios (unknown guidance or metric)Post-treatment surveys for probationary teachers Case studies - site observations of participant teachers

Student achievement gains CSAP statewide standardized assessmentsParticipant feedbackNarrative retrospective feedback from participant teachersObservation protocol (PDOP - streamlined version) Pre-/post-treatment teacher assessmentsYear-end post-treatment teacher assessmentsParticipant feedbackSummer institute surveyNarrative retrospective feedback from participant teachersPre-/post-treatment surveys of participant teachers in math & scienceParticipant teacher journalsClassroom snapshot observation protocol

Student achievement gains FCAT (statewide standardized assessment)Pre/post treatment teacher knowledge assessmentClassroom observationsSurveys of Enacted Curriculum (SEC)

Pre-/post-teacher knowledge assessment - issues of numbers (program created)

Pre-/post-teacher knowledge assessment - geometry & measurements (program created)Pre-/post-teacher knowledge assessment - probability & statistics (program created)Pre-observation protocol Observation protocol Surveys of Enacted Curriculum (SEC)ISATS (statewide standardized assessment)Direct Mathematics Assessment (grades K-2 mathematics inventory)Surveys of participant teachersParticipant teacher reflection logsVideotaped and on-site observations of classroom instructionIndiana Curriculum Framework (statewide student assessment)Evaluator-created assessment aligned with state assessment

iCATS: Beyond the Textbook (IN)

Evaluation Designs of Programs

Fidelity in the implementation of PD activities

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Northeast Front Range (CO)

Change in instructional practices

Gains in teacher knowledge

Change in instructional practices

Change in instructional practices

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Change in instructional practices

Rocky Mountain Middle School (CO)

Teacher Quality Enhancement/Secondary Teacher Enhancement

Project - TQE/STEP (CO)

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Change in instructional practices

Launch II (FL)

Developing Mathematical Thinking - DMT (ID)

Student achievement gains

Student achievement gains

Created by CCSSO, 2007NSF Grant No. GREC0438359 1 of 4

Table 3

Program Outcomes to Be Measured Instruments Post-treatment surveys of participant teachersParticipant teacher reflections logs

Gains in teacher knowledge Participant teacher assessmentChange in instructional practices Classroom observationsStudent achievement gains Student assessment based on ISTEP (statewide standardized assessment)

Attitude SurveysAttendance @sessions by WKU or GREC supervisors

Gains in teacher knowledge U. of Louisville (DTAMS) teacher assessment-Algebraic ThinkingObservation protocol (based on KY Teacher Evaluation Process) for participant teachers Participant teacher reflection logs Documentation reviews from site visits

Student achievement gains End-of-course assessmentsPost-treatment attitude surveys of participant teachersTeam meeting minutes Group reflection logsWKU discussions notes with teamsPraxis (national standardized teaching licensure exams)WKU faculty-created assessmentsMentor and mentee participant teacher reflection logsIndividual growth plans (Based on NSDC standards)Individual growth plans (Based on NSDC standards)Classroom observations (no protocol) of participant teachersWKU mentor-generated checklistStatewide standardized student assessmentPre-/post-unit student assessments (Teacher-designed)

Gains in teacher knowledge U. of Massachusetts - Pre-/post-teacher assessment of participant teachers (Concept mapping)Classroom observation protocol (COP)Focus groups of 9th graders (college-bound and non-college bound students)Document reviews of curriculaProgram specific student assessment-9th grade algebra Statewide standardized student assessments - sections covering patterns, relations, functions, algebra concepts, and mathematical communicationFocus groups of 9th graders (college-bound and non-college bound students)Mid-treatment surveys of participant teachersPost-treatment surveys of participant teachersParticipant teacher feedbackPre-/post-treatment teacher assessments of participant teachersPost-treatment retention assessment of participant teachersMid-treatment surveys of participant teachersPost-treatment surveys of participant teachersYear-end, post-treatment reflection surveys of participant teachersMCAS (statewide standardized assessment)District-created "Classroom Performance System" assessment Galileo (state-sponsored standardized student assessment)Classroom observation (no protocol)Pre-/post-treatment surveys of trainer teachersDocumentation of training of teachers by trainer teachers

Gains in teacher knowledge Pre-/post-treatment assessment of participating trainer teachersPre-/post-treatment surveys of trainer teachersVermont Institute interview & observation protocol

Student achievement gains MCAS (statewide standardized assessment)Participant teacher survey (based on re-design of 2000 National Survey of Science and Mathematics Education)End-of-course survey of participant teachersParticipant teacher survey (based on re-design of 2000 National Survey of Science and Mathematics Education)Teacher assessment (instructor-designed) of participant teachersEnd-of-course grades of participant teachers in graduate courses Participant teacher survey (based on re-design of 2000 National Survey of Science and Mathematics Education)Lesson plan analysis rubric

Student achievement gains Terra Nova test scores (standardized student assessment)

Mathematics Access and Teaching in High Schools -

MATHS (ME)

Teachers as Leaders and Learners - TaLL (NJ)

Fidelity in the implementation of PD activitiesMaking Math More

Meaningful - M4 (IN)

Fidelity in the implementation of PD activities

Change in instructional practices

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Change in instructional practices

Change in instructional practices

Fidelity in the implementation of PD activitiesCoalition for Higher

Standards Math Partnership Program -

Lesley/MassInsight (MA)

EduTron (MA)

Fidelity in the implementation of PD activities

Change in instructional practices

Student achievement gains

Gains in teacher knowledge

Change in instructional practices

Student achievement gains

Math/Science Teachers Engaged as Mentors -

MS*TEAMS (KY)

Fidelity in the implementation of PD activities

Student achievement gains

Change in instructional practices

Gains in teacher knowledge

High School Mathematics Alliance Grant (KY)

Created by CCSSO, 2007NSF Grant No. GREC0438359 2 of 4

Table 3

Program Outcomes to Be Measured Instruments Observation of design team meetings (evaluator-created guidelines)Summer institute observation (evaluator-created guidelines)Post-Big Idea Module (BIM) creation surveys of BIM design teamBIM design team journals Pre-post summer institute participant assessments Summer institute participant questionnaires Post-summer institute presentation impact surveys of BIM design team and institute participantsBIM Design artifacts (audio and photographic records and documents)Pre-/post-summer institute assessments Observation of design team meetings (evaluator-created guidelines)Summer institute observation (evaluator-created guidelines)Pre-/post- BIM team meetings assessmentsPost-summer institute presentation impact surveys of BIM design team and institute participants

Change in instructional practices Summer institute observation (evaluator-created guidelines)Fidelity in the implementation of PD activities Pre-/post-treatment teacher assessments of participant teachers

U. of Louisville (DTAMS) teacher assessment-GeometryPenn State U. - Process skill test for scienceUniversity course instructor-created teacher assessmentsEnd-of-course grades of participant teachers in graduate courses End-of-grade student assessmentPenn State U. - Process skill test for scienceMonitoring asynchronous Blackboard (PD facilitator and participant teacher forum and courses' visibility portal)Pre-treatment survey of participant teachersPre-/post-teacher assessments of participant teachers (Face-to-face treatment only)Facilitators InterviewsFocus group of participant teachersParticipant teacher survey (online 2006, based on some of the criterion from IMPDE rubric)Pre-/post-teacher assessments of participant teachers (Face-to-face treatment only)Interviews of treatment facilitators

Change in instructional practices Post-treatment surveys of participant teachersFidelity in the implementation of PD activities Observations (without protocol)

Gains in teacher knowledge Pre-/post-treatment teacher assessments of participant teachers

Pre-/post-program attitude surveyAt-term and post-term analysis of each unit of instruction -NCTM Middle School and High School StandardsPre-/post-institute and end-of-term workshop surveys (Treatment Group only)Interviews

Praxis-Middle School Math Test (national standardized teaching licensure exams)

U. of Louisville (DTAMS) pre-/post-treatment assessment (Treatment Group only)

Surveys of Enacted Curriculum (SEC)Treatment group's journal reflectionsAnalysis of submitted responses by teachers (Treatment Group only) for two problems per term (summer, fall, winter, spring)InterviewsPre-post institute and end-of-term workshop surveys (Treatment Group only)Horizon's Reformed Teacher Observation Protocol (RTOP)Surveys of Enacted Curriculum (SEC)Technology Enhanced Student Assessment (TESA) (statewide standardized assessments (Student Treatment and Control Groups)Oregon State Problem Solving Guide (on student work samples of students of Treatment Group)

BreakThrough Mathematics (OH)

Pursuing Excellence in Middle School Math & Science - Nash-Rocky

Mount (NC)

Consortium for New Explorations in Coherent

Teacher Education - CONNECT-ED (NJ)

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Student achievement gains

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Gains in teacher knowledge

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Comprehensive, Data-Based Professional

Development for Middle School Mathematics

Teachers (OH)

Change in instructional practices

Student achievement gains

High Desert (OR)

Created by CCSSO, 2007NSF Grant No. GREC0438359 3 of 4

Table 3

Program Outcomes to Be Measured Instruments Pre-/post- participant and comparison teacher surveys (evaluator designed)Lesson plans from Participant and Comparison teachersInterview protocol for sample follow-up interviews with participating teachersPre-/post-treatment teacher assessment (project-developed)Lesson plans from Participant and Comparison teachersInterview protocol for sample follow-up interviews with participating teachers Lesson plans from Participant and Comparison teachersClassroom artifacts3 student work samples, one of each performance level (Low, Med, Hi)-Performance) per teacherPre-/post-treatment student assessment in science (Northwest Evaluation Association designed)Three student work samples, one of each performance level (Low, Med, Hi)-Performance) per teacherOregon statewide standardized student assessmentSummer institute observation protocol (PDOP)Teacher surveysObservation of planning meetingsEnd-of-workshop survey (distributed to Teacher Leaders and to attending administrators)Principal surveysProfessional Development Database (documents PD sessions, including PD activities led by Teacher Leaders)University Faculty Focus Group Protocol K-12 Instructional Staff Focus Group Protocol U. of Michigan (CKT-M measures) teacher content knowledge assessmentsProfessional Development Database - Teacher LeadersClassroom observation protocol (adopted from Horizon/LSC)RMC teacher survey Observing Student Discourse Protocol - 2nd Tier Research Design (developed from Horizon/LSC COP) 2nd Tier - Research DesignRMC Teacher Leader Survey (external evaluator-created) RMC Teacher survey (external evaluator-created) RMC Principal survey (external evaluator-created) Oregon State Mathematics Assessment - statewide standardized student assessmentHorizon/LSC COP Observing Student Discourse Protocol High School Mathematics Course Enrollment Data - captures changes in percentage of traditionally underrepresented and underserved high school students enrollment and successfully completion of challenging and advanced math coursesRMC Teacher Surveys (external evaluator-created) RMC Principal Surveys (external evaluator-created) Pre-/post-treatment surveys to select teacher coaches (adopted from Horizon)Teacher coaches' journalsHorizon/LSC survey of teachers Focus groups of participant teacher coachesPACT (statewide standardized assessment) Curriculum calibration/rubrics (based on state standard for math/science)Observation of courses (no protocol)Interviews of the instructors (IHE faculty)Interview of participant teachers

Post-course satisfaction survey of participant teachers

Pre-/post-treatment surveys of participant teachersObservation of treatment (no protocol)Focus group of administratorsSite visits of district & schools of participant teachersU. of Michigan (CKT-M measures) teacher content knowledge assessmentsClassroom observation protocol (COP-CETP) of participant teachersInformal interviews of mentor and mentees teachersPre-/post-treatment surveys of participant teachersClassroom observation protocol (COP-CETP) of participant teachersInterviews of mentor and mentee teachersRecorded responses in electronic bulletin boards of teachers

Change in instructional practices

Student achievement gains

Oregon Mathematics Leadership Institute - OMLI

(OR)

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Change in instructional practices

Wisconsin Academy Staff Development Initiative

Retention and Renewal - R & R (WI)

Fidelity in the implementation of PD activities

Change in instructional practices

Student achievement gains

Mathematics and Science Coaching Initiative - South Carolina Coaching Initiative

(SC)

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Fidelity in the implementation of PD activities

Northwest Wisconsin Partnership for

Mathematics and Science Education - NW Wisconsin

(WI)

Willamette Watershed Project (OR)

Student achievement gains

Fidelity in the implementation of PD activities

Gains in teacher knowledge

Change in instructional practices

Created by CCSSO, 2007NSF Grant No. GREC0438359 4 of 4

Table 4

Program Report date (4) Student Achievement (1) PD Activities (2) Teacher Knowledge (3) Instructional Practices # of Teachers

• Student achievement linked to treatment teachers

• Measured through teach survey on student

• Change from 2004 to 2005• Treatment vs. Control Groups• No student achievement scores reported

• Measured through teachersurveys

• Measured through teacherknowledge test

• Students with advanced courses by district

• Measured through survey of IHE staff

• Analyzed pre-post,gains per course reported

• Student achievement by year for student of new Gr. 6-10 teachers

• Measured through survey of new teachers

• Measured through Praxis, Place

• Reported on Baseline Year - 2005

• Retention rate of new teachers/yr.

• Measured through exams of new teachers

Science Lead Teacher (DE) • Report, no date - None - - None - - None - - None - - None -Science Coalition Courses (DE) • October 2000 - None - - None - - None - - None - - None -

• 161 total elementary & middle school teachers

• Year 1 (N=30), Year 2 (N=55), Year 3 (N=76)

• ISAT planned • Measured through teachermath knowledge inventory

• Direct Math Assessment (K-2) • Analyzed pre-post gains

• Student achievement gains of treatment teachers vs. of control teachers

• 430 Gr. 4 teachers, 60 comparison teachers

• Reported pre-post tests for Gr. 4 & 5, Year 1 & Year 2

• 430 Gr. 5 teachers, 60 comparison

• 2004-05-Year 1 report

• Based on Interviews w/ teachers

• 2005-06-Year 2 report

• Based on Everyday Math professional development & implementation

High School Mathematics Alliance Grant (KY) • No report - None - - None - - None - - None - - None -

• Descriptive report of evaluator • 21 teachers

• Interviews with teachers • Learning teams

Program Evaluation Report Results, as of December 2006

• September 2006

• March 2006 - Yr 2 Report (Jan. Dec. 2005)

• May 2006 - Yr 2 Report

• October 2006 - Year 2 Report

Teacher Quality Enhancement/Secondary Teacher Enhancement Project - TQE/STEP

(CO)

• Reported change in student achievement from 2004 to 2005, and '05 to '06 of students from treatment math teachers, no comparison group ; ISTEP grade 3-8

• Reported on 2005 to 2006 school mean in math and in science vs. district mean

• Measured through survey of teachers on PD activities

• Measured through pre-post teacher knowledge test

Launch II (FL)

Northeast Front Range (CO) • Measured through teachersurvey by course

• Measured through pre-post test by course

• 175 math & science teachers

• Measured through teachersurvey on instructional practices

Rocky Mountain Middle School (CO)

• 60 elementary teachers• Measured through SEC surveys on instructional practices

• 136 treatment teachers/yr• Examined impact through focus group

• Measured through survey of instructional practices

• 364 teacher leaders

• 50 elementary math teachers

- None -Math/Science Teachers Engaged as Mentors - MS*TEAMS (KY)

- None -

- None - • Based on observations of 26 teachers

- None -

Making Math More Meaningful – M4 (IN)

• 2005-06-Year 2 Report

Developing Mathematical Thinking - DMT (ID)

iCATS: Beyond the Textbook (IN) - None -

• 2006 - None -

• 2006-Year 2 Report

- None - - None -

- None -

Created by CCSSO, 2007NSF Grant No. REC0438359 1 of 3

Table 4

Program Report date (4) Student Achievement (1) PD Activities (2) Teacher Knowledge (3) Instructional Practices # of Teachers

• Measured through algebra concepts test -- Gr. 9

• Based on classroom observation, analyzing types of instruction

• Linked to treatment teachers • Level of cognitive activity

• Analyzing student achievement change from 2005 to 2007

• Reported through student focus groups

• Measured through MA teacher licensing test (MTEL)• Pre-post teacher knowledge test

• Reported on student achievement for 2004, baseline year

• Measured through teacherknowledge test

• Based on intervews of coaches, teachers

• Math teachers vs. control teachers, by school

• Treatment teachers vs. control teachers

• Based on Classroom observations

• Student achievement linked to teachers• Analyzing change from 2004to 2005, and 2005 to 06 • Treatment vs. control groups • Grades

Consortium for New Explorations in Coherent Teacher Education - CONNECT-

ED (NJ)

• October 2004, pilot project - None - • Based on interviews,

observations - None - - None - • 40 treatment teachers in10 design teams

• Observations of workshops• Based on teacher focus groups• Based on open-response teacher survey

BreakThrough Mathematics (OH) • June 2006 - None -

• Measured through teacherfollow-up survey on perceptions, satisfaction on program

- None -

• Measured through survey on applying newly acquired knowledge/skills to practices

• 194 teachers taking online math course

Comprehensive, Data-Based Professional Development for Middle School

Mathematics Teachers (OH)

• Fall 2006-Year 2 report

• Percent proficient by school in math and/or science, baseline from 2004-05 results

• Surveys for each session teacher satisfaction - None - • Classroom observ and

interviews by coaches - None -

• Measured treatment teacher knowledge through DTAMS

• Measured through SEC, RTOP

• Compared Praxis results between treatment vs. control teachers

• Needs assessment

• 24 elementary math teachersHigh Desert (OR) • August 2006

• Reported on student achievement change and comparing treatment vs. control groups, based on results in 2004 to 2005

• 35 teachers and coaches all levels

• Measured through teachersurvey on practices and attitudes

• Based on grades in math courses

• 20 middle school math teachers treatment group

• Measures on teacher survey

- None -

- None -

• 44 elementary & middle school math teachers- None -

- None -

Coalition for Higher Standards Math Partnership Program - Lesley/MassInsight

(MA)• 2006

Pursuing Excellence in Middle School Math & Science – Nash-Rocky Mount (NC)

EduTron (MA)

Mathematics Access and Teaching in High Schools - MATHS (ME)

Teachers as Leaders and Learners - TaLL (NJ)

• 100 middle school & high school math teachers

• School level data 05 math and science

• Analysis of teacher assignments for increased math understanding using 5 levels

• Student achievement district level change from 2003 to 2006

- None -

• 100 elementary & middle school math teachers as coaches in 4 school districts

- None -

• Teacher participation rate

• May 2006 final report (2004-05)

• August 2006 (2004-06 implementation period)

• October 2006

• 2006

Created by CCSSO, 2007NSF Grant No. REC0438359 2 of 3

Table 4

Program Report date (4) Student Achievement (1) PD Activities (2) Teacher Knowledge (3) Instructional Practices # of Teachers

• Reported on student achievement in science based on test specific to the project.• Compared treatment & control group, using pre-post testing

• Measured through teachersurvey

• Measured through instructional practice surveyon sample of teachers

• Measured through teach content knowledge test, reported pre-post results

• Based on classroom observation

• 33 science and math teacher coaches• 600+ teachers

• Measured through teachersurvey

• Measured through interviews

• Evaluation report on PD

• 40 teachers/year

• Teacher mentors

23 total 12 with student achievement data

17 since January 2006

6 reports with treatment vs. control comparison

- None - - None -

13 on PD activitiesNumber of Evaluation Reports 12 on teacher knowledge

13 on instructional practices

• 30 middle school math and science teachers

• No student achievement data reported

• Qualitative report on PD activities - None -

• Plan measures of instructional practices in 2006 (Cohort 2) & 2007 (Cohort 3)

• Measured through teachersurvey (Horizon, dev.) to track instructional change

• 2005 • No student achievement data reported

• Measured through teacherself-rating of knowledge gains

- None -

• 2007 report on Fall 2003-Spring 2006

Wisconsin Academy Staff Development Initiative Retention and Renewal – R & R

(WI)

Mathematics and Science Coaching Initiative – South Carolina Coaching

Initiative (SC)

Northwest Wisconsin Partnership for Mathematics and Science Education – NW

Wisconsin (WI)

• No student achievement data reported

Willamette Watershed Project (OR)

Oregon Mathematics Leadership Institute - OMLI (OR)

• 15 treatment teachers, 15 control teachers• Analyzed lesson plans

• Measured through teacherknowledge test linked to student achievement test

- None -

• 264 School leadership Team members (2 teachers & 1 administrator per team)

- None -• No student achievement data reported• May 2006

• 2004-05-Year 2 Exec. Summary

• October 2006

Created by CCSSO, 2007NSF Grant No. REC0438359 3 of 3

Evaluation Study Reports Reviewed BreakThrough Math McKnight, K. (2006, June). Evaluation summary report for teacher follow-up survey. Ohio Department

of Education BreakThrough Math project. Arlington, VA: LessonLab Research Institute, Pearson Achievement Solutions.

Comprehensive, Data-Based PD Rusch, T. L., Farrell, A. M., & Reed, M. K. (2006, May 11). Comprehensive, data-based professional

development. Final report, narrative & data. Report to the Ohio Department of Education Middle Grade Mathematics Professional Development. Dayton, OH: Wright State University.

CONNECT-ED Lord, B. (2004, October). CONNECT-ED pilot project. Final evaluation report. Report to Rider

University SELECT, Bristol-Myers Squibb Center for Science Teaching and Learning. Newton , MA: Center for Leadership and Learning Communities, Education Development Center.

DMT Brendefur, J., Espinoza, I., & Pfiester, J. (2006). Developing mathematical thinking Project Micron

Foundation grant and mathematics and science partnership grant. Technical report – year 2, July 1, 2004-June 30, 2006. Boise, ID: Boise State University.

EduTron Chen, A. (2006). Intensive immersion institute in mathematics for grades 4-8 teachers. Report to the U.S.

Department of Education, Mathematics and Science Partnerships. Winchester, MA: EduTron Corporation.

Chen, A. (2006, Fall). Annual report addendum. Report to Massachusetts Department of Education

Massachusetts Math and Science Partnerships Program. Winchester, MA: EduTron Corporation. High Desert Niess, M. L. (2005). Oregon ESEA Title IIB MSP: Central Oregon consortium. Report to the U.S.

Department of Education, Mathematics and Science Partnerships. Corvallis, OR: Department of Science & Mathematics Education, Oregon State University.

iCATS: Beyond the Textbook Evansville-Vanderburgh School Corporation Integrating Curriculum and Technology Specialists. (2004,

May). Ed tech grant posttest data. (Year 1). Unpublished manuscript. Evansville, IN: Author. Evansville-Vanderburgh School Corporation Integrating Curriculum and Technology Specialists. (n.d.).

EdTech grant posttest data – Year 2. Unpublished manuscript. Evansville, IN: Author. Evansville-Vanderburgh School Corporation Integrating Curriculum and Technology Specialists. (n.d.).

EVSC EdTech grant – beyond the textbook [Video]. Retrieved March 12, 2007, from http://www.evscicats.com/edtech/

LAUNCH II Howard, M. (2006, September). Project LAUNCH II external evaluation report. Unpublished manuscript. Schmidt, D. L. (2006). Project LAUNCH II. Report to the U.S. Department of Education, Mathematics

and Science Partnerships. Fort Myers, FL: Whitaker Center at Florida Gulf Coast University.

Lesley/MassInsight Coalition for higher standards math partnership program. (2006). Report to the U.S. Department of

Education, Mathematics and Science Partnerships. Cambridge, MA: Lesley University. Meyers, B. (2006, January 19). Advanced math coaching in urban school districts. Report to the

Massachusetts Board of Higher Education, Improving Teacher Quality Higher Education Partnership Grant Program. Unpublished manuscript.

M4 Colonis, M., & Rodriguez, E. (n.d.). M4 – making math more meaningful mathematics and science

partnership grant k-9 mathematics initiatives. Evaluation report I, 2005-06 academic year. Report to the Superintendent, Crawford Community School Corporation and Grant Coordinator, Crawford Community School Corporation. Unpublished manuscript.

MATHS Rosenblum, J. (2006, August). Mathematics: Access and teaching in high schools – MATHS, 2004-06

evaluation report. Augusta, ME: Maine Mathematics and Science Alliance. Rosenblum, J. (2005, September). Mathematics: Access and teaching in high schools – MATHS, 2004-05

evaluation report. Augusta, ME: Maine Mathematics and Science Alliance. MS*TEAMS Fiene, J. (2006). Tiered mentoring project year 2 report. Report to the Green River Regional Educational

Cooperative. Unpublished manuscript. Nash-Rocky Mount Hankerson, L. (2006, October 4). Pursuing excellence in middle school math & science. Project

Narrative. Report to the U.S. Department of Education, Mathematics and Science Partnerships. Nashville, NC: Nash-Rocky Mount Schools.

Perry, M. & White, C. T. (2006, October 4). Pursuing excellence in middle school math & science Project

Profile. Report to the U.S. Department of Education, Mathematics and Science Partnerships. Nashville, NC: Nash-Rocky Mount Schools.

Perry, M. (2005, July 28). Pursuing excellence in middle school math & science Project Narrative. Report

to the U.S. Department of Education, Mathematics and Science Partnerships. Nashville, NC: Nash-Rocky Mount Schools.

White, C. T. (2007, January 2). MSP impact and effectiveness of the mathematics and science coaching

program. Unpublished manuscript. White, C. T. (2006, October 30). Percent proficient on Spring 2006 EOG reading and math tests &

alternate assessments. (Data table). Unpublished manuscript. Northeast Front Range META Associates. (2006, March). Northeast Front Range math/science partnership (MSP) to increase

teacher competence in content. Year 2 evaluation report: January 1, 2005-December 31, 2005. Golden, CO: Author.

NW Wisconsin Wilsman, M. J. (2005, December). Summary evaluation report for the northwest Wisconsin mathematics

and science partnership. Unpublished report, University of Wisconsin-Madison.

OMLI Weaver, D. (2006, May). Oregon mathematics leadership institute. Spring 2006 evaluation report. Report

to the Oregon State University, Department of Mathematics. Portland, OR: RMC Research Corporation.

R & R Greenseid, L., & Lawrenz, F. (2005, June 9). WASDI retention and renewal program evaluation: Second

year executive summary. Unpublished report, University of Minnesota, Minneapolis. Rocky Mountain Middle School Heath, B. P. (2006, May 18). Rocky Mountain middle school mathematics and science partnership.

Annual evaluation report: Year 2 final. Wilmington, NC: East Main Educational Consulting, LLC.

Heath, B. P. (2006, March). Rocky Mountain middle school mathematics and science partnership.

Formative evaluation report for 2005 summer academy and structured follow-up (draft). Wilmington, NC: East Main Educational Consulting, LLC.

Science Coalition Courses and Science Lead Teacher Noble, A. J., Boudreau, C., & Smith, A. (2000, October). Local systemic change initiative evaluation

report. Publication No. T00.014.1. Newark, DE: Delaware Education Research & Development Center, University of Delaware.

Wood, R. E., & Collette, J. W. (n.d.). The Delaware LSCI: A systemic partnership to improve the

teaching and learning of science for all children. Final report to the National Science Foundation. Unpublished manuscript.

South Carolina Coaching Initiative Michel, Y. (2007). Report of teacher surveys in coached schools 2003. Report to the South Carolina State

Department of Education Math and Science Unit. Unpublished manuscript. TaLL Grip, R. S. (2006, October). Year-end evaluation of the teacher as leaders and learners program. Mount

Laurel, NJ: Statistical Forecasting, LLC. Grip, R. S. (2005, October). Year-end evaluation of the teacher as leaders and learners program. Mount

Laurel, NJ: Statistical Forecasting, LLC. TQE/STEP JVA Consulting, LLC. (2006, October). Teacher quality enhancement grant. Annual evaluation progress

report: Year 2. Denver, CO: Author. JVA Consulting, LLC. (2005, December). Teacher quality enhancement grant. Annual evaluation

progress report: Year 1. Denver, CO: Author. Willamette Watershed Hansen, J. B. (2006, October 17). Evaluation of the Willamette Valley Watershed Partnership project –

2005-06. Year two external evaluation report. Monmouth, OR: The Teacher Research Institute’s Evaluation Research Group Office, Western Oregon University.

Hansen, J. B. (2005, September). Evaluation of the Willamette Valley Watershed Partnership project – 2004-05. Year one external evaluation report. Monmouth, OR: The Teacher Research Institute’s Evaluation Research Group Office, Western Oregon University.

Willamette Educational Service District. (2006, October 17). Willamette Valley Watershed Partnership

project. 2005-06 final report, part 2: Narrative responses. Unpublished manuscript.

References Banilower, E. R., Boyd, S. E., Pasley, J. D., & Weiss, I. R. (2006, February). Lessons from a decade of

mathematics and science reform: A capstone report for the Local Systemic Change through Teacher Enhancement Initiative. (Research Monograph-Prepublication copy). Chapel Hill, NC: Horizon Research, Inc. Retrieved March 21, 2006, from http://www.pdmathsci.net/reports/capstone.pdf

Birman, B. F. & Porter, A. C. (2002). Evaluating the effectiveness of education funding streams. Peabody

Journal of Education, 77(4), 59–85. Cohen, D. K., & Ball, D. L. (1999). Instruction, capacity, and improvement (CPRE Research Report No.

RR-043). Philadelphia: University of Pennsylvania, Consortium for Policy Research in Education.

Cohen, D. K., & Hill, H.C. (2001). Learning policy: When state education reform works. New Haven,

CT: Yale University Press. Corcoran, T., & Foley, E. (2003). The promise and challenge of evaluating systemic reform in an urban

district. Research Perspectives on School Reform: Lessons form the Annenberg Challenge. Providence, RI: Annenberg Institute at Brown University.

Council of Chief State School Officers (2006). Rubric for professional development program review.

Retrieved March 05, 2007, from http://www.ccsso.org/content/PDFs/Revised%20Rubric%20%2D%20rev022406.xls

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional

development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112.

Educational Testing Service. (n.d.). Pre-professional skills test: Mathematics (0730). Sample test items

and answers. Retrieved March 6, 2007, from http://www.ets.org/Media/Tests/PRAXIS/pdf/0730.pdf

Frechtling, J. (2001). What evaluation tells us about professional development programs in math and

science. In C. R. Nesbit, J. Wallace, D. K. Pugalee, A. Miller, & DiBiase, W. J. (Eds.), Developing teacher leaders: Professional development in science and mathematics. Columbus, OH: Education Resources Information Center.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective: Results from a national sample of teachers? American Educational Research Journal, 38(4), 915-945.

Garet, M. S., Birman, B. F., Porter, A. C., Desimone, L., & Herman, R. (with Yoon, K. S.). (1999).

Designing effective professional development: Lessons from the Eisenhower program and technical appendices (Report No. ED/OUS99-3). Washington, DC: American Institutes for Research. (ERIC Document Reproduction Service No. ED442634)

Grant, S. G.. Peterson, P. L., & Shojgreen-Downer, A. (1996, Summer). Learning to teach mathematics

in the context of systemic reform. American Educational Research Journal, 33(2), 509-41. Hiebert, J. (1999, January). Relationships between research and the NCTM standards. Journal for

Research in Mathematics Education, 30(1), 3-19. Horizon Research, Inc. (2000, November 30). Inside the classroom observation and analytic protocol.

(COP). Chapel Hill, NC: Author. Retrieved March 16, 2007, from http://www.horizon-research.com/instruments/clas/cop.pdf

Kennedy, M. M. (1998, April). Form and substance in inservice teacher education. Paper presented at the

annual meeting of the American Educational Research Association, San Diego, CA. Learning Mathematics for Teaching (LMT) Project. (2005). Content knowledge for teaching mathematics

measures (CKT-M measures). Mathematics released items. [Test items]. Ann Arbor, MI: Consortium for Policy Research in Education (CPRE). Retrieved March 6, 2007, from http://sitemaker.umich.edu/lmt/files/LMT_sample_items.pdf

Lieberman, A. (1996, November). Creating intentional learning communities. Educational Leadership,

54(3), 51-55. Loucks-Horsley, S., Hewson, P., Love, N., & Stiles, K.E. (1998). Designing professional development for

teachers of science and mathematics. Thousand Oaks, CA: Corwin Press. Piburn, M., Sawada, D., Falconer, K., Turley, J. Benford, R., Bloom, I. (2000). Reformed teaching

observation protocol (RTOP). ACEPT IN-003. Phoenix, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers. Retrieved, March 16, 2007, from http://physicsed.buffalostate.edu/AZTEC/RTOP/RTOP_full/

University of Louisville Center for Research in Mathematics and Science Teacher Development. (n.d.).

Diagnostic teacher assessments in mathematics and science: Middle school mathematics. Geometry and measurement assessment (version 6.3) [Test items]. Louisville, KY: Author.

Weiss, I. R., Banilower, E. R., McMahon, K. C., & Smith, P. S. (2001). Report of the 2000 national

survey of science and mathematics education. Chapel Hill, NC: Horizon Research, Inc.

Errata On page 7 of Table 1 – The number of educators directly served by the OMLI project is 280 for 3 years. Participation in the OMLI School Leadership Team is a three-year commitment. On page 25 – The OMLI Student Discourse Observation Protocol was developed by the RMC Research Corporation with advice and input from the OMLI project leadership. The Student Discourse Observation Protocol drew from previous works including those from Horizon Research’s work with the Local Systemic Initiative.