the effects of math pathways and pitfalls on students’ … · 2013. 8. 2. · [mpp nsf final...

The Effects of Math Pathways and Pitfalls on Students’ Mathematics Achievement

National Science Foundation Final Report

August 29, 2007

Joan I. Heller Heller Research Associates

Deborah A. Curtis

San Francisco State University

Sophia Rabe-Hesketh University of California, Berkeley

Carol J. Verboncoeur

Heller Research Associates

[MPP NSF Final Report 070829.doc] Printed 8/29/07 i


Executive Summary

This study addressed two sets of questions. First, the evaluation was designed to measure the impact

of Mathematics Pathways and Pitfalls (MPP) on the mathematics that second-, fourth-, and sixth-grade students learn. The specific research questions that were addressed are: (a) What is the impact of MPP on students’ knowledge of the mathematics topics addressed, compared to that of students using the regular math curriculum? and (b) How equitable is the impact of MPP on students’ mathematics knowledge across levels of English language proficiency and entering mathematics ability? To contribute to the interpretation of the results, the research also examined the fidelity of lesson implementation as enacted within MPP classrooms, compared to the structure and processes that were intended by the curriculum designers. Questions that were addressed are: (a) How closely does MPP as enacted follow the structure, content, and discourse processes that were intended by the curriculum designers? and (b) How does MPP as enacted in classrooms that had greater student math score gains compare with MPP in classrooms with lower student gains? Study Design

A cluster-randomized experimental design was implemented in five school districts across the nation

over a two-year period. In the first year of the study, second-, fourth-, and sixth-grade teachers were randomly assigned within their school districts in roughly equal numbers to either an experimental group or a control group. The experimental group teachers were taught how to implement MPP during a six-hour professional development (PD) session in the summer. Project consultants, trained by project staff, conducted the PD. In the first year of the study, experimental group teachers substituted MPP for a portion of their regular mathematics curriculum. The control group teachers used their regular mathematics curriculum, and received whatever professional development they normally were provided during that year. A total of 99 teachers and 1,971 students participated in the first year of the study.

In the second year of the study, control group teachers received four days of PD and then substituted

MPP for a portion of their regular mathematics curriculum. The experimental group teachers from the first year of the study had the option of teaching MPP again in the second year. These teachers were not obligated to do so, however, and because many elected not to participate in the second year of the study, results from the first year only are presented in this report. Methods

Teachers completed a background questionnaire, an implementation questionnaire, and an end-of-year evaluation questionnaire. The MPP Pitfalls Quiz, which was given at the beginning and end of the school year, was the primary instrument used to measure students’ mathematical knowledge. A separate MPP Pitfalls Quiz was developed for each grade level, each of which was aligned with MPP for that grade level. Items on the MPP Pitfalls Quizzes assess concepts and procedures that are known to cause difficulty for students as identified from the research literature and prominent assessments such as the NAEP and TIMSS. The tests contain one or more items that relate directly to the content of each lesson, and a few additional items that were indirectly related to the lesson content. Most items on these tests were multiple-choice format, with one correct answer. At least one of the choices for items in the multiple-choice format contained a common misconception that students have with regard to the concept being assessed. In addition to the MPP Pitfalls Quizzes, standardized mathematics achievement test score data were collected.

ii

Results

Because of the hierarchical structure of the data, with students “nested” in classrooms, multilevel models, also known as hierarchical linear models, were used to analyze the MPP and standardized achievement test data. These analyses showed that student math performance in MPP classes was higher than in non-MPP classes for all three grade levels. With respect to the impact of MPP on students’ mathematics knowledge across levels of English language proficiency, we found that for second and fourth grades, MPP impacted ELL and non-ELL students equally. The effect size statistics (ESS) for second and fourth grade were .43 and .66, respectively. For sixth grade, MPP had a greater treatment effect for ELL students (ESS = .74) than non-ELL students (ESS = .28). There were no differences between the effectiveness of MPP for students with mathematically stronger versus weaker entering knowledge except at fourth grade, where MPP was more effective for children who had higher pretest scores. On standardized mathematics tests, for the fourth and sixth-grade students, no statistically significant difference was found between the means of control and experimental group students. (No analysis was performed for the second-grade students because standardized-test-score data were not available.)

The positive impact of MPP on student mathematics performance, as measured by the MPP Pitfalls Quizzes, is consistent with results of an earlier pilot study of MPP materials by Heller, Gordon, Paulukonis, and Kaskowitz (2000). Because the current study was based on a more rigorous research design (i.e., a cluster randomized design) than the one used in the Heller et al. pilot study, the results of the current study can be viewed as even stronger evidence of the effectiveness of the MPP materials.

With respect to fidelity of lesson implementation, analysis of classroom audio recordings and teacher questionnaires revealed that (a) almost all teachers implemented every major component and intended discourse process of the lessons; (b) teachers made some minor modifications to the lesson structures—namely some steps or prompts were left out more than others, particularly in lower-scoring classes; (c) some of the tools for building extended student talk about math, such as the Discussion Builders, are spontaneously used by teachers and students, even for lesson segments that are not guided by specific prompts in the teaching guide, and during class time on subjects other than math; (d) in classes with higher-scoring students, there was more use of Discussion Builders by both teachers and students, were asked to explain their thinking less frequently than in lower-scoring classes but more often talked about the math among themselves, and gave longer responses about the math.

Teachers expressed strongly positive opinions about the value of the program, including that their students understood the math topics in the lessons better than students in past years, that MPP helped most of their students learn the math concepts and prevent pitfalls, and their students really liked MPP. Overall, the teachers strongly agreed that they would love to use MPP again next year, and students in their schools would benefit greatly if all of the teachers used Math Pathways and Pitfalls.

[MPP NSF Final Report 070829.doc] Printed 8/29/07 iii

Table of Contents Math Pathways and Pitfalls................................................................................................................................ 1 Review of the Research on Academic Language, Discourse, and Equity....................................................... 3 Pilot Study on the Impact of MPP Instructional Materials .............................................................................. 5 Research Questions ............................................................................................................................................. 5 Methods................................................................................................................................................................ 6 Teacher Participation in Professional Development ......................................................................................... 9 Data Analysis..................................................................................................................................................... 11 Results................................................................................................................................................................ 13 Analysis of the MPP Pitfalls Quizzes .............................................................................................................. 19 Analysis of the Standardized Mathematics Achievement Tests .................................................................... 24 Implementation Study Results.......................................................................................................................... 28 Conclusions ....................................................................................................................................................... 39

iv

List of Tables Table 1. Theoretical Foundation for Design Features and Expected Student Benefits

of Math Pathways and Pitfalls Model.................................................................................... 2 Table 2. Synopses of Selected Lessons................................................................................................. 3 Table 3. Experimental Design of Two-Year Study With Pre-Post Teacher and Student

Assessments ............................................................................................................................. 6 Table 4. Schedule of Data Collection.................................................................................................... 8 Table 5. Number of Teachers per Group by District and Grade ....................................................... 14 Table 6. Number of Students per Group by District and Grade ........................................................ 14 Table 7. Number of Teachers Who Identified their Schools as Urban, Suburban, or

Rural, by Treatment Group ................................................................................................... 15 Table 8. Teacher Gender and Ethnicity............................................................................................... 16 Table 9. Teacher Mathematics Education and Training .................................................................... 16 Table 10. Teacher Mathematics Teaching Experience ........................................................................ 17 Table 11. Student ELL Status by Treatment Group ............................................................................. 18 Table 12. Student Gender by Treatment Group.................................................................................... 18 Table 13. Student Ethnicity by Treatment Group................................................................................. 19 Table 14. MPP Pitfalls Test Reliability Analysis (Year 1 and Year 2 Data Combined) ................... 20 Table 15. MPP Test Scores by Treatment Group and Grade .............................................................. 20 Table 16. MPP Test Scores by Treatment Group and ELL Status and Grade ................................... 21 Table 17. Multilevel Model Results Using MPP Pitfall Tests as the Outcome Variable .................. 22 Table 18. Multilevel Analysis: Adjusted Mean Posttest Scores of the Control Group

versus Experimental Group by ELL Status.......................................................................... 23 Table 19. Name of Standardized Test and Metric Used by District.................................................... 25 Table 20. Number of Students with Valid Standardized Mathematics Achievement

Data by Grade and District.................................................................................................... 26 Table 21. Descriptive Statistics for Standardized Tests by Treatment Group and Grade ................. 27 Table 22. Percent of Lesson Components Completed in Higher- and Lower-Scoring

Classrooms ............................................................................................................................. 29 Table 23. Percent of Lower- and Higher-Scoring Classrooms in which Components of

MPP Lesson 6 Were Present in Audio Recordings ............................................................. 29 Table 24. Length of MPP Lessons in Lower- and Higher-Scoring Classrooms ................................ 31 Table 25. Mean Percent and Frequency of Lower- and Higher-Scoring Classroomsa in

which Components of MPP Lesson Were Present in Audio Recordings .......................... 32 Table 26. Percent of Teachers Reporting How Students Use of MPP Discussion

Builders in Lower- and Higher-Scoring Classrooms .......................................................... 33 Table 27. Teachers’ Use of MPP in Lower- and Higher-Scoring Classrooms .................................. 35 Table 28. Teachers’ Self-Reported Use of MPP in 2003-04 End-of-Year Questionnaire ................ 36 Table 29. Teachers’ Ratings of MPP in End-of-Year Questionnaire.................................................. 37

[MPP NSF Final Report 070829.doc] 1 of 43 Printed 8/29/07


This study was designed to evaluate how the Mathematics Pathways and Pitfalls (MPP) lessons

impact the mathematics that students learn, and the equity of learning across groups of students at different levels of language proficiency. The MPP instructional materials take the unique approach of not only fostering correct ways to represent and reason about mathematical concepts, but also explicitly calling students' attention to common pitfalls and misconceptions. MPP also provides lesson-specific assistance in both the student materials and the teacher's guides for learning how to use mathematical vocabulary and symbols, present complete and coherent explanations orally and in writing, and participate in mathematical discourse. Students learn to present, expand, justify, and prove or disprove mathematical ideas in paired, small-group, and whole-class settings. An important goal of the lessons is to help students become careful critics of their own thinking and take a proactive stance toward their own learning.

Math Pathways and Pitfalls

Math Pathways and Pitfalls (MPP) for K–7 students was developed and field-tested with grants from the NSF (ESI 9911374) and Stuart Foundation. The program has broad appeal, especially in the existing climate of accountability, since it addresses some of the toughest math concepts and associated learning pitfalls culled from the research literature and from national and international assessments. The mathematical topics for grades K–3 focus on developing whole number concepts and operations, while the topics in grades 4–7 focus on developing rational number concepts. These supplementary materials address the need for improving instruction, regardless of the core instructional materials being used.

MPP consists of video and print materials which include: (a) eight units, one each for grades K–7, in

English and Spanish, each with 10 to 12 core lessons and follow-up mini lessons for students; (b) teaching guides for each lesson as well as each mini lesson; (c) four videos—two professional development videos for teachers and two for students—that model how to present and discuss mathematical ideas); and (d) Pitfalls Quizzes for each grade to assess math learning.

Each MPP lesson uses a consistent, easy-to-follow format and includes sections that (a) introduce key

words and symbols; (b) promote discussion about two excerpts of student dialogue: one that contains a correct example of student thinking and another that contains a pitfall in thinking; (c) provide teacher-guided and individual practice; and (d) reinforce each concept through follow-up mini lessons, one requiring responses to multiple-choice questions and the other eliciting written explanations of a mathematical idea.

Theoretical Framework for Math Pathways and Pitfalls

An extensive review of the research literature identified fundamental representations and approaches to developing mathematical concepts, as well as common misconceptions and conceptual “snags”, which we call “pitfalls”. Specifically, the primary-grade lessons on number and operation concepts draw primarily on that of Carpenter and Moser (1983), Fuson (1992), Griffin (1998), and Sowder (1992). The lessons in the upper elementary grades draw on the rational number research of Behr, Lesh, Post, and Silver (1983), Carraher (1996), Moss and Case (1999), Parker and Leinhardt (1995), and Wearne and Hiebert (1989).

The framework in Table 1 describes the theoretical foundation underlying the development of MPP.

The columns identify the critical features of the materials, the theory underlying the feature, and the

2

expected student benefits. In the left-hand margin are the key foci—mathematics, and academic language, discourse, and equity—that drove the development of the materials.

Table 2 provides a synopsis of selected lessons to give examples of the mathematical concepts and

pitfalls targeted by MPP.

Table 1 Theoretical Foundation for Design Features and Expected Student Benefits of Math Pathways and Pitfalls Model

Critical Feature of MPP Model Theoretical Basis Expected Student Benefits

Mat

hem

atic

s

• Explicit strategies motivate students to become careful critics of their own thinking, justify ideas logically, and question the validity of ideas.

• Pitfalls related to important mathematical concepts are used as a springboard for inquiry and learning.

• Lessons stimulate creative solutions to non-routine problems and use of a variety of representations.

• Lessons build on prior math concepts and connect to related concepts within each unit and from grade to grade.

• Successful students develop intentional learning strategies for knowledge-related goals. Unsuccessful students focus on surface features (Scardamalia & Bereiter, 1983).

• Cognitive dissonance stimulates spontaneous inquiry and meaning construction (Festinger 1957; Borasi, 1994).

• Inability to solve problems with misleading features is symptomatic of fundamental misunderstandings (Moss & Case, 1999).

• A spiral curriculum links new and prior learning to achieve knowledge breadth and depth and facilitate extrapolation (Bruner, 1960, 1966).

• Students become increasingly independent mathematics learners, elevate the quality of their work, and monitor their own thinking for pitfalls.

• Students acquire “habits of mind” that incorporate inquiry and critical thought.

• Students gain complex understandings that adapt to different contexts and are resilient to misleading cues.

• Learning is cumulative, generative, and strengthened from grade to grade.

Lang

uage

, Dis

cour

se, a

nd E

quity

• Lessons model inventive student ideas and logical reasoning. Discussion Builders model ways to build on or disagree with an idea respectfully.

• Students prove or disprove the validity of mathematical statements.

• Lessons introduce math vocabulary and symbols and point out language pitfalls.

• Teaching guides suggest ways to make mathematical discourse accessible to students and achieve broad participation.

• Cognitive apprenticeship and scaffolding support the new cognitive behaviors and patterns of discourse (Brown, Collins, Duguid, 1989; Gibbons, 2002).

• Knowledge is socially constructed, with discourse playing a major role in developing meaning (Vygotsky, 1962; Cobb, Wood, & Yackel, 1993).

• Attending to language and status issues enhances discourse participation (Cohen, 1982; Khisty, 1995; Secada, 1992).

• Students build their capacity to think inventively and reason logically.

• Students are open to presenting mathematical ideas and examining their validity with their peers.

• Students are prepared for the discourse expected in demanding curricula and advanced mathematics.

• Students, regardless of their language background or social status, increase their contributions to mathematical discourse.


Table 2 Synopses of Selected Lessons

Naming Equal Fractions Infinite Names for Equivalent Fraction Amounts

This lesson is an opportunity to help students realize that a region can be divided into an infinite number of equal parts and that an infinite number of fractions can name the shaded amount. To find other fraction names, Teresa (a fictional student in the lesson) models the idea that you can divide the same region into more or fewer equal parts, as long as the ratio of the shaded amount to the whole amount remains the same. Visualizing more or fewer subdivisions when naming fraction amounts using any model is a valuable strategy that helps children think about equivalent fractions.

Pitfall: Students count number of parts shaded for the numerator and the number of parts not shaded for the denominator (instead of comparing a part to the whole).

Decimals Are Fractions, Too Using Reasoning to Convert Fractions to Decimals

In this lesson, a fictional student, Tom, models a way for students to use reasoning to find an equivalent decimal in tenths for 1/5. First he thought about the whole, or 1, as 10 tenths. He then thought about how many tenths would match the area of each of the 5 fifths the rectangle was divided into. He discovered that 1/5 is equal to 2 tenths and wrote 0.2. If he renamed the whole as 100 hundredths and it was divided into 5 parts (or fifths), then each part would be 20 hundredths or 0.20. Likewise, if the whole was 1,000 thousandths, 1/5 would be 200 thousandths or 0.200.

Pitfall: Students use the digits in the fraction to make 1/5 into .15 instead of the correct equivalent of 0.2.

Percent Names for Shaded Areas Using Reasoning to Name Percents

In this lesson, Brendon models how to name the percent for a shaded area that is not divided into 100 parts. First he thought about the whole area as 100%. Then, he used reasoning to think about the percent for each of 10 equal parts given that the whole is 100%. So, if 100% were split equally into 10 parts, each part would have a value of 10%. So 4 of 10 equal parts would represent 40%. Brendon’s method of first finding the amount for one equal part can be used to change any fraction amount into a percentage. A related fundamental understanding is that percentage divides a whole amount into hundredths. This means that 1 hundredth represents 1%, 10 hundredths represents 10%, 125 hundredths represents 125%, and so on.

Pitfall: Students think that 4 shaded parts (of 10 equal parts) are equal to 4%, even though each part is 10%, not 1% of the whole.

Review of the Research on Academic Language, Discourse, and Equity

Language is central to all learning since it plays an important role in the way concepts are formed, held in memory, and used in reasoning (Pimm, 1987; Vygotsky, 1978). Yet there is considerable evidence that most mathematical instruction in the United States is characterized by little verbalization. For example, the TIMSS Video Study (Stigler, Gonzales, Kawanaka, Knoll, & Serrano, 1999) revealed that in the United States, 78% of the mathematical concepts in lessons were simply stated by the teacher rather than developed through explanations or discussion of examples. In a study of bilingual classrooms, Khisty (1992) documented that when teachers introduce mathematics concepts they often teach by giving

4

a few typical examples with little or no discussion of the mathematical ideas behind the examples. The few verbal explanations that are provided are often ambiguous, incorrect, or inappropriate. Teachers often use vocabulary or symbols assuming that they are meaningful to students, even when words or symbols have multiple mathematical meanings or have very different meanings from common speech. This language-impoverished approach to the instruction of mathematical concepts is not working well for most students, but the negative impact is particularly acute for non-native English-speaking students who need to learn to navigate the specialized language of mathematics and do so in their second language.

Given the prevailing mode of recitation-style instruction, it is not surprising that conceptual

understanding in mathematics, regardless of students' language background, is much weaker than procedural fluency (National Research Council, 2001). Even on relatively straightforward mathematics problems, students' understanding easily caves into pitfalls. For example, on the National Assessment for Educational Progress (NAEP) only 35% of the U.S. 13-year-olds chose the correct response when asked for a number between .03 and .04. Similarly, only about 25% of the U.S. sixth graders correctly responded 60% when asked to complete the number sentence: .6 = ____%. The most likely incorrect response for this problem is 6%, which is a prevalent pitfall identified in research (Moss & Case, 1999; Parker & Leinhardt, 1995). The research literature and national and international assessments provide many similar examples, especially in the realm of rational numbers (Behr, Lesh, Post, and Silver, 1983; Carraher, 1996; Moss & Case, 1999; Parker & Leinhardt, 1995; and Wearne & Hiebert, 1989). What is disconcerting is that students don't just make a mistake; their lack of conceptual understanding prohibits them from realizing that their incorrect responses do not make sense, even when it is pointed out. This feeble conceptual base leaves students unprepared to tackle higher mathematics.

The research literature in language acquisition and ELL instruction points to specific ways to tailor

instruction so that understanding of content in English is enhanced. These strategies include (a) providing explicit discussion and preview of vocabulary and lesson structure, (b) building on students’ previous knowledge, (c) using discourse markers (i.e., "next,” "after"), (d) using visual aids, and (e) helping students develop the ability to regulate their own thinking. (Chamot & O’Malley, 1994; Echevarria, 1998; Echevarria & Graves, 1998; Gersten, 1996; Short & Echevarria, 1999; Wong-Fillmore, 1982). Typical mathematics instruction does not effectively utilize these practices, or if it does, the ideas are often misapplied. When teachers try to lower the linguistic complexity of a task, they often also lower the cognitive demand of the instruction. This results in watered-down mathematics instruction, which can only widen the already large achievement gap. This is just one example of how academic language, discourse, and equity are interrelated.

Academic language has been defined in the literature in terms of vocabulary, syntax, discourse, and

language functions as they cut across different contexts of use (Butler & Bailey, 2002; Chamot & O'Malley, 1994; Cummins, 1980; Solomon & Rhodes, 1995). Discourse plays a central role in developing academic language and in promoting equitable learning. For example, Khisty (1995), in a study of mathematical language and discourse notes how a simple concept such as talk can either empower students or disenfranchise students. She finds that active dialogue plays an important role in giving students access to higher cognitive levels of mathematics, and can increase equity in mathematics learning. Other researchers note that student-to-student interaction is most effective when students actively provide explanations to each other (Webb, 1985, 1989); and when they communicate about, in, and with mathematics (Brenner, 1998). Teacher-to-student interaction is most effective, both for academic language development and concept development in a content area, when teachers communicate with students slightly above their level of competence and mediate interaction so that students have opportunities to produce extended stretches of academic discourse (Gibbons, 2002).

English Language Learners in some schools receive instruction from teachers trained in Specially

Designed Academic Instruction in English (SDAIE or sheltered instruction) or in bilingual strategies.


However, many of these students spend their time in mainstream classes not designed to meet their needs (McKeon, 1994). Their teachers have good intentions, but little training in adapting their instruction so that the mathematics content remains rigorous while academic language is incrementally developed. The Math Pathways and Pitfalls materials offer support to overburdened teachers by embedding the discourse and language acquisition strategies identified from the research literature cited above directly into the lessons. The intended goal is for these strategies to become part of regular classroom instruction though practice with MPP. In addition, the teaching guides provide mathematically robust examples, and explicit discussion probes, so that teachers can guide students towards increasingly sophisticated levels of mathematics understanding and discipline-specific use of academic language. Sentence stems written on posters model appropriate language for students to use as they learn to participate in mathematical discourse.

Pilot Study on the Impact of MPP Instructional Materials

Initial Pilot Study (1998-99). Prototype materials were pilot-tested with a diverse group of 233

students whose teachers replaced 11 hours of related instruction with project lessons. Utilizing a quasi-experimental design with statistical controls, results indicated that students exposed to MPP materials exhibited greater gains in rational number knowledge than a similar group of students exposed to regular standards-based instructional materials during the same time period (Heller, Gordon, Paulukonis, & Kaskowitz, 2000). Impact estimates were pronounced, with an overall effect size estimate of 0.59 standard deviations. The results were most noticeable for students who performed at low and medium levels in mathematics at the beginning of the academic year, with effect sizes ranging from 0.68 to 0.90. Consequently, the gap between less and more able students decreased for the project group and increased for the comparison group. In addition, scores improved significantly by the posttest among both native English speakers and students who were designated by their district as ELL. This was the case although all the materials were pilot-tested in English.

Research Questions

This study addressed two sets of questions. First, the evaluation was designed to measure the impact

of the Mathematics Pathways and Pitfalls (MPP) lessons on the mathematics that students learn. The questions that were addressed are:

1. What is the impact of MPP on students’ knowledge of the mathematics topics addressed, compared to that of students using the regular math curriculum?

2. How equitable is the impact of MPP on students’ mathematics knowledge across levels of

English language proficiency and entering mathematics ability?

To contribute to the interpretation of the results, the research also examined the fidelity of lesson implementation as enacted within MPP classrooms, compared to the structure and processes that were intended by the curriculum designers. Questions that were addressed are:

3. How closely does MPP as enacted follow the structure, content, and discourse processes that were intended by the curriculum designers?

4. How does MPP as enacted in classrooms that had greater student math score gains compare with

MPP in classrooms with lower student gains?

6

Methods

This study was conducted in second-, fourth-, and sixth-grade classrooms in five school districts across the country. Altogether, teachers from 40 schools participated in the study. The number of teachers per school participating in this study ranged from 1 to 3. On average, there were 1.5, 1.7, and 1.4 teachers per school in second, fourth, and sixth grades, respectively.

As shown in Table 3, an experimental design was implemented over a two-year period. In the spring

of 2003, teachers were randomly assigned within their school district to either an experimental group or control group. Random assignment was done after receiving informed consent from the teachers. Randomization was stratified by grade level within each school district. If there was more than one teacher from the same school and grade level, these teachers were randomly assigned to either the experimental or control group.

Table 3 Experimental Design of Two-Year Study With Pre-Post Teacher and Student Assessments Year 1 Year 2

Random-

ization Summer

2003 Fall 2003

Spring 2004

Summer 2004

Fall 2004

Spring 2005

Teachers A: Experimental R X1 O O O O B: Control R O O X2 O O

Students A: Experimental NR O X O O X O B: Control NR O O O X O

X1 – One day of professional development X2 – Four days of professional development

In the summer of 2003, the experimental group teachers were taught how to implement MPP during one day of professional development (PD). In the first year of the study, experimental group teachers substituted seven MPP lessons for a portion of their regular mathematics curriculum. The control group teachers used their regular mathematics curriculum, and received whatever professional development they normally were provided during that year.

In the second year of the study, both the Year 1 control group and Year 1 experimental group teachers

implemented seven MPP lessons. The previously-control-group teachers received four days of professional development on the use of MPP in the classroom, and the experimental group teachers received no additional professional development.

Districts

Five school districts in California, Missouri, and Arizona served as research sites. Three of these were districts in urban or urban fringe communities, one was in a suburban community, and one site served several small rural schools spread over a large geographic area. These sites were selected to provide a balance of urban, rural, and suburban populations, as well as diversity in the economic, ethnic, and language backgrounds of students.


A Memorandum of Understanding was negotiated with each district outlining the responsibilities for them and the project. Each district had a site coordinator who had the responsibility of recruiting teachers to volunteer for the study from their district. They also were responsible for handling the logistics of the professional development meetings with teachers, communicating with teachers, and interfacing with project staff. They were also charged with interfacing with their district to provide the project with student scores from district-administered standardized achievement test. Participants

The goal for the number of elementary classrooms in the study was 100, with about equal numbers of teachers in grades 2, 4, and 6. Depending on the size of the district(s), coordinators at each site were charged with recruiting between 15 and 40 teachers. A total of 99 teachers participated in the first year of the study, and 41 continued to the second year of the study. Each site coordinator recruited teachers from his or her district to request voluntary participation in the study. In some districts all of the teachers were in elementary schools, while in others, the sixth-grade teachers were in middle schools. If a teacher was in middle school, only the data for the first class period in his or her weekly schedule was included.

The project staff asked coordinators to make every effort to recruit a diverse group of teachers in

terms of their background, experience, and teaching philosophy. In order to claim that MPP is practical and effective for a variety of teachers, it was important that the study be conducted with a variety of teachers and classrooms.

Site coordinators in each district first met with school administrators to get their commitment, then

met with teachers to solicit volunteers. The project provided site coordinators with a set of presentation slides and handouts for both meetings. Recruitment meetings were held in the spring of 2003. The purpose of the teacher recruitment meetings was to provide information about the goals and activities of the MPP program and the research study. Teachers who volunteered signed consent forms, which informed them of their rights and responsibilities as research participants.

At the recruitment meeting, teachers were informed that they would receive a stipend to participate

based on the number of hours of project activities they completed and the district hourly rate. In Year 1, teachers in the project group received a stipend of $350, and teachers in the control (wait-listed) group received a stipend of $200. In Year 2, teachers who had been wait-listed the first year received a stipend of $400, and teachers who elected to continue from the first-year project group were paid a stipend of $250. Teachers who participated in a separate implementation study in the second year received an additional $500 stipend.

In spring 2005, participants were recruited to participate in the implementation study, a descriptive

study of MPP lesson enactment in the classroom. Program staff invited 17 teachers to take part in the study, all of whom were Group A teachers who both taught MPP in Year 1 of the project and also volunteered to teach MPP in Year 2 of the project. Of those invited, 11 teachers agreed to participate. Teacher Attrition

Some Year 1 teachers in both the experimental and control groups chose not to participate in the

second year of the study. The actual number of participating teachers is presented in Table 4. Continuing in the second year of the study was considered optional for teachers in the experimental group, and only a subset of the original teachers in that group continued for various reasons not related to the study. For example, one of the study coordinators left their district prior to the second year of the study. As a result, communication with the teachers in that district was difficult and several teachers did not continue with the project during Year 2.

8

Furthermore, when this study began, funding was assured for only one year. Districts and teachers

were informed that the second year of the study would only be conducted if additional funding were procured. For this reason, Year 2 of the project depended on the commitment of the teachers who did not know if the study would continue into the second year. While the second year of funding did come through, we do not know how many of the Year 1 teachers were lost because of the uncertainty about Year 2. Official word of funding for the second year did not come until May 3, 2004. Because of this uncertainty, dates for professional development for the second summer were tentative. Teachers from the control group may not have reserved time to attend the MPP summer PD since funding was not assured and could have dropped from the study as a result.

An attrition rate can introduce a confound if the teachers who continue in Year 2 differ from those

who dropped out. We conducted some preliminary analyses to see if there were systematic differences, and found evidence that the two groups did differ. Most notably, we found that teachers who dropped out tended to have students with lower MPP pretest scores in Year 1 than teachers who continued. Thus, we had evidence that teachers were not dropping out in the second year by chance alone, and that the random assignment of teachers to groups in the first year would not hold in the second year. For this report, we therefore focus on the first year of data for the study. Instruments and Data Collection

As shown in Table 4, in both the 2003-04 and 2004-05 years, participating teachers’ students took

project-developed pre- and post-tests, and standardized achievement test scores were obtained from the district. In both years, teachers completed questionnaires about their relevant background and about their implementation of MPP.

Table 4 Schedule of Data Collection

Instrument Sample 2003-04 2004-05 Students

MPP Pitfalls Quiz pretest All students Sept/Oct 2003 Sept/Oct 2004 MPP Pitfalls Quiz posttest All students May 2004 May 2005 Standardized achievement test (as a covariate) All students Spring 2003 Spring 2004 Standardized achievement test (as an outcome measure)

All students Spring 2004 Spring 2005

Teachers Teacher Information Form All teachers Fall 2003 Fall 2004 End-of-year MPP Questionnaire All teachers July 2004 July 2005 Implementation Study Questionnaire – Lessons 5 and 6

Sample of 11 teachers - Spring 2005

Classrooms Classroom audio recording – Lesson 6 Sample of 11 teachers - Spring 2005

MPP Pitfalls Quizzes. Items on the MPP Pitfalls Quizzes assess mathematics concepts that are known to cause difficulty for students as identified from the research literature and prominent assessments such as the NAEP and TIMMS. The tests contained one or more items that relate directly to the content of each lesson and a few additional transfer items that were indirectly related to the lesson content.


For both years of the study, all children in the study received an MPP Pitfalls Quiz as a pretest at the beginning of the academic year, and the same MPP Pitfalls Quiz as a posttest at the end of the academic year. For each grade level, a separate MPP Pitfall Test was developed (see quizzes for grades 2, 4, and 6 in Appendix A). Each test was developed to match the MPP lessons for that grade level.

Most items on these tests were multiple-choice format, with one correct answer. At least one of the

choices for items in the multiple-choice format contained a common misconception that students have with regard to the concept being assessed. A few open-ended items were included on the tests, and student responses to these open-ended items were scored as either correct or incorrect. For each MPP Pitfalls Quiz, we calculated the number of items answered correctly for each student, and then converted these raw scores to percentage correct. Thus, a student’s score could range from 0 (0% of items answered correctly) to 100 (100% of items answered correctly). On the second-, fourth-, and sixth-grade quizzes, there were 18, 17, and 20 items, respectively.

Mathematics Standardized Achievement Tests. Districts were asked to provide standardized-

mathematics-test-score data for all students participating in the study. For each student in the study, districts were asked to provide end-of-year standardized-test-score data for the student’s previous grade level (which served as the covariate) as well as the student’s current grade. For second grade students, standardized-test-score data were only obtained for the end of the second grade. (No standardized-test-score data were obtained for the end of first grade for these students because first-grade students are not typically given standardized tests). For fourth-grade students in the study, standardized-test-score data were obtained for the end of third grade and the end of fourth grade, and for sixth-grade students, standardized-test-score data were obtained for the end of fifth grade and the end of sixth grade.

MPP Questionnaires. Teachers were asked to complete a background questionnaire at the beginning

of each year, providing information about their education, teaching experience, and current teaching context as well as teacher and student demographics (see Appendix B). An end-of-year questionnaire was given to all teachers in spring of each year (see Appendix C). This questionnaire was designed to obtain teachers’ ratings of agreement with statements regarding their use of each component of the MPP materials, their own and their students’ responses to the lessons, and their overall evaluation of the curriculum. Finally, an implementation questionnaire was administered at the end of the 2004-05 year to a subset of 11 teachers in grades 2, 4, and 6 Lessons 5 and 6 to obtain more specific information about how teachers implemented MPP (see Appendix D).

Classroom audio recordings. The 11 teachers who completed the implementation questionnaires were

also asked to collect audio recordings of all discourse in their classrooms during Lesson 6 in spring 2005. Teachers were each provided with an audio recorder and asked to collect a complete recording of all parts of the lesson. Teachers were told that the purpose of asking them to audio tape the lesson was to study how the lesson was implemented. The audio recorder was to be placed around the teacher's neck so it would clearly record what the teacher was saying.

Teacher Participation in Professional Development

Teachers were randomly assigned in roughly equal numbers to either the experimental or control group during the spring of 2003. Since the study was being conducted with three grade levels of classrooms, separate summer professional development trainings were conducted for each of these grades. Thus during the summers of 2003 and 2004, three professional development trainings were held at each of three sites, each within driving distance of the districts in the study. In 2004, however, the number of teachers at one professional development site was so small that grade levels were combined for parts of the professional development that overlapped.

10

Experimental Group

Year 1: During the summer of 2003, the experimental group attended a one-day, six-hour introduction to MPP that was conducted by project consultants, trained by project staff. Most of the professional development consultants had used MPP in their classrooms as teachers. During this brief professional development training, teachers (a) received an introduction to the goals and purpose of MPP, (b) observed a video of a class participating in an MPP lesson, and (c) participated in a short practicum of how to teach an MPP lesson. Project staff carefully designed the agenda and activities for this meeting.

During the school year of 2003-04, the experimental group attended two, two-hour meetings after

school. One was held during the month of December or January, and the other held in May. The purpose of the first meeting was primarily to check in informally with teachers about the lessons they had taught so far and have them look through and discuss the next set of lessons they would be teaching. The site coordinator conducted these first meetings. The second meeting was a wrap-up.

Year 2: Teachers in the experimental group were invited, but not obligated, to teach MPP during the

2004-05 school year. Seventeen teachers chose this option. They were also invited, but not obligated to attend the four-day professional development during the summer. Two teachers chose this option. Control Group

Year 1: Teachers in the control group were waitlisted to receive training for the second year of the study, and participated only in data collection activities in Year 1 of the study. They did not participate in any professional development in the first year.

Several steps were taken to ensure that being exposed to the MPP materials through the experimental

group teachers did not contaminate the control group. First, face-to-face meetings were held with both the control group teachers and the teachers using project materials. In these meetings, a project representative discussed how participation in this project came with a professional obligation to assist the project in giving the materials a fair test. Project consultants who led the professional development explained why this is important and gave explicit instructions for not sharing any of the materials with anyone else and for not looking at the pretests or posttests prior to or following the assessment. To emphasize the importance of this request, both teachers using the project materials and the control group teachers were asked to sign an affidavit (see Appendix E).

Year 2: During the summer of 2004-05, teachers in the control group attended four six-hour days of

professional development. Project consultants who were trained by project staff conducted this professional development. At two sites, the professional development was scheduled over a period of four consecutive days. The professional development at one site was scheduled for two, two-day sessions. Teachers met each day for six hours.

The content of the professional development training for these teachers consisted of specific activities

that are part of a course developed for teachers, plus a one-day introduction to MPP. During the first three days of the training, teachers used a set of professional development materials that also had guides for the professional development instructors. During these three days, specific tasks were used that provided practice-based experiences to guide teachers in self-assessment, analysis and reflection, case discussion, and discussion of mathematical background and research readings. The final day of the training was essentially identical to the one-day training that teachers in the experimental group received in Year 1, introducing them to MPP. The mathematical topic of training for teachers matched the topic of the MPP lessons they would teach: grade 2 (whole numbers and operations), 4 (fractions), or 6 (percents).


Teacher Implementation of MPP

In the summer of 2003, experimental group teachers received a binder with three MPP lessons as part of their summer professional development training. At an after-school meeting held in December or January of Year 1 of the study, these teachers received a second binder with four additional MPP lessons. In the front of each of these two binders was a suggested schedule for teaching the lessons. Teachers recorded in their binders the dates they actually did the lessons. Each MPP lesson consisted of a two-period core lesson and two follow-up mini lessons. The entire lesson took approximately two hours of instructional time and was taught once per month. Seven lessons were taught over the course of the school year for a total of approximately 15 hours of instructional time.

The schedule was the same for both years of the study. As mentioned, only teachers in the

experimental group taught MPP in 2003-2004. Teachers who were in the control group in the first year of the study taught MPP in 2004-2005. MPP was optional for the experimental group teachers in year 2004-2005.

Data Analysis

The teacher questionnaires were analyzed using basic descriptive statistics. For the mathematics achievement data, it is not appropriate to use linear regression because students

within the same classroom cannot be assumed to be independent of one another. This is because there are likely to be many classroom effects and characteristics that the students share in common, for instance the teacher. The data are hierarchical, or nested, in that students exist within classrooms. We have data on both these levels—on students (e.g., score on previous standardized math test, whether or not the student is an English Language Learner), and on their classrooms (e.g., experimental group). (Because there were so few teachers per school, with many schools only having one teacher participating, we did not include the students’ school as a level in these analyses). Using ordinary regression would yield incorrect standard errors. In particular, since treatment varies between classrooms, the standard error of the treatment effect estimate and the associated p-values would be too low (e.g., Snijders & Bosker, 1993). Multilevel models, also known as hierarchical linear models (HLM) are designed to analyze relationships among precisely these kinds of nested data (Raudenbush & Bryk, 2002).

The multilevel analyses were performed for each grade level separately. Furthermore, for each grade

level, the MPP Pitfalls Quiz data were analyzed separately from the standardized mathematics achievement test data. Thus, for each grade level, two sets of analyses were performed: one using the MPP Pitfalls posttest scores as the outcome variable, and one using the standardized mathematics achievement scores obtained at the end of the student’s current grade level as the outcome variable.

Although multilevel models were used to analyze both the MPP Pitfalls Quiz data and the

standardized achievement test data, the approach taken to analyze the two achievement measures were not exactly the same. Most notably, in the analysis of the MPP Pitfalls Quiz data, we were able to evaluate the effect of ELL status on mathematics achievement. For the standardized-test data, we were not able to evaluate the effect of ELL, because there was more missing standardized-test data, resulting in smaller sample sizes and fewer ELL students in the samples.

In the next section, the multilevel model approach used to analyze the MPP Pitfalls Quiz data will be explained in detail. The multilevel model approach used to analyze the standardized achievement data, which was actually a simplified approach of that used for the MPP Pitfalls Quiz data, will then be explained in a later section.

12

Multilevel Statistical Analyses of the MPP Pitfalls Quiz Data

For each grade level, the full multilevel model consisted of four predictor variables: MPP Pitfalls pretest scores (X1), plus three dummy-variable predictors, as follows:

X2. Experimental group dummy variable. The experimental group variable was coded as ‘0’ if the student was in the control group, and ‘1’ if the student was in the experimental group. X3. English language learner (ELL) dummy variable. The ELL dummy variable was coded as ‘0’ if the student was not an English language learner, and ‘1’ if the student was an English language learner. X4. ELL-by-experimental group interaction. In order to find out if the MPP curriculum affected ELL and non-ELL students differently, an interaction term was included in the model. This interaction term was constructed by multiplying the experimental group comparison dummy variable (X2) by the ELL dummy variable (X3). Lastly, in order to find out if students’ initial performance level (as measured by the MPP Pitfalls

pretest) had a differential effect on how they performed in the experimental groups, a pretest by experimental group interaction term was also evaluated. (These interactions are sometimes referred to as “aptitude-by-treatment” interactions). This pretest by treatment interaction term, labeled X5, was constructed by multiplying the pretest score variable (X1) by the experimental group dummy variable (X2). If this pretest by experimental group interaction is statistically significant for a given grade level, we have evidence that the MPP curriculum is effecting students differently depending on their initial level of performance. All models included a random intercept for classroom.

Centering the MPP Pitfalls Pretest Data

To aid in interpreting the multilevel model results, the MPP Pitfalls pretest scores were centered to have a mean of zero. For each grade level, this centering was done by subtracting the pretest mean from each student’s pretest score. By centering the data, we were able to obtain adjusted posttest means, the estimated posttest means for students with mean pretest scores, where the posttest means are adjusted based on centered pretest scores. By centering the pretest scores, the adjusted posttest means can be interpreted on the scale of the original metric.

A Description of the Multilevel Analyses for the MPP Pitfalls Quizzes

For each grade level, several multilevel analyses were performed. For all the analyses, MPP Pitfalls posttest scores were used as the outcome variable.

For the first analysis, four predictor variables were included in the model:

1. X1. MPP Pitfalls Quiz pretest scores (centered), 2. X2. the experimental group dummy-variable, 3. X3. the ELL-status dummy-variable, and 4. X4. the ELL by experimental group interaction term.

If the treatment-by-ELL interaction term (X4) was statistically significant for a given grade level, then

two simple-effect comparisons were made. First, a comparison was made between ELL students in the control group and ELL students in the experimental group, controlling for all other variables in the model.


Second, a comparison was made between non-ELL students in the control group and non-ELL students in the experimental group, controlling for all other variables in the model.

If the treatment-by-ELL interaction term (X4) was not statistically significant for a given grade level,

then the multilevel model was simplified by dropping this interaction term, and re-analyzing the data using a model with only three predictor variables, namely (X1) to (X3) from the above list.

As a secondary analysis, we were interested in evaluating the pretest by experimental group

membership interaction term (X5). If the experimental group by ELL interaction term (X2) was statistically significant in the previous analysis, then an HLM analysis was performed by adding the pretest by group interaction (X 5) to the full-model, as follows:

1. X1. Pitfalls Quiz pretest scores (centered), 2. X2. the experimental group dummy variable, 3. X3. the ELL-status dummy variable, 4. X4. the ELL by experimental group interaction term, and 5. X5. the Pretest by experimental group interaction term.

If the experimental group by ELL interaction term (X4) was not significant for a given grade level,

then this term was dropped from the model, and the Pretest by experimental group interaction term (X5) was evaluated in this simplified model:

1. X1. Pitfalls Quiz pretest scores (centered), 2. X2. the experimental group dummy variable, 3. X3. the ELL-status dummy variable, 4. X4. the Pretest by experimental group interaction term.

Multilevel Software. The statistical package Stata, published by StataCorp (2005), was used for the

multilevel analyses (Rabe-Hesketh & Skrondal, 2005). For these analyses, parameters were estimated by maximum likelihood estimation using the “xtmixed” command. All multilevel analyses are based on listwise deletion of missing data. To obtain adjusted means and standard errors, the Stata command “adjust” was used. To obtain the simple effect comparisons, the Stata command “lincom” was used.

Calculating Effect Size Statistics

If a dummy-variable predictor was statistically significant, then an effect size statistic (ESS) was calculated. For each grade level, this ESS was calculated by dividing the estimated regression coefficient obtained in the multilevel analysis by the standard deviation of the posttest scores for all subjects in that grade level.

Results

School, Teacher , and Student Sample Sizes

Altogether, there were 40 schools that participated in the study. A summary of the number of teachers and students in the study is presented in Tables 5 and 6, respectively. For Year 1, there were 32, 38, and 29 teachers altogether in second-, fourth-, and sixth-grade classes, respectively, and 577, 812, and 582 students. Only students who had both pretest and posttest MPP test data were included in these analyses.

14

Table 5 Number of Teachers per Group by District and Grade

Year 1 Year 2

District Control group Experimental

group

Extended pd group (control

group in Year 1)

Continuing MPP teachers

(experimental group in Year 1)

Grade 2 1 5 5 3 3 2 3 3 1 1 3 1 2 0 2 4 5 4 0 0 5 2 2 1 0 Total 16 16 5 6

Grade 4 1 8 6 7 6 2 3 4 3 0 3 3 3 2 2 4 5 3 1 0 5 2 1 1 0 Total 21 17 14 8

Grade 6 1 4 5 3 3 2 3 4 1 0 3 3 1 1 0 4 2 3 0 0 5 2 2 0 0 Total 14 15 5 3 Table 6 Number of Students per Group by District and Grade

District Control group

Experimental group (one Day of PD)

Extended PD group (control

group inYear 1)

Continuing MPP teachers

(experimental group in Year 1)

Grade 2 1 108 101 68 66 2 53 44 17 13 3 9 40 - 34 4 84 62 - - 5 38 38 18 -

Total 292 285 103 113


Grade 4 1 186 136 141 125 2 57 70 52 - 3 63 56 38 38 4 111 64 11 - 5 42 27 24 -

Total 459 353 266 163 Grade 6

1 91 105 98 70 2 54 69 14 - 3 42 13 17 - 4 44 59 - - 5 53 52 - -

Total 284 298 129 70

The number of teachers who identified their schools as urban, suburban, or rural is presented in Table 7. Most schools were described as “suburban” (56.6% of teachers) or “urban” (27.3% of teachers); approximately 15% were rural.

Table 7 Number of Teachers Who Identified their Schools as Urban, Suburban, or Rural, by Treatment Group

Setting Control Group

Experimental Group Total

Urban N 16 11 27 Pct 31.4% 22.9% 27.3% Suburban N 26 30 56 Pct 51.0% 62.5% 56.6% Rural N 9 6 15 Pct 17.6% 12.5% 15.2% Other N 0 1 1 Pct 0.0% 2.1% 1.0% Total N 51 48 99 Teacher Demographics and Mathematics Background

Information on teachers’ gender and ethnicity is presented in Table 8. The majority of teachers in the study were women (89.8%), and most teachers identified themselves as “White” (73.1%) or “Black/African American” (18.3%).

16

Table 8 Teacher Gender and Ethnicity

Grade 2 Grade 4 Grade 6 Total Demographic category Cntrl Exp Cntrl Exp Cntrl Exp

Teacher Gender Male Pct 0.0 6.3 20.0 11.8 14.3 6.7 10.2 N 0 1 4 2 2 1 10 Female Pct 100.0 93.8 80.0 88.2 85.7 93.3 89.8 N 16 15 16 15 12 14 88

Teacher Ethnicity White Pct 87.5 68.8 68.4 81.3 76.9 53.8 73.1 N 14 11 13 13 10 7 68 Black/African American Pct 12.5 18.8 26.3 12.5 7.7 30.8 18.3 N 2 3 5 2 1 4 17 Latino/Spanish/Hispanic Pct 0.0 6.3 0.0 0.0 7.7 7.7 3.2 N 0 1 0 0 1 1 3 Asian/Southeast Asian Pct 0.0 0.0 5.3 0.0 7.7 7.7 3.2 N 0 0 1 0 1 1 3 American Indian or Pct 0.0 0.0 0.0 6.3 0.0 0.0 1.1 Alaskan Native N 0 0 0 1 0 0 1 Other Pct

N 0.0 0

6.3 1

0.0 0

0.0 0

0.0 0

0.0 0

1.1 1

Descriptive statistics on teachers’ mathematics training are presented in Table 9. Most teachers (71.7%) indicated that they had some college mathematics coursework, and 27.3% said they had a bachelor’s degree in mathematics or graduate-level mathematics coursework. Nearly half of the teachers (42.4%) had between 3 and 6 days of mathematics PD in the previous xx years, and close to one-quarter of the teachers (23.2%) had 7 days or more of mathematics PD.

Table 9 Teacher Mathematics Education and Training

Grade 2 Grade 4 Grade 6 Total Math background Cntrl Exp Cntrl Exp Cntrl Exp

Formal math education High school math courses Pct 0.0 0.0 0.0 0.0 0.0 6.7 1.0 N 0 0 0 0 0 1 1 Some college math courses Pct 68.8 100.0 70.0 64.7 78.6 46.7 71.4 N 11 16 14 11 11 7 70 BA or BS in Math Pct 18.8 0.0 10.0 17.6 0.0 20.0 11.2 N 3 0 2 3 0 3 11 Graduate level coursework Pct 12.5 0.0 20.0 17.6 21.4 26.7 16.3 in math N 2 0 4 3 3 4 16


Math professional development None Pct 6.3 12.5 10.0 11.8 0.0 13.3 9.2 N 1 2 2 2 0 2 9 Up to 2 days Pct 25.0 31.3 30.0 35.3 28.6 0.0 25.5 N 4 5 6 6 4 0 25 3 to 6 days Pct 56.3 50.0 35.0 29.4 35.7 46.7 41.8 N 9 8 7 5 5 7 41 7 days or more Pct 12.5 6.3 25.0 23.5 35.7 40.0 23.5 N 2 1 5 4 5 6 23

Additional information on teacher’s mathematics background and teacher training is presented for quantitative variables in Table 10. Because many of these variables were skewed, we will summarize these statistics using medians instead of means. The typical teacher in this study had 7 years of prior teaching experience altogether, and had taught mathematics at any grade level for roughly the same amount of time. Furthermore, the typical teacher in this study had been teaching mathematics at his or her current grade level for 3 years. Lastly, the typical teacher reported teaching 5 mathematics classes per week. The number of mathematics classes taught per week varied according to grade level.

Table 10 Teacher Mathematics Teaching Experience

Grade 2 Grade 4 Grade 6 Total Teaching experience Cntrl Exp Cntrl Exp Cntrl Exp Years taught prior to this one Mean 6.94 8.19 11.38 10.29 8.18 11.87 9.58 Median 8.00 6.00 8.00 4.00 6.50 13.00 7.00 SD 3.97 5.42 9.80 9.60 7.34 8.52 7.90 N 16 16 21 17 14 15 99 Years taught math at any grade level Mean 7.31 8.00 11.10 9.59 8.32 9.67 9.12 Median 7.50 6.00 8.00 4.00 6.50 6.00 7.00 SD 3.63 5.33 9.12 8.54 7.05 8.85 7.40 N 16 16 21 17 14 15 99 Years taught math at current grade level Mean 4.80 3.78 6.65 3.35 4.86 5.73 4.91 Median 3.00 3.00 3.50 3.00 3.00 4.00 3.00 SD 3.82 3.59 7.24 2.15 5.74 5.23 5.01 N 16 16 21 17 14 15 99 Number of math classes taught per week Mean 4.87 5.67 5.00 12.80 8.86 7.73 7.34 Median 5.00 5.00 5.00 5.00 5.00 5.00 5.00 SD .35 2.58 .00 28.30 7.45 7.22 12.04 N 16 16 21 17 14 15 99 Student Demographics

Information on students’ ELL status and gender are presented in Tables 11 and 12. The percent of ELL students in this study was 17.8%, 18.3%, and 16.6%, for second-, fourth-, and sixth-grade students, respectively. In terms of gender, roughly half of the students were boys, and half were girls, as would be expected.

18

Table 11 Student ELL Status by Treatment Group ELL Status Control Experimental Total

Grade 2 ELL Pct 14.1% 21.8% 17.8% Not ELL Pct 85.9% 78.2% 82.2% Total N 284 271 555

Grade 4 ELL Pct 16.7% 20.6% 18.3%

Not ELL Pct 83.3% 79.4% 81.7% Total N 449 306 755

Grade 6 ELL Pct 11.8% 21.4% 16.6%

Not ELL Pct 88.2% 78.6% 83.4% Total N 279 276 555

Table 12 Student Gender by Treatment Group Gender Control Experimental Total

Grade 2 Boy Pct 51.7% 49.5% 50.6% Girl Pct 48.3% 50.5% 49.4% Total N 292 285 577

Grade 4 Boy Pct 53.7% 48.5% 51.5%

Girl Pct 46.3% 51.5% 48.5% Total N 454 324 778

Grade 6

Boy Pct 49.3% 51.3% 50.3%

Girl Pct 50.7% 48.7% 49.7% Total N 280 279 559

Information on students’ ethnicity is presented in Table 13. Across grade levels, approximately 40% of the students were European American. There were roughly equal percentages of African American and Latino students in each grade. In the second grade, the percentage of African American and Latino students was 30.3% and 16.5% respectively; in the fourth grade, the percentage of African American and Latino students was 26.5% and 19.7% respectively; and in the sixth grade, the percentage of African American and Latino students was 24.6% and 26.6% respectively.


Table 13 Student Ethnicity by Treatment Group Ethnicity Control Experimental Total

Grade 2 White Pct 36.3% 37.9% 37.1% Black/African American Pct 35.3% 25.3% 30.3% Asian/Southeast Asian Pct 5.8% 7.0% 6.4% Latino/ Spanish-Origin/ Hispanic Pct 16.1% 16.8% 16.5% Native American Pct 3.1% 6.3% 4.7% All other responses Pct 3.3% 6.7% 5.0% Group Total N 292 285 577

Grade 4 White Pct 35.3% 45.5% 39.6% Black/African American Pct 31.6% 19.4% 26.5% Asian/Southeast Asian Pct 3.5% 1.8% 2.8% Latino/Spanish-Origin/ Hispanic Pct 20.5% 18.5% 19.7% Native American Pct 4.9% 11.1% 7.5% All other responses Pct 4.2% 3.7% 4.1% Group Total N 453 325 778

Grade 6 White Pct 42.5% 43.0% 42.7% Black/African American Pct 20.4% 28.5% 24.6% Asian/Southeast Asian Pct 4.6% 3.7% 4.2% Latino/ Spanish-Origin/Hispanic Pct 19.6% 21.5% 20.6% Native American Pct 4.6% 0.0% 2.2% All other responses Pct 8.3% 3.2% 5.8% Group Total N 280 298 578

Analysis of the MPP Pitfalls Quizzes

Measurement Analyses of the MPP Pitfalls Quizzes

A summary of basic measurement analyses for the MPP achievement tests is presented in Table 14. Cronbach’s alpha—a measure of internal consistency—was roughly equal to .80 for most of the tests. (The test that was the exception in terms of these high reliability coefficients was the fourth-grade pretest, where the reliability coefficient was .42. This was probably due to the overall difficulty of this test, where the average item difficulty was .21).

Multilevel Analyses of the MPP Pitfalls Quizzes

Descriptive statistics for MPP test scores are presented in Tables 15 and 16. The statistics in Table 14 indicate that for each grade level, the mean change from pretest to posttest was greater in the experimental group than in the control group.

20

Table 14 MPP Pitfalls Test Reliability Analysis (Year 1 and Year 2 Data Combined) Measure Grade 2 Grade 4 Grade 6

Pretest Number of items 18 17 20 Average item difficulty .40 .21 .41 Cronbach’s alpha .82 .42 .78 Number of subjects 794 1241 781

Posttest Number of items 18 17 20 Average item difficulty .67 .40 .54 Cronbach’s alpha .85 .80 .82 Number of subjects 794 1241 781

Table 15 MPP Test Scores by Treatment Group and Grade Treatment Group N Pretest Posttest Change

Grade 2 Control 284 Mean 41.10 62.34 21.24 SD 21.00 22.44 19.91 Experimental 271 Mean 41.06 71.67 30.61 SD 22.10 23.01 20.09 Total 555 Mean 41.08 66.90 25.82 SD 21.53 23.18 20.52

Grade 4 Control 449 Mean 20.33 31.80 11.46 SD 10.22 15.31 15.16 Experimental 306 Mean 21.76 45.92 24.16 SD 10.71 22.38 20.53 Total 755 Mean 20.91 37.52 16.61 SD 10.43 19.75 18.60

Grade 6 Control 279 Mean 39.32 49.14 9.82 SD 20.80 22.08 16.68 Experimental 276 Mean 41.12 56.99 15.87 SD 19.67 22.44 16.17 Total 555 Mean 40.22 53.05 12.83 SD 20.25 22.58 16.69 Note. This table only includes data for students whose district-identified English language proficiency status was available.


Table 16 MPP Test Scores by Treatment Group and ELL Status and Grade Treatment Group ELL? N Pretest Posttest Change

Grade 2 Control Yes 40 Mean 37.36 59.44 22.08 SD 17.02 20.64 18.55 No 244 Mean 41.71 62.82 21.11 SD 21.55 22.73 20.15 Experimental Yes 59 Mean 27.97 65.63 37.66 SD 18.05 22.64 19.66 No 212 Mean 44.71 73.35 28.64 SD 21.78 22.88 19.81



Multilevel analysis results are presented in Tables 17 and 18. This table contains adjusted posttest means, which can be helpful in interpreting the multilevel analysis results. For the analysis where the experimental group by ELL interaction term was statistically significant, these adjusted means were obtained from the model with the significant interaction term. For the analyses where the experimental group by ELL interaction term was not statistically significant, these adjusted means were obtained from the simplified model, which did not include the interaction term. The standard errors are based on the multilevel analyses. For making statements about differences between the experimental and control group, we reported estimated regression coefficients, which correspond to the difference between adjusted means, and z-tests based on them.

22

Table 17 Multilevel Model Results Using MPP Pitfall Tests as the Outcome Variable Fixed Effects b SEb Z p > |z|

Grade 2 Intercept (Average Teacher Posttest Achievement) 61.12 2.29 26.68 .000 Pretest 0.57 0.04 15.17 .000 Treatment Group (Experimental vs. Control) 10.14 3.24 3.13 .002 ELL Status -1.68 2.63 -0.64 .524 Treatment x ELL Status - - - -

Grade 4 Intercept (Average Teacher Posttest Achievement) 32.29 2.02 15.98 .000 Pretest 0.56 0.06 9.95 .000 Treatment Group (Experimental vs. Control) 13.10 3.09 4.25 .000 ELL Status -2.02 2.09 -0.96 .335 Treatment x ELL Status - - - -

Grade 6 Intercept (Average Teacher Posttest Achievement) 49.84 2.02 24.70 .000 Pretest 0.74 0.04 20.84 .000 Treatment Group (Experimental vs. Control) 6.29 2.87 2.19 .029 ELL Status -8.57 3.27 -2.63 .009 Treatment x ELL Status 10.20 4.38 2.33 .020 Random Effects

Grade 2 Intercept (Variance between teachers = SD (constant))

8.07

Level 1 (Variance within teachers = SD (residual))

16.55

Intraclass correlation 0.33 Grade 4

Intercept (Variance between teachers = SD (constant))

8.48


15.07

Intraclass correlation 0.36 Grade 6

Intercept (Variance between teachers = SD (constant))

6.59


14.39

Intraclass correlation 0.31 Note. The intraclass correlation coefficient was obtained by dividing the variance between teachers by the sum of the variance between teachers and variance within teachers.


Table 18 Multilevel Analysis: Adjusted Mean Posttest Scores of the Control Group versus Experimental Group by ELL Status

Cntrl Exp N of N of Treatment by

ELL Interaction Treatment Effect ELL Effect Pretest by Treatment

ELL? Group Group Tchrs Students Sig? Sig? Sig? Sig?

Grade 2 31 555 No Yes No No

Yes Adj. Mean 59.44 69.58

SE (3.17) (3.11)

No Adj. Mean 61.12 71.26

SE (2.29) (2.38)

Grade 4 36 755 No Yes No Yes


SE (2.63) (2.87)

No Adj. Mean 32.29 45.40

SE (2.02) (2.40)

Grade 6 29 555 Yes N/A N/A No


SE (3.46) (3.04)

No Adj. Mean 49.84 56.13

SE (2.02) (2.04)

24

For Grades 2 and 4, no statistically significant treatment-by-ELL interactions were found. For both Grades 2 and 4, a statistically significant main effect for treatment was found—the adjusted posttest mean for students in the experimental group was higher than the adjusted posttest mean for students in the control group (estimated regression coefficient b = 10.14, z = 3.13, p = .002, ESS = 0.43 for Grade 2; and b = 13.10, z = 4.25, p < .001, ESS = 0.66 for Grade 4). For second grade students, there was a 10.14 difference between the experimental and control groups adjusted posttest means. This difference can be thought of as the value added by being in the experimental group, after controlling for pretest scores and ELL status. For fourth grade, there was a 13.10 difference between the experimental and control groups adjusted posttest means. Finally, for sixth grade, there was a statistically significant treatment-by-ELL interaction (b = 10.20, z = 2.33, p = .020). Because the interaction term was statistically significant, simple effects were analyzed. For non-ELL students, a statistically significant difference was found between the experimental and control groups (b = 6.29, z = 2.19, p = .029, ESS = 0.28). For ELL students, a statistically significant difference was also found between the two groups (b = 16.49, z = 3.58, p < .001, ESS = 0.74). Thus, the value added for non-ELL students by being in the experimental group was smaller (6.29) than the value added by being in the experimental group for ELL students (16.49), after controlling for the other variables in the model.

To summarize, statistically significant differences favoring the experimental group were found for all

three grade levels. For second and fourth grade, no statistically significant treatment-by-ELL interaction was found. The effect size statistics for these grade levels were 0.43 and .061, respectively, favoring the experimental group. For sixth grade, a statistically significant treatment-by-ELL interaction was found. The difference between the experimental and control groups was statistically significant for both non-ELL and ELL students (effect size statistics equal 0.28 and 0.74, respectively), so that the value added by being in the experimental group was higher for ELL than non-ELL students.

Analyses of the pretest-by-treatment interactions revealed a statistically significant difference in

Grade 4 (b = .239, z = 2.09, p = .036) only. This finding indicates that the MPP intervention was more effective for fourth-grade children who had higher pretest scores than for children who initially had lower pretest scores.

Analysis of the Standardized Mathematics Achievement Tests

The MPP Pitfalls Quizzes were constructed to directly assess the impact of MPP. As such, the

analysis of the MPP Pitfalls Quiz data is of primary interest. Standardized mathematics achievement data was also collected and analyzed, to see if a more global effect of MPP could be seen. All districts were asked to provide standardized mathematics achievement test score data for all grade levels. Three of the five districts provided these data for all three grade levels (see Table 19).

As shown in Table 19, the districts varied in terms of which standardized test they used. Three

districts used the Stanford Achievement Test, 9th Edition (SAT-9), one district used the Missouri Assessment Program Test (MAP), and one district used both the Terra Nova test (TN) and MAP. The districts also varied in terms of the metric they used to report the test scores. Some districts provided national percentile ranks (NPR), others provided scaled scores (SS), and others provided normal curve equivalents (NCE).


Table 19 Name of Standardized Test and Metric Used by District

Grade 2 Grade 4 Grade 6 District Pretest Posttest Pretest Posttest Pretest Posttest 1 Provided

Data? No Yes Yes Yes Yes Yes

Name of Test

- SAT-9 SAT-9 SAT-9 SAT-9 SAT-9

Metric NPR NPR NPR NPR NPR 2 Provided

Data? No No No Yes Yes Yes

Name of Test

- - - MAP MAP TN

Metric SS SS NCE 3 Provided

Data? Yes Yes Yes Yes Yes Yes

Name of Test

SAT-9 SAT-9 SAT-9 SAT-9 SAT-9 SAT-9

Metric NCE NCE NCE NCE NCE NCE 4 Provided

Data? No No No Yes No Yes

Name of Test

- MAP - MAP - MAP

Metric SS SS SS 5 Provided

Data? No Yes Yes Yes Yes Yes

Name of Test

- SAT-9 SAT-9 SAT-9 SAT-9 SAT-9

Metric SS SS SS SS SS Note. “No” indicates that the school district did not provide standardized test score data for a given grade level, and “Yes” indicates that the district did provide standardized test score data.

For these analyses, only fourth-grade and sixth-grade data were analyzed. For the fourth-grade

analysis, the end-of-fourth-grade standardized total math score served as the posttest variable, and the end-of-third-grade standardized total math score served as the pretest. Likewise, for the sixth-grade analysis, the end-of-fourth-grade standardized total math score served as the posttest variable, and the end-of-third-grade standardized total math score served as the pretest. In most districts, students in first grade are not given standardized tests. Because we did not have pretest data for the second grade students, we decided not to analyze the second-grade student data using multilevel models. Sample Size and Missing Data

Table 20 provides information on the number of students who had standardized mathematics achievement data available. As is evident from this table, there was a fair amount of missing standardized-test-score data, either because it was missing district-wide, or because specific students were not tested. Data from some districts were not available for several reasons, including (a) some grade levels were not tested in some districts, (b) a computer system change in one district between years 1 and

26

2 of the project interfered with getting the data, and (c) one of the district coordinators responsible for providing the student assessment scores left the district too late for a replacement to be found.

Table 20 Number of Students with Valid Standardized Mathematics Achievement Data by Grade and District

Grade 2 Grade 4 Grade 6 District Pretest Posttest Pretest Posttest Pretest Posttest

1 Valid N - 132 163 184 93 110 Missing - 77 159 138 103 86 2 Valid N - 0 0 126 101 112 Missing - 97 127 1 22 11 3 Valid N - 49 107 111 29 8 Missing - 0 12 8 29 28 4 Valid N - 0 0 142 0 64 Missing - 146 175 33 103 39 5 Valid N - 75 56 66 79 92 Missing - 11 13 3 26 13

For consistency, only fourth-grade and sixth-grade students who were in the MPP analyses were

included in the standardized-test-score analysis. In addition, for the standardized-test-score analyses, only students with both “pretest” and “posttest” standardized-test scores were included.

Because of the smaller sample sizes, there were many classrooms that had no ELL students. Thus, for

the standardized-test-score analyses, the ELL status variable was not evaluated. Transforming the Standardized-Test Scores to the Same Metric

Because different standardized achievement tests were used, and because the test score data were reported in different metrics, the first step in the data analysis was to transform all data to standard scores, also known as z-scores. For each district and grade level, “pretest” scores were transformed to z-scores using the mean and standard deviation of the pretest for that district and grade level. Likewise, for each district and grade level, “posttest” scores were transformed to z-scores using the mean and standard deviation of the posttest for that district and grade level. Recall that z-scores have a mean and standard deviation of zero and one, respectively. Thus, for fourth-grade students in District 1, for example, the mean and standard deviation of the pretest z-scores scores would be zero and one respectively, and the mean and standard deviation of the posttest z-scores scores would also be zero and one respectively.

Individual z-scores tend to be very small values, ranging mostly between

!

±3 . Because it is difficult to look at tables of results with such small values, we transformed these z-scores to T-scores, with a mean of 50 and a standard deviation of 10, using the formula

!

Ti = 50 +10(zi) . Thus, for example, for fourth-grade students in District 1, the mean and standard deviation of the pretest T-scores would be 50 and 10 respectively, and the mean and standard deviation of the posttest T-scores would also be 50 and 10 respectively.

There are two issues to note in interpreting these standardized-test-score results. First, because the

standardized-test scores were transformed to T-scores within each district, any treatment effect needs to be interpreted as the impact on student performance relative only to the performance of students in the district. Second, because different tests were used and different metrics reported, we cannot look at


“growth” over the course of a year. For any given grade level and district, the mean pretest score will be 50, and the mean posttest score will be 50. Thus, for the standardized-test-score analyses, it is not meaningful to evaluate the difference between the mean pretest and mean posttest scores. Multilevel Analyses of the Standardized Achievement Test Results

Two multilevel analyses of the standardized-test scores were performed, one for the fourth-grade

students, and one for the sixth-grade students. For each of these multilevel analyses, the pretest score served as a level-1 covariate, and the experimental group served as the level-2 predictor. For both the fourth-grade and sixth-grade multilevel analyses, to aid in the interpretation of the multilevel analysis results, the pretest variable was centered by subtracting the pretest mean from each student’s pretest score.

Descriptive statistics for the standardized-test-score analyses are presented in Table 21. For the

fourth-grade students, no statistically significant difference was found in the adjusted posttest means between the experimental and control groups (b = 1.09, z = 0.63, p = 0.528). Likewise, for the sixth-grade students, no statistically significant difference was found in the adjusted means between the experimental and control groups (b = .102, z = 0.07, p = 0.946). (Recall that for the second grade students, no multilevel analyses were performed).

Table 21 Descriptive Statistics for Standardized Tests by Treatment Group and Grade Treatment Group Pretest Posttest

Grade 2 Control Mean 51.1 SD 9.1 N 137 Experimental Mean 48.7 SD 10.7 N 119

Grade 4 Control Mean 48.5 48.4 SD 9.9 10.0 N 186 186 Experimental Mean 52.2 52.2 SD 9.8 9.5 N 131 131

Grade 6 Control Mean 51.8 51.3 SD 9.8 9.6 N 158 158 Experimental Mean 47.7 48.4 SD 9.7 10.2 N 124 124 Note. Standardized test scores were converted to T-scores with a mean and SD of 50 and 10, respectively for both the pretest and posttest. For these standardized test score analyses, one cannot evaluate growth from pretest to posttest.

28

Implementation Study Results

As reported above, analyses of students’ Pitfalls Quiz scores showed that students in the experimental groups at all three grade levels outperformed the control group students. Attributing these differences to MPP requires evidence that establishes that the lessons were implemented in experimental classrooms. In addition, it is important to look for systematic differences in lesson implementation in higher- versus lower-scoring classrooms, in order to understand the conditions that enhance students’ math learning. Therefore, the research also examined the fidelity of lesson implementation as enacted within MPP classrooms, compared to the structure and processes that were intended by the curriculum designers. Teacher questionnaires and audio-recorded classroom discourse were analyzed to address the following questions:

• How closely does MPP as enacted follow the structure, content, and discourse processes that were intended by the curriculum designers?

• How does MPP as enacted in classrooms that had greater student math score gains compare with

lessons in classrooms with lower student gains?

Lesson Components Implemented

Completion of major components within the lessons. Each MPP lesson includes several components that are done over a period of two days. Table 22 lists these components, and shows the proportion of components in Lessons 5 and 6 that were reported as completed by the 11 teachers in the implementation study. This table also disaggregates the data by student scores on the MPP Pitfalls Quizzes, showing results separately for “higher-scoring” than “lower-scoring” classes. Classes were considered higher-scoring if they had a mean gain from pre- to posttest of at least 15 percentage points and a post mean score of at least 40% correct, in contrast to lower-scoring classrooms that did not meet those cutoffs. As shown in Table 22, almost all teachers in the implementation study reported completing all parts of the lessons on both days of each lesson, with just one notable exception. The last component of the Day 1 lessons, Look Back, in Lesson 5, was not done in two of the lower-scoring classes and one of the higher-scoring classes. (Data were not available for this component of Lesson 6 because of an omission on the questionnaire.) The omission of this component may be attributable in part to teachers’ running out of time to complete the last part of the Day 1 lesson. Another possible explanation is that, compared to other components of the lesson, the Look Back component is not emphasized as strongly in the teachers' introduction to MPP.

Audio recordings of Lesson 6 in all of these classrooms were also analyzed for presence or absence of lesson components, providing additional evidence in support of teachers’ self-reports (see Table 23). Because of technical problems, some lessons were not recorded for some teachers. However, as the teachers also reported on the questionnaires, almost all of the teachers were observed to complete all of the major parts of the lesson. Once again, the Look Back component was most frequently skipped. In addition, several other components were also not always completed, particularly in lower-scoring classrooms.

Audio recordings were also examined in more detail to determine how completely some of the multi-

step lesson components were implemented. These analyses also indicate that the teachers did not adhere rigidly to the lesson “scripts”, instead they varied them to some extent. For example, on Day 1, across all classes, roughly half of the prompts in Discussing the OK and Discussing the Pitfall were completed. Although the numbers are very small, for Lesson 6 it is worth noting that higher-scoring classrooms completed more parts of these two discussions, which are core activities of the MPP approach.


Table 22 Percent of Lesson Components Completed in Higher- and Lower-Scoring Classroomsa

Low (n = 5) High (n = 6) Total (N = 11) Lesson component Lesson 5 Lesson 6 Lesson 5 Lesson 6 Lesson 5 Lesson 6

Day 1 Purpose 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Math Words 100.0% 100.0% 83.5% 100.0% 90.9% 100.0% Starter Problem 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Discussing the OK 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Discussing the Pitfall 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Things to Remember 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Look Backb 60.0% - 83.5% - 72.7% -

Day 2 Review of Day 1 100.0% 100.0% 83.3% 100.0% 90.9% 100.0% Our Turn 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Your Turn 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Note. Data reported by teachers in Implementation Study Questionnaire. aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct. bItem not included on Implementation Study Questionnaire for Lesson 6

Table 23 Percent of Lower- and Higher-Scoring Classroomsa in which Components of MPP Lesson 6 Were Present in Audio Recordings

Lesson component Low (n = 5)

High (n = 6)

Total (N = 11)

Day 1 Discussion Builders 100.0%

(n = 4) 100.0%

(n = 4) 100.0%

(N = 8) Purpose 80.0%

100.0%

(n = 5) 90.0%

(N = 10) Math Words 80.0% 100.0%

(n = 5) 90.0%

(N = 10) Starter Problem 80.0% 100.0%

(n = 5) 90.0%

(N = 10) Discussing the OK 100.0% 100.0% 100.0%

Mean percent of lesson prompts presentbc Mean (SD)

52.0% (24.3%)

59.0% (23.0%)

55.8% (22.7%)

Discussing the Pitfall 50.0% (n = 4)

100.0% (n = 5)

77.8% (N = 9)

Mean percent of lesson prompts present (of five possible)bd

Mean (SD)

60.0% (56.6%)

88.0% (17.9%)

80.0% (30.6%)

Things to Remember 100.0% (n = 4)

33.3% 60.0% (N = 10)

Mean percent of lesson prompts present (of three possible)b

Mean (SD)

50.0% (19.2%)

100.0% (0.0%)

66.7% (29.8%)

Look Back 25.0% (n = 4)

66.7% 50.0% (N = 10)

30

Day 2 Review of Day 1 100.0% 100.0% 100.0% Our Turn 100.0% 100.0% 100.0%

Percent who completed all three problems 100.0% 83.3% 90.9% Percent who completed all three problems 80.0% 83.3% 81.8%

Your Turn 100.0% (n = 4)

100.0% (n = 4)

100.0% (N = 8)

Percent who completed all three problems 75.0% (n = 4)

75.0% (n = 4)

75.0% (N = 8)

Percent who completed all three problems 75.0% (n = 4)

66.7% (n = 3)

71.4% (N = 7)

Note. No classroom completed all of the lesson components. aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct. bMean percent only in classes where component ws present. cNumbers of components in Discussing the OK for each grade: Grade 2 = 17, Grade 4 = 18, and Grade 6 = 20. dNumbers of components in Discussing the Pitfall is five for all grades.

The Day 1 activity of reviewing Things to Remember at three points in the lesson was partially

implemented as well, but with lower-scoring classrooms completing more of these reviews. On Day 2, all lessons observed began with a review of the previous day’s conclusions, and then in the

Our Turn and Your Turn sections, the classes worked on problems like those discussed on Day 1. More of the classes completed all three Our Turn problems than the Your Turn, and, in general, only some classes completed all of the practice problems.

Length of Lessons

Questions about lesson length were included on the Implementation Study Questionnaire (see Table 24). MPP lessons are designed to be completed in less than two class periods – roughly one period on Day 1, and a shorter time on Day 2. Teachers’ reports indicate that lesson length approximates these intentions. Teachers estimated that the Core Lesson (including both Day 1 and Day 2) for Lesson 5 averaged close to one hour (57.3 minutes), and for Lesson 6, approximately 20 additional minutes (81.6 minutes). Time required for Lesson 5 ranged from less than one class period to two periods; time for Lesson 6 ranged from one class period to more than two.

Across both the Core Lessons and each of the Mini Lessons, there is a consistent trend of lessons

taking longer in the lower-scoring classes than the higher-scoring.

Discourse Processes

Audio recordings of lessons and teachers’ questionnaire responses provide a variety of information about the interactions that comprise MPP. Table 25 shows teacher and student discourse behaviors that were observed in the classroom recordings. Although the number of classrooms is small and it is not possible to reach conclusions about differences with any certainly, we point out here some of the strongest patterns that were observed.


Table 24 Length of MPP Lessons in Lower- and Higher-Scoring Classroomsa

Low (n = 5)

High (n = 6)

Total (N = 11) Questionnaire item

Lesson 5 Lesson 6 Lesson 5 Lesson 6 Lesson 5 Lesson 6 1.3. About how long did you take to teach

the core lesson, which does not include the mini–lessons? Note: A class period is about 45–50 minutes.

a. Less than 1 class period 20.0% 0.0% 16.7% 0.0% 18.2% 0.0% b. 1 class period 20.0% 60.0% 50.0% 33.3% 36.4% 45.5% c. > 1 ≤ 2 class periods 60.0% 20.0% 33.3% 66.7% 45.5% 45.5% d. > 2 class periods 0.0% 20.0% 0.0% 0.0% 0.0% 9.1%

2.3a. Core Lesson - Approximate minutes Mean 60.0 89.0 55.0 75.5 57.3 81.6 SD 19.0 34.0 27.6 30.3 23.1 31.2 Median 70.0 90.0 42.5 72.5 45.0 75.0 Range 35-80 50-125 30-90 43-120 30-90 43-125

2.3b. Mini Lesson 1 - Approximate minutes Mean 25.0 28.8 20.0 16.7 22.0 21.5 SD 15.8 13.1 7.1 2.6 10.9 10.0 Median 22.5 32.5 20.0 15.0 20.0 17.5 Range 10-45 10-40 10-30 15-20 10-45 10-40

2.3c. Mini Lesson 2 - Approximate minutes Mean 26.3 22.5 19.2 16.7 22.0 19.0 SD 14.9 11.9 7.4 2.6 10.9 7.7 Median 25.0 22.5 17.5 15.0 20.0 15.0 Range 10-45 10-35 10-30 15-20 10-45 10-35

Note. Data reported by teachers in Implementation Study Questionnaire. aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct.

Number of follow-up prompts. To achieve in-depth student discussions and explanations, it is essential that the teacher follow up on student responses with prompts that build on or require elaboration of the students’ original responses. As shown in Table 25, the mean number of follow-up prompts was considerably greater during the Discussing the OK component (mean of 43.6 follow-up prompts per lesson) than in the Discussing the Pitfall (mean of 16.8 per lesson). Furthermore, the number of follow-up prompts during Discussing the OK in lower-scoring classrooms was greater than in higher-scoring classrooms (52.0 vs. 36.7). Although the reverse was true for Discussing the Pitfall (12.3 in lower-scoring classrooms vs. 20.4 in higher), these means are closer and less notable. This finding might be expected, however, since all lessons have roughly twice as many prompts for Discussing the OK component as for Discussing the Pitfall. Thus there are fewer prompts to follow up on.

Number of follow-up prompts. To achieve in-depth student discussions and explanations, it is

essential that the teacher follow up on student responses with prompts that build on or require elaboration of the students’ original responses. As shown in Table 25, the mean number of follow-up prompts was considerably greater during the Discussing the OK component (mean of 43.6 follow-up prompts per

32

lesson) than in the Discussing the Pitfall (mean of 16.8 per lesson). Furthermore, the number of follow-up prompts during Discussing the OK in lower-scoring classrooms was greater than in higher-scoring classrooms (52.0 vs. 36.7). Although the reverse was true for Discussing the Pitfall (12.3 in lower-scoring classrooms vs. 20.4 in higher), these means are closer and less notable. This finding might be expected, however, since all lessons have roughly twice as many prompts for Discussing the OK component as for Discussing the Pitfall. Thus there are fewer prompts to follow up on.

Table 25 Mean Percent and Frequency of Lower- and Higher-Scoring Classroomsa in which Components of MPP Lesson Were Present in Audio Recordings

Low High Total Lesson component Mean (SD) Mean (SD) Mean (SD)

Day 1 Discussing the OK (n = 5) (n = 6) (N = 11)

Number of follow-up prompts 52.0 (24.6) 36.7 (17.5) 43.6 (21.4) Number of Discussion Builders used by teacher 4.0 (3.5) 2.7 (2.9) 3.3 (3.1) Number of Discussion Builders used by students 1.4 (1.7) 1.8 (1.5) 1.6 (1.5) Times students were asked to talk with neighbor 2.4 (0.9) 2.7 (3.6) 2.5 (2.6) Times students were asked to show/explain their ideas 6.8 (4.3) 4.0 (4.4) 5.3 (4.4) Rating of length of student responses in discussionsb 2.4 (0.8) 3.2 (1.0) 2.8 (0.9)

Discussing the Pitfall Number of follow-up prompts 12.3 (17.0) 20.4 (20.0) 16.8 (18.1) Number of Discussion Builders used by teacher 0.0 (0.0) 2.2 (2.3) 1.2 (2.0) Number of Discussion Builders used by students 0.0 (0.0) 1.2 (1.6) 0.7 (1.3) Times students were asked to talk with neighbor 1.8 (2.9) 1.0 (1.2) 1.3 (2.0) Times students were asked to show/explain their ideas 0.3 (0.5) 1.0 (1.4) 0.7 (1.1) Rating of length of student responses in discussionsb 2.0 (1.8) 4.5 (0.7) 3.4 (1.8)

Day 2 Our Turn

Number of Discussion Builders used by teacher 8.6 (7.8) 8.5 (5.4) 8.5 (6.2) Number of Discussion Builders used by students 2.8 (2.8) 2.5 (3.0) 2.6 (2.8) Times students were asked to talk with neighbor 5.0 (2.6) 8.5 (9.6) 6.9 (7.2) Times students were asked to show/explain their ideas 7.8 (4.6) 5.0 (3.0) 6.3 (3.9) Rating of length of student responses in discussionsb 1.9 (1.2) 2.6 (0.9) 2.3 (1.1)

Your Turn (n = 3) (n = 4) (N = 7) Number of Discussion Builders used by teacher 1.7 (2.1) 4.8 (6.4) 3.4 (5.0) Number of Discussion Builders used by students 2.0 (3.5) 3.8 (5.6) 3.0 (4.5) Times students were asked to talk with neighbor 0.0 0.0 0.0 Times students were asked to show/explain their ideas 0.0 0.8 (1.0) 0.4 (0.8) Rating of length of student responses in discussionsb 1.0 (0.8) 1.6 (1.3) 1.3 (1.0)

aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct. b Length of student responses rated on a scale from 1-Few if any “moderate-to-long” student responses, to 5-Many or mostly “moderate-to-long” student responses.


Number of Discussion Builders. MPP incorporates Discussion Builders to support teachers and students in achieving extended classroom discourse about the math in each lesson. As shown in Table 25, the number of times that teachers used Discussion Builders averaged just over 3 during Discussing the OK, and fewer than 2 during Discussing the Pitfall. In contrast, the teacher used Discussion Builders more than 8.5 times per lesson during the Our Turn segments. This pattern also holds for the number of times students used Discussion Builders in these three parts of the lessons. During the Your Turn segments, Discussion Builders were used approximately as frequently as during Discussing the OK. There are few notable differences between lower- and higher-scoring classes in Discussion Builder use, except during Your Turn when discussions in higher-scoring classes included more of these sentence types than the lower-scoring classes. This finding is encouraging, since it indicates that the Discussion Builders are spontaneously used, even for lesson segments that are not guided by specific prompts in the teaching guide (such as the Our Turn). Since one of the goals of MPP is for students to become more proactive in their learning, this finding could be an indication that the Discussion Builders may play a role in that desired behavior.

Teachers were also asked about their use of Discussion Builders on the Implementation

Questionnaire. As shown in Table 26, the majority of teachers reported that their students use the Discussion Builders one to five times during a typical day. More teachers of lower-scoring students than higher-scoring perceived their students as using Discussion Builders more than five times a day (40% vs. 16.7%). More than 80% of the teachers reported that their students use these sentence forms both in math and other subject areas.

Table 26 Percent of Teachers Reporting How Students Use MPP Discussion Builders in Lower- and Higher-Scoring Classroomsa

Questionnaire item Low

(n = 5) High

(n = 6) Total

(N = 11) 1. When do your students use the Discussion Builders?

a. The children use the Discussion Builders in math and other subject areas.

80.0% 83.3% 81.8%

b. The children use the Discussion Builders only in math. 0.0% 16.7% 9.1% c. The children seldom use the Discussion Builders in any subject area. 20.0% 0.0% 9.1%

2. Estimate about how often you think the students use the Discussion Builders during a typical day.

a. More than 15 times a day 0.0% 0.0% 0.0% b. Between 5 and 15 times a day 40.0% 16.7% 27.3% c. 1 to 5 times a day 60.0% 66.7% 63.6% d. Never 0.0% 16.7% 9.1%

Note. Data reported by teachers in Implementation Study Questionnaire. aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct.

Times students were asked to talk with their neighbor. Periodically during the lessons, teachers are

urged to have students talk about the math with one another. Analysis of the audio recordings indicate that this occurred approximately 2-3 times during Discussing the OK, and 1-2 times during Discussing the Pitfall, but most frequently by far (close to 7 times per lesson, on average) during Our Turn on Day 2. As would be expected, students never were asked to talk among themselves during Your Turn. Students talked among themselves more often in higher-scoring classrooms during Our Turn (8.5 vs. 5.0 times per lesson, on average), but otherwise lower- and higher-scoring classes were very similar.

34

Times students were asked to show or explain their math ideas. A key aspect of the MPP approach is fostering students’ abilities to articulate their math thinking. Teachers explicitly asked students to explain their ideas primarily during Discussing the OK (mean of over 5 times per lesson) and Our Turn (mean of over 6 times per lesson). During both of these segments, students in lower-scoring classes were asked to explain their thinking more frequently than those in higher-scoring classes.

Length of student responses in discussions. The length of students’ statements in each audio

recording was rated on a scale from 1 - “Few if any ‘moderate-to-long’ student responses,” to 5 – “Many or mostly ‘moderate-to-long’ student responses.” These ratings were assigned within each lesson component. As shown in Table 25, during every lesson component, students in higher-scoring classes gave longer responses than those in lower-scoring classes. Overall, students in lower-scoring classes were rated approximately 2 on average, whereas those in higher-scoring classes were rated 3 or higher. Teachers’ Reports on Lesson Implementation

Teachers reported the following on questionnaires (see Tables 27 and 28). Use of Getting Started tasks. As shown in Table 28, teachers at all three grade levels agreed on

average with a statement that they used the MPP Getting Started tasks at the beginning of the year to help their students learn how to use the Discussion Builders. Second grade teachers agreed most strongly with this statement.

Use of Mini-Lessons. Teachers at all three grade levels agreed strongly with statements that they used

all of the Mini-Lesson 1 and 2 lessons, involving multiple-choice questions and explanations in writing from their students (see Table 28).

Use of the Mathematical Background section. As shown in Tables 27 and 28, the majority of

teachers reported reading closely the Mathematical Background section of the teaching guide for Lessons 5 and 6, and reading this section for almost every lesson. Most teachers of higher-scoring classes reported reading this section closely for both Lesson 5 and 6 (with the highest percentage for Lesson 6), whereas most teachers of lower-scoring classes just skimmed the Mathematical Background section, or did not read it, particularly for Lesson 6.

Questions in the teaching guide. Questionnaire responses summarized in Tables 27 and 28 indicate

clearly that teachers use both the questions in the teaching guide and some of their own when conducting discussions during MPP lessons.

Teacher behavior during students’ paired discussions. Almost all of the teachers reported that when

their students were talking among themselves about the math, the teachers walked around and sometimes interacted with students about their work.


Table 27 Teachers’ Use of MPP in Lower- and Higher-Scoring Classroomsa

Low (n = 5) High (n = 6) Total (N = 11) Questionnaire item Lesson 5 Lesson 6 Lesson 5 Lesson 6 Lesson 5 Lesson 6

3. How did you use the Mathematical Background section of the teaching guide for Lesson 5/6?

a. I read it closely prior to teaching the lesson. 60.0% 20.0% 66.7% 83.3% 63.6% 54.5% b. I skimmed it prior to teaching the lesson. 40.0% 60.0% 33.3% 16.7% 36.4% 36.4% c. I did not read it. 0.0% 20.0% 0.0% 0.0% 0.0% 9.1% d. Other 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

4. How did you use the questions in the teaching guide for Lesson 5/6? a. I used all or most of the questions in the teaching guide and none of my

own. 0.0% 0.0% 16.7% 16.7% 9.1% 9.1%

b. I used all or most of the questions in the teaching guide and some of my own.

100.0% 80.0% 83.3% 66.7% 90.9% 72.7%

c. I used few or none of the questions in the teaching guide and mostly used ones I made up.

0.0% 20.0% 0.0% 16.7% 0.0% 18.2%

d. Other 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 5. What did you do while your students were in paired discussions?

a. Walked around and sometimes interacted with students about their work.

100.0% 80.0% 100.0% 100.0% 100.0% 90.9%

b. Walked around but seldom interacted with students about their work. 0.0% 20.0% 0.0% 0.0% 0.0% 9.1% c. I did not walk around or interact with students about their work but

waited for them to finish before proceeding. 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%

d. The students did not have paired discussions. 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Note. Data reported by teachers in Implementation Study Questionnaire. aHigher-scoring classroom = a mean gain from pre- to posttest of ≥15 percentage points and a post mean score of ≥40% correct,

36

Table 28 Teachers’ Self-Reported Use of MPP in 2003-04 End-of-Year Questionnaire

End-of-year questionnaire item Grade 2

(n = 18) Grade 4 (n = 25)

Grade 6 (n = 13)

Total (N = 56)

Teacher Use of MPP Materials

14. I used most of the questions in the teaching guides, along with some of my own, to help me conduct the discussions with my class.

Mean (SD)

Range

8.5 (1.7) 4-10

8.5 (1.6) 4-10

8.8 (1.4) 6-10

8.6 (1.6) 4-10

15. I made up most of my own questions to conduct the discussion with my class, instead of using those in the teaching guide.

Mean (SD)

Range

3.1 (2.5) 1-10

4.4 (2.7) 1-10

3.1 (2.6) 0-8

3.7 (2.7) 0-10

18. I used the Getting Started tasks at the beginning of the school year to help my students learn how to use the Discussion Builders.

Mean (SD)

Range

8.2 (2.4) 1-10

6.6 (3.7) 0-10

7.3 (2.9) 0-10

7.3 (3.2) 0-10

22. I used all of the Mini-Lesson 1 lessons (multiple-choice questions) with my students. (Write a 10 if you don't think you skipped any.)

Mean (SD)

Range

9.1 (1.6) 5-10

9.4 (1.3) 4-10

9.5 (1.0) 7-10

9.4 (1.3) 4-10

23. I used all of the Mini-Lesson 2 lessons (requiring explanations in writing) with my students. (Write a 10 if you don't think you skipped any.)

Mean (SD)

Range

8.9 (1.8) 4-10

9.2 (1.7) 4-10

9.4 (1.2) 6-10

9.1 (1.6) 4-10

25. I read the Mathematical Background section of the teaching guide for almost every lesson. (Write a 10 if you don't think you skipped any.)

Mean (SD)

Range

8.8 (1.7) 4-10

8.8 (2.0) 4-10

8.8 (2.5) 1-10

8.8 (2.0) 1-10

Note. Ratings on 10-point scale from 1-Don’t agree at all, to 10-Strongly agree. Teachers’ Evaluation of MPP

Teachers were asked to rate the impact and value of MPP on the end-of-year questionnaire. Results

are provided in Table 29. Impact on student learning. For the most part, teachers agreed rather strongly with a variety of

statements that MPP was helpful for their students (see Table 29). These statements included opinions that, after completing the set of lessons this year, their students understood the math topics in the lessons better than students in past years, all of their students are making fewer pitfalls, and MPP helped most of their students learn the math concepts and prevent pitfalls. The one statement that teachers at all grade levels disagreed with is that the lessons were helpful for their non-native English-speaking students. These opinions are contradicted by the fact that the lessons actually were at least as effective for ELLs’ math achievement as for English-proficient students. Teachers’ ratings with respect to special needs students were stronger than for ELL students.

Interestingly, sixth-grade teachers agreed but less strongly that the lessons were helpful for high-

performing students, whereas fourth-grade teachers agreed less strongly that they were helpful for low-performing students. Second-grade teachers agreed but less strongly than the other grades with a statement that their students understood the math topics in the lessons better than students in past years.

Teachers’ opinions of students’ opinions. Teachers at all grade levels agreed strongly that their

students really liked Math Pathways and Pitfalls and that the format of the lessons was easy for most of their students to follow. Second and fourth grade teachers agreed that their students really liked the video of other students doing a Math Pathways and Pitfalls lesson, but sixth-grade teachers disagreed with this statement.


Table 29 Teachers’ Ratings of MPP in End-of-Year Questionnaire

Questionnaire Item Grade 2

(n = 18) Grade 4 (n = 25)

Grade 6 (n = 13)

Total (N = 56)

Impact on Student Learning 4. These lessons were helpful for my high-performing

students. Mean (SD)

Range

7.0 (2.4) 2-10

8.2 (1.5) 5-10

5.9 (2.5) 2-9

7.3 (2.2) 2-10

5. These lessons were helpful for my low-performing students.

Mean (SD)

Range

6.7 (2.6) 3-10

5.8 (2.8) 1-10

7.9 (1.7) 4-10

6.6 (2.6) 1-10

6. These lessons were helpful for my non-native English-speaking students.

Mean (SD)

Range

4.5 (3.6) 0-9

3.4 (3.3) 0-9

3.5 (4.1) 0-10

3.8 (3.6) 0-10

7. These lessons were helpful for my special needs students. Mean (SD)

Range

5.2 (3.1) 0-9

5.6 (2.8) 0-9

6.2 (3.7) 0-10

5.6 (3.1) 0-10

8. When working on the Starter Problems, many of my students made the same or similar pitfalls to the ones in the lessons.

Mean (SD)

Range

7.1 (2.2) 4-10

8.3 (1.8) 3-10

6.7 (2.8) 2-10

7.5 (2.2) 2-10

9. After completing the set of lessons this year, my students understood the math topic in the lessons better than students in past years.

Mean (SD)

Range

5.8 (2.2) 0-10

7.0 (2.4) 0-10

6.8 (2.0) 3-10

6.6 (2.3) 0-10

10. After completing the set of lessons this year, all of my students are making fewer pitfalls.

Mean (SD)

Range

6.1 (1.3) 4-8

7.2 (1.5) 4-10

6.6 (2.3) 2-10

6.7 (1.7) 2-10

12. The Discussion Builders were very important in helping my students learn how to discuss mathematics.

Mean (SD)

Range

7.4 (2.5) 3-10

7.2 (2.5) 1-10

9.2 (1.3) 6-10

7.8 (2.4) 1-10

13. The Discussion Builders helped my students learn to discuss other content areas besides math.

Mean (SD)

Range

7.2 (2.5) 2-10

7.5 (2.4) 1-10

8.5 (2.8) 0-10

7.6 (2.5) 0-10

20. The video played an important role in helping my students learn how to do the Math Pathways and Pitfalls lessons.

Mean (SD)

Range

7.3 (2.8) 0-10

7.0 (2.9) 0-10

4.6 (4.0) 0-10

6.5 (3.3) 0-10

24. The Mini-Lessons that I used helped my students consolidate their understanding.

Mean (SD)

Range

7.6 (2.0) 3-10

7.9 (1.2) 5-10

8.7 (1.5) 5-10

8.0 (1.6) 3-10

30. I believe that the Math Pathways and Pitfalls lessons helped most of my students learn the math concepts and prevent pitfalls.

Mean (SD)

Range

7.0 (1.9) 4-10

7.9 (1.8) 3-10

7.3 (2.0) 2-10

7.5 (1.9) 2-10

Students’ Opinions 1. Most of my students really liked the Math Pathways and

Pitfalls lessons. Mean (SD)

Range

7.3 (1.5) 4-10

7.5 (2.2) 0-10

6.9 (2.0) 4-9

7.3 (1.9) 0-10

2. The format of the lessons was easy for most of my students to follow.

Mean (SD)

Range

7.7 (1.9) 3-10

7.1 (2.3) 1-10

8.6 (1.7) 5-10

7.7 (2.1) 1-10

19. My students really liked the video of other students doing a Math Pathways and Pitfalls lesson. (Write NA if you didn't show the video.)

Mean (SD)

Range

7.5 (2.6) 0-10

6.9 (3.3) 0-10

4.2 (3.9) 0-10

6.5 (3.4) 0-10

38

Questionnaire Item Grade 2

(n = 18) Grade 4 (n = 25)

Grade 6 (n = 13)

Total (N = 56)

Teachers’ Opinions of MPP 3. The teaching guide is clearly organized and easy to

follow. Mean (SD)

Range

7.4 (2.1) 3-10

7.0 (2.7) 2-10

8.8 (1.5) 5-10

7.6 (2.4) 2-10

11. The language support provided in the teaching guide and in the student lessons (for example, the Math Words) was helpful.

Mean (SD)

Range

8.3 (1.6) 5-10

7.7 (1.6) 2-10

7.6 (2.2) 3-10

7.9 (1.8) 2-10

16. Prior to the Math Pathways and Pitfalls lessons, I conducted thought-provoking math discussions with my class several times a week.

Mean (SD)

Range

5.1 (2.8) 1-9

5.6 (2.4) 1-10

6.6 (2.7) 2-10

5.7 (2.6) 1-10

17. I learned a lot about using thought-provoking discussions to teach math through the Math Pathways and Pitfalls lessons.

Mean (SD)

Range

8.1 (1.6) 5-10

7.5 (1.4) 4-10

7.9 (1.5) 6-10

7.8 (1.5) 4-10

21. The language support for students provided in the teaching guides and in the student lessons helped me prepare students for the special mathematical vocabulary and symbols in the lesson.

Mean (SD)

Range

8.2 (1.8) 5-10

7.3 (1.5) 4-10

6.7 (2.4) 2-10

7.4 (1.9) 2-10

26. Substituting a Math Pathways and Pitfalls lesson for two of my regular lessons about once a month worked well.

Mean (SD)

Range

5.5 (3.1) 1-10

6.1 (3.1) 0-10

5.4 (3.0) 1-10

5.8 (3.0) 0-10

27. I would rather teach the whole set of Math Pathways and Pitfalls lessons as a single unit, rather than throughout the year.

Mean (SD)

Range

6.2 (3.6) 1-10

6.7 (3.1) 1-10

6.1 (3.8) 1-10

6.4 (3.4) 1-10

28. I would love to use the Math Pathways and Pitfalls lessons again next year.

Mean (SD)

Range

7.9 (2.1) 3-10

8.3 (1.7) 5-10

7.3 (2.4) 3-10

7.9 (2.0) 3-10

29. The students in our school would benefit greatly if all of the teachers used the Math Pathways and Pitfalls lessons.

Mean (SD)

Range

7.2 (1.6) 4-10

7.4 (2.2) 3-10

7.2 (2.0) 2-10

7.3 (2.0) 2-10

31. The Math Pathways and Pitfalls lessons supported hard-to-teach topics in our math textbook.

Mean (SD)

Range

6.7 (2.1) 1-10

7.5 (2.3) 0-10

7.8 (2.2) 2-10

7.3 (2.2) 0-10

32. The Math Pathways and Pitfalls lessons went beyond our textbook in a positive way.

Mean (SD)

Range

7.6 (2.0) 5-10

7.9 (2.8) 0-10

7.8 (2.2) 2-10

7.8 (2.4) 0-10

33. I feel that my teaching has improved as a result of using the Math Pathways and Pitfalls materials.

Mean (SD)

Range

7.5 (2.2) 3-10

8.0 (1.6) 5-10

7.0 (2.4) 2-10

7.6 (2.0) 2-10

Note. Ratings on 10-point scale from 1-Don’t agree at all, to 10-Strongly agree. Teachers’ overall opinions of MPP. The teachers also agreed strongly with strong statements about

the overall value of MPP. These include: • I would love to use Math Pathways and Pitfalls again next year. • The students in our school would benefit greatly if all of the teachers used Math Pathways and

Pitfalls. • Math Pathways and Pitfalls supported hard-to-teach topics in our math textbook. • Math Pathways and Pitfalls went beyond our textbook in a positive way.


• I feel that my teaching has improved as a result of using the Math Pathways and Pitfalls materials.

• I learned a lot about using thought-provoking discussions to teach math through Math Pathways and Pitfalls.

Teachers at all grade levels, but especially at sixth grade, agreed strongly that the teaching guide is

clearly organized and easy to follow. The only statement about which the teachers approached neutrality is that substituting a Math

Pathways and Pitfalls lesson for two of their regular lessons about once a month worked well. The teachers agreed, instead, that they would rather teach the whole set of Math Pathways and Pitfalls lessons as a single unit, rather than throughout the year.

Summary

With respect to fidelity of lesson implementation, analysis of classroom audio recordings and teacher

questionnaires revealed that (a) almost all teachers implemented every major component and intended discourse process of the lessons; (b) teachers made some minor modifications to the lesson structures—namely some steps or prompts were left out more than others, particularly in lower-scoring classes; (c) some of the tools for building extended student talk about math, such as the Discussion Builders, are spontaneously used by teachers and students, even for lesson segments that are not guided by specific prompts in the teaching guide, and during class time on subjects other than math; (d) in classes with higher-scoring students, there was more use of Discussion Builders by both teachers and students, were asked to explain their thinking less frequently than in lower-scoring classes but more often talked about the math among themselves, and gave longer responses about the math.

Teachers expressed strongly positive opinions about the value of the program, including that their

students understood the math topics in the lessons better than students in past years, that MPP helped most of their students learn the math concepts and prevent pitfalls, and their students really liked MPP. Overall, the teachers strongly agreed that they would love to use MPP again next year, and students in their schools would benefit greatly if all of the teachers used Math Pathways and Pitfalls.

Conclusions

Using a project-developed Pitfalls Quiz as the measure of mathematics achievement, this experiment

found that student math performance in MPP classes was higher than in non-MPP classes for all three grade levels. For second and fourth grades, MPP benefited ELL and non-ELL students equally. The effect-size statistics (ESS) for second and fourth grades were .43 and .66, respectively. For sixth grade, MPP had a greater treatment effect for ELL students (ESS = .74) than non-ELL students (ESS = .28). In evaluating how equitable the impact of MPP was on students’ mathematics knowledge across levels of entering math knowledge, the study found no difference in the effectiveness of MPP for mathematically stronger versus weaker students except at fourth grade, where MPP was more effective for children who had higher pretest scores than for children who had lower pretest scores. However, the grade 4 quiz was very difficult and, accordingly, less reliable than the second- and sixth-grade tests.

For the district-administered standardized achievement tests, no statistically significant differences

were found in the adjusted standardized achievement test posttest means between the experimental and control groups for either the fourth-grade students or the sixth-grade students. (No standardized achievement test data were available for second grade students.)

The findings of positive impact of MPP on student mathematics performance across grades, levels of

40

English proficiency, and entering mathematics ability are consistent with an earlier study of MPP materials by Heller, Gordon, Paulukonis, and Kaskowitz (2000). Because the current study was based on a more rigorous research design (i.e., a cluster randomized design) than the one used in the Heller et. al. study, the results of the current study can be viewed as even stronger evidence of the effectiveness of the MPP materials.

Comparing MPP Pitfalls Quiz and standardized achievement test results. Although statistically

significant results were found for all three grade levels on the MPP Pitfalls Quizzes, no statistically significant results were found using standardized achievement tests as the outcome variable. This disparity might be due to the fact that the MPP Pitfalls Quizzes were designed to assess the rational number topics covered by the MPP lessons, whereas the standardized achievement tests assess a more global construct of mathematical achievement, so may not have been instructionally sensitive enough to detect differences between the MPP and non-MPP groups.

Implementing cluster randomized designs in education. Cluster randomized designs are a powerful

way of evaluating the impact of a given educational intervention on student learning. The random assignment of teachers to MPP and non-MPP groups is an important element in the internal validity of this study.

There are many logistical challenges to implementing a cluster randomized design in education. First,

random assignment of teachers requires uniformity of schedule, district policy, and preferences across many difference school and district contexts. Because the real world of education is so complex, there were many challenges involved in implementing and maintaining the research design. For example, group assignment dictated the timing of professional development sessions for teachers in a given group, and teachers’ schedules were often in conflict with the project’s. Teachers and site coordinators are highly mobile, resulting in considerable attrition. Furthermore, this study was carried out in several states, and these states differed in terms of the standardized achievement tests they used. Because school district officials are reluctant to add any additional standardized testing requirements over and above the tests they currently use, we had to rely on the standardized achievement test data provided by each district.

In addition, because the study was conducted in multiple districts, a great deal of effort was required

to get formal consent from each district to conduct the study. Because this study was conducted in school districts that were distant from each other, the project depended upon local school and district personnel to implement the research design. The study was vulnerable to the ongoing availability of these coordinators—when they moved on, communication with teachers in the district became highly problematic. In addition, a significant amount of time was devoted to coordinating logistical issues with school personnel representing the various school sites.

Limitations. Although the MPP materials were found to have a positive impact on student learning as

measured by the MPP Pitfalls Quiz, several limitations of the study should be noted. First, because of the large number of teachers who dropped out in the second year of the study, the data from the second year of the study were not considered usable, and we were therefore unable to compare the impact on student learning of one day versus four days of teacher professional development in MPP. Second, the standardized achievement test data were problematic. Different school districts provided different standardized tests to the researchers, and these different tests do not all measure the same underlying constructs. Moreover, there was a fair amount of missing standardized-test data, making the results of the analyses of these tests difficult to interpret. Finally, in terms of generalizability of the findings to other students, this study was implemented in five school districts across the country. Although every effort was made to select districts with diverse student bodies, caution is still needed in generalizing these results to other students. In addition, teachers participating in the study were volunteers and may not represent the full spectrum of teachers.


References

Behr, M., Lesh, R., Post, T., & Silver, E., (1983). Rational number concepts. In R. Lesh & M. Landau (Eds.). Acquisition of mathematics concepts and process (pp. 91-126) New York: Academic Press.

Borasi, R., (1994). Capitalizing on errors as “springboards for inquiry.” Journal for Research in Mathematics Education, 25 (2) 166–208.

Brenner, M. E. (1998). Development of mathematical communication in problem solving groups by language minority students. Bilingual Research Journal, 22.

Brown, J. S., Collins, A., & Duguid, P.(1989). Situated cognition and the culture of learning. Educational Researcher, (18) 1, 32-42.

Bruner, J.S. (1960). The process of education. Cambridge MA: Harvard University Press.

Bruner, J.S. (1966). Toward a theory of instruction. Cambridge MA: Harvard University Press.

Butler, F. A., & Bailey, A. L. (2002, Spring). Equity in the assessment of English language learners K-12. Idiom, 32(1), 1, 3. [Available from New York State TESOL at http://www.nystesol.org/]

Carpenter, T.P., & Moser, J.M. (1983). The acquisition of addition and subtraction concepts. In R. Lesh and M. Landau (Eds.) The acquisition of mathematical concepts and processes, (pp. 7–44). New York: Academic Press.

Carraher, D. W. (1996). Learning about fractions. In: L. P. Steffe, P. Nesher, G. A. Goldin, P. Cobb, & B. Greer (Eds.), Theories of mathematical learning (pp. 241–266). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Chamot, A.U., & O'Malley, J. M. (1994). The CALLA handbook: Implementing the cognitive academic language learning approach. Reading, MA: Addison-Wesley Publishing Company.

Cobb, P., Wood, T.L, & Yackel E. L. (1993). Discourse, mathematical thinking and classroom practice. In E. A. Forman, N. Minick, and C. A. Stone (Eds.). Contexts for learning: Sociocultural dynamics in children’s development (pp. 91-119). New York: Oxford University Press.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: Lawrence Erlbaum.

Cohen, E.G. & Intili, J.K., (1982), "Interdependence and management in bilingual classrooms", Final report II (NIE contract #NIE-G-80-0217), Stanford: Stanford University Center for Education Research Collier, V. (1995). Acquiring a second language for school. Directions in Language and Education, 1:4. Washington, DC: The National Clearinghouse for Bilingual Education. Available: http://www.ncbe.gwu.edu/ncbepubs/directions/04.htm

Cummins, J. (1980). The construct of proficiency in bilingual education. In J. E. Alatis (Ed.), Georgetown University Round Table on Languages and Linguistics: Current Issues in Bilingual Education, 81-103.

Echevarria, J. (1998, December). Teaching language minority students in elementary schools. CREDE Research Brief No. 1.

42

Echevarria, J., & Graves, A. (1998). Sheltered content instruction: Teaching English-language learners with diverse abilities. Boston: Allyn & Bacon.

Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row, Perterson & Company.

Fuson, K. C., (1992). Research on learning and teaching addition and subtraction of whole numbers. In G. Leinhardt, R. T. Putnam, and R.A. Hattrup (Eds.). The analysis of arithmetic for mathematics teaching (pp. 53-187). Hillsdale, NJ: Erlbaum.

Gersten, R. (1996). Literacy instruction for language minority students: The transition years. The Elementary School Journal 96 (3), 228-244.

Gibbons, P. (2002). Scaffolding language, scaffolding learning. Portsmouth, NH: Heineman.

Griffin, S. (1998, April). Fostering the development of whole number sense. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Heller, J. I., Gordon, A., Paulukonis, S., & Kaskowitz, S. (2000). Mathematics Case Methods Project discussion materials for students: Initial evaluation results. Unpublished manuscript.

Khisty, L. (1995). Making inequality: Issues of language and meanings in mathematics teaching with hispanic student. In W. G. Seced, Elizabeth Rennema, and Lisa Byrd Adajian (Eds.). New direction for equity in mathematics education (279-297). New York: Cambridge University Press.

Khisty, L. (1992, August). A naturalistic look at language factors in mathematics teaching in bilingual classrooms. Proceedings of the Third National Research Symposium on Limited English Proficient Student Issues: Focus on Middle and High School Issues. Washington D.C.

McKeon, D. (1994). When meeting common standards is uncommonly difficult. Educational Leadership, 51 (8).

Moss, J., & Case, R. (1999). Developing children's understanding of the rational numbers: A new model and an experimental curriculum. Journal for Research in Mathematics Education, (30) 2, 122-147.

National Research Council. (2001). Adding it up: Helping children learn mathematics. J. Kilpatrick, J. Swafford, and B. Findell, (Eds.) Mathematics Learning Study Committee, Center for Education, Division of Behavioral and Social Sciences and Education, p. 118. Washington, DC: National Academy Press.

Parker, M., & Leinhardt, G. (1995). Percent: A privileged proportion. Review of Educational Research, 65, (4), 421-481.

Pimm, D. (1987). Speaking mathematically: Communication in mathematics classrooms. London: Routledge,

Rabe-Hesketh, S., & Skrondal, A. (2005). Multilevel modeling using stata. College Station, TX: Stata Press.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis (2nd ed.). Thousand Oaks, CA: Sage.

Scardamalia, M. and Bereiter, C. (1983). The development of evaluative, diagnostic, and remedial capabilities in children composing. In M. Martlew (Ed.). The psychology of written language: A developmental approach (pp. 67-95). New York, NJ: John Wiley and Sons.


Secada, W. G., (1992) Race, ethnicity, social class, language, and achievement in mathematics. In D. Grouws (Ed.) Handbook of research on mathematics teaching and learning (pp. 623-660). New York: Macmillan.

Short, D. J. & Echevarria, J. (1999). The sheltered instruction observation protocol: A tool for teacher-researcher collaboration and professional development (Educational Practice Report No. 3). Santa Cruz, CA: University of California, Santa Cruz, Center for Research on Education, Diversity & Excellence.

Snijders, T.A.B., & Bosker, R. L. (1993). Standard errors and sample sizes for two-level research. Journal of Educational Statistics, 18(3), 237-259.

Solomon, J., & Rhodes N. (1995). Conceptualizing academic language (Research Rep. No.15). Santa Cruz: University of California, National Center for Research on Cultural Diversity and Second Language Learning.

Sowder, J. T. (1992). Making sense of numbers in school mathematics. In G. Leinhardt, R. Putnam, & R.A. Hattrup (Eds.). Analysis of teaching arithmetic for mathematics teaching (pp. 1-51). Hillsdale, NJ: Erlbaum.

StataCorp (2005). Stata statistical software: Release 9. StataCorp LP, College Station, TX.

Stigler, J.W., Gonzales, P. Kawanaka, T., Knoll, S., & Serrano, A. (1999). The TIMSS Videotape Classroom Study: Methods and finding from an exploratory research project on eighth-grade mathematics instruction in Germany, Japan, and the United States. Washington, DC: National Center for Education Statistics. Available: http://nces.ed.gov/timss. [July 10, 2001]

Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Vygotsky, L. S. (1962). Thought and language. Edited and translated by Eugenia Hanfmann and Gertrude Vakar. Cambridge, MA, NY: MIT Press; John Wiley.

Wearne, D., & Hiebert, J. (1989). Cognitive changes during conceptually based instruction on decimal fractions. Journal of Educational Psychology, 81 (4), 507–513.

Webb, N. (1989). Peer interaction and learning in small groups. International Journal of Educational Research, 13, 21-39.

Webb, N. (1985) Student interaction and learning in small groups: A research summary. In R. Slavin, S. Sharan, S. Kagan, R. Lazarowitz, C. Webb, & R. Schmuck (Eds.) Learning to cooperate, cooperating to learn (pp. 147 - 176). New York, Plenum.

Wong-Fillmore, L. (1982). Language minority students and school participation: What kind of English is needed? Journal of Education, 164, 143–156.

Student name ____________________________________ - or - Student ID # _______________

Teacher _______________________________________________ Date _____________________

Pitfalls QuizFall 2003

Grade 2 Quiz page 1 Mathematics Case Methods Project/WestEd © 2003

Fill in the circle next to the answer you choose.

1. Which point shows the number 23?

point W

point X

point Y

point Z

2. 4 + 58 =

62

98

63

458

3. 293 + 7 =

2910

993

300

2937


4. Find the difference.

3

7

13

25

5. 798 + 10 =

708

898

808

7918

6. What number is 10 more than 482?

483

4812

582

492


7. What number is 10 less than 821?

820

831

721

811

8. 4 + 5 = ___ + 3

2

6

9

12

9. 8 + 1 = ___ + 3 + 2

4

5

9

10


10. 5 + ___ = 12 – 4

3

7

8

11

11.

73

83

713

63

12. Find the difference.

35 – 29 =

14

6

64

7


Write the Answer

13. Estimate what number point B shows.

Answer: ____________

14. Draw a point on the number line to show the number 35.

15. Find the difference between 13 and 7.

Answer:_______________


16.49 + 3 = _____________

17. Ann’s aunt is 23 years old. Her uncle is 31. What is the differencein their ages?

Answer:____________

18. Jorge has 15 turtles and 27 fish. How many animals does hehave in all? Show your work.

Answer:______________


Teacher _______________________________________________ Date _____________________

Pitfalls Quiz - FractionsFall 2003


Fill in the circle next to the answer you choose.

1. What fraction of the circle is shaded?

3

5

5

8

5

3

3

8

2. Which number should go in the box?

3

4 =

20

5 15

12 19


3. What fraction of the square is shaded?

1

16

2

5

1

8

2

7

4. Which of these pictures shows

1

4 shaded?

only picture A only picture C

only picture B both A and B show

1

4


5. Which point shows

1

4?

point A point C

point B point D

6. Draw and label a point on the number line to show

1

3 .


7. What fraction of the set of triangles is shaded?

3

8

5

8

3

5

5

3

8. Which picture shows

2

3 shaded?

not given


9. Which shows

16

3 written as a mixed number?

1

6

33

1

6

5

1

316

1

3

10. Which shows 4

2

3 written as an improper fraction?

6

3

14

3

42

3

8

3


11. Which number is equal to this mixed number?

6

5

4

11

4

65

4

6

4

57

1

4

12. Which fraction is greater?

3

6 or

8

18

3

6They are equal

8

18


13. Which number is greater? 4

1

3 or

12

3

4

1

3They are equal

12

3

14. Which fraction is more than

5

8?

5

9

7

15

5

6

2

5


15.

3

10 +

2

5 =

1

2

1

3

7

10not given

16.

1

2 +

2

3 =

7

121

1

6

3

51

1

3


17. Patty bought

3

8 yard of gold chain and

1

4 yard of silver chain.

What is the total length of her chains?

5

8 yards

1

3 yards

3

32 yards

4

12 yards


Teacher _______________________________________________ Date _____________________

Pitfalls Quiz – PercentsFall 2003


Circle the letter next to the answer you choose.

1. What percent of the rectangle is shaded?

6% 60%

46% 64%

2. What percent of the rectangle is shaded?

10% 150%

15% 1.5%


3. What percent of this diagram is shaded?

25% 75%

34% 60%

4. What percent of the picture is shaded?

14% 33%

25% 39%


5. What percent of the picture is shaded?

35% 65%

60% 70%

6. Estimate the percent of students who don’t have pets.

80% 30%

60% 90%


7. About what percent of the students are in the 5th and 6th

grades combined? ,

75% 47%

63% 50%

8. What percent is equal to

5

8?

16% 58%

85% 62.5%



910

?

9% 90%

0.9% 910%


8

4 ?

84% 2%

50% 200%

11. Which set of numbers is in order from least to greatest?

6% 0.6

1

60.6

1

66%

6%

1

60.6

1

66% 0.6


12. Which set of numbers is in order from least to greatest?

0.5

1

828%

1

828% 0.5

1

80.5 28% 28%

1

80.5

13. Which of the following numbers is closest to 1?

0.12 97% 1.2

56

0.12 1.2

97%

56

14. What is 5% of 120?

6 60

24 115


15. My brother has 150 cd's. 60% of them are rap music. Howmany the cd's are rap?

100 90

60 40

16. What is 30% of 170?

140 17

510 51

17. What is 300% of 12?

3 3.6

36 300



18

45?

25% 2.5%

4% 40%

19. 12 is 25% of what number?

2 48

3 60

20. In a recent survey, 7 out of every 20 kids chose chocolate astheir favorite ice cream. What percent of the kids preferchocolate ice cream? Show your work.

Answer:____________%

Teacher Information Form Fall 2003

Math Case Methods Project, WestEd Teacher Information Form [030718.ns] 1

First name:_______________________________ Last name:___________________________________ Mailing address: ______________________________________________________________________ City, state, zip: _______________________________________________________________________ Day phone: _____________________________ Evening phone: _________________________________ Email: ______________________________________________ Do you check email regularly?_________ School name: __________________________________ District________________________________ 1. Which of the following best describes the setting of your school or district?

1 Urban

2 Suburban

3 Rural

4 Other: ________________________________

2. Which grade(s) do you currently teach? ______________________________________________

3. How many years have you taught prior to this school year? _________ years

4. How many years have you taught math at any grade level? _________ years

5. How many years have you taught math at your current grade level? _________ years

6. What math textbook or curriculum do you currently use? _____________________________________

7. Which of the following best describes your formal math education? (Check highest level.)

1 High school math courses

2 Some college math courses

3 B.A. or B.S. degree in math

4 Graduate level coursework or degree in math

8. Approximately how much time have you spent participating in math professional development programs during the last three years?

1 None

2 Up to 2 days (16 hours or less)

3 3 to 6 days (17-48 hours)

4 7 days or more (Please specify approximate number of hours: _______ hours)

9. Type of teaching credential: __________________________________________

Teacher Information Form [030716.ns] 2

10. Your gender: ___________________________

11. Your ethnic identity (check one or more)

1 American Indian or Alaskan Native

2 Asian (Chinese, Japanese, Korean, Asian Indian, other Asian)

3 Black or African American, non-Hispanic

4 Filipino 5 Latino, Spanish-Origin, Hispanic

6 Pacific Islander (Native Hawaiian, Guamanian, Samoan)

7 Southeast Asian (Cambodian, Laotian, Vietnamese, other Southeast Asian)

8 White

9 Other: _________________________

12. Do you teach in a self-contained classroom?

1 Yes

2 No

3 Not applicable

13. Number and breakdown of students:

(a) (If yes to #12) Number of students in class: (a) _________

(b) (If no to #12) Number of classes you teach each week: (b) _________

(c) (If no to #12) Average number of students you teach each week:

Approximate number of your students in each of the following categories (if none, write “none”):

(d) Special Education or Resource: (c) _________

(e) GATE: (d) _________

(f) Free or Reduced Lunch: (e) _________

(g) English Language Learner: (f) _________

Approximate number of your students in each of the following ethnic groups (if none, write “none”):

(h) American Indian or Alaskan Native: (g) _________

(i) Asian (Chinese, Japanese, Korean, Asian Indian, Other Asian): (h) _________

(j) Black or African American, non-Hispanic: (i) _________

(k) Filipino: (j) _________

(l) Latino, Spanish-Origin, Hispanic: (k) _________

(m) Pacific Islander (Native Hawaiian, Guamanian, Samoan): (l) _________

(n) Southeast Asian (Cambodian, Laotian, Vietnamese, Other Southeast Asian): (m) _________

(o) White: (n) _________

(p) Other: (o) _________

Math Pathways and Pitfalls – July 2004 1

Math Pathways and Pitfalls Questionnaire For Teachers Using the Lessons During 2003-2004

Did we accomplish our goals to make lessons that work for kids and teachers? This questionnaire will help us find out. It takes only 15 to 20 minutes, so please do it now! Then put it in the self-addressed envelope and mail it back. We count on teachers like you to help us make a program that works, so thank you very much for your help! DIRECTIONS: Write a number from 1 to 10 in each blank to tell how much you agree with the statements below. Write NA for statements that don't apply. You will be identified only by a number, so please be candid.

1 2 3 4 5 6 7 8 9 10 Don't agree at all Neutral Strongly agree

Rating 1. Most of my students really liked the Math Pathways and Pitfalls lessons. ______

2. The format of the lessons was easy for most of my students to follow. ______

3. The teaching guide is clearly organized and easy to follow. ______

4. These lessons were helpful for my high-performing students. ______

5. These lessons were helpful for my low-performing students. ______

6. These lessons were helpful for my non-native English-speaking students. ______

7. These lessons were helpful for my special needs students. ______

8. When working on the Starter Problems, many of my students made the same or similar pitfalls to the ones in the lessons.

______

9. After completing the set of lessons this year, my students understood the math topic in the lessons better than students in past years.

______

10. After completing the set of lessons this year, all of my students are making fewer pitfalls.

______

11. The language support provided in the teaching guide and in the student lessons (for example, the Math Words) was helpful.

______

12. The Discussion Builders were very important in helping my students learn how to discuss mathematics.

______

13. The Discussion Builders helped my students learn to discuss other content areas besides math.

______

14. I used most of the questions in the teaching guides, along with some of my own, to help me conduct the discussions with my class.

______

15. I made up most of my own questions to conduct the discussion with my class, instead of using those in the teaching guide.

______


16. Prior to the Math Pathways and Pitfalls lessons, I conducted thought-provoking math discussions with my class several times a week.

______

17. I learned a lot about using thought-provoking discussions to teach math through the Math Pathways and Pitfalls lessons.

______

18. I used the Getting Started tasks at the beginning of the school year to help my students learn how to use the Discussion Builders.

______

19. My students really liked the video of other students doing a Math Pathways and Pitfalls lesson. (Write NA if you didn't show the video.)

______

20. The video played an important role in helping my students learn how to do the Math Pathways and Pitfalls lessons.

______

21. The language support for students provided in the teaching guides and in the student lessons helped me prepare students for the special mathematical vocabulary and symbols in the lesson.

______

22. I used all of the Mini-Lesson 1 lessons (multiple-choice questions) with my students. (Write a 10 if you don't think you skipped any.)

______

23. I used all of the Mini-Lesson 2 lessons (requiring explanations in writing) with my students. (Write a 10 if you don't think you skipped any.)

______

24. The Mini-Lessons that I used helped my students consolidate their understanding.

______

25. I read the Mathematical Background section of the teaching guide for almost every lesson. (Write a 10 if you don't think you skipped any.)

______

26. Substituting a Math Pathways and Pitfalls lesson for two of my regular lessons about once a month worked well.

______

27. I would rather teach the whole set of Math Pathways and Pitfalls Lessons as a single unit, rather than throughout the year.

______

28. I would love to use the Math Pathways and Pitfalls lessons again next year.

______

29. The students in our school would benefit greatly if all of the teachers used the Math Pathways and Pitfalls lessons.

______

30. I believe that the Math Pathways and Pitfalls lessons helped most of my students learn the math concepts and prevent pitfalls.

______

31. The Math Pathways and Pitfalls lessons supported hard to teach topics in our math textbook.

______

32. The Math Pathways and Pitfalls lessons went beyond our textbook in a positive way.

______

33. I feel that my teaching has improved as a result of using the Math Pathways and Pitfalls materials.

______


Let's Hear from You

Please write a few sentences about something that was important to you about using the Math Pathways and Pitfalls lessons. Perhaps you have a story about a particular student or lesson; or you can talk about something that you or your students learned; or you may have something to say about the usability of the lessons. (Attach another sheet if you'd like.) We also value your input for improving the program. Please write a few sentences about what didn't work for you or your students, and if possible, provide suggestions for improvement.

1

February 9, 2005 Dear Math Pathways and Pitfalls Implementation Study Teachers: We wish to thank you again for agreeing to participate in the additional implementation study. Here are the directions we'd like you to follow: 1. We are sending you a questionnaire to complete IMMEDIATELY

AFTER you teach MPP Lesson 5, including the mini-lessons. 2. Please collect the YOUR TURN pages for Lesson 5 from two of

your students. (Just select two from a stack at random.) 3. Return the YOUR TURN pages to us in the enclosed envelope along

with the completed questionnaire. As a reminder, prior to April 1, we will send additional instructions for how to audiotape Lesson 6.

Thank you from all of us for your contribution to the teaching profession and to our program. Sincerely, Carne Barnett-Clarke, Director Alma Ramirez, Co-Director

300 Lakeside Drive 25th Floor, Oakland, CA 94612; Phone (510) 302-4253; Fax (510) 302-4242; [email protected]

2

Math Pathways and Pitfalls Questionnaire For the 2005 ADDITIONAL Implementation Study

MPP Lesson 5 Title DON'T SQUEEZE THE DIGITS

Teacher ____________________ Grade ___2____Date __________

How do teachers prepare to teach the lessons? How easy are the lessons to use? This 20 minute questionnaire will help us find out. We count on teachers like you to help us make a program that works so, thank you very much for your help! DIRECTIONS: Complete this questionnaire RIGHT AFTER YOU TEACH LESSON 5, including the MINI - LESSONS! Mail it back in the self-addressed envelope. Part 1 - Check the response that most closely corresponds to what you did with Lesson 5. Refer to a copy of the lesson and teaching guide for Lesson 5 as needed.

1. How did you use the Mathematical Background section of the teaching guide for Lesson 5?

I read it closely prior to teaching the lesson. I skimmed it prior to teaching the lesson. I did not read it. Other, please explain: _______________________________________________

2. How did you use the questions in the teaching guide for Lesson 5? I used all or most of the questions in the teaching guide and none of my own. I used all or most of the questions in the teaching guide and some of my own. I used few or none of the questions in the teaching guide and mostly used ones I made up. Other, please explain: _______________________________________________

3

3. About how long did you take to teach the core lesson, which does not include the

mini–lessons? Note: A class period is about 45–50 minutes. Less than 1 class period 1 class period 2 class periods Other, please explain: _____________________________________________

4. What did you do while your students were in paired discussions? Walked around and sometimes interacted with students about their work. Walked around but seldom interacted with students about their work. I did not walk around or interact with students about their work but waited for them to finish before proceeding. The students did not have paired discussions.

5. When do your students use the Discussion Builders?

The children use the Discussion Builders in math and other subject areas. The children use the Discussion Builders only in math. The children seldom use the Discussion Builders in any subject area.

6. Estimate about how often you think the students use the Discussion Builders during a

typical day. More than 15 times a day Between 5 and 15 times a day 1 to 5 times a day Never

4

Part 2 - Let's Hear from You - Free Response Please be candid in your responses. This will help us ensure that the program works for more teachers and students! Please refer to the Lesson 5 materials as needed. 1) If you read the Mathematical Background section of the teaching guide, please

explain if and how it was helpful. If you did not, please explain why. 2) If you added your own questions to the discussion prompts provided for Lesson 5,

please list the ones you remember, and say a bit about how you felt they helped the discussion.

3) Date of core lesson Approximate number of minutes

Date of mini-lesson 1 Approximate number of minutes Date of mini-lesson 2 Approximate number of minutes

Write NA in the blank if you didn't do a mini-lesson and briefly explain why you did

not do it:

5

4) Below is a list of each of the core lesson parts. Put a check in the box to show which

of the following lesson parts you taught:

Day 1

Purpose Math Words Starter Problem Discussing the OK Discussing the Oops Things to Remember

Day 2

Review of Day 1 Our Turn Your Turn

If you did not teach all the parts of the lesson, please explain why for each one you did not teach.

5) Did you find the transparency with the Purpose, Math Words and Starter Problem

helpful? Explain how. If you did not use the transparency, please explain why. 6) The teaching guide suggests that students work in pairs on the first Our Turn problem

and that the class discuss 1 or 2 solutions before moving on to the second problem, and so on.

Did you follow this suggested procedure? If so, what did you think about it? If not, please explain what you did differently and why.

6

7) The teaching guide suggests that students work individually for about 5 minutes on the Your Turn problems. Then the teacher goes over the problems with the class, and the students revise in ink if needed. Did you follow this suggested procedure? If so, what did you think about it? If not, please explain what you did differently and why.

8) How did the paired discussions work with the students? If you did not have students

discuss some ideas in pairs, please explain why. 9) If you thought the Discussion Builders were helpful, please explain what impact they

had on your students or the discussion. If you didn't think they were helpful, explain why.

10) Please write a few sentences about something that was important to you about Lesson

5. Perhaps you have a story about a particular student with this lesson; or you can talk about something that you or your students learned; or you may have something to say about the usability of the lesson. (Attach another sheet if you'd like.)

Mathematics Pathways and Pitfalls Materials Protection Agreement [put line break here]

and Affidavit of Test Administration

By signing below, I agree to the following provisions: I will administer the Pitfalls Quiz, a math assessment for students, without providing assistance except as noted in the instructions. I will not view the test prior to or after administering it. I will not copy or reproduce the test or any questions on the test. I will not grade, score, or assess the students’ performance on the test. I will return the tests, unscored, to WestEd. Any Math Pathways and Pitfalls materials developed by the Mathematics Case Methods Project/WestEd are privileged and confidential and may not be used by anyone without express written consent from the Mathematics Case Methods Project/WestEd. In particular, I will not share the materials, print or video, with other teachers until the field test has been completed and I receive the published set of materials as part of my remuneration.

Printed Name Site

Signature Date

the effects of math pathways and pitfalls on students’ … · 2013. 8. 2. · [mpp nsf final...

Documents