curriculum implementation in secondary mathematics...

41
DRAFT Curriculum Implementation in Secondary Mathematics Classrooms 1 Title: The Development of Multiple Measures of Curriculum Implementation in Secondary Mathematics Classrooms: Insights from a Three-Year Curriculum Evaluation Study Author Team: James E. Tarr, Melissa D. McNaught, and Douglas A. Grouws Complete Mailing Addresses: James E. Tarr Department of Learning, Teaching and Curriculum 303 Townsend Hall University of Missouri Columbia, MO 65211-2400 Phone: 573.882.4034 Fax: 573.882.4481 E-mail: [email protected] Melissa D. McNaught Department of Teaching and Learning Mathematics Education N287 Lindquist Center University of Iowa Iowa City, Iowa 52242 Phone: 319.335.5433 Fax: 319.335.5608 E-mail: [email protected] Douglas A. Grouws Department of Learning, Teaching and Curriculum 303 Townsend Hall University of Missouri Columbia, MO 65211-2400 Phone: 573.884.5982 Fax: 573.882.4481 E-mail: [email protected] Abbreviated Title: Curriculum Implementation in Secondary Mathematics Classrooms

Upload: others

Post on 07-Jul-2020

15 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

1

Title: The Development of Multiple Measures of Curriculum Implementation in Secondary Mathematics Classrooms: Insights from a Three-Year Curriculum Evaluation Study Author Team: James E. Tarr, Melissa D. McNaught, and Douglas A. Grouws Complete Mailing Addresses: James E. Tarr Department of Learning, Teaching and Curriculum 303 Townsend Hall University of Missouri Columbia, MO 65211-2400 Phone: 573.882.4034 Fax: 573.882.4481 E-mail: [email protected] Melissa D. McNaught Department of Teaching and Learning Mathematics Education N287 Lindquist Center University of Iowa Iowa City, Iowa 52242 Phone: 319.335.5433 Fax: 319.335.5608 E-mail: [email protected] Douglas A. Grouws Department of Learning, Teaching and Curriculum 303 Townsend Hall University of Missouri Columbia, MO 65211-2400 Phone: 573.884.5982 Fax: 573.882.4481 E-mail: [email protected] Abbreviated Title: Curriculum Implementation in Secondary Mathematics Classrooms

oscar
Typewritten Text
Page 2: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

2

Introduction

In comparative curriculum research, achievement scores provide only a partial picture of

student knowledge acquisition. Factors such as students’ opportunity to learn and how teachers

present mathematical content also impact what students learn and how well they learn it.

Therefore, attention to curriculum implementation is a high priority in the Comparing Options in

Secondary Mathematics: Investigating Curriculum (COSMIC) project. In the remainder of this

chapter, the importance of examining curriculum implementation and the methods the project

uses to garner this information are described and discussed. We begin with an overview of the

context in which the instruments were developed, the COSMIC project, and then describe our

conceptual approach to instrument development, which includes the use of two data collection

perspectives and the gathering of data in several grain sizes. In the third section of the chapter,

we outline our instrument development process, describe the instruments developed, and provide

validity and reliability data. In the final two sections, we report on how the instruments have

been used, provide some illustrative findings, and suggest future directions.

The COSMIC Project

The goal of the COSMIC project is to examine student mathematical learning associated

with secondary mathematics curriculum programs of two types – a subject-specific approach,

where students follow a course sequence of Algebra I, Geometry, and Algebra II, and an

integrated content approach, where students follow a course sequence of Integrated I, Integrated

II, and Integrated III. The study is situated in 11 high schools in five states where each school

offers students a choice of either curricular approach, with more than 100 teachers and 5,800

students participating in the first two years of the project. Student learning over a three-year

period is tracked using standardized measures of achievement along with project-designed tests

Page 3: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

3

to assess depth of knowledge, skills acquisition, ability to reason mathematically, and conceptual

development. Moreover, we pay careful attention to teachers’ implementation of curricular

materials in order to draw causal inferences between curriculum type and student learning.

The research questions related to the implementation component of the study are:

(1) How do teachers use textbooks with different approaches to content organization in

the ongoing process of mathematics teaching?

(2) What is the relationship between curriculum implementation and student learning?

To answer these questions, the project has developed and used multiple measures of curriculum

implementation, including teacher surveys, classroom observation tools, Textbook-Use Diaries,

and Table of Contents Records.

The Importance of Measuring Curriculum Implementation

One impetus for undertaking the COSMIC project was the call for more comparative

curriculum research put forth in the National Research Council report On Evaluating Curricular

Effectiveness: Judging the Quality of K-12 Mathematics Evaluations (NRC, 2004). The

recommendations in the report provided the guidelines for the development of our

implementation measures. The COSMIC project used the NRC report authors’ definition of

curriculum as the student textbooks and auxiliary materials that accompany them, including the

teacher guides. The representative textbook series of the integrated mathematics curriculum

approach in this study is Contemporary Mathematics in Context (Coxford et al., 2003), also

known as the Core-Plus Mathematics Project (hereafter Core-Plus). Several subject-specific

textbook series, all with similar content organizations, represented the subject-specific

Page 4: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

4

curriculum approach (see COSMIC Technical Report 1.8 for additional details about the

textbooks included in the study1).

To develop an understanding of the learning that occurs through the use of a particular

curriculum, researchers must consider the way in which the curriculum materials are used in the

classroom, including how closely aligned classroom instruction is to the intent of the textbook

authors. The NRC (2004) refers to this alignment as “implementation fidelity” and describes it as

“a measure of the basic extent of use of the curricular materials” (p. 114). George, Hall, and

Uchiyama (2000) emphasize that assessing implementation is key to productively evaluating

curriculum programs. The NRC expanded on this admonition, stating that evaluations of

curricula require a measure of implementation in order to draw conclusions regarding their

influence on student achievement. More specifically, they stated,

Evaluations should present evidence that provides reliable and valid indicators of the

extent, quality, and type of the implementation of the materials. At a minimum, there

should be documentation of the extent of coverage of curricular materials (what some

investigators referred to as ‘opportunity to learn’) and the extent and type of professional

development provided. (p. 194)

Teachers are active decision makers with regard to how and when mathematical content

is taught and, as such, they are influenced by the instructional materials available to them as well

as events that occur within the mathematics classroom (Ben-Peretz, 1990; Clandinin & Connelly,

1992; Clarke, Clarke, & Sullivan, 1996; Remillard, 1999; Remillard and Bryans, 2004). For

these reasons, the ways in which teachers use the same curriculum vary. The need to document

teacher use of curriculum materials is supported by research suggesting that curriculum

1 Available by request from the first author.

Page 5: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

5

implementation is an uneven process within and across schools (Grouws, 1992; Grouws &

Smith, 2000; Jackson, 1992; Kilpatrick, 2003; Senk & Thompson, 2003). Kilpatrick argued,

Two classrooms in which the same curriculum is supposedly being ‘implemented’ may

look very different; the activities of teacher and students in each room may be quite

dissimilar, with different learning opportunities available, different mathematical ideas

under consideration, and different outcomes achieved. (p. 473)

The next section outlines our development of implementation tools and the steps we took

to ensure that we collected data to assess implementation from multiple perspectives using

different lenses and various grain sizes. The focus was on developing multiple measures with

known validity and high reliability.

Conceptualizing Implementation Fidelity: Two Lenses

There are varying perspectives on the issue of curriculum implementation. While some

argue that a truly faithful implementation is not possible (Remillard, 2005), there is substantial

agreement among researchers that it is an important variable to consider in analyzing data from

curriculum studies. In the COSMIC project, we conceptualized implementation fidelity along

two dimensions, content and presentation.

Through the content fidelity lens, we examine what mathematics content in the intended

curriculum (the textbooks in this case) was taught as part of classroom instruction. We

conceptualized this dimension as lying on a continuum from using the curriculum content

exactly as it is written in the textbook to the other extreme of regularly skipping content or

substituting content for what is in the textbook. In COSMIC, content fidelity is assessed at both

the course level and the lesson level. At the course level, we broadly examine what content from

Page 6: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

6

the textbook was used on a daily basis across the school year. At the lesson level, we look in-

depth at the extent of textbook use during a specific lesson or in consecutive lessons.

Through the presentation fidelity lens, we examine how textbook lessons are presented to

students and assess the way students are expected to engage with the textbook material during

the mathematics class period. For example, textbooks that are purportedly based on research

regarding how students learn mathematics and embody recommendations of the National

Council of Teachers of Mathematics’ Standards documents (NCTM, 1989; 1991; 1995; 2000)

often call for students to work in small group settings, to engage in discussion of ideas, to

discover skills and procedures, and so on. In our project, we assess each of these facets of

implementation through classroom observations, and therefore presentation fidelity was

measured at the lesson level, but not at the course level.

Textbook Authors’ Conceptualizations of Implementation Fidelity

Prior to the development of tools to measure curriculum implementation, we engaged in

discussions about how validity of such instruments could be achieved. As a project team, we

certainly could have outlined what we believe constitutes a faithful implementation of a textbook

curriculum, but would those who designed and developed the textbooks share our conceptions of

implementation fidelity? We had some misgivings about our capacity to design valid measures

of teachers’ enactment of the written curriculum without consideration of the textbook authors’

perspective. Consequently, we decided to “go to the source” and directly interview textbook

authors about their conceptualizations of implementation fidelity with respect to both content

and presentation. We selected the Project Director of the integrated textbook series as well as an

author of one of the subject-specific textbook series, namely one who consistently served as a

member of the authorship team through numerous editions of its publication. Both authors were

Page 7: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

7

granted consent from their respective publishing companies; as such, we considered them to be

“spokespersons” for their products. However, we acknowledge that it is plausible that different

members of each authorship team could have provided alternative conceptions of

implementation fidelity.

In preparation for our author interviews, we examined the various components of

curriculum programs, including student and teacher materials. Additionally, we identified the

structural components within each textbook, including the organizational structure such as units,

lessons, and elements that comprise individual lessons. Our analysis was specific to the two

types of textbooks, subject-specific and integrated, and yielded distinctive structures for each.

In the integrated curriculum studied, the textbooks are composed of Units broken down

into Lessons that contain multiple Investigations. Each Lesson is structured similarly utilizing

the following components: Launch, Explore, Share and Summarize, and Apply, with the

accompanying activities spanning multiple days. Exercises, known as MORE (Modeling,

Organizing, Reflecting, Extending) problems accompanying each Lesson, are typically assigned

over a period of days. These assignments offer students additional opportunities to apply, reflect

on, and extend their knowledge of the concepts developed during the Investigations within the

specific Lesson. Each of these Lesson components serves a distinctive purpose, as depicted in

Table 1.

! INSERT TABLE 1 – LESSON COMPONENTS "

The subject-specific curriculum typically includes an Algebra I, Geometry, and Algebra

II textbook. The books are organized by chapters consisting of 6 to 15 single day lessons, with

each lesson organized into three main components. Although the labels the textbook authors give

Page 8: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

8

to each component vary, they can be logically categorized as (a) Lesson Preview; (b) Teach; and

(c) Practice and Apply. The purpose of each lesson component is presented in Table 1.

Based on the structural components identified, we devised a separate interview protocol for

an author of the Core-Plus materials and for an author of a popular Algebra I textbook. During

our interviews with the textbook authors, we asked questions that focused on the authors’ views

about a faithful implementation of the curriculum materials and gathered information about how

the authors envisioned the lesson components being enacted. For example, for both textbook

authors we asked: “If you were to visit a classroom in which a teacher was implementing your

curriculum materials ‘faithfully,’ what would you expect to observe?” and “What might you

observe that would lead you to conclude, ‘That is not what we intended when we wrote these

textbooks’?” We also inquired about each of the lesson components in Table 1. For example, we

asked the author of Core-Plus, “Are teachers expected to explicitly state the lesson objectives?

Why or why not?”, “How are teachers expected to deal with vocabulary? Should they explicitly

define words that may be new to the students?”, and “What is meant by ‘Teacher is director and

moderator’ (p. 12 of the Implementation Guide)?” Our overarching goal was to develop a

classroom visit protocol that contained specific observable teacher behaviors that textbook

authors would expect to see in a faithful implementation of their curriculum materials. We found

the textbook authors were forthright in offering their visions of curriculum enactment, and both

emphatically agreed that no curriculum is “teacher proof.” From both interviews, we recognized

the importance of classroom observations to assess alignment with textbook authors’

conceptualizations of implementation fidelity.

Instruments to Measure Implementation Fidelity

Page 9: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

9

Evidence concerning implementation can be, and should be, measured from several

perspectives. In the COSMIC project, we developed instruments that separately reflected the

perspectives of the researcher and the teacher. In doing so, we were able to gather data that

allowed us to examine patterns in implementation within and across different perspectives.

Researcher Perspective

We used the Classroom Visit Protocol (CVP) to document the use of materials and

classroom activities from the researcher perspective. The CVPs we developed took account of

previous work in describing classroom instruction (e.g., Romberg & Shafer, 2003; Tarr, Reys,

Reys, Chávez, Shih, & Osterlind, 2008), but were more specific in nature than the tools used in

previous research. For example, we had separate CVPs for the integrated curriculum and the

subject-specific curricula, each designed to embody the components of their respective lesson

structures (discussed subsequently in this section).

Although there were separate CVPs for each curriculum type, there were common

elements in the CVPs, too. For example, in the first of three parts of the form, during each

classroom visit, the observer recorded anecdotal evidence of particular presentation features

being implemented and noted grouping of students, examples used from the textbook, questions

posed by the teacher, student responses, homework assignments, and use of assessments. At the

conclusion of the class period, the researcher used these data to complete the second and third

parts of the CVP, namely, a Lesson Summary form and a Classroom Learning Environment

scale.

! INSERT FIGURE 1 – EXCERPT OF OBSERVATION PROTOCOL "

Page 10: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

10

Lesson Summary. Based on the information garnered from the interviews with the

authors and examination of the curriculum materials specific to each sample site2, we developed

the Lesson Summary Form to document specific classroom events and activities such as

interactions between the teacher and students, interactions among students, instructional

materials used by the teacher or students, and exercises or examples worked on or discussed. An

excerpt of the CVP for the integrated curriculum appears in Figure 1. Note that in the Launch

component of Core-Plus the textbook author expected to see four behaviors in a “faithful”

implementation of a Launch. We generated similar behavioral checklists from author interviews

for each of the lesson components denoted in Table 1. These measures require a dichotomous

judgment by the observer: “Did I see this action?” or “Did I not see this action?” The checklist

thus provides a low inference measure of fidelity to the structural components of the lesson

elements. Analyses of the checklists provide a means of judging whether appropriate attention is

given by the teachers to various lesson parts when examined within and across lessons.

As part of the researcher perspective we also developed high inference measures of

content fidelity and presentation fidelity. These measures focused, respectively, on the content

taught aspect of the curriculum and on the pedagogical aspect of the curriculum. We wanted a

numeric assessment and found that we could reliably measure these aspects of the enacted

curriculum if we coupled observer training with a carefully developed rubric for each scale.

Thus, we developed descriptions of “high,” “moderate,” and “low” Content Fidelity ratings and

Presentation Fidelity ratings (Table 2) that represented holistic judgments of the observed

behaviors. Rather than offer three distinctive descriptions (“high,” “moderate,” and “low), we

2 The curriculum for the subject-specific course was taught from one of several textbooks, among these the most widely used were the Glencoe Publishing Company textbook, Algebra I (Holliday, et al., 2005) and McDougal Littell, Algebra I (Larson, et al., 2004).

Page 11: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

11

settled on a 5-point scale, such that ratings of 2 and 4 could serve as compromise positions

between adjacent anchors. We provided more detailed descriptions and examples for observers

in the coding training manual.

! INSERT TABLE 2 – RUBRIC FOR CONTENT & PRESENTATION FIDELITY "

Classroom Learning Environment. Our experience in previous curriculum evaluation

projects led to the decision to document selected elements of the classroom learning

environment. Our rationale for doing so was based on two factors. First, in an earlier study (Tarr,

Reys, Reys, Chávez, Shih, & Osterlind, 2008) classroom learning environment factors offered

predictive power in analyses of student achievement data. Second, we sought to document

particular features of the classroom learning environment that are considered important across

different types of curriculum (e.g., development of conceptual understanding). In the COSMIC

Classroom Learning Environment measure, there are 10 elements that collectively represent the

classroom environment. These 10 elements are classified into three themes: Reasoning about

Mathematics, Students’ Thinking in Instruction, and Focus on Sense-making. Using a well-

defined rubric for each of the 10 elements, observers rendered ratings from 1 to 5, with a 1

indicating the absence of the feature and a 5 indicating a strong presence of the feature during the

observed lesson. These ratings offered a common measure across curriculum types. However,

because the Classroom Learning Environment is curriculum independent, for the purpose of this

paper we have limited our discussion of it.

Teacher Perspective

To gain the teacher perspective, we had teachers report information by completing two

surveys, Textbook-Use Diaries, and a Table of Content Record. These instruments, described in

the sections that follow, served different, yet related, purposes.

Page 12: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

12

Teacher Surveys. Each teacher completed two written surveys adapted from the 2000

National Survey of Science and Mathematics Education (Weiss, Banilower, McMahon, & Smith,

2001). The first survey, titled Initial Teacher Survey, was completed at the beginning of each

school year and was used to gather teacher demographic and background data as well as teacher

beliefs about teaching and learning and teacher involvement in professional development. In

analyses of student achievement data, we statistically controlled for a variety of teacher-level

variables such as the number of years of teaching experience, years teaching the textbook

curriculum, and quantity and nature of professional development. Data from the Initial Teacher

Survey were collected because such teacher-level variables have the potential to explain

variation in student outcomes. The second survey, titled Mid-course Teacher Survey, was

completed at in the middle of the year gathering data about teacher use of curriculum materials.

More specifically, teachers were asked about the use of their particular textbook during

instruction, perception of the quality of the textbook, use of key instructional practices, use of

graphing calculators, assignment of homework, and assessment practices.

Table of Contents Record. The Table of Contents Record is a textbook-specific indicator

of content implementation that mirrors the table of contents of the particular textbook. For each

section of each Chapter, teachers are asked to indicate whether the content of each section

(Investigation) was (a) taught primarily from the textbook; (b) taught from the textbook with

some supplementation; (c) taught primarily from alternative(s) to the textbook; or (d) not taught.

A sample of part of a Table of Contents Record for the integrated textbook appears in Figure 2.

These records provide an indication of content coverage and the extent to which the teachers

utilized their textbook during instruction. More specifically, this instrument enables us to

determine what mathematics content students were afforded opportunities to learn, and whether

Page 13: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

13

some mathematics content is disproportionately emphasized or skipped. Moreover, such

opportunity-to-learn data is particularly useful in analyses of student achievement data.

! INSERT FIGURE 2 – SAMPLE TABLE OF CONTENTS RECORD "

Textbook-Use Diary. The Textbook-Use Diary serves as a daily record of how teachers

engaged with curriculum materials during instruction. Questions pertaining to the subject-

specific curricula (Figure 3) ask for information about examples the teacher used during

instruction, textbook problems assigned to the students, and printed materials other than the

textbook (e.g., auxiliary curriculum resources, teacher-developed worksheets) used during

instruction. Questions pertaining to the Core-Plus curriculum ask for information regarding

which portions of the Think About This Situation, Investigations, and Checkpoint were used

during instruction and the assignment of On Your Own and Modeling-Organizing-Reflecting-

Extending (MORE) problems. Going beyond the Table of Contents Record, which reveals

information about what mathematics students are afforded opportunities to learn, the Textbook-

Use Diary provides information about the emphasis teachers placed on the content. Specifically,

comparisons can be made between the number of days a teacher spends on a particular section

and the number of instructional days recommended by authors in the scope and sequence section

of the textbook.

! INSERT FIGURE 3 – SAMPLE TEXTBOOK-USE DIARY "

Pilot Testing the Classroom Visit Protocol

Initially, a subject-specific curriculum protocol and an integrated curriculum protocol were

piloted in Algebra I and Integrated Course 1 classrooms, respectively. In early phases of piloting,

our primary objective was to determine whether our extensive observation protocol was

manageable. More specifically, we sought to determine whether it was possible for an observer

Page 14: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

14

to attend to all of the features of curriculum implementation without the use of video or

audiotape recordings of instruction. To gauge classroom observers’ facility with completing the

CVP, over the course of one month three COSMIC researchers simultaneously viewed the same

four videotaped lessons, with each observer taking field notes independently. Immediately

following each lesson, we completed the observation protocol, subsequently comparing codes

and providing commentary about the strengths and limitations of the CVP.

After extensive deliberation, we concluded that the observation tool, in its present state,

required too many sophisticated judgments for observers to make. Accordingly, we made

substantial adjustments and refinements to the CVPs and User’s Guides. We reduced the number

of coding options, thereby limiting the number of features of curriculum implementation to

which the observers would have to attend. We also wrote supplemental narrative for the User’s

Guides to draw clearer distinctions between many of the available coding options.

After completion of the pilot testing, we consulted with several mathematics educators and

educational researchers who were not otherwise involved with the COSMIC project. Our

reviewers included two former presidents of the National Council of Teachers of Mathematics

and two researchers with expertise in teaching and teacher education. They were asked to

provide feedback on the revised protocols and User’s Guides. Specifically, we asked:

• Is the protocol a manageable tool for a trained classroom observer?

• Is there an adequate level of specificity in the User’s Guide so that the observer will

understand what notations he/she should be marking on the form?

• Will the data we collect from the forms be useful in making judgments about extent

of implementation?

• Are we missing some important concepts that should be measured?

Page 15: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

15

• Are there some areas measured that in your opinion are not particularly relevant or

helpful for our purposes?

Although the reviewers’ comments were exceedingly positive, there were key suggestions that

necessitated revisions to the protocol. For example, one reviewer suggested that we document

the seating configuration because Core-Plus is designed for students to work in cooperative

learning groups. Therefore, if students were seated in rows, this feature would compromise the

notion of a “faithful” implementation of Core-Plus. Another reviewer suggested that we record

whether interactive software was used by the teacher and by students during mathematics

instruction. Finally, a reviewer suggested that we document the extent to which students were

engaged (i.e., “on task”) because it was conceivable that a teacher could demonstrate all aspects

of faithful implementation and yet have many students off-task during the lesson. These

modifications were made to the CVP and the User’s Guide was updated accordingly.

Training on Use of the Classroom Visit Protocol

Preliminary training on use of the CVP

Training for the project team on the preliminary version of each of the two CVPs (one for

Core-Plus and one for subject-specific textbooks) was conducted in a two-day session at the start

of our project. Prior to the training, we distributed copies of each CVP and accompanying User’s

Guide to all members of the COSMIC research team. Also prior to training, developers of the

CVP coded the lesson, shared codes, and negotiated consensus codes that essentially represented

the “key,” that is, a target set of “correct” codes for those in training. At the training, one of the

developers of the CVPs used a PowerPoint presentation to provide an overview of the Core-Plus

curriculum, identifying the overall structure of the curriculum and components common to all

Investigations that comprise each Lesson. A second developer then provided an overview of the

Page 16: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

16

essential elements of the integrated CVP. Following this discussion, the project team viewed a

videotape of one Core-Plus lesson. Subsequently, this procedure was used to train observers to

use the subject-specific CVP.

After the observers viewed a lesson in a training session, one of the facilitators led

discussions of how codes should be assigned. This step-by-step process involved toggling back

and forth between the CVP and the User’s Guide, identifying the most appropriate code and

justifying its assignment by explicit reference to narrative in the User’s Guide. Moreover,

facilitators provided explanations to help those in training understand why other available codes

were not appropriate. This laborious induction to the use of the CVP spanned most of the first

day of training, but was a necessary step in learning the coding scheme.

After bringing closure to the first lesson, the project team viewed a second videotape and

coded the lesson individually before the coding “key” was shared. Relatively low consistency

was observed across researchers and in relation to the “key,” and, consequently, the development

team concluded that it was necessary to scale back the specificity of the CVP. In short, for each

aspect of a given lesson, the preliminary version offered too many options to code in a reliable

manner. This notion was confirmed on Day 2 of training, when researchers experienced similar

struggles as they attempted to assign codes for two lessons using subject-specific curriculum

materials. As a result, we streamlined the CVPs to focus on only the most critical elements of

curriculum implementation, and subsequently planned a second phase of training.

Final training on use of the CVP

Training on the final version of the CVPs involved viewing videotapes of four

mathematics lessons in order to gauge the consistency of codes rendered by COSMIC project

personnel. Specifically, two subject-specific lessons (in Algebra 1) were shown to the project

Page 17: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

17

team in one training session. One week later, two Core-Plus lessons were viewed, coded, and

discussed. We acquired the videos of the Core-Plus lessons from the curriculum developers; one

video represented the case of a high-fidelity teacher and the second video represented the case of

a lower-fidelity teacher.

In an effort to simulate an actual classroom visit, each video ran uninterrupted so that the

project team members could experience the recording of field notes in real time. After each

lesson, individual members of the project team completed the Lesson Summary Form, consulting

the User’s Guide as needed. Next, each researcher assigned one code to each of the 10 elements

that comprise the Classroom Learning Environment. When all coding was completed, discussion

commenced and focused on the codes of each researcher relative to one another. Feedback from

the research team led to modest changes in the User’s Guides in order to further enhance the

reliability of coding and attain a high level of coding consistency, as described in the following

section.

Establishing Coding Reliability

The reliability study of the Classroom Visit Protocol (CVP) coding was a two-stage

process. In addition to collecting reliability data during training to determine whether observed

lessons could be coded consistently, we gauged ongoing reliability through the double coding of

selected lessons during classroom visits during the data collection phase of the study.

Exploratory Reliability Study

As noted earlier, initial reliability was gauged in the final training on use of the CVP.

With respect to the Lesson Summary section, relatively high inter-rater reliability was attained.

In particular, researchers assigned identical codes to 93% of items comprising the Lesson

Summary Form for the subject-specific CVP. Similarly high inter-rater reliability was evident on

Page 18: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

18

the Lesson Summary Form for the CVP for Core-Plus, with agreement reached on 90% of all

codes including dichotomous codes (e.g. observed/not observed) and the five-point fidelity

ratings scale. Across the two types of CVPs, agreement was reached in the assignment of 92% of

codes, lending credence to the notion that this portion of classroom visit data could be coded

reliably.

There was somewhat less agreement on the codes assigned by researchers in relation to

the Classroom Learning Environment. In particular, researchers assigned the same code at a 72%

rate on the two subject-specific CVPs, 69% on the two Core-Plus CVPs, and 70% overall. When

inconsistencies were observed, rubrics in the User’s Guides were read aloud and discussed in

order to negotiate the optimal code for the given classroom element. Researchers’ initial codes

differed by no more than ± 1 from the negotiated code (on a five-point scale) in 94% of all cases

(98% for subject-specific, 90% for integrated). Given the relatively high consistency in coding

across observers in other parts of the instrument and the high inference nature of this measure,

the decision was made to proceed with data collection in the classrooms of teacher participants in

the COSMIC project.

Confirmatory Reliability Study

During data collection, a confirmatory reliability test was conducted by double coding 15

lessons, chosen based on feasible observation schedules. Individual researchers took their own

field notes; immediately following the lesson each researcher worked in isolation to complete the

protocol including the content and presentation fidelity judgments. When researchers completed

all coding, they compared codes, negotiating disagreements until all were resolved. Consensus

codes were used in subsequent analysis of implementation data, but the set of original assigned

codes were used to gauge the ongoing reliability of the classroom visit coding.

Page 19: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

19

! INSERT TABLE 3 – CODING RELIABILITY "

As depicted in Table 3, remarkably high consistency was evident in the codes researchers

assigned to the Lesson Summary, with agreement observed for more than 94% of codes. The

reliability of coding Content Fidelity and Presentation Fidelity was similarly high. With regard to

Content Fidelity, 14 of 15 (or 93%) rating pairs from the two observers were identical. In the one

instance when the two observers disagreed, their individual ratings were within one unit of each

other. The results for Presentation Fidelity revealed 10 of 15 (or 67%) rating pairs were identical,

with the remaining 5 pairs all within one unit of each other. Higher reliability of codes was

observed for the Lesson Summary than for the Classroom Learning Environment. However,

although there was exact agreement in only about two-thirds of codes in the Classroom Learning

Environment, more than 92% codes differed by no more than 1 (on a 5-point scale), providing

evidence of relatively high inter-rater reliability for this component of the Classroom Visit

Protocol.

Analyses of Implementation Data

Although we continue to analyze implementation data, our completed analyses have

yielded several interesting findings. We organize this section around the data sources and offer

insights into the processes of making sense of such voluminous data.

Classroom Visit Protocols

Each of the teachers in the study was observed at least three times during the school year,

for a total of 325 observations completed during the first two years of data collection. The

relatively few teachers who participated in both years were observed three times each year.

During classroom observations, observers recorded judgments regarding the degree to which the

textbook influenced the content taught and the manner of presentation of the mathematics

Page 20: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

20

lessons. The individual ratings for the overall Content Fidelity and Presentation Fidelity ratings

were aggregated across visits for each teacher to provide a mean Content Fidelity and

Presentation Fidelity rating for each teacher.

With respect to Content Fidelity, our analysis revealed that the content of lessons was

primarily attributable to the textbook. Across 109 teachers, the mean Content Fidelity rating was

3.67 (on a 5 point scale). Moreover, our data indicate that Content Fidelity ratings were similarly

high regardless of whether the teacher was teaching from an integrated or a subject-specific

textbook and the means were not significantly different across textbook types. The mean overall

Presentation Fidelity rating was 3.11 across the 109 teachers, which was significantly lower and

statistically different than the Content Fidelity ratings. This difference suggests that the manner

in which the lesson was taught was less consistent with the authors’ expectations than was the

content of lessons taught. Furthermore, the average Presentation Fidelity rating for teachers of

the integrated curriculum was 2.91, significantly lower than the mean rating of 3.28 for teachers

of the subject-specific curriculum.

The correlation between the Content Fidelity and Presentation Fidelity ratings was 0.50, a

moderate relationship between the two dimensions. The magnitude of the correlation between

Content and Presentation Fidelity suggests that these two dimensions of implementation should

be examined separately. The substantial variation we found on both of these dimensions across

teachers (1.02 SD for Content Fidelity and 0.96 SD for Presentation Fidelity) underscores why

attention to multi-dimensions of fidelity is warranted.

Background Characteristics

As noted earlier, all teachers who participated in the COSMIC study completed an Initial

Teacher Survey in which they reported numerous background characteristics, including: number

Page 21: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

21

of years of experience teaching mathematics; beliefs about teaching and learning mathematics;

familiarity and agreement with NCTM Standards; implementation of NCTM Standards; the

amount of time allotted for and focus of professional development; the impact professional

development has on teachers’ instructional practices; and the use of technological tools to

supplement mathematical instruction. We collected such data, in part, in order to examine

associations between implementation fidelity and teacher characteristics. However, given the

focus of this chapter on the development of measures of curriculum implementation, we forego

the reporting of such findings in this chapter.

Table of Contents Records

A total of 182 Table of Contents Records were collected. Three indices were developed

to capture the nature and extent of textbook use: (1) Opportunity to Learn index; (2) Extent of

Textbook Implementation index; and (3) Textbook Content Taught index.

Opportunity to Learn (OTL) index. The OTL index indicates whether the mathematical

content contained within the textbook lessons3 was or was not taught. The OTL index is

computed by summing the frequency of occurrence of content taught (reported across all

textbook lessons on a Table of Contents Record) and then dividing by the total number of lessons

included in the particular textbook. The OTL index essentially represents the percentage of the

content in the textbook that students were provided an opportunity to learn.

As an example, Teacher 26, who taught from the integrated curriculum, reported 29

Investigations taught primarily from the textbook, 11 Investigations taught from the textbook

with some supplementation, 9 taught primarily from an alternative source, and 28 not taught out

of a total of 77 Investigations. Thus, the OTL index is calculated as follows:

3 Although we use “lessons,” in the case of the integrated curriculum the unit of analysis is an Investigation; several Investigations comprise one “Lesson”.

Page 22: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

22

!

OTL =29 +11+ 9

77•100 = 63.64

! INSERT FIGURE 4 – SAMPLE OTL GRAPH "

It should be noted that when interpreting OTL indices, an index of 63 does not imply that the

coverage corresponds to the first 63% of the textbook, as can be seen in Figure 4. For this teacher

implementing the integrated textbook, we see that many of the Investigations that were not

taught occurred midway through the textbook.

Extent of Textbook Implementation Index (ETI). The ETI index is determined by

weighting each of the first three options provided to the teachers on the Table of Contents

Record. The largest weight was given when the first option was identified for a lesson. That is,

when lesson content was taught primarily from the textbook, it was given a weight of 1. Content

not taught was given a weight of 0. The two options in between, content taught with

supplementation and content taught primarily from an alternative source were assigned weights

of

!

23 and

!

13 , respectively. The index was then calculated by summing the weights across textbook

lessons and then dividing by the number of lessons contained in the particular textbook. The

quotient was then multiplied by 100 giving the ETI index a scale ranging from 0 to 100. An

index of 100 would represent that every lesson contained in the textbook was taught directly

from the textbook and done so without supplementation or use of alternate sources. An index of

0 would indicate that no lessons from the textbook were taught. Using the previous example

(Teacher 26), the ETI index is calculated as follows:

ETI =1 29( ) + 2

311( ) + 1

39( ) + 0 28( )

77!100 = 51.08

! INSERT FIGURE 5 – SAMPLE ETI GRAPH "

Page 23: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

23

This index indicates the degree to which the content contained in the lessons was taught directly

from the textbook. Note the heavy shading in Figure 5 takes on a different connotation than in

Figure 4. In particular, the heavy shading indicates only those Investigations that were directly

taught from the integrated textbook. The dark gray shading indicates Investigations taught with

supplementation, light gray shading shows Investigations taught from another source, and

unshaded cells indicate Investigations not taught. The first bar in Figure 5 depicts the manner in

which Teacher 26 taught each individual textbook Investigation. When these Investigations are

grouped together in the second bar by method taught, the figure reveals the proportion of the

content of the Investigations that was taught directly from the text, taught with some

supplementation, taught primarily with alternatives to the textbook, or not taught.

Textbook Content Taught Index (TCT). The TCT index differs from the ETI index by

considering only those lessons where content was taught in some manner, thereby ignoring

content students were not given the opportunity to learn (see Figure 6). The lessons were

weighted in the same manner as in the ETI, but the index was calculated by dividing by the

number of lessons reported as being taught in any manner and again multiplied by 100. The

index is reported on a scale ranging from 0 to 100. An index of 100 would indicate that all

lessons were taught using only the textbook. Thus, indices less than 100 indicate the extent to

which textbook lessons taught were supplemented or replaced. Ultimately, this index reports the

extent to which teachers, when teaching textbook content, followed their textbook, supplemented

their textbook lessons, or used altogether alternative curricular materials. The computation of the

TCT index for Teacher 26 is shown below.

TCT =1 29( ) + 2

311( ) + 1

39( )

29 +11+ 9!100 = 80.27

! INSERT FIGURE 6 – SAMPLE TCT GRAPH "

Page 24: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

24

Teacher 26 represents the case of a teacher with markedly different ETI and TCT indices,

51.08 and 80.27, respectively. However, the decisions of other teachers regarding textbook use

resulted in remarkably similar ETI and TCT indices. For example, Teacher 44, who taught from

the subject-specific curriculum, “covered” 76 of 80 lessons in her textbook. As depicted in

Figure 7, she taught 38 lessons directly from her textbook, 32 lessons from the textbook with

some supplementation, 6 lessons primarily from alternative sources, and 4 lessons were not

taught, resulting in an ETI of 76.67. Because the textbook was the primary source for precisely

one-half (38 of 76) of the textbook lessons she taught, black comprises exactly one-half of the

bar in the TCT graph in Figure 8. Because so few lessons were omitted by Teacher 44, the TCT

index of 80.70 is quite similar to the 76.67 score on the ETI. In fact, for any teacher who omits

no textbook lessons, the ETI and TCT indices will be equal.

! INSERT FIGURE 7 – SAMPLE ETI GRAPH "

! INSERT FIGURE 8 – SAMPLE TCT GRAPH "

Textbook-Use Diaries

A total of 219 Textbook-Use Diaries have been collected. These diaries can be analyzed

to describe the extent and nature of the use of the curriculum materials over 15 consecutive days

of instruction. Researchers can perform a number of analyses, including examining how many

instructional days were necessary to “cover” the specified lesson, determining what homework

problems were assigned, and assessing the degree of supplementation used during the lesson.

Instructional days. When coding diaries with respect to how many instructional days

were used to cover a specified chapter or unit, we accounted for variation in class period formats

across schools. In this study, a day is defined as a class period in a typical 7-period day with

these periods ranging in length from 47 to 60 minutes. For those teachers on a block schedule,

Page 25: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

25

the periods ranged from 85 to 90 minutes. Conventionally, a block period is considered as two

periods in a 7-period school day. Thus, the mean number of days allocated to a lesson will be

reported in terms of a 7-period day and data for those teachers who reported their days on a block

schedule are doubled. The distributions of days allocated to the lesson can be determined along

with the means and standard deviations. Despite the fact that textbook authors recommend

teachers spend a certain number of instructional days on the particular lessons upon which the

Textbook-Use Diaries were based, an initial examination of the diaries revealed that teachers

often do not abide by this recommendation (McNaught, 2009). For example, for a particular

integrated lesson on which the authors recommended 12 days of instruction, the number of

instructional days devoted to the lesson varied from 2 to 30 days. By requesting data on a

common set of lessons across the sites, this analysis sheds light on the variability among teachers

within and across schools in regard to the pace at which the curriculum unfolds for particular

mathematical content.

Homework Assignments. With respect to the assignment of homework, both types of

curricula include problems that are intended to be out-of-class exercises to help students

reinforce and extend the knowledge they acquired during the classroom portion of the lesson.

Although the authors offer recommendations regarding the types of problems and the length of

the assignments, one analysis to date indicates that teachers do not closely abide by these

recommendations. For example, few teachers using the integrated curricula assigned the number

of problems recommended by the textbook authors, typically assigning fewer problems than

recommended (McNaught, 2009).

Degree of supplementation. Textbook-Use Diaries can be used to study the regularity

with which the textbook is used by the teacher to teach particular mathematical content and the

Page 26: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

26

extent to which the textbook is supplemented. The number of days a teacher used the textbook is

determined by counting the days the teacher reported using any problems from the textbook,

including days when a teacher reported using both the textbook and some supplements. Each

entry that did not indicate any sort of textbook use (e.g., using only a teacher-made worksheet)

would be coded as no textbook use. This information can then be used to calculate a frequency of

use index by dividing the total days the textbook was used by the total days of instruction. The

degree of supplementation can then be reported by providing percentages with regard to three

categories: (1) use of textbook only; (2) use of supplements only; and (3) use of a mix of

textbooks and supplements.

Conclusion

This chapter has focused on describing the development and nature of instruments that

can be reliably used to assess curriculum implementation. The instruments developed are

valuable tools in characterizing the enacted curriculum, but no single tool or collection of tools

can portray all aspects of how a teacher implements a curriculum program in the classroom.

Thus, researchers need to develop other tools that target specific aspects of implementation that

our tools do not. Furthermore, our instruments would benefit from refinements that will help us

move beyond measures of implementation to develop an understanding of how teacher decisions

about curriculum implementation are made and to identify factors that dictate or at least

influence this decision-making. Although many paths can be taken in this follow-up research,

our current efforts are directed at developing surveys and interview protocols that can be used to

ascertain how teachers make decisions about which textbook lessons to skip and what textbook

lessons require the use of supplemental material or replacement units. Moreover, we expect to

explore the complex interaction between implementation fidelity, teacher characteristics, and

Page 27: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

27

student learning in mathematics. Such findings have the potential to inform teacher education,

curriculum development, and professional development as well as spawn numerous follow-up

studies in mathematics education.

In conclusion, given the importance of measuring curriculum implementation, we hope

other researchers will move forward with the development of additional innovative tools to

measure fidelity of implementation as well as find creative ways to use the instruments we have

developed.

Notes

This paper is based on research conducted as part of the Comparing Options in Secondary

Mathematics: Investigating Curriculum (COSMIC) project, a research study supported by the

National Science Foundation under grant number REC-0532214. Any opinions, findings, and

conclusions or recommendations expressed in this paper are those of the authors and do not

necessarily reflect the views of the National Science Foundation.

Portions of this paper were presented at the annual meeting of the American Educational

Research Association in Denver, Colorado in May 2010.

References

Ben-Peretz, M. (1990). The teacher-curriculum encounter: Freeing teachers from the tyranny of

texts. Albany, NY: State University of New York Press.

Clanindin, D.J., & Connelly, F.M. (1992). Teacher as curriculum maker. In P. Jackson (Ed.), The

handbook of research on curriculum (pp. 363-396). New York: Macmillan.

Clarke, B., Clarke, D., & Sullivan, P. (1996). The mathematics teacher and curriculum

development. In A. J. Bishop, K. Clements, C. Keitel, J. Kilpatrick, & C. Laborde (Eds.),

International handbook of mathematics education (pp. 1207–1233). Dordrecht, The

Page 28: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

28

Netherlands: Kluwer.

Coxford, A. F., Fey, J., Hirsch, C. R., Schoen, H., Burrill, G., Hart, E., et al. (2003).

Contemporary mathematics in context. New York, NY: Glencoe/McGraw-Hill.

George, A. A., Hall, G. E., & Uchiyama, K. (2000). Extent of implementation of a Standards-

based approach to teaching mathematics and student outcomes. Journal of Classroom

Interaction, 35(1), 8-25.

Grouws, D. A. (Ed.). (1992). Handbook of research in mathematics teaching and learning. New

York: Macmillan.

Grouws, D. A., & Smith, M. S. (2000). NAEP findings on the preparation and practices of

mathematics teachers. In Results from the seventh mathematics assessment of the

National Assessment of Educational Progress (pp. 107–139). Reston, VA.

Holliday, B., Cuevas, G., Moore-Harris, B., Carter, J. A., Marks, D., Casey, R. M., Day, R., &

Hayek, L. M. (2005). Algebra 1. New York, NY: Glencoe/McGraw-Hill.

Jackson, P. W. (Ed.). (1992). Handbook of research on curriculum: A project of the American

Educational Research Association. New York: Macmillan.

Larson, R., Boswell, L., Kanold, T. D., & Stiff, L. (2004). Algebra 1. Evanston, IL: McDougal

Littell.

Kilpatrick, J. (2003). What works? In S. L. Senk & D. R. Thompson (Eds.), Standards-based

school mathematics curricula. What are they? What do students learn? (pp. 471-493).

Mahwah, NJ: Lawrence Erlbaum.

McNaught, M.D. (2009). Implementation of integrated mathematics textbooks in secondary

school classrooms. Unpublished doctoral dissertation, University of Missouri, Columbia.

Page 29: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

29

National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for

school mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics. (1991). Professional standards for teaching

mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics. (1995). Assessment standards for school

mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics. (2000). Principles and standards of mathematics.

Reston, VA: Author.

National Research Council. (2004). On evaluating curricular effectiveness: Judging the quality

of K-12 mathematics evaluations. Washington, DC: The National Academies Press.

Remillard, J. T. (1999). Curriculum materials in mathematics education reform: A framework for

examining teachers; curriculum development. Curriculum Inquiry, 29(3), 315-342.

Remillard, J. T., & Bryans, M. B. (2004). Teachers’ orientations toward mathematics curriculum

materials: Implications for teacher learning. Journal for Research in Mathematics

Education, 35, 352-388.

Remillard, J. (2005). Examining key concepts in research on teachers' use of mathematics

curricula. Review of Educational Research, 75(2), 211-246.

Romberg, T. A., & Shafer, M. (2003). Mathematics in Context (MiC): Preliminary evidence

about student outcomes. In S. Senk, D. Thompson (Eds), Standards-based school

mathematics curricula: What are they? What do students learn? (pp. 224-250). Mahwah,

NJ: Lawrence Erlbaum Associates, Inc.

Senk, S. L. & Thompson, D. R. (2003). Standards-based school mathematics curricula: What

are they? What do students learn? Mahwah, New Jersey: Lawrence Erlbaum Associates.

Page 30: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

30

Tarr, J.E., Reys, R.E., Reys, B.J., Chávez, Ó., Shih, J., & Osterlind, S. (2008). The impact of

middle-grades mathematics curricula and the classroom learning environment on student

achievement. Journal for Research in Mathematics Education , 39(3), 247-280.

Weiss, I. R., Banilower, E. R., McMahon, K. C., & Smith, P. S. (2001). Report of the 2000

National Survey of Science and Mathematics Education. Chapel Hill, NC: Horizon

Research, Inc.

Page 31: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

31

Table 1 Lesson elements of integrated and subject-specific textbooks

Subject-Specific Lesson Element Purpose

Integrated Lesson Element Purpose

Lesson Preview To introduce the instructional objective for the class period, typically through a real-world context to illuminate the relevancy of the mathematics in the section.

Launch To generate student interest and provide a context for the Lesson, and to enable teachers to informally assess students’ prior knowledge.

Explore To investigate the mathematics of the Lesson in small groups, to gather data, look for patterns, construct models and meanings, and make and verify conjectures.

Teach To present new mathematical content, typically through worked examples closely tied to the lesson objectives.

Share and Summarize

To use whole class discussion of student ideas to review the content of the Investigation, amplify the mathematical ideas and reinforce connections made during the Investigation.

Practice and Apply

To work independently on exercises to achieve proficiency, typically by mirroring the worked examples from the Teach component of the lesson.

Apply To reinforce student learning through individual practice: On Your Own, a series of questions to assess student understanding of the content of the Investigation; and MORE problems, exercises to be worked out-of-class.

Page 32: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

32

Table 2 Content and Presentation Fidelity Scales

1 2 3 4 5 Lower Content Fidelity Moderate Content Fidelity Higher Content Fidelity

The content of the enacted curriculum was largely inconsistent with the written curriculum. The textbook was not the primary source of the lesson content because of omissions, significant modifications, and/or supplementation.

The content of the enacted curriculum was moderately consistent with the written curriculum. Although the textbook was a source of some of the lesson content, other portions of the lesson could not be attributed to the textbook.

The content of the enacted curriculum was consistent with the written curriculum. The textbook was the primary source of the lesson content with little or no deviation or supplementation.

Lower Presentation Fidelity Moderate Presentation Fidelity Higher Presentation Fidelity The presentation of the enacted curriculum was not consistent with the expectations of the textbook authors. During the lesson, the teacher implemented actions/activities that were not recommended and/or neglected to implement actions/activities that were advised or recommended. The teacher placed disproportionate emphasis on particular lesson components at the expense of others.

The presentation of the enacted curriculum was moderately consistent with the expectations of the textbook authors. During the lesson, the teacher either implemented some actions/activities that were not recommended or neglected to implement actions/activities that were advised or recommended. The teacher generally placed appropriate emphasis on each lesson component.

The presentation of the enacted curriculum was consistent with the expectations of textbook authors. During the lesson, the teacher implemented recommended actions/activities and refrained from actions/activities that were not advised or recommended. The teacher placed appropriate emphasis on each lesson components.

Page 33: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

33

Table 3 Percentage of coding agreements for each protocol element, by curriculum type and overall, in data collection phase

Protocol Element Subject-Specific Integrated Overall

Coding Matches

Total Codes Reliability Coding

Matches Total Codes Reliability Coding

Matches Total Codes Reliability

Lesson Summary 277 300 .923 566 594 .953 843 894 .943

Content Fidelity 14 15 .933

Content Fidelity (±1) 15 15 1.000

Presentation Fidelity 10 15 .667

Presentation Fidelity (±1) 15 15 1.000

Classroom Learning Environment 40 60 .667 66 110 .600 106 170 .624

Classroom Learning Environment (±1) 56 60 .933 101 110 .918 157 170 .924

Page 34: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

34

Figure 1. Excerpt from integrated Classroom Visit Protocol.

Page 35: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

35

Unit 1 Matrix Models

Taught primarily from

Core-Plus textbook

Taught from Core-Plus textbook with

some supplementation

Taught primarily from alternative(s) to Core-Plus

Did not teach

content

Lesson 1 Building and Using Matrix Models

Inv1

There's No Business Like Shoe Business ! ! ! !

Inv 2 Analyzing Matrices ! ! ! !

Inv 3 Combining Matrices ! ! ! !

Figure 2. Excerpt from Table of Contents Record for integrated textbook.

Page 36: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

36

Glencoe Geometry Textbook-Use Diary

Instructions: For the first regular Geometry class you teach, record the instructional activities that correspond to Chapter 3. Log activities for the first 15 days only. A sample entry is provided below.

Date Section

Number of examples used from textbook

Number of examples used from

other sources

Homework Assignment

(include page numbers)

Oct 7 3.1 4 1 Pages 134-137

#3-36 even; 44-47

Figure 3. Excerpt from Textbook-Use Diary for subject-specific textbook.

Page 37: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

37

Figure 4. Opportunity to Learn (OTL) index for a teacher implementing an integrated textbook. Note: Black shading indicates one of the first three options on the TOC reported: (1) content taught primarily from textbook; (2) content taught from the textbook with some supplementation; (3) content taught primarily from an alternative source. No shading indicates option (4) content not taught.

Page 38: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

38

Figure 5. Extent of Textbook Implementation (ETI) index for a teacher implementing an integrated textbook.

Page 39: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

39

Figure 6. Textbook Content Taught (TCT) index for a teacher implementing an integrated textbook.

Page 40: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

40

Figure 7. Extent of Textbook Implementation (ETI) index for a teacher implementing a subject-specific textbook

Page 41: Curriculum Implementation in Secondary Mathematics Classroomscosmic.missouri.edu/bookchapter/TarrMcNaughtGrouws2010.pdf · DRAFT Curriculum Implementation in Secondary Mathematics

DRAFT

Curriculum Implementation in Secondary Mathematics Classrooms

41

Figure 8. Textbook Content Taught (TCT) index for a teacher implementing a subject-specific textbook