developing a writing observation...

DEVELOPING A WRITING OBSERVATION PROTOCOL 1

Developing a Writing Instruction Observation Protocol:

Implications for Practice, Research, and Policy

Andrea Winokur Kotula and Cynthia Mata Aguilar

Education Development Center, Inc.

Terrence Tivnan

Harvard University Graduate School of Education


Suggested citation:

Kotula, A. W., Aguilar, C. M., & Tivnan, T. (2014). Developing a writing instruction

observation protocol: Implications for practice, research, and policy. Waltham, MA:

Education Development Center, Inc.

Copyright © 2014 by Education Development Center, Inc.

Education Development Center, Inc. is a global nonprofit organization that creates learning

opportunities for people around the world, empowering them to pursue healthier, more

productive lives. For more information, visit edc.org.


Abstract

The focus of this article is on the design of a Writing Instruction Observation Protocol to capture

the details of writing lessons; the components of effective writing instruction, as indicated by

research; the practices that teachers are using when they teach writing; and the effectiveness of

those practices, as documented by classroom observers using our protocol. This singular focus

on writing was essential, not only because it could inform our research but also because of the

Common Core State Standards’ focus on writing, and the potential need in the field for such a

protocol. Over a two-year period, we observed fourth-and fifth-grade writing instruction in 259

classrooms in 56 randomly selected treatment and control schools in six districts. What we

discovered in the classroom can guide teacher preparation programs, shape professional in-

service training, direct curriculum, influence policy, and ultimately support the development of

improved writing skills among students.


Developing a Writing Instruction Observation Protocol:

Implications for Practice, Research, and Policy

If today’s students expect to compete in a global economy, they must possess the skills to think

critically and creatively, solve problems, collaborate, and especially communicate (Partnership

for 21st Century Skills, 2011). “Writing has never been more important than in this digital age. It

is almost inconceivable to achieve academic success without good writing skills” (College Board

Advocacy and Policy Center, 2010, p. 2). Writing is essential in the business sector as well (cf.

Levy & Murnane, 2004; Murnane & Levy, 1996; National Governors Association Center for

Best Practices, 2005), with many arguing that along with being extremely important for daily job

performance (National Commission on Writing, 2003), good writing affects hiring and

promotions (ACT, 2005).

However, “despite the existence of solid models of how to proceed . . . writing is often

poorly taught. The reality is that a lot of writing instruction is perfunctory” (College Board’s

National Commission on Writing, 2006, p. 12). Teachers in grades 4-6 report that they devote

only about 1.25 hours per week on writing instruction; their students spend about two hours each

week writing. Moreover, these teachers indicate that they felt unprepared to teach writing after

graduating from their teacher preparation programs (Gilbert & Graham, 2010). While much has

been written about what constitutes “good” reading instruction, writing instruction has taken a

back seat. The National Reading Panel (National Institute of Child Health and Human

Development, 2000) has found more than 100,000 published studies about reading. In contrast,

the landmark meta-analysis of writing research by Graham and Perin (2007a, 2007b) identifies


approximately 600 documents on writing instruction for students in grades 4–12. In fact, writing

has been called “the neglected element of American school reform” (National Commission on

Writing, 2003, p. 9).

The 10 College and Career Readiness Anchor Standards for Writing, part of the Common

Core State Standards for English Language Arts & Literacy in History/Social Studies, Science,

and Technical Subjects (CCSS) (National Governors Association Center for Best Practices,

Council of Chief State School Officers, 2010), become increasingly challenging as students

progress in school. In the early grades, students begin to learn how to write opinion pieces,

informational/explanatory texts, and narratives. However, in grade six, for example, they must

write not only opinion pieces, but arguments that link a concluding statement to the argument

and that incorporate claims, logic, evidence, and credible sources. By the time they reach grades

eleven to twelve, students need to “write arguments to support claims in an analysis of

substantive topics or texts, using valid reasoning and relevant and sufficient evidence” (p. 45).

They are expected to develop counterclaims, include words, phrases, clauses, and syntax that

“link the major sections of the text, create cohesion, and clarify the relationships between

claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims”

(p. 45). Writing under pressure adds another dimension to the task, and students must “have the

flexibility, concentration, and fluency to produce high-quality first draft text under a tight

deadline as well as the capacity to revisit and make improvements to a piece of writing over

multiple drafts when circumstances encourage or require it” (p. 41).

Student Writing Outcomes

While calls to improve student writing are not new, the stakes are far higher today than ever

before. However, across all grades, only about one-quarter of the students in our nation’s schools


are proficient in writing. The most recent National Assessment of Educational Progress (NAEP)

Writing Assessment at grades 8 and 12 (National Center for Education Statistics [NCES], 2012)

found that only 27% of students at both grades performed at or above the proficient level (“solid

academic performance”) in writing; just 54% of eighth graders and 52% of twelfth graders

performed at the basic level, and 20% of eighth graders and 21% of twelfth graders obtained

writing scores below this level.

The NAEP most recently assessed fourth-grade student writing in 2002. Findings from

this assessment revealed that only 28% of fourth graders obtained scores at or above the

proficient level. Fully 58% of fourth graders scored at the basic level—revealing only a “general

grasp” of the writing task—with 14% demonstrating writing skills below this level (Persky,

Daane, & Jin, 2003).

The writing of students among certain racial and ethnic groups suffers even more. In

2011, 34% of white eighth-grade students obtained scores at or above the proficient level, while

only 14% of Hispanic and 11% of black students did so; similarly, 35% of white twelfth-grade

students obtained scores at or above the proficient level compared to only 11% of Hispanic and

9% of black twelfth graders (NCES, 2012). There was also a significant gap in writing

performance among different racial and ethnic groups for fourth graders in 2002. Whereas 34%

of white students obtained scores at or above the proficient level, only 17% of Hispanic and 14%

of black students scored at those levels (Persky et al., 2003).

While one would expect student writing proficiency to improve over time, findings from

the NAEP (Persky et al., 2003) suggest that it does not. By twelfth grade, one in five students

still cannot compose texts without a significant number of grammar, spelling, and punctuation

errors (National Commission on Writing, 2003). Even among college-bound students, just over


one in three high school graduates (36%) cannot meet the readiness benchmarks set for college-

level writing (ACT, 2013).

Overview of Research

In response to the national student writing proficiency crisis, we engaged in a four-year research

project funded by the US Department of Education, Institute of Education Sciences, to study a

writing curriculum for fourth- and fifth-grade classrooms in six school districts. As part of that

study, we developed a Writing Instruction Observation Protocol (see Appendix) for classroom

observations specific to writing instruction. In this article, we detail our design of the protocol—

among the first such instruments in the field and the first that is based on research, was shown to

have moderate to high inter-rater reliability, and was used in a study with large numbers of

teachers to gauge its feasibility. We also examine one set of findings from our research: the

strategies that our study’s teachers used when they taught writing—as documented by classroom

observers using our protocol—and the effectiveness of those strategies, as determined by

research on the components of effective writing instruction. We examined the observations

across treatment conditions to capture a “snapshot” of current writing instruction in our study

schools. What we have discovered in the classroom, if validated in future studies, can guide

teacher preparation programs, shape professional in-service training, influence policy, direct

curriculum, and ultimately support the development of improved writing skills among students.

Method

Participants

We observed writing instruction of 259 teachers (203 fourth-grade classes and 56 fifth-grade

classes) in 56 randomly selected treatment and control schools in six school districts in

Massachusetts (see Table 1). Districts ranged in size from 13,373 students to 4,496 students.


Districts are classified by NCES locale codes according to their distance to urbanized areas. Five

of our study districts are considered “suburbs, large,” or a “territory outside a principal city and

inside an urbanized area with population of 250,000 or more.” The sixth district is considered a

“city, small,” or a “territory inside an urbanized area and inside a principal city with population

less than 100,000” (NCES, n.d.) All are Title I districts.

There was a high percentage of low-income students in the study (those eligible for free

or reduced lunch), 76%. Thirty-two percent received Limited English Proficiency (LEP) services

during the study or up to two years prior to it, and 17% were enrolled in special education during

either or both years of the implementation. Twenty-five percent of the students were White, 52%

Hispanic, and 6% Black. Teacher experience varied from beginning to veteran teachers with over

20 years of experience within and across schools and districts, and participating teachers taught

between one and five writing classes in their schools.

Procedure

The data collection for the study took place over two years. In Year 1, we studied the

students in fourth-grade classrooms. In Year 2, we studied a second cohort of fourth-grade

classrooms and followed the first cohort of students into their fifth-grade classrooms. We

observed all of the fourth-grade teachers who taught writing in Year 1 (2010—2011) in five

districts in Massachusetts, or 121 lessons, with a few exceptions because of scheduling or

availability issues.

In Year 2 of the implementation (2011–2012), one district dropped out of the study due to

competing interventions; thus, we observed teachers in four of the Year 1 districts engage a new

cohort of fourth-grade students in writing lessons, and we also observed fourth-grade teachers’

writing lessons in a new district that joined the study. In addition, we observed fifth grade


teachers. We observed all of the new fourth- and fifth-grade teachers of writing in Year 2 and

about 40% of the returning fourth-grade teachers. Of the 178 lessons observed in Year 2, 54 were

new fourth grade teachers, 96 were new fifth grade teachers, and 28 were returning fourth grade

teachers. We determined the order in which we observed teachers by randomizing within each

district within time blocks. If a teacher taught more than one section, we observed one of the

classes, making adjustments to the random order of observations for geographical constraints.

Five districts participated each year—or a total of six districts across the two years.

Measures

Development of an observation protocol for writing lessons. To capture the details of

the writing lessons, we needed a teacher observation protocol that focused specifically on writing

instruction. When it became apparent that no such protocol existed for our purposes, we created

our own. In the process of developing it, we needed to (1) identify the elements of “good”

writing instruction; (2) examine national and state writing standards, frameworks, and

assessments; and (3) examine other observation protocols for formatting suggestions.

Elements of good writing instruction. Over the past two decades, a series of meta-

analyses and additional research studies have found that teachers need to learn how to teach

writing effectively (Coker & Lewis, 2008; Graham & Perin, 2007a). Unfortunately, teachers and

curriculum designers tend to believe that writing instruction is easier than it is (Hillocks, 2008).

The research identified the following effective teaching practices that support positive writing

outcomes:

Providing explicit instruction as part of a systematic writing curriculum that uses a

process writing approach (Graham, McKeown, Kiuhara, & Harris, 2012; Graham &


Perin, 2007a, 2007b), maintaining a balance among materials, activities, learning tasks,

and teacher instruction (Hillocks, 2008)

Scaffolding students’ writing practices to help them while they are learning to be better

writers (Graham & Perin, 2007a, 2007b)

Teaching writing strategies for the writing process, including such stages as planning,

drafting, revising, and editing (Bui, Schumaker, & Deshler, 2006; Graham et al., 2012;

Graham & Perin, 2007a, 2007b; Hillocks, 1986). Providing comments about student

writing paired with an opportunity for revision. (Hillocks 1986 found that students gain

almost twice as much on measures of their writing as those who just receive comments

for the teacher.)

Building students’ capacity to evaluate their own writing using guides or scales (Hillocks,

1986)

Providing instruction of prewriting, summarization, and sentence combining skills

(Graham et al., 2012; Graham & Perin, 2007b; Hillocks, 1986).

Ensuring that students have time to study and emulate good models of writing (Graham

& Perin, 2007b)

Engaging students in inquiry activities (Graham & Perin, 2007b; Hillocks, 1986)

Helping students to set product goals (Graham et al., 2012; Graham & Perin, 2007b)

Teaching text structure, transcription (spelling, handwriting, keyboarding), and self-

regulation (e.g., goal setting, self-assessment); using imagery and creativity; assessing

student writing and providing feedback; and increasing time spent writing (Graham et al.,

2012)


Providing opportunities for peers to assist each other in collaborative writing (Graham et

al., 2012; Graham & Perin, 2007b). Engaging students in student-led small-group

discussions (Hillocks, 1986, 2008).

Using word processing equipment (Graham et al., 2012; Graham & Perin, 2007a, 2007b)

Including multiple purposes for writing, using varied genres, and reflecting real-world

writing needs (Coker & Lewis, 2008; Graham & Perin, 2007a, 2007b)

Focusing on the quality of the writing, not just on the lower-level skills of writing

mechanics or word count (Deatline-Buchman & Jitendra, 2006)

All of the above findings on the elements of good writing instruction informed our design of

items for our Writing Observation Protocol.

National and state standards, frameworks, and assessments. To ensure that our Writing

Instruction Observation Protocol reflected expectations for “good writing” in the fourth and fifth

grades as set by national and state standards and assessments, we compared the fourth-grade

NAEP writing objectives and scoring rubrics to the writing standards and scoring rubrics from

our study state’s English language arts curriculum the Massachusetts English Language Arts

Curriculum Framework. The NAEP is generally considered the gold standard writing

assessment. We also examined the scoring rubrics from the five states with the highest fourth-

grade NAEP writing scores in 2002.

We found a very close alignment between the Massachusetts standards and the NAEP

objectives. In both, students need to write for a variety of purposes; use both narrative and

informational formats; write for different audiences; make use of various stimulus materials;

employ different stages of a writing process that includes revision; demonstrate organization in


their writing; and apply standard English conventions (Massachusetts Department of Education,

2001; Persky et al., 2003).

Our examination of the NAEP and MCAS scoring rubrics indicated that the rubrics

disagree only on the role of spelling in good writing and writing instruction. The NAEP employs

one six-point holistic rubric, while MCAS uses a six-point rubric for “topic/idea development”

and a four-point rubric for “standard English conventions.” The NAEP includes spelling only as

part of conventions—which is just one element of the holistic score. MCAS does not assess

spelling on its rubric for “standard English conventions” and only considers spelling if those

errors interfere with communication. We did not include a spelling item in our observation

protocol.

We examined the writing scoring rubrics of the five states with the highest 2002 NAEP

writing scores for three reasons: to determine how they define good writing; to see if they share

skill foci with NAEP or MCAS, or both; and to ascertain if there are additional essential areas

that they assess. We found close alignment between the NAEP, the MCAS, and the writing

scoring rubrics of the five states—Connecticut, New York, North Carolina, Delaware, and

Massachusetts. (Note that Delaware assesses writing at the fifth—not fourth—grade level. Also,

New York does not assess writing from a stand-alone prompt. Fourth graders in New York

answer two extended-response questions: one each in the reading and listening sections.) The

writing scoring rubrics for the NAEP and the top-scoring states were perfectly aligned in the skill

areas of Organization, Details, Sentence Structure, Word Choice, Revision, and Editing. We

have already discussed the alignment with Massachusetts. The differences among the other states

are as follows:

Mechanics: not assessed in Connecticut and New York.


Spelling: only assessed in North Carolina.

Grammar and Usage: not assessed in Connecticut or New York.

Audience and Purpose: assessed in New York and Delaware.

Because of the close alignment of the scoring rubrics, we did not add items for mechanics,

spelling, grammar and usage, or audience and purpose to our protocol.

Sample classroom observation protocols. To inform the format of our Writing

Instruction Observation Protocol, we reviewed numerous classroom observation protocols to

identify their structure, categories, and rating systems. While the list is not comprehensive, a few

examples follow:

The Oregon Teacher Observation Protocol (OTOP) (Wainwright, Flick, & Morrel,

2003), which consists of 10 items that are scored holistically along a 1–4 scale, or N/O

when not observed. We found the use of a holistic scale especially helpful.

PEP Observation Protocol (Howard, Shoemaker, Peach, & Tomlinson, 2007), which is

an adaptation of the OTOP and adds two capsule description items that summarize the

lesson. We incorporated the check-off format of the capsules in some of our items.

English Language Observation Development Tool (Sapien-Melchor, 2006), which targets

English language development classes but provides rich detail in a check-off format.

Four Dimensions of Classroom Practice (Disston, 1997), which begins each item with

“How does the teacher,” a question that guided our team’s focus toward what both the

teacher and the student are doing during the lesson.

In summary, developing our Writing Instruction Observation Protocol was a recursive

process; we continuously cross-checked the sources to determine the appropriate placement and

wording of items. In addition, we added items the research literature identifies as good


instruction, although neither NAEP nor our study state measures them (e.g., planning, revising,

and providing feedback to students).

The observation protocol. Our Writing Instruction Observation Protocol includes a

series of questions and checklist items that represent relatively objective considerations about the

organization of the class, the way teachers explain lessons and interact with students, and the

types of interactions and feedback included in lessons. The observer’s task is to notice and record

these activities during writing lessons. The protocol includes the following four sections

(Sections 2 and 3 constitute the two primary blocks of the instrument):

Section 1—Introduction: This brief section focuses on the teachers, how they introduce

the lessons, and how set the stage for the day’s work. The section addresses three

questions fundamental to any lesson: (1) “What instruction is the teacher providing?”;

(2) “What are students doing during skill/strategy instruction/practice?”; and (3) “What

is the teacher doing while the students are working?”

Section 2—Skills/Strategies Instruction/Practice: Section 2 provides 14 questions about

what the teachers are doing during this phase of the lesson; what the students are doing

and how they are organized; and whether activities involve the whole class, small

groups, or individual students. If the lesson focuses on composing, and not on

“instruction/practice,” the observer has a space for checking off the section as “not

applicable.” The observer also notes whether or not the teacher gives students feedback

on their writing as part of the skills/strategies instruction/practice and, if so, the kinds of

feedback.

Section 3—Composing: Section 3 consists of 10 questions on the portion of the lesson

that involves student writing and composing. In this section, the observer notes how the


students are working as they compose and in what part of the composing process they

are engaged. As in Section 2, the observer has a space to check off the section as “not

applicable”—if the lesson focuses on instruction/practice instead of composing—and

space to document if the teacher provides feedback and, if so, what kinds of feedback.

Section 4—Closure: Section 4 contains miscellaneous questions about whether the

teacher provides closure to the lesson and assigns homework that involves writing.

Section 4 also asks the observer to document what other adults in the room, aside from

the teacher, are doing during the lesson, and provide space for the titles of other adults.

The protocol provides space at the end of the page for the observer’s comments.

All sections have subcategories for the observer to check off. Some sections record the

number of minutes spent on the task and include a category for “other.” Throughout, space is

included for observer comments as needed.

We designed the protocol so that an observer could easily record whether the teacher

provides instruction in skills or strategies, practice, or both and whether the lesson includes

composing. The protocol also enables the observer to determine what the teacher and students

are doing during different parts of the lesson. We recognized that some features of the questions

in the protocol are not appropriate or necessary to all lessons. For example, a writing session that

focuses entirely on completing a draft might not include an instructional component and, in that

case, guided practice would not be relevant. Similarly, it might be unnecessary to show an

example or model a skill or strategy for a lesson that is a continuation from a previous day.

Protocol rating scale. After completing Sections 1 through 4 and documenting the

details of the writing lesson, observers use the following scale to rate the effectiveness of nine


specific features of the lesson (e.g., “How effective was the lesson objective?” “How effective

was the feedback?”):

1 = not effective

2 = somewhat effective

3 = effective

4 = very effective

Observers indicate if they did not observe a category or if it did not apply to the lesson.

Then, to capture the effectiveness of the writing instruction, observers provide an overall rating,

which enables them to rank lessons along the same four-point Likert scale.

Observers were trained not to average the individual category ratings because some

categories are more essential to a specific lesson than others. For a number of reasons—different

lessons have different objectives, students vary in their knowledge and skills, some teachers use

a writing curriculum and others fashion their own, lessons may share characteristics and yet

differ notably in their overall effectiveness—the category ratings were to be used as guidelines

and the overall rating scored holistically. The observers provided the summary rating of the

overall effectiveness of the writing session using the same 4-point scale of effectiveness.

We relied on the observer’s overall rating rather than on an average of the nine individual

ratings for two reasons. First, an average of the nine ratings might not capture the variability

among lessons of each category’s importance. A second reason for relying on the observer’s

overall rating rather than on an average of the nine individual ratings emerged when we looked at

the results. As it turned out, the ratings of the nine categories of lesson effectiveness were highly

consistent with one another and also highly correlated with the overall rating. The internal

consistency reliability (Cronbach’s coefficient alpha) for a composite variable based on an


average of the nine ratings was .87, indicating that the lessons rated as more effective on one

dimension (e.g., effectiveness of the lesson objective) also tended to be rated as more effective

on other dimensions (e.g., feedback or classroom climate). In addition to this relatively high level

of internal consistency, the correlation between the average of these ratings and the observer’s

overall rating was .86 (p < .001); in other words, the observers’ ratings of overall effectiveness

were also consistent with their ratings based on the separate features of instruction. We found

both the overall summary rating and the ratings of separate features to be useful parts of the

observation protocol and believe they help the observers. Moreover, the separate ratings could be

valuable in enabling observers to examine specific features of the lessons. In principle, using an

average of several ratings provides an assessment of effectiveness that is potentially more fine-

grained, but the single overall rating offers a simple and easy-to-use assessment with virtually

identical results.

Observer training. During the initial implementation of the classroom observations,

observers received a full-day training on the use of the protocol conducted by the principal

investigator and site coordinator, both experienced literacy researchers. Because of the

importance of consensus on terminology, our research team created common definitions for

observers to apply to the following terms: “direct vs. indirect instruction,” “discussion,”

“modeling vs. examples (models),” “feedback,” “guided practice,” “lesson integrity,” and

“classroom climate.” For example, we defined “guided practice” as “a purposeful means by

which students can learn, practice, and/or apply new skills/strategies.” To be considered using a

guided practice, a teacher needs to be guiding the students—walking around and helping or

leading a whole-class or small-group practice. Reviewing a worksheet or homework was not

considered guided practice.


Inter-rater reliability. As part of the process of developing the observation protocol,

multiple observers viewed a sample of writing lessons at the beginning of each year so we could

assess the levels of rater agreement and reach an acceptable level of inter-rater reliability. A

member of our research team observed classrooms with the new observers, comparing and

discussing results after each observation. Two observers rated 34 lessons and three observers

rated 24 lessons for a total of 58 lessons. Each rated the lessons independently, with discussions

occurring only after the lessons and the ratings were completed. We randomly selected

classrooms in order to ensure a variety of settings in this phase of the study.

The observers tended to show high levels of agreement when rating the objective features

of the lessons—whether the teacher introduced the lesson (from 89 to 96% for the three

questions) or whether the instruction included feedback to students about their writing (91%), for

example. We examined the percentage of agreement, using Cohen’s kappa to adjust for

agreement “by chance” when analyzing the classroom activities variables. Sections 1-4 contained

30 questions that included some subquestions, resulting in a total of 39 items for analysis. For

these 39 items, the inter-rater agreement averaged more than 90% (mean = 91, median = 93, SD

= 5.3). We analyzed the effectiveness ratings by determining the percentage of agreement, the

correlations of the ratings, and the intraclass correlation. For the nine ratings that focused on the

effectiveness of the instruction, the inter-rater agreement was still adequate, with an average

percentage of exact agreement of just less than 80% (mean = 78, median = 78, SD = 5.9). For the

summary rating of the overall effectiveness of the lesson, the raters showed exact agreement of

82%, and the average correlation of the raters’ summary ratings was also adequate (r = .73).

Administering the protocol. Each year after we completed our inter-rater reliability

study, two observers conducted the remaining observations, one per classroom. Before we


conducted any visits, we sent a letter reviewing the procedures and assuring teachers and

principals of the confidentiality of the data we collected. To minimize scheduling conflicts, we

informed teachers, principals, and district liaisons one week prior to the proposed class visit. Our

observers did not share the protocol or provide feedback to the teachers.

Results and Discussion

Descriptive statistics and overall reliability information on the ratings of lesson

effectiveness are presented in Table 2. Observers rated most lessons as “effective” (about 72%

of all lessons), while they rated about 11% as “very effective” and 18% as “somewhat effective.”

Introducing the Lessons and “Skills/Strategies Instruction Practice”

Table 3 presents information on how frequently teachers—across treatment and control

schools—used a variety of instructional features while introducing the lessons and during the

“Skills/Strategies Instruction/Practice” portion of the lessons. This table indicates the overall

frequency of each feature (the “most popular” features are listed first) and also shows the relative

frequency of the features of the lessons rated as “very effective,” “effective,” and “somewhat

effective.” We created three sets of classrooms based on their effectiveness rating and compared

them with a series of cross tabulations. Chi-square tests of independence were used to identify

which differences were statistically significant.

Introducing the lessons. One of the significant findings was in the way that teachers set

the stage for the day’s work and introduced the writing lessons. In almost all cases, they provided

an explicit statement of the lesson objective(s) (in 91% of all of the lessons observed). However,

they did so in 100% of the lessons rated as “very effective,” but in only 82% of the lessons rated

as “somewhat effective.” Similarly, teachers linked a previous lesson to a current lesson in 96%

of the lessons rated as “very effective,” but in only 65% of the lessons rated as “somewhat


effective.” Activating prior knowledge was also a strategy found significantly more often in

lessons rated as “very effective.” We observed it in 74% of those rated “very effective,” but in

only 46% of the “somewhat effective” lessons. These three features of lessons—explicit

objectives, connections with earlier lessons, and activation of students’ prior knowledge—were

the clearest features that distinguished the lessons that were given the highest overall ratings

from lessons given ratings of only “somewhat effective.”

Skills and strategy instruction and/or practice. While significant variation appeared in

the portion of the lessons that focused on the instruction and practice of skills and strategies, the

most common approach emphasized direct skill or strategy instruction (96% of the lessons) and

practice (94%) during class. In many lessons, teachers also provided examples, asked open-

ended questions, explained how the skills would be helpful, and modeled the process of using the

skills. These approaches to explaining, questioning, providing models, and modeling, while

observed in 74%, 59%, 50%, and 47% of the lessons, respectively, were observed only slightly

more often in the highest-rated lessons; and the differences were statistically significant only in

the case of “providing examples of writing related to the lesson”; teachers provided examples in

89% of the lessons rated “very effective” compared to only 63% of the lessons rated as

“somewhat effective.” The use of discussion, small-group work, or computers (i.e.,

keyboarding), as identified by the literature as important elements of effective writing

instruction, was rare.

It was not common for teachers to engage in class discussions (14% in small groups; 8%

whole class).Teachers modeled the skill or strategy they were teaching in fewer than half of the

lessons (47%). Because this practice was observed only slightly more often in the highest-rated

lessons, our findings did not indicate a statistically significant difference on the effectiveness of


teacher modeling as an instructional practice. However, 89% of the lessons in which teachers

provided examples of the focal skill or strategy were rated “very effective,” strengthening the

evidence in favor of using that practice in writing instruction.

In summary, direct instruction was the predominate approach to teaching writing skills

or strategies, and teachers usually provided time to practice the relevant skills or strategies. They

often posed open-ended or probing questions and elicited student responses. In general, our

findings point to the effectiveness of three specific practices in teaching writing skills and

strategies: working directly with individual students, providing individualized feedback, and

providing examples of the skill or strategy that is the focus of the lesson.

Student activity. During the “Skills/Strategies Instruction/Practice” phase of the lesson,

students tended to work alone (72%) and write on paper (76%), often giving oral responses to

teacher questions (63%). They worked less frequently in small groups (13%) or with partners

(26%), and we rarely observed them using computers (1%).

Teacher activity. When students were engaged in seatwork, teachers generally circulated

around the classroom (76%), often stopping to engage with individual students (77%), and

addressed the whole class (70%). We found that when teachers worked with individual students,

the effectiveness rating of the instruction increased: 82% of the lessons rated “very effective”

included teachers working with individual students compared to 59% of the lessons rated

“somewhat effective.” It was much less likely for teachers to work with small groups of students

during the “Skills/Strategies Instruction/Practice” phase of the writing lessons (14%).

Feedback to students. During the “Skills/Strategies Instruction/Practice” phase of the

lessons, most teachers provided feedback to students about their writing (98%); this feedback

was mostly verbal (94%), sometimes given to individuals and sometimes to the whole class. We


saw the same pattern when teachers gave feedback to individual students as we did when they

worked with individuals. Teachers provided feedback to individual students in about 90% of the

lessons rated “effective” or “very effective” compared to 68% of the lessons rated “somewhat

effective.” Teachers very rarely gave written feedback or any formal grades to students during

this part of the lessons. In sum, our findings show that the effectiveness of lessons increased

when teachers both work individually with students and provide them with individualized

feedback.

Student composing and writing in class. In addition to lessons on writing skills and

strategies, most of the lessons that we observed also included time for students to write and

compose in class. Table 4 presents the results from this phase of the lessons in order of

frequency, along with the relative frequencies for the lessons rated as “very effective,”

“effective,” and “somewhat effective.” Students benefited from the same kinds of instruction in

this part of the lesson that effectively supported their efforts to learn skills and strategies. For

instance, 83% of the lessons rated “very effective” included teacher-provided examples of

writing that related to instruction compared to only 29% of the lessons rated as “somewhat

effective.” Interestingly, this was not a common strategy, occurring in less than one-third of the

lessons. Similarly, highly rated lessons were also more likely to have teachers actively model

writing as part of the lesson; 33% that were rated “very effective” compared to 7% that were

rated “somewhat effective,”

Student activity. Similar to what we saw when observing the skills and strategies portion

of the lessons, students almost always worked alone (98%) and wrote on paper (97%) during the

composing part of the lesson. We occasionally saw them working with partners (12%) or in

small groups (8%), but the predominant mode was for them to write on their own. The fact that


they almost always worked on their writing tasks alone during the composing part of the lessons

cannot be taken as indicative of any inadequacy in the instruction; writing is in and of itself a

solitary activity. However, given clear direction and rubrics, small-group work may well succeed

at any grade level, and discussion and collaborative writing have been noted in the literature as

important components of good writing instruction. Because of its limited use in our study (8%),

we can draw no conclusions about its effectiveness.

In the lessons that we observed, students generally wrote drafts (78%) about a specific

topic (76%) and occasionally engaged in the planning part of the writing process (23%) or in

revising earlier work (15%). We very rarely observed students using computers, tablets, and/or

laptops to compose (1%). Word processing is an essential writing practice, and it is a concern

that it was not observed in more lessons.

Teacher activity. Because the focus was on having students work on drafts, teachers tended to

circulate around the classroom (93%) and work with individual students (94%) during the

composing phase of the lessons. Teachers occasionally addressed the whole class (33%),

although this was less common while students were composing, and they sometimes worked

with small groups of students (13%).

Teacher feedback to students. Virtually all teachers provided some type of feedback on

writing (97%) to at least some of the students, almost always in the form of individual comments

given verbally (91%) as the teachers moved about the room. Much less frequently, teachers

provided whole-class feedback (34%). Written feedback was also rare (17%).

Sharing writing. Students frequently shared their writing (in 74% of the lessons). Sharing

usually involved individual students reading their compositions aloud to the whole class (68%);

only rarely did we see students sharing with partners (8%) or in small groups (3%).


Writing genres. In most cases (81%), students worked on imaginative or literary writing.

Writing in these genres was very common in the most effective lessons (95% of the “very

effective” lessons compared to 72% of the “somewhat effective” lessons). But many lessons also

focused on informational writing (43%), and over the course of the school year most classes

undertook a variety of writing tasks and topics, such as poetry, opinion pieces, or responding to

literature. For some lessons, teachers used a mixed approach in which some students wrote

informational reports while others composed fictional or imaginative pieces. The fact that

imaginative or literary writing was the most common mode of writing in the lessons we observed

suggests potential challenges for teachers and students alike as they come to terms with the

CCSS. While the CCSS were not fully implemented at the time of these observations, they had

been published, and our study state’s schools have agreed to adopt them. Although it should be

noted that the fourth-grade statewide composition test writing prompt was a narrative format, this

still does not prepare students for the more demanding genres that constitute the backbone of the

CCSS: the expectation that by sixth grade students will be able to write opinion and

informational pieces that establish a clear, supporting point of view. Students who are not

proficient in writing informational text will not be able to meet the new standards. Even more

importantly, they will not be prepared for higher education or the workplace, in which

informational text predominates.

Closure. About two-thirds (68%) of the lessons came to a formal end with the teachers

summarizing some of the key ideas that were covered. Teachers provided closure in 81% of the

lessons rated “very effective” compared to 56% of the lessons rated “somewhat effective.” The

assignment of writing instruction homework was very rare—in only 8% of the lessons we

observed. However, we are not aware of any research that demonstrates a relationship between


the assignment of homework and effective writing. Of that 8%, homework involving writing was

commonly used as a way to give more practice in specific skills (50%) and to provide practice

with other kinds of writing (31%).

Study Limitations

One limitation of this study is that we only observed each teacher once during the study, with the

exception of about 40% of the returning fourth grade teachers in Year 3. Consequently, we may

not have adequately captured typical teaching practices in all cases. Our hope is that because we

observed teachers in random order and because we observed so many lessons, we were able to

obtain a “snapshot” of all the instruction. However, we note that it is possible that we

overestimated or underestimated the prevalence of some of the teaching practices. A second

limitation is that the classroom observers were trained by the same research team that created the

observation protocol. Therefore, the observers rated the effectiveness of each lesson according to

the nine features that the team selected and deemed important and were influenced by the other

features on the protocol that they observed or did not observe. While we think that all the items

on the protocol are driven by the research literature on writing and effective instruction, we may

have inadvertently introduced error into the ratings by our own beliefs. Third, we are aware that

the relationships we found between the features of instruction and the effectiveness ratings were

determined through a series of cross tabulations and that these correlations do not necessarily

imply causality. Finally, although the items in our protocol are based on what the professional

literature identifies as good writing instruction, we suggest the need for an analysis of student

outcome data to explore the predictive validity of the protocol.

Implications for Practice, Policy, and Research


We created the Writing Instruction Observation Protocol to capture the details of lessons

that focus specifically on writing instruction in the intermediate grades. This singular focus on

writing was essential not only because of the needs of our research study but also because of the

importance of writing in general—both in school and in adult life—and because of the emphasis

on writing in the CCSS. Researchers, principals, writing coaches, and other teacher leaders need

such a protocol to evaluate and recommend practices, as do teachers themselves if they intend to

examine their own classroom effectiveness.

The Writing Observation Protocol makes it possible to accurately observe how teachers

engage in a very challenging task: helping students improve their writing. The NAEP results, the

multiple reports citing the urgency of improving writing instruction in our schools, and the new

rigor within the CCSS all compel us to examine our findings with an eye toward practice, future

research, and policy. The ability to write a personal narrative, once the centerpiece of many state

assessments at the fourth-grade level, is now not enough to demonstrate writing proficiency and

growth. Our project’s 259 classroom observations provide baseline data for the state of writing

instruction in today’s classrooms. Below are some implications and recommendations based on

our findings.

Practice. The following practices—which our observers rated as effective, based on the

findings from the literature that inform our protocol—should be included daily in writing

instruction:

Explicitly state the objective of the writing session, explain how the lesson connects to

previous instruction—even to instruction from prior years—and activate students’ prior

knowledge about the topic.


During the instruction or practice of skills or strategies, work with individual students and

provide feedback to them about their writing. Feedback needs to be focused, clear, and

timely. Written feedback can provide students with documentation to which they can

refer when reviewing their writing.

When students are composing, provide examples of the skill or strategy that is the focus

of the lesson or of the type of writing. Engage students in a discussion about the

challenges and successes they experience when using the examples to inform their own

writing.

Model how to compose the type of writing that is required of students. Help students

understand and talk about the different types of writing and the approaches to

implementing each of them.

We also recommend research-based practices that we did not observe or only observed

rarely:

Include more informational/expository writing instruction and practice.

Work in dyads or small groups, as collaboration has been found to be an effective

instructional practice.

Provide small-group or whole-class discussions in which students reflect on what their

peers and their teachers have said and build on the discussion with new comments.

Use computers and/or tablets in the composing process.

Research. Our findings indicate many implications for further research. Although there

was no significant difference among the various effectiveness ratings for many of the items on

our protocol, additional research with a larger and especially more varied sample is needed to

test this out. For example, how important is it to tell students how or why what they are learning


will help them as writers? As well, are lessons more effective when the teacher asks open-ended

and/or probing questions? How important is it for students to share their writing? Are lessons

more effective when students share their writing with the whole class than when they work with

partners or in small groups? Are discussions about writing in small groups more effective than

they are with the whole class? While the amount of instructional time for writing was not a focus

of our study, what exactly is the optimal amount of writing instruction and practice per day or

week?

Research has identified some effective practices. However, that research is scant. We

would like to see more research designed to discover additional effective practices, along with a

concerted effort by researchers to partner with school districts in randomized control trials to

determine which writing practices are truly effective. Moreover, some of the current decisions

about effectiveness have been based on a limited number of studies, several of which were of

poor quality. It is important to determine—in real classrooms—which combinations of practices

help students become better writers. Finally, while we identified several clusters of instructional

features, we do not know if any particular combination leads to more effective lessons. There

was no single pattern or approach that was consistently associated with higher observer ratings of

overall effectiveness. Are there particular clusters that help to improve student writing more than

other clusters? Additional research might tell us.

Policy. Several policy implications can be drawn from our findings. School

administrators need to review their current writing curricula to ensure that they reflect the CCSS

and the research on writing. By fourth grade, students are required to write opinion papers on

literature and expository texts; their curriculum needs to support the progressive development of

these skills. Thus may be important for administrators to examine the types of writing instruction


provided in the early grades to ensure it includes more than lessons on writing personal narrative.

It is also essential that districts and schools provide professional development that focuses on

building teachers’ knowledge and expertise in teaching writing, particularly in using the

practices that we (and others) have found to be effective. This training needs to be intensive and

ongoing, with mentoring by trained personnel.

To enable these changes, school leaders will need to create a culture of writing by

allowing staff to examine student writing, perhaps with a structured protocol. They might also

consider cultivating in-house teacher-leaders who can support all teachers in writing across the

curriculum. School leaders of all kinds could take advantage of the protocol used in this study to

examine and inform writing instruction in their classrooms.

Finally, to bring about significant change in teaching practices, colleges and universities

with teaching preparation programs will need to stress writing instruction, with an emphasis on

the CCSS and research on effective writing practices.


NOTE:

The research reported here was supported by the Institute of Education Sciences, US

Department of Education, through Grant Number R305A090479 to Education Development

Center, Inc. The opinions expressed are those of the authors and do not represent views of the

Institute or the US Department of Education.


References

ACT. (2005). Crisis at the core: Preparing all students for college and work. Washington, DC:

Author. Retrieved from http://www.act.org/research/policymakers/pdf/crisis_report.pdf

ACT. (2013). The condition of college and career readiness 2013. Washington, DC: Author.

Retrieved from http://www.act.org/research/policymakers/cccr13/index.html

Bui, Y. N., Schumaker, J. B., & Deshler, D. D. (2006). The effects of a strategic writing

program for students with and without learning disabilities in inclusive fifth grade classes.

Learning Disabilities Research & Practice, 21(4), 224–260.

Coker, D., & Lewis, W. (2008, spring). Beyond writing next: A discussion of writing research

and instructional uncertainty. Harvard Educational Review, 78(1), 231–251.

College Board Advocacy and Policy Center. (2010). Teachers are the center of education:

Writing, learning and leading in the digital age. New York: College Board.

College Board’s National Commission on the Writing. (2006). Writing and school reform.

New York: College Board.

Deatline-Buchman, A., & Jitendra, A. K. (2006, winter). Enhancing argumentative essay writing

of fourth-grade students with learning disabilities. Learning Disability Quarterly, 29, 39–54.

Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to

broaden pre-service teacher development. (Master’s project) MACSME Program,

University of California, Berkeley.

Gilbert, G., & Graham, S. (2010). Teaching writing to elementary students in grades 4–6: A

national survey. The Elementary School Journal, 110(4), 494-518.


Graham, S., McKeown, D., Kiuhara, S., & Harris. K. R. (2012). A meta-analysis of writing

instruction for students in the elementary grades. Journal of Educational Psychology,

104(4), 879–896.

Graham, S., & Perin, D. (2007a). What we know, what we still need to know: Teaching

adolescents to write. Scientific Studies of Reading, 11(4), 313–335.

Graham, S., & Perin, D. (2007b). Writing next: Effective strategies to improve writing of

adolescents in middle and high schools – A report to Carnegie Corporation of New York.

Washington, DC: Alliance for Excellent Education.

Hillocks, G., Jr. (1986). Research on written composition: New directions for teaching.

Urbana, IL: ERIC Clearinghouse on Reading and Communication Skills and the National

Conference on Research in English.

Hillocks, G., Jr. (2008). Writing in secondary schools. In C. Bazerman (Ed.), Handbook of

research in writing: History, society, school, individual, text (pp. 311–329). New York:

Lawrence Erlbaum.

Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007). PEP observation protocol.

Appalachian Mathematics and Science Partnership, Partnership Enhancement Program.

Retrieved from http://www2.research.uky.edu/amsp/pub/

Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc

Levy, F., & Murnane, R. (2004). The new division of labor: How computers are creating the

next job market. Princeton, NJ: Princeton University Press.

Massachusetts Department of Education. (2001). Massachusetts English language arts

curriculum framework. Malden, MA: Author.


Murnane, R., & Levy, F. (1996). Teaching the new basic skills: Principles for educating

children to thrive in a changing economy. New York: Free Press.

National Center for Education Statistics. (n.d.). Common core of data: Identification of rural

locales. Retrieved from http://nces.ed.gov/ccd/rural_locales.asp

National Center for Education Statistics. (2012). The Nation's Report Card: Writing 2011

(NCES 2012–470). Washington, DC: U.S. Department of Education, Institute of Education

Sciences. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf

National Commission on Writing. (2003). The Neglected R: The need for a writing revolution.

Retrieved from http://www.host-collegeboard.com/advocacy/writing/publications.html

National Governors Association Center for Best Practices. (2005). Reading to achieve: A

governor’s guide to adolescent literacy. Retrieved from http://www.nga.org/cms/home/nga-

center-for-best-practices/center-publications/page-edu-publications/col2-content/main-

content-list/reading-to-achieve-a-governors-g.html

National Governors Association Center for Best Practices, Council of Chief State School

Officers. (2010). Common core state standards for English language arts & literacy in

history/social studies, science, and technical Subjects. Washington DC: Author.

National Institute of Child Health and Human Development (NICHD). (2000). Teaching

children to read: An evidence-based assessment of the scientific research literature on

reading and its implications for reading instruction: Reports of the subgroups (NIH

Publication No. 00-4754). Washington, DC: U.S. Government Printing Office.

Partnership for 21st Century Skills. (2011). Framework for 21

st century learning. Washington,

DC: Author. Retrieved from http://www.p21.org/overview/skills-framework


Persky, H. R., Daane, M. C., & Jin, Y. (2003). The nation’s report card: Writing 2002 (NCES

2003-529). U.S. Department of Education. Institute of Education Sciences. National Center

for Education Statistics. Washington, DC: Government Printing Office.

Sapien-Melchor, R. (2006, January). English language development observation tool. San

Diego County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/

ELDobservationtool.pdf

Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for

assessment of instructional practices in standards-based teaching. Journal of Mathematics

and Science: Collaborative Explorations, 6, 21–46.

Table 1

Schools and Classroom Sample for the Observations of

Writing Instruction Lessons

Frequency Percent

Teachers observed

259

Grade level 4 203 78.4

5 56 21.6

School district A 5 schools

B 9 schools

C 15 schools

D 14 schools

E 7 schools

F 6 schools

Total 56 schools

School year 2010–2011 119 45.9

2011–2012 140 54.1


Table 2

Descriptive Statistics for the Summary Ratings of the

Effectiveness of the Lessons

Rated item* Observed

n of

Lessons**

Mean SD Min Max

1. Lesson objective 231 2.88 .54 1 4

2. Activation of prior

knowledge

219 2.81 .60 1 4

3. Skill/strategy

instruction

232 2.07 .60 1 4

4. Models or modeling 196 2.92 .69 1 4

5. Guided practice 224 2.94 .60 1 4

6. Feedback 250 2.93 .52 1 4

7. Academic challenge 255 2.98 .44 2 4

8. Lesson integrity 257 2.93 .68 1 4

9. Classroom climate 257 3.04 .74 1 4

Composite (mean of all

nine ratings) a

258 2.92 .42 1.4 4.0

Single item overall rating

of effectiveness b

257 2.93 .53 2 4

*Rating scale for each item: 1 = not effective; 2 = somewhat effective; 3 = effective; 4 =

very effective.

** Observed number of lessons varies slightly because certain ratings were not applicable

for some lessons. The composite rating was based on the mean of the relevant ratings.

a. Cronbach’s alpha (internal-consistency reliability) for composite scale = 0.87.

b. Correlation of composite rating and single-item rating = 0.86, p < .0001.


Table 3

Features of Writing Instruction during the Introduction and the

Skills and Strategies Phase of the Lessons

Percentage of Lessons in Which Feature Was Noted

Phase of

the

Writing

Lesson

Observed Variable Across All

Lessons

Lessons

Rated

Somewhat

Effective

Lessons

Rated

Effective

Lessons

Rated

Very

Effective

Introducing the Writing Lesson

Teacher explicitly stated

the writing session’s

objective

91 82 92 100 *

Teacher explained how

previous lessons relate

to today’s writing

session

80 65 80 96 *

Teacher activated prior

knowledge relevant to

today’s writing session

63 46 66 74 *

Skills and Strategies During

Instruction/Practice

Teacher provided direct

skill/strategy instruction

96 90 97 96

Teacher provided

students with in-class

time to practice the

focus skills/strategies.

94 95 93 100

Teacher provided

example of writing

related to the instruction

and/or practice.

74 63 75 89 *

Teacher asked open-

ended and/or probing

questions.

59 65 56 78


Table 3




or why skills/ strategies/

processes will help

students as writers (the

“why” of instruction).

50 39 51 58

Teacher modeled the

process of using focus

skills/ strategies.

47 48 46 58

Teacher provided

indirect skill/strategy

instruction.

19 21 20 19

Teacher engaged

students in small group

discussion.

14 20 11 23

Teacher adjusted whole

class and/or small group

skill/strategy instruction

and/or practice based on

observations of

perceived student needs.

12 5 14 11

Teacher engaged

students in whole class

discussion.

8 15 5 19

What Were Students Doing?

Student Activities During

Instruction

Students worked on

paper.

76 64 79 70

Students worked alone. 72 56 76 67 ~

Students gave oral

responses.

63 73 60 70

Students worked with

class.

60 64 59 67

Students wrote single

words or phrases.

37 33 38 41

Students wrote single

sentences (not

connected).

27 20 31 19

Students marked text. 27 30 27 19


Table 3



Students worked with a

partner.

26 18 26 33

Students wrote more

than one sentence

(connected text).

25 15 28 19


small group.

13 23 11 15

Students worked in

“other” way

12 10 12 26

Students worked on

computer.

1 0 1 4

What Was the Teacher Doing?

During student work

time, the teacher

worked with

individuals.

77 59 80 82 *

During student work

time, the teacher

circulated around the

room.

76 62 79 78 ~

During student work

time, the teacher

addressed whole class.

70 72 68 74

During student work

time, the teacher

worked with small

groups.

14 18 14 7

During student work

time, the teacher did

“other.”

4 5 4 4

Teacher Feedback to Students

During the Lessons

Teacher gave students

feedback on their

writing or oral

responses.

98 100 98 96

If teacher gave 96 97 95 96


Table 3



feedback, was the type

verbal?

If teacher gave


in comments?

94 92 95 96

If teacher gave

feedback, was format

individual?

87 68 91 89 *

If teacher gave


whole class?

72 76 69 82

If teacher gave


small group?

14 21 15 4 ~

If teacher gave


in writing?

7 5 9 4

If teacher gave


in ‘other’ (explain)?

3 8 2 0

If teacher gave


in grades?

0 0 0 0

If teacher gave


in rubrics?

0 0 0 0

~ p < .10.

* p < .05, comparisons across the lessons rated somewhat effective, effective, and very

effective.


Table 4

Features of Writing Instruction during the Composing Phase of the Lessons

Percentage of Lessons in Which Feature Was Noted

Phase of

the

Writing

Lesson

Observed Variable Across

All

Lessons

Lessons

Rated

Somewhat

Effective

Lessons

Rated

Effective

Lessons

Rated

Very

Effective

Composing and writing

instruction: Student composing

and writing in class

Teacher provided

example of writing

related to the

composition lesson.

31 29 29 83 *


or why

skills/strategies/processes

will help students as

writers (the why of

instruction)?

28 38 22 40

Teacher modeled

relevant process of

writing.

18 7 22 33 ~

Teacher adjusted whole

class and/or small group

instruction based on

observations of perceived

student needs.

12 15 12 6

Students during the composing

time

Students worked alone. 98 95 98 100

Students worked on

paper.

97 95 97 100

Students worked with a

partner.

12 13 13 5


small group.

8 5 7 20

Students worked with 7 5 8 5


Table 4


class.

Students worked on

computer.

1 0 2 0

Students worked in

‘other’ way (explain).

1 3 1 0

Student tasks during composing

time

Students wrote to a topic. 78 80 80 57 ~

Students drafted

composition.

76 80 77 62

Students

planned/organized

composition.

23 28 22 30

Students revised

composition.

15 20 12 24

Students edited

composition.

6 10 6 5

Students gave or received

feedback on composition.

3 3 3 5

Students completed final

copy.

3 3 4 5

Students reflected on the

what-why-how of the

lesson.

1 0 2 0

Students freewrote.

0 0 1 0

Teachers during the composing

time

During student

composition time, the

teacher worked with

individuals.

94 95 94 91

During student


teacher circulated around

the room.

93 93 94 91

During student


teacher addressed whole

33 43 32 24


Table 4


class.

During student


teacher worked with

small groups.

13 13 12 19

During student


teacher did ‘other’

(explain).

2 0 4 0

Providing feedback on writing

Teacher gave students

feedback on their writing.

97 98 97 95

If teacher gave feedback,

was format individual?

94 93 95 91


was the type verbal?

91 93 90 91


was the type in

comments?

90 88 91 90


was format whole class?

34 45 30 43


was the type in writing?

17 18 17 19


was format small group?

12 10 12 19


was the type in ‘other’

(explain)?

1 2 0 5


was the type in grades?

0 0 0 0


was the type in rubrics?

0 2 0 0

Sharing of writing

Teacher had students

share their writing.

74 67 75 82


share their writing with

the whole class.

68 62 68 77


Table 4



share their writing with a

partner.

8 8 9 5


share their writing with a

small group.

3 0 4 4

Writing genres

Students are working in

imaginative/literary

genre.

81 72 81 95 ~

Students are working in

informational/expository

genre.

43 48 43 25

End of lesson steps

Teacher provided closure

to lesson.

68 56 69 81 ~

If teacher assigned

homework that involved

writing, the purpose was

to practice the focus of

targeted instruction.

50 60 50 0

If teacher assigned


writing, the purpose was

to engage in other kinds

of writing.

31 40 30 0

Teacher assigned


writing.

8 12 7 4

~ p < .10.

* p < .05, comparisons across the lessons rated somewhat effective, effective, and very

effective.

Appendix

Writing Instruction Observation Protocol

All observation questions refer to the writing instruction

Introduction

Does the Teacher:

1. Explicitly state, verbally, the writing session’s objective? Yes No

2. Explicitly explain how previous lessons relate to today’s writing session? Yes No

3. Activate prior knowledge relevant to today’s writing session? Yes No

Skills/Strategies Instruction/Practice

(___ minutes) __N/A

Does the Teacher:

4. Provide direct skill/strategy instruction? Yes No

5. Provide indirect skill/strategy instruction? Yes No

6. Provide an example of writing related to the instruction and/or practice? Yes No

7. Model the process of using focus skills/strategies? Yes No

8. Explain how or why skills/strategies/processes will help students as writers (the why of

instruction)? Yes No

Date ________________________ District ________________ School _________________

Teacher _____________________ Grade _________________ Observer _______________

Topic ______________________________

Length of Lesson: ___ Minutes ___ New Lesson ___ Continued Lesson


9. Ask open-ended and/or probing questions? Yes No

10. Engage the students in whole class discussion? Yes No

11. Engage the students in small group discussion? Yes No

12. Provide students with time, in class, to practice the focus skills/strategies? Yes No

13. Adjust whole class and/or small group skill/strategy instruction and/or practice based on

observations of perceived student needs? Yes No

14. What are the students doing during skill/strategy instruction/practice?

____ writing single words or phrases

____ writing single sentences (not connected)

____ writing more than one sentence (connected text)

____ marking text

____ oral responses

15. As part of the skills/strategies instruction and/or practice, students are working:

(Check all that apply)

___alone ___with a partner ___with a small group ___with the class

___ on paper ___ on computer ___ other (explain):________________________

16. What does the teacher do while students are working? (Check all that apply.)

___ circulates around the room

___ works with individuals

___ works with small groups

___ addresses whole class

___ other (explain): _____________________________________________________________


17. Does the teacher give students feedback on their writing or oral responses as part of the

skills/strategies instruction and/or practice? Yes No

If yes (check all that apply):

Format Types of Feedback

___ individually ___ comments ___ written ___ verbal

___ small group(s) ___ grades

___ whole class ___ rubrics ___ other (explain): ________________

Composing

(____minutes) ___N/A

Does the Teacher:

18. Provide an example of writing related to the lesson on composition? Yes No __ N/A

19. Model the relevant process of writing? Yes No __ N/A

20. Explain how or why skills/strategies/processes will help students as writers (the why of

instruction)? Yes No __ N/A

21. Adjust whole class and/or small group instruction based on observations of perceived student

needs? Yes No __ N/A

22. Have students working (Check all that apply)

___alone ___with a partner ___with small group ___with class

___ on paper ___ on computer ____other (explain): ______________________

23. In what part of the composing process are students engaged? (Check all that apply)

___planning/organizing ___drafting ___revising ___editing

___providing and/or receiving feedback ___ completing final copy

___ writing to a topic ___ freewriting ___ reflecting on what-why-how of lesson


24. What does the teacher do while students are composing?

___ circulates around the room ___ works with individuals

___ works with small groups ___ addresses the whole class

___ other (explain): _____________________________________________

25. Does the teacher give students feedback on their writing? Yes No

If yes (check all that apply:

Format Types of Feedback

___ individually ___ comments ___ written ___ verbal

___ small group ___ grades

___ whole class ___ rubrics ___ other (explain):

__________________________________

26. Does the teacher have students share their writing? Yes No

If yes (check all that apply):

____with a partner

____with small group

____whole class

27. What genre of writing are students working on today?

___ imaginative/literary type: _______________________________________

___ informational/expository type: _______________________________________

Miscellaneous

28. Does the teacher provide closure to the lesson? Yes No


29. Does the teacher assign homework that involves writing? Yes No

If yes, the purpose of the homework is

___ to practice the focus of targeted instruction

___ to engage in other kinds of writing

30. What are other adults in the room besides the teacher doing during the lesson?

______________________ __________________________________________________________

Title

______________________ __________________________________________________________

Title

______________________ __________________________________________________________

Title

Additional comments:

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________

__________________________________________________________________________________


31. Overall, how would you rate the effectiveness of this writing session? 1 2 3 4

Categories* 0=Not

Observed

1=Not

Effective

2=Somewhat

Effective

3=Effective

4=Very

Effective

N/A

Lesson Objective 0 1 2 3 4

Activation/Use of Prior

Knowledge

0 1 2 3 4

Direct Skill/Strategy

Instruction

0 1 2 3 4

Models and/or

Modeling

0 1 2 3 4

Guided Practice 0 1 2 3 4

Feedback 0 1 2 3 4

Academic Challenge 0 1 2 3 4

Lesson Integrity 0 1 2 3 4

Classroom Climate 0 1 2 3 4

* These categories were developed primarily from our observation protocol. Additional guidance was

derived from:

Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to broaden pre-

service teacher development. (Master’s project) MACSME Program, University of California,

Berkeley.


Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007. PEP observation protocol.

Appalachian Mathematics and Science Partnership, Partnership Enhancement Program. Retrieved

from http://www2.research.uky.edu/amsp/pub/

Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc

Sapien-Melchor, R. (2006, January). English language development observation tool. San Diego

County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/

ELDobservationtool.pdf

Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for assessment of

instructional practices in standards-based teaching. Journal of Mathematics and Science:

Collaborative Explorations, 6, 21–46.