Progress Monitoring of Oral Language Using the Narrative Language Measures

Douglas B. Petersen, PhD, CCC-SLPUniversity of Wyoming

Objectives

• Describe the key features of a curriculum-based measure• Explain why narrative assessment is suitable for progress

monitoring• Describe the purposes of the NLM• Administer and score different subtests of the NLM• Describe how the NLM can be integrated into a multi-tiered

system of support for language

Defining the Problem

• "How we define a problem usually determines how we analyze it. It sends us in a particular direction.

• And how we analyze a problem—the direction we take —absolutely determines whether we find a solution and what the quality of that solution is.” -Morgan D. Jones

National Assessment of Educational Progress (NAEP, 2011)

Broad View of Reading

Reading

Word recognitionLinguistic Comprehension

Perfetti, 1985

Narrow View of reading

Reading Language

Kamhi, 2009

Word recognition

Linguistic Comprehension

Common Core State Standards

• “Children’s oral language competence is strongly predictive of their facility in learning to read and write…

• listening and speaking vocabulary and even mastery of syntax set boundaries as to what children can read and understand no matter how well they can decode.” (Appendix A, pg. 26)

Problem

Language Difficulty

Reading Comprehension Problems

Academic Failure

Direction - Solution

• To identify students who experience language difficulty so we can intervene and promote academic success as early as possible

• Language Difficulty– Not just a language disability– Includes English language learners– Includes cultural diversity

Multi-tiered System of SupportAs

sess

men

t AssessmentA

sses

smen

t Assessm

ent

A Paradigm Shift

• Purpose has shifted from eligibility and classification to early detection, prevention and progress monitoring.

Not QUALIFY-PLACE-SERVEBut TEST-TEACH-TEST

Purposes of Assessment

• Screening (Universal)– Helps answer “Who is at risk?”

• Diagnosis and Eligibility– Helps answer “Does this student have a disability?”

• Instructional/Intervention Planning– Helps answer “What and how should we teach?”

• Progress Monitoring– Helps answer “Is our intervention working?”and “What, if anything, needs to be adjusted?”

Curriculum Based Measurement

• Specific set of procedures for measuring student academic growth

• Designed to evaluate the effectiveness of instruction

• Primarily used for – screening – progress monitoring

• Often interpreted to mean the same thing as General Outcome Measurement

(Deno, 2003)

CBM Drives RTI

• Screening (Benchmark Assessment)– Universal benchmark screening (e.g., Fall, Winter, Spring)– Students’ scores are compared to a performance criterion– Identifies children who may require intensified services or more

frequent and targeted monitoring

• Progress monitoring– Regular sampling of performance (e.g., weekly, monthly)– Data are used to make decisions about

• Adjustments in intervention• Entry into and exit from tiers• Need for intensive “special” education

Requirements for Curriculum Based Measurement

1. Strong reliability and validity

2. Time efficient/easy to administer/score

3. Alternate forms of standardized tasks

4. Measure socially important outcomes

5. Sensitive to growth due to intervention

-Deno, 2003

Requirements for Language RTI

1. An effective, manualized language curriculum that is easy to use and sufficiently flexible to address the language needs of diverse children

2. Valid, reliable, easy to use, assessment tools with equivalent forms for frequent sampling of students’ language skills and measuring growth over time.

+

Claiming the Job

• “There is an urgent need for a simple, quick, standardized measure of language that can be administered repeatedly…in a manner akin to the continuous ‘dibbeling’ that is used to monitor progress in code learning.”

• “It behooves the profession and research community to claim the job of developing a set of very brief, standardized benchmark measures of vocabulary, grammar, and narrative…”

Ukrainetz, 2006

Narrative Language

• Narrative – an account of causally-related events in temporal order (aka. a story)

• Narrative language is associated with academic achievement.

• Narrative proficiency at age 5 is one of the best predictors of future reading comprehension (Catts, Fey, Tomblin, & Zhang, 2002; Dickinson & McCabe, 2001; Griffin, Hemphill, Camp, & Wolf, 2004.)

Narrative Language

• Narratives are socially important.• Both language comprehension and production

can be measured through narration.• Narration is a authentic, functional integration

of language• Narratives

…can be the intervention target…can be the intervention context

Reading Standards for Literature

Analysis of Narratives

• Story Grammar Analysis– Entire story is the unit of analysis– Story Grammar

• Linguistic Complexity Analysis– Words and clauses are the units of analysis– Linguistic Complexity

Story Grammar

• Story Grammar: Main parts of the story– causally and temporally related

• Reveal the plot or point of the story

• Reinforced and tested in American schools

Character Problem Feeling Action Ending

Setting End Feeling

Story Grammar

Linguistic Complexity

• The storyteller must create a context for the listener almost exclusively through the use of sophisticated, complex language.

Linguistic Complexity

• Similar to written language

– Adverbs (e.g., quickly)– Temporal subordination (e.g., When he was walking…)– Elaborated noun phrases (e.g., The huge dog that was

scary)– Mental & linguistic verbs (e.g., decided, considered)– Conjunctions (e.g., and, but, because) – Dialogue (e.g., He said, “I need a band-aide!”)

Quality Narratives

• A Quality Narrative…– includes important Story Grammar elements that create an

episode (plot)• Initiating Event (problem)• Attempt• Consequence

– includes complex language features that convey causal and temporal relations between key events (Ukrainetz, 2006).

Quality is impacted by components and the clarity of the relations between components

Narrative Language Measures (NLM)

• Three Subtests– Test of Narrative Retell (TNR)– Test of Personal Generation (TPG)– Test of Story Comprehension (TSC; Preschool only)

• Preschool, Kindergarten, First, Second, Third– 25 equivalent stories per grade level• 9 Benchmark stories (3 Fall, 3 Winter, 3 Spring)• 16 Progress Monitoring stories

Equivalent Features of the NLM

• Consistent across:– Story Grammar– Linguistic Complexity (in extreme detail)– Consistent vocabulary complexity with limited

overlapping vocabulary (across stories and across grade levels)

Technical Adequacy

• Because the TNR is a new measure, empirical investigations of its technical adequacy are ongoing.

• The following table outlines the available psychometric characteristics.

Evidence Preschool TNR Kindergarten TNR

Alternate Form Reliability: Bivariate Pearson Correlations

M = .77, p < .001 M = .75, p < .0001

Fidelity of Administration 91% 97%Real-Time Inter-rater Reliability 96% 94%Criterion-Related Evidence of Validity

Renfrew Bus Story: Strong Correlation

Frog Where Are You:No significant difference in MLU, NDW/TNW, Subordination, Story Grammar

Standardized Features of the NLM

• Standardized administration procedures – TNR– TPG– TSC (Preschool only)

• Standardized scoring guidelines– Real time– Audio file– Transcript

Standardized Administration

• Select the appropriate level• Benchmark TNR Purpose: compare children to an age or grade

criterion• Progress Monitoring TNR Purpose: show an individual child’s

language growth over time Materials and Setting

• The booklets are designed to be administered in order• The progress monitoring stories can be administered in a

different order or a story can be skipped. • Stories should never be repeated

Standardized Administration

Examiner says, “I’m going to tell you a story. Please listen carefully. When I’m done you are going to tell me the same story. Are you ready?”

Examiner reads the selected story word for word in a moderate pace with normal inflection.

Examiner says, “Thanks for listening. Now you tell me that story.”

Examiner should wait 10-15 seconds before providing encouragement.

If the child is reluctant, examiner encourages by saying “It’s okay. Just do your best.”

If the child asks for help, examiner says, “I can’t help but you can just tell the parts you remember.”

Examiner listens and makes only neutral comments such as “uh-huh” and “okay.”

Examiner may not repeat the story or any part of it, but can repeat the directions and provide encouragement as needed.

When the child appears to be finished, the examiner says, “Are you finished?”

If the child is not finished, the examiner should let the child continue.

If the child does not tell a story even after encouragement the examiner says, “That’s okay. Thanks for listening to my story.”

TNR:P with Pictures

• Lay out pictures before beginning the story.• As you read the story, point to the picture that

depicts the part of the story.

Time Efficient, Economical

Brief Administration • Three benchmark TNRs take about 5 minutes • A single TNR for progress monitoring takes less

than 2 minutes.

Reduced Scoring Time• Scoring can be done in real-time while the child is

retelling the story. • Scoring can be done by listening to an audio

recording

TNR Scoring in Real-Time

TNR Scoring in Real-Time

• Um, This kid, um Shane was getting ready for school because he was late and he couldn’t find his shirt so then he asked his mom to help him find it and she told him it was on the dryer and he got it and he was happy.

TNR Scoring in Real-Time

12 18

2

4

Some TNR Scoring Guidelines

• While listening to a child retell a story, the examiner cues in on key words or phrases provided in the scoring form.

• To be scored, the child’s utterances do not need to match the samples in the scoring rubric exactly or word for word; however they should be very close and contain the same amount of correct information.

• To earn 2 points, the utterance should be able to stand alone. The scorer should not ‐need to have shared knowledge of the story to correctly interpret the utterance.

• When children say something other than the exact words and phrases contained in the scoring form, examiners should write in what they say.

• It is helpful to underline or write words and phrases as children say them and once the administration is complete, rate the words or phrases as 2, 1, or 0 based on the general scoring guidelines.

Pgs. 7 & 9

Test of Personal Generation (TPG)

• Assesses language using a conversation elicitation technique – (adapted from McCabe & Rollins, 1994).

• Examiner elicits personal stories while playing with a young child in a very natural manner.

• Provide at least 3 opportunities.

• Most difficult narrative task we use.

TPG Variation

• Immediately following a child’s retell narrative (TNR), elicit a personal generation by asking, “Has something like that ever happened to you?”

• Samples a different dimension of their narrative skills in a very efficient manner– Content limitations– Motivation limitations

TPG Transcription

• C (Mm) my mom helped me in the bike.• E Tell me, keep going!• C Because I fall one day.• E Keep going!• C (And) and my mom say, “Just do it like this”. • C “And you go now”.• C (And) and my mom say, “Let’s do a race!” • C (And) and I won.

Benchmark Assessment (TNR)

• Benchmark testing for all students– Use student’s best/median score out of 3 administrations – Use a criterion score (level of performance) to assign level of risk

for each student (e.g., at-risk, some risk, no risk)– Use gain scores (or lack of gain) to identify students who would

benefit from more intensive intervention

Progress Monitoring

What About Diagnosis?

• Limited growth or response to instruction over time is a strong indicator of language impairment.– Chanthongthip, Petersen, Spencer, (2012)

What About Intervention Planning?

• While some tools are not capable of informing instruction, the NLM measures can directly inform a child’s language needs.

• Goals can be derived directly from the results of the NLM assessments.

• Narratives…can be the intervention target…can be the intervention context

Informing Instruction

=2 points =1 point =0 points

Informing Instruction

=2 points =1 point =0 points

Overview of Multi-tiered

Narrative Language

Intervention

Multi-tiered Language Instruction

• Multi-tiered Curriculum– Manualized– Flexible

• Key Features– Carefully structured stories– Engaging visual materials– Explicit teaching procedures– Fun and motivating

Engaging Visual Material

• Pictures for each story – 5 panels

• Story Grammar Icons– Character– Setting– Problem– Feeling

– Action– Ending– End Feeling

Character Problem Feeling Action Ending

Setting End Feeling

Story Champs - Tiers

• Large Group• Small Group• Individual

• Intensity increases as group size decreases– More opportunities to respond – More explicit and individualized targets – More specialized prompting – More contingent feedback– More frequent sessions– Increasing duration– More reliance on instructors with greater

expertise

Teaching Procedures

• Based on the effective teaching literature and principles of instruction– Frequent opportunities to respond– Explicit and individualized targets– Systematic scaffolding– Corrections– Least restrictive prompting– Curriculum-based measurement

Test of Narrative RetellM

ean

Score

s

Group x TimeF(2, 68) = 3.6, p = .03

d= .80

F(1, 68) = 4.1, p = .046, d = 0.49

F(1, 68) = 4.45, p = .039 d = 0.51

ANCOVA

Tota

l Sco

res

50-minutes Individualized Intervention: Treatment & Control Groups

Causality: Language Impairment Story Grammar: Language Impairment

Pretest Posttest0

0.5

1

1.5

2

2.5

3

3.5

4

Pretest Posttest0

0.20.40.60.8

11.21.41.61.8

2

Pretest Posttest0

0.5

1

1.5

2

2.5

3

3.5

4

Story Grammar: Typical LanguageCausality: Typical Language

0

Treatment GroupControl Group

Pretest Posttest0

0.5

1

1.5

2

LARGE GROUP

Systematic ScaffoldingSignalsStory Gestures

LARGE GROUPSystematic ScaffoldingSignalsStory GesturesQuestioningIndividual Turns

LARGE GROUP

Systematic ScaffoldingSignalsStory GesturesQuestioningIndividual TurnsLeast Restrictive PromptsCorrections

SMALL GROUPSeating/RotationPersonal Stories

Story Games

Scaffolding

CorrectionsPromptingDifferentiationVariations

INDIVIDUALPersonal Stories

Pictography

Scaffolding

Over-correctionsPromptingIndividualize Targets

Big Picture Recommendations

For more information, contact:

Doug Petersen: dpeter39@uwyo.eduTrina Spencer: tds95@nau.edu