The Problem of Comprehension: The Problem of Comprehension:

A Potential Barrier to Reading Curriculum-Based A Potential Barrier to Reading Curriculum-Based Measurement as a Progress Monitoring and Measurement as a Progress Monitoring and Screening Instrument Screening Instrument

Mark R. Shinn, Ph.D.

Professor and School Psychology Program

National Louis University, Skokie, IL

markshinn@icloud.com

http://markshinn.org

February 21st, 2013

1 of 6 members of Technical Review Panel, National Center for Student Progress Monitoring, USDE/OSEP2003-2007

Editor and Contributor to 2 Major Texts on CBM

Author of More than 75 Refereed Journal Articles and Book Chapters on the Topic of CBM, Progress Monitoring, and Screening

My Area of Expertise

Mark R. Shinn, Ph.D. Serves as a Paid Consultant for Pearson Assessment for their AIMSweb product that provides CBM assessment materials and organizes and report the information from 3 tiers, including RTI. He provides technical support and training.

Mark R. Shinn, Ph.D. Serves as a Consultant for Cambium/Voyager/Sopris for their Vmath product, a remedial mathematics intervention but has no financial interests. He helped them develop their progress monitoring system.

Mark R. Shinn, Ph.D. Serves as a Consultant for McGraw-Hill Publishing for their Jamestown Reading Navigator (JRN) product and receives royalties.He helped them develop their progress monitoring system.

Mark R. Shinn, Ph.D. Serves as a Member of the National Advisory Board for the CORE (Consortium on Reaching Excellence) and receives a stipend for participation. He provides training and product development advice.

DisclosureDisclosure

QuickTime™ and aH.264 decompressor

are needed to see this picture.

Learning to Read is Critical for School (and Life) Success

Frequent Formative Assessment Like Weekly Use of Reading Curriculum-Based Measurement (R-CBM) is Among our Most Powerful Tools!

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.

And the Number 1 Most Powerful TEACHING Variable

What about comprehension?

What about “word callers?”

I know this kid who...reads “fluently” but

doesn’t understand a THING they read...

BUT...

Will Be

Available in

•pdf format

•iBook format

Presentation is Based on the Following White Paper in PRODUCTION

A “glossy” and official Pearson version will be finished soon and sent to you.

Shinn, M.R. (2013). The problem of comprehension: A potential barrier to Reading Curriculum-Based Measurement (R-CBM) as a progress monitoring and screening instrument. Minneapolis, MN: Pearson Assessment.

References on CBM Criterion Related Validity

Deno, S. L., Marston, D., Shinn, M. R., & Tindal, G. (1983). Oral reading fluency: A simple datum for scaling reading disability. Topics in Learning and Learning Disability, 2, 53-59.

Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239-256.

Fuchs, L. S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.

Hamilton, C., & Shinn, M. R. (2003). Characteristics of word callers:  An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

Miura Wayman, M., Wallace, T., Ives Wiley, H., Ticha, R., & Espin, C. (2007). Literature synthesis on curriculum-based measurement in reading. The Journal of Special Education, 41, 85-120.

Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 458-478.

10

Accessing Reading Materials

markshinn.org1. Click on the Downloads for

Professionals Icon

2. Click on the Presentations and Handouts Folder

3.Click on AIMSweb Webinar R-CBM and the Problem of Comprehension (Sponsored by Pearson) 2013 Folder

10

Big Ideas

1. When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general

reading skills.

2. A strong empirical relation has been demonstrated between R-CBM and measures of general reading

and reading comprehension (RC).

3. Persuasive empirical data are not always persuasive of teacher opinion.

4. General reading skills are necessary, but not sufficient for comprehension.

5. RC is not a single “thing."  Judgments are based on:

1. What a student is expected to read.

2. How the student will be assessed.

6. RC is influenced by a number of important variables, including

1. General reading ability

2. Language, especially vocabulary and familiarity with cultural idioms

3. Knowledge, especially content knowledge and knowledge of the world

4. Metacognitive strategies, including motivation and interest

2 Other Big Ideas We Can’t Discuss Today

1. If educators are concerned about specific student’s RC, then we must have a systematic

assessment process to address these concerns, starting with ruling out general reading skill

deficits, but also including interviews, observations, and diagnostic assessment.

See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.

Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD:

National Association of School Psychologists.

• If students have low general reading skills, RC strategy intervention should be time limited and

the use of such strategies should be actively taught, expected, supported, and rewarded in Tier

1, general education and/or content area classes.

See: Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension

strategies. American Educator, Winter, 39-50.

2 Other Big Ideas We Can’t Discuss Today

1. If educators are concerned about specific student’s RC, then we must have a systematic

assessment process to address these concerns, starting with ruling out general reading skill

deficits, but also including interviews, observations, and diagnostic assessment.

See: Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.

Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 671-698). Bethesda, MD:

National Association of School Psychologists.

• If students have low general reading skills, RC strategy intervention should be time limited and

the use of such strategies should be actively taught, expected, supported, and rewarded in Tier

1, general education and/or content area classes.

Willingham, D.T. (2006/07). The usefulness of Brief instruction in reading comprehension strategies.

American Educator, Winter, 39-50.

The Problem of “Fluency”

Experienced--and Often Powerful--Teachers Push Back When R-CBM is (Inappropriately) Interpreted and Communicated as “Fluency”

How Did This Happen?

– Errors in Judgment(s), Confusing Constructs with Behaviors Tested

– Tendency toward Reductionism

An Example of Reading Curriculum-Based Measurement (R-CBM)

It was a pretty good composition. I felt proud knowing it was the best one at my school. After I’d read it five times, I was impatient to start reading it out loud. I followed the book’s directions again. First I read the composition out loud without trying to sound impressive, just to hear what the words sounded like.

QuickTime™ and aH.263 decompressor

are needed to see this picture.

Billy, 4th Grader

Phonemic Awareness

Alphabetic Understanding

Fluency

Vocabulary

Comprehension

Oversimplifying the NRP

Strands are NOT BOXES-

Low Scores “in the Box” Led to

TEACHING the Things in the Box

SEPARATELYhttp://www.nationalreadingpanel.org/

Phonemic Awareness

Phonics

Fluency

Vocabulary

Comprehension

ALL These Skills

General Reading Skill

Consequences of (Mis) Interpretation as Fluency

Teacher Push Back and Shallow InterventionIf R-CBM Interpreted As

FluencyIf Interpreted R-CBM(Correctly) As

General Reading Ability

Goal is to Read Fast Goal is to Read WELL

Interventions Emphasize Speed

Interventions Emphasize Quality

Interventions are “Slices” or Bandaids

Interventions are Integrated or Bandages

Research Evidence of “Fluency-Driven” Intervention or

“Reading Faster”?

Our results indicate that repeated reading does not qualify as an evidence-based or promising practice for students with or at risk for learning disabilities

(p. 276)Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading

interventions for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263-281.

Implementing repeated reading and wide reading interventions without more formative intervention is not likely to be valuable (p. 9)

Wexler, J., Vaughn, S., Roberts, G., & Denton, C. A. (2010). The efficacy of repeated reading and wide reading practice for high school students with severe reading disabilities. Learning Disabilities Research & Practice, 25, 2-10.

Potentially Severe Educational Need

A Student with a Significant Reading Discrepancy

Poor Reading Quality as Well as Quantity

Reads Accurately?

Reads Efficiently with Automaticity?

Reads with Expression (Prosody)?

Effective Strategy for Unknown Words?

Errors Distort or Preserve Meaning?

Self Corrects Errors (Comprehension Self-Monitoring)?

Adjusts Pace When Text Difficulty Changes?

QuickTime™ and aH.264 decompressor

are needed to see this picture.

A Poor High School Reader

55

Grade 8 Material < 10th percentile at beginning of Grade 8

It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming

Comprehension Measure Criterion Measure Correlation

R-CBM SAT Word Study .80

SAT Comprehension 0.91

Question Answering SAT Word Study .66

SAT Comprehension .82

Recall SAT Word Study .58

SAT Comprehension 0.70

Cloze SAT Word Study .71

SAT Comprehension 0.72

Fuchs, L.S., Fuchs, D., & Maxwell, L. (1988). The validity of informal reading comprehension measures. Remedial and Special Education, 9, 20-28.

It’s Not an Empirical Issue--The Evidence is Compelling and Overwhelming

Shinn, M.R., Good, R.H., Knutson, N., Tilly, W.D., & Collins, V. (1992). Curriculum-Based reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21(3), 458-478.

.90.88 .85 .71 .72

But, I KNOW and HAVE SEEN Word Callers!!!!!!!

The Research

The idea of word callers has gained popularity despite a lack of evidence that applies “to an appreciable number of poor readers” (Stanovich, 1986, p. 372).

Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360-406.

But, I KNOW and HAVE SEEN Word Callers!!!!!!!

The Practice

31 of 75 Third Grade Teachers responded to a postcard put in their box asking them

“if you teach a third-grade student who can read fluently, but has difficulty comprehending text.”

41%!Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers'

judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

But, I KNOW and HAVE SEEN Word Callers!!!!!!!

Teachers Identified PAIRS of Kids in Their Classrooms:

Hamilton, C., & Shinn, M.R. (2003). Characteristics of word callers: An investigation of the accuracy of teachers' judgments of reading comprehension and oral reading skills. School Psychology Review, 32, 228-240.

Word Callers (WC) Similarly “Fluent” Peers

Read Fluently, But Didn’t Comprehend

Read Equally as Fluently (Well) As WC, But DID COMPREHEND

Teachers Predicted BOTH Pair’s R-CBM, Maze, and Oral Comprehension Question Answering Skills--THEN Both Pairs WERE TESTED

The “Word Caller”

Teachers Were Partially Right!

WCs “Comprehended” Less Well

WC

Peers

But WRC Was ALSO MUCH Lower

WC

Peers

Almost 2 Standard Deviations Lower

Almost 1.3 Standard Deviations Lower

So Where Do This Leave Us...

1. Reading Comprehension is Complex and We Tend to Oversimplify and Overgeneralize

• Reading is Necessary, But Not Sufficient for Understanding

• Reading Comprehension Comes From Reading, But is MORE Than Reading

“Comprehension” is Oversimplified

A-Rod hit the cover off of the ball, but ended the game with a 6-4-3 double play.

Factual:

Who is A-Rod?

What does “6” mean?

Inferential:

Why would people from Beantown celebrate this?

Why would this event mean different things in June than October?

Another Common Example

173-word paragraph with 14 sentences of words that are very short (about 4 letters) and rated as late Grade 6 by Flesch-

Kincaid

The procedure is actually quite simple. First, you arrange things into

different groups. Of course, one pile may be sufficient depending on how

much there is to do. If you have to go somewhere else due to lack of

facilities that is the next step, otherwise you are pretty well set. It is

important not to overdo things. That is, it is better to do too few things at

once than too many. In the short run, this may not seem important, but

complications can easily arise. A mistake can be expensive as well. After

the procedure is completed, one arranges the materials into different

groups again. Then they can be put into their appropriate places.

Eventually, they will be used once more and the whole cycle will have to

be repeated. At first, the whole procedure will seem complicated. Soon,

however, it will become just another fact of life. It is difficult to foresee

any end the necessity for this task in the immediate future, but then one

can never tell.

Let Me Add Some Information

The procedure is actually quite simple. First, you arrange things into different groups. Of

course, one pile may be sufficient depending on how much there is to do. If you have to go

somewhere else due to lack of facilities that is the next step, otherwise you are pretty well set.

It is important not to overdo things. That is, it is better to do too few things at once than too

many. In the short run, this may not seem important, but complications can easily arise. A

mistake can be expensive as well. After the procedure is completed, one arranges the

materials into different groups again. Then they can be put into their appropriate places.

Eventually, they will be used once more and the whole cycle will have to be repeated. At first,

the whole procedure will seem complicated. Soon, however, it will become just another fact of

life. It is difficult to foresee any end the necessity for this task in the immediate future, but

then one can never tell.

How to Wash Clothes

Reading Reading ComprehensionComprehension

KnowledgeKnowledge Fluency*Fluency*We Refer to It as We Refer to It as

General Reading SkillsGeneral Reading Skills

MetacognitionMetacognition

LanguagLanguagee

• • ProsodyProsody• • Automaticity/RateAutomaticity/Rate

• • AccuracyAccuracy• • DecodingDecoding

• • Phonemic AwarenessPhonemic Awareness

• • Oral Language SkillsOral Language Skills• • Knowledge of Language Knowledge of Language

StructuresStructures• • VocabularyVocabulary

• • Cultural InfluencesCultural Influences

•• Life ExperienceLife Experience• • Content KnowledgeContent Knowledge• • Activation of Prior Activation of Prior

KnowledgeKnowledge• • Knowledge about Knowledge about

TextsTexts

• • Motivation & Motivation & EngagementEngagement

• • Active Reading Active Reading StrategiesStrategies

• • Monitoring StrategiesMonitoring Strategies• • Fix-Up StrategiesFix-Up Strategies

*modified slightly from presentations by Joe Torgesen, Ph.D. Co-Director, Florida Center for Reading Research; www.fcrr.org

For Some, the Hardest Thing

They’ll Ever Do

The Easiest Thing To Teach

The Longer It Takes...

The Bigger Deficits Here

And Here

And the MOST Unmotivated

Here

Oral Reading is the EASIEST to Measure--Let’s Get This Down

and Add MORE Tools

Comes From Reading, But Is Not Reading

Reading Early, Well, and Wide (EWW) is Critical for Reading

Comprehension.

See: Hunter, P.C. (2012). It's not complicated! What I know for sure about helping our students of color

become successful readers. New York, NY: Scholastic.

Cunningham, A.E., & Stanovich, K.E. (1998). What reading does for the mind. American Educator, 8-15.

Denton, C.A., & Vaughn, S. (2010). Preventing and remediating reading difficulties: Perspectives from

research. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in

a three-tier model, including RTI (pp. 469-500). Bethesda, MD: National Association of School

Psychologists.

Linan-Thompson, S., & Vaughn, S. (2007). Research-based methods of reading instruction for English

language learners Grades K-4. Alexandria, VA: Association for Supervision and Curriculum

Development.

http://www.youtube.com/watch?feature=player_embedded&v=RiP-ijdxqEc

Comes From Reading, But Is Not Reading

Very Few Students Receive Evidence-Based or Sufficiently Explicit/Systematic

Comprehension Strategy Instruction, Especially in Navigating

Expository/Informational Text and Vocabulary, Learning the “Right Words” the

“Right Way.”

This is an INSTRUCTIONAL PROBLEM, Not An Assessment Problem!See: Feldman, K., & Kinsella, K. (2005). Narrowing the language gap: The case for explicit vocabulary

instruction. In Scholastic (Ed.), Read About: . New York, NYKamil, M.L., Borman, G.D., Dole, J., Kral, C.C., Salinger, T., & Torgesen, J. (2008). Improving Adolescent

Literacy: Effective Classroom and Intervention Practices: A Practice Guide. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Educational Sciences, U.S. Department of Education.

Marchand-Martella, N.E., Martella, R.C., Modderman, S.L., Petersen, H., & Pan, S. (2013). Key areas of effective adolescent literacy programs. Education and Treatment of Children, 36, 161-184.

Torgesen, J., Houston, D.D., Rissman, L.M., Decker, S.M., Roberts, G., Vaughn, S., . . . Lesaux, N. (2007). Academic literacy instruction for adolescents: A guidance document from the Center on Instruction. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

We Need to Expand Our Assessment Repertoire

We Need to Add a Variety of Ways to Authentically Assess Comprehension in the

Materials We Expect Students to be Able to Read and UNDERSTAND

See:

Howell, K. W. (2008). Best practices in Curriculum-Based Evaluation and advanced reading. In A.

Thomas & J. Grimes (Eds.), Best practices in school psychology IV (pp. 671-698). Bethesda, MD:

National Association of School Psychologists.

Howell, K.W., & Nolet, V. (1999). Curriculum-based evaluation: Teaching and decision making (3rd ed.). Atlanta, GA: Wadsworth Publishing Company.

Big Ideas

1. When students read passages aloud for 1 minute, we are obtaining a holistic assessment of general

reading skills.

2. A strong empirical relation has been demonstrated between R-CBM and measures of general reading

and reading comprehension (RC).

3. Persuasive empirical data are not always persuasive of teacher opinion.

4. General reading skills are necessary, but not sufficient for comprehension.

5. RC is not a single “thing."  Judgments are based on:

1. What a student is expected to read.

2. How the student will be assessed.

6. RC is influenced by a number of important variables, including

1. General reading ability

2. Language, especially vocabulary and familiarity with cultural idioms

3. Knowledge, especially content knowledge and knowledge of the world

4. Metacognitive strategies, including motivation and interest