USING FORMATIVE AND BENCHMARK ASSESSMENTS IN AN RTI SYSTEM

Abby PotterEducation Consultant, Title I School Support Team

John HumphriesSchool Psychologist, Student Services/Prevention & Wellness

Agenda: 60 minutes

Abby & John: Introduction and Background

John: Benchmark Assessment—Screening Abby: Formative (Ongoing) Assessment John: Benchmark Assessment—Progress

Monitoring Abby & John: Q & A, Discussion as time

allows

Response to Intervention (RtI)

RtI is a process forachieving higher levels of academic and behavioral

success for all students through:

High Quality Instructional Practice

Continuous Review of Student Progress (multiple measures)

Collaboration

Response to Intervention (RtI)

A Systemic Approach forConstant Inquiry

Continuous Review of Student Progress

To Assess:• How all students are performing (screening)• How they are responding to differentiated core instruction

(ongoing assessment)• How they are responding to intervention/additional supports

(monitoring progress)

Topics for Assessment Types

Definitions Purposes/Rationale Strengths & Limitations Common features Research Resources for getting started

BALANCED ASSESSMENT

Continuous Review of Student Progress

6

Benchmark

Balanced Assessment System

Key Components: Continuum of assessments Multiple users Multiple information sources, used to

create a complete picture of student progress

Each assessment type has a primary purpose, as well as strengths and limitations

Balanced Assessment System

Formative Benchmark Summative

Daily Ongoing Evaluation Strategies Periodic Diagnostic/Progress Assessments Large-Scale Standardized Assessments

Immediate Feedback Multiple Data Points Across Time Annual Snapshot

Student-Centered Classroom/School-Centered School /District/State-Centered

8

CLASSROOM INTERIM LARGE SCALE What do I learn from this assessment?

What learning comes next for this student? How are students progressing? How well is this program working?

How are schools, districts, and/or states progressing?

Who will use the information?

Primary Users Students Teachers

Primary Users Students Teachers/ Teaching teams Parents Schools/Districts

Primary Users Secondary Users Schools/Districts Teachers State Parents Federal Students Independent Evaluators

What is the frequency of the assessment?

Short Cycle - occurring within and between lessons

Medium Cycle - occurring within and between instructional units

Long Cycle – occurring annually or bi-annually

What action will be taken with the results?

Within the instructional activity, information is used to change or adjust teaching

Students receive frequent and meaningful feedback on their performance

Teachers engage students in the monitoring of their own learning

Re-teach decisions

Professional learning communities use data to identify strengths and gaps in instruction and curriculum

Curriculum may be changed/refined Teachers may modify instruction for

individuals and groups of students based on their progression towards outcome goals (selected, targeted options)

Strategic long term evaluation of curriculum/programming

Determination of Adequate Yearly Progress

What professional development is needed for the primary users?

Need for a solid understanding of learning progressions and the feedback loop

Concrete examples of formative assessment strategies

Collaborative work time to share effective teaching strategies

Opportunities to practice and perfect usage Collaboration time plan for assessments

If district or school developed – identification of learning outcomes and time to develop items

If using a commercial product – time to learn features of the tool and how the results link to district curriculum and state standards

Time to disaggregate and interpret data Collaborative work time to share effective

teaching strategies

Understanding and interpretation of large scale assessment

Time to disaggregate data and evaluate progress at building level and grade level

Opportunities for revising instructional practices and/or curriculum

What is the connection to content standards?

Lessons are connected to the broader curriculum Curriculum is linked to grade level standards Grade level standards are linked to the framework

Examples Examples: feedback questioning, interviews, exit questions observations discussions un-graded class work

Examples: end of unit assessments district assessments (6-traits) benchmark assessments commercial products (See

www.studentprogress.org))

Examples: WKCE WAA NAEP MAPS?

Summative/ Large-Scale

Purpose : To determine how students in schools,

districts, and states are progressing

To inform curriculum and instruction

To determine Adequate Yearly Progress (AYP)

Benchmark Assessment

Purpose: To determine to what extent all students

are progressing (screening)

To determine how well additional supports or services are working before too much time passes (monitoring progress)

Formative Assessment

Purpose:

To consider what learning comes next for students

To improve learning while there is still time to act – before the graded event

Current Practices

What are you doing now to assess your students?

How is it working? WKCE and AYP WKCE Definition of Proficiency Differences WKCE and NAEP Achievement Gaps

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 20140%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

f(x) = 0.000285714285714286 x + 0.860666666666668R² = 0.0171428571428512

f(x) = − 0.00342857142857143 x + 0.458666666666667R² = 0.280519480519481

f(x) = 0.00514285714285715 x + 0.508666666666667R² = 0.15779220779221

% 4th Grade Proficient/Advanced WKCE Reading Statewide

White

Linear (White)

Black

Hispanic

SwD

Linear (SwD)

Econ. Dis.

ELL

Linear (ELL)

NCLB requires 100% by 2014.Does not include WAA.Data from WINNS, 9/10/08

RtI: From SPED to Gen Ed to Every Ed

Perspective: “The Art and The Science of Teaching”

Why RtI? Why now? One difference is the major advances in assessment and intervention technologies that allow us to make better decisions and intervene more appropriately. Moving from “true” CBM to more standardized measures.

Benchmark Assessment:Screening

Definitions Purposes/Rationale Strengths and Limitations Common features Research Resources for getting started: Academics

& Behavior

Screening: Definition

Screening is characterized by fast, inexpensive, repeatable data collection about critical skills, beliefs, or behaviors.

Screening usually identifies students who need further assessment or provides information for future planning activities.

Screening: Purposes/Rationale

The purpose of screening is to identify students who are “at-risk” of a poor outcome

Rationale: Use a screener with strong statistical properties along with other data to identify students you want to learn more about

Don’t wait until it’s too late. WKCE is a poor screener for this reason.

Screening: Strengths & Limitations

By definition, easy, quick, repeatable

Immediate results Guide

programming Predictive validity

Diagnostically Guiding

instruction Administrators Teachers

Absent good PM and Formative Asmt.

Statistical limitations

Strengths Limitations: How Misused

Selected Research on Screening

Jenkins, J. R., Hudson, R. F., & Johnson, E. S. (2007). Screening for service delivery in an RTI framework: Candidate measures. School Psychology Review, 36, 560-82.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103) New York: Macmillan.

Riedel, B. W. (2007). The relationship between DIBELS, reading comprehension, and vocabulary in urban first-grade students. Reading Research Quarterly, 42, 546–567.

Ritchie, K. D., & Speece, D. L. (2004). Early identification of reading disabilities: Current status and new directions. Assessment for Effective Intervention, 29(4), 13–24.

Snellen Eye Chart (1862).

World Health Organization: Principles of Screening (1968)

The condition should be an important health problem. There should be a treatment for the condition. Facilities for diagnosis and treatment should be available. There should be a latent stage of the disease. There should be a test or examination for the condition. The test should be acceptable to the population. The natural history of the disease should be adequately understood. There should be an agreed policy on who to treat. The total cost of finding a case should be economically balanced in

relation to medical expenditure as a whole. Case-finding should be a continuous process, not just a one time

project. Wilson JMG, Jungner G. Principles and Practice of Screening for

Disease. WHO Chronicle 1968;22(11):473

Resources for Screening

BASC CBCL Office Referrals Teacher Nomination TeenScreen GAIN-SS Online SOS

Go to National Center on Student Progress Monitoring

Also see The ABCs of CBM by Hosp etc.

MAP?

Behavioral Screening Academic Screening

Formative (Ongoing) Assessment

Definitions Purposes/Rationale Strengths and Limitations Common features Research Resources for getting started: Academics

& Behavior

Formative (Ongoing) Assessment

Definition:“Formative assessment is an intentional and

systematic process used by teachers and students during instruction that provides feedback to adjust on-going teaching and learning to improve students’ achievement of the intended instructional outcomes.”

CCSSO, 2007

Formative (Ongoing) Assessment

intentional systematic process feedback adjust on-going intended instructional outcomes

Purpose:• To consider what learning comes next

for the student• To improve learning while there is still

time to act – before the graded event

Formative (Ongoing) Assessment

Formative (Ongoing) Assessment

Examples: • Teacher observations• Teacher questioning & class discussions• Analysis of student work (graded & non-

graded)• Exit questions• Teacher feedback• Student self-assessment• KWLs• Student Journals

Strengths:• Informs day-to-day instruction• Informs intervention• Instant information• Student self-assessment• Provides information about on-going student progress• Designed & evaluated by those who know the students

best• Provides a huge volume of qualitative, descriptive data

Formative (Ongoing) Assessment

Formative (Ongoing) Assessment

Limitations: Time Informal/not standardized Overabundance of information May be challenging to ‘grade’ When used to the exclusion of other types

of assessment

Formative (Ongoing) Assessment

Essential components of effective formative assessment:

Learning Progressions: clearly articulate the sub-goals of the ultimate learning goal

Learning Goals and Criteria for Success: clearly identified and communicated to students

Descriptive Feedback: provided to studentswith evidence-based feedback that is linked to the intended instructional outcomes and criteria for success.

CCSSO, 2008

Formative (Ongoing) Assessment

Essential components of effective formative assessment (continued):

Self- and Peer-Assessment: important for providing students an opportunity to think metacognitively about their learning.

Collaboration: A classroom culture in which teachers and students are partners in learning should be established.

CCSSO, 2008

Formative (Ongoing) Assessment

Research•Inside the Black Box: Raising Standards Through Classroom AssessmentBy Paul Black and Dylan Wiliam (1998)

•New assessment beliefs for a new school missionBy Rick Stiggins (2004)

•Implementing Formative Assessment at the District Level: An Annotated Bibliography (New England Comprehensive Center)

Formative (Ongoing) Assessment

Resources for getting started: Academics & Behavior

Set learning goals and criteria for success

Select assessment techniques (teacher and students)

Determine how feedback is provided Organize information from formative

assessment (teacher and students)

Formative (Ongoing) Assessment

“Assessment FOR learning turns the classroom assessment process and its results into an instructional intervention designed to increase, not merely monitor, student learning.”

Richard Stiggins

Benchmarks: Progress Monitoring

Definitions Purposes/Rationale Strengths and Limitations Common features Research Resources for getting started: Academics

& Behavior

Progress Monitoring: Definition Progress monitoring (PM) is a

scientifically-based practice used to assess student performance and evaluate the effectiveness of instruction.

PM: Purposes/Rationale

PM has two purposes: Determine whether students are

progressing appropriately from additional supports and intervention

Build more effective supports and interventions

Rationale: Use PM to closely monitor whether what we’re doing is effective!

PM: Strengths & Limitations

High frequency Sensitive to

change Guide

programming more than screening

May have to make your own PM tools

Improper tools give invalid, unreliable results

Statistical limitations

Used in isolation

Strengths Limitations: How Misused

Research on Progress Monitoring

A substantial research literature Support a wide range of educational

decisions Beginning in 1977 as Data-Based Program

Modification (Deno & Mirkin, CEC) "Developments in Curriculum-Based

Measurement" by S.L. Deno, 2003, The Journal of Special Education, 37. 3., 184-192.

Progress Monitoring and SLD Federal regulations currently require: Information demonstrating that the student

received repeated assessments of achievement reflecting student progress §300.309(b)(2)

Data-based documentation is “…an objective and systematic process of documenting a child’s progress.” “…data documenting a child’s progress are systematically collected and analyzed…” Comments, Page 46657

National Perspective

President Obama on Monday: “…far too few districts are emulating the example of Houston and Long Beach, and using data to track how much progress a student is making and where that student is struggling - a resource that can help us improve student achievement.”

Resources for PM

Frequency of difficulties in school

Self-rating Parent/teacher

rating Determined by

treatment providers

Go to National Center on Student Progress Monitoring

Also see The ABCs of CBM by Hosp

Behavioral PM Academic Screening

Summary

Everyone has an important role in selecting assessments for RtI schools

Reading specialists have expertise in teaching and assessing reading skills

Teachers have expertise in aligning assessment with curriculum

School psychologists have expertise in tests and measurement for academics and behavior

How Does it Fit Together?

Addl.Diagnostic

Assessment

InstructionResults

Monitoring

IndividualDiagnostic

IndividualizedIntensive

weekly

All Students at a grade level

Fall Winter Spring

UniversalScreening

None ContinueWithCore

InstructionGuided byFormative

Assessment

SUMMATIVE•Grades•Discipline•AYP Measures

GroupDiagnostic

SmallGroupDifferen-tiatedBy Skill

2 times/month

Step 1Step 2 Step 3 Step 4

SupplementalServices

1-5%

5-10%

80-90%CoreInstruction

IntensiveSupport

Courtesy of Dave Tilley, Heartland AEA

Questions in Test Selection

How does this map to our data system? Does this test have adequate technical properties

for our intended use? Reliability & Validity Frequency Scale Alignment with our curriculum

How will we use the collected data? Don’t use a test outside of its how intended.

If using for SLD, federal regulations § 300.304 require use “for the purposes for which the assessments or measures are valid and reliable”

Achieving Balance

Thinking about your assessment system… Is your system balanced?

If not, are you okay with imbalance? How do the assessments support and

inform one another? Do all users know the purpose, strengths

and limitations of the assessments? What do you do with the results of the

different assessments? Does everyone play an important role? How deal with disagreement? Collaborate!

Myths and Misperceptions

Myth: “Running Records have no role in RtI” Myth: “DIBELS does not work—no research” Myth: “PM is SPED, no need in Gen Ed” Myth: “WKCE is worthless” Myth: “We already assess too much and now

RtI is going to make us assess more” Myth: “School Psychologists will rule the world

and take over reading programming” Myth: “Reading Specialists will rule the world

and take over school psychology programming”

Additional Research Sources

Response to Intervention: Research for Practice. 2007, NASDSE

www.rti4success.org www.rtinetwork.org (NASP and IRA)

Response to Intervention: A Research Review. Hughes & Dexter

Contact Information

Abby Potter aubree.potter@dpi.wi.gov (608) 267-7338

John Humphries john.humphries@dpi.wi.gov (608) 266-7189

USING FORMATIVE AND BENCHMARK ASSESSMENTS IN AN RTI SYSTEM

Abby Potteraubree.potter@dpi.wi.gov

John Humphriesjohn.humphries@dpi.wi.gov

(800) 441-4563 dial 6