dynamic instructional formative feedback (diff) pilot study...mix of item types, task model, dok,...
TRANSCRIPT
DYNAMIC INSTRUCTIONAL FORMATIVE FEEDBACK
(DIFF)PILOT STUDY
Val Verde Unified School District
Michael R. McCormick
Superintendent
June 20, 2016
CAST OF CHARACTERS
Michael R. McCormick, Superintendent
Jennifer M. Doskocil, Elementary Coordinator
Sandy Sanford EdD, Assessment Consultant
Pete Goldschmidt PhD, Research Consultant
VAL VERDE USD
Perris, Riverside County, CA
19,841 students
74% Hispanic
15% African American
6% White
82% Poverty
22% English Learners (30% at K)
13% Special Ed
OUR NEED
Transition to a Formative approach to assessment as opposed to a Competition approach
Transition to an individual Teacher-Driven approach to assessment as opposed to a District-Dictated approach
John Hattie
James Popham
Rick Stiggins
“Accountability is something that is left when
responsibility has been subtracted.”Pasi Sahlberg, 2011
OUR BRAND OF “FORMATIVE”
Both...
Short-Cycle “Pure” Formative Assessments, plus...
Medium-Cycle Assessments (≈Monthly) treated, analyzed, & with results employed “Formatively”
PILOT RESEARCH QUESTIONS
Will teachers given full authority to choose all assessment items (Control Group) OR teachers choosing half the assessment items with an expert choosing the other half (Treatment Group) build assessments that align better to Smarter Balanced specifications with respect to mix of DOK, Item Types, Task Models, & Claims?
Which Group will perform better on the SA?
TARGETED PROFESSIONAL DEVELOPMENT
3 hours training on the value of
DOK mix
Item Type mix
Task Model mix
Claim mix (Math)
Continual emphasis via Instructional Coaches
Reminders & emphasis via Candidate Item Lists with each Assessment Cycle
EXPERIMENT (SCOPE)
5th Grade Mathematics—all students
Medium Cycle Assessments (≈Monthly)
5 Assessment Cycles
2-3 Content Standards per Testlet
Testlets (one per cycle; always 8 Items)
To be used Formatively
EXPERIMENT (ITEM ALIGNMENT)
All Items created by Assessment Consultant (aka “Expert”)
Each item created to exactly match appropriate Smarter Balanced Item Specification
Mix (Item Type, Task Model, DOK, & Claim) as close as possible to SA variety
EXPERIMENT (CANDIDATE LIST)
Expert builds Candidate List of 20 items in advance of each Assessment Cycle. Items are authored in EADMS
2-3 Focus Content Standards per Cycle
Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment
Cover sheet (Group A or B specific) describing process with the following sheets showing each item with indication of standard, item type, Task Model, DOK, and Claim/Target
EXPERIMENT (GROUPS)
12 Elementary Schools (randomly parsed)
6 in Control Group (A)—26 teachers (23 net)
Each teacher chooses 8 items from Candidate List
Expert builds Testlets
6 in Treatment Group (B)—26 teachers (26 net)
Each teacher chooses 4 items from Candidate List
Expert chooses 4 items (to match SA specs)
Expert builds Testlets
EXPERIMENT (ADMINISTRATION)
Candidate Item List
Published Teachers Study List
Group AChoose 8 Items
Group BChoose 4 Item
Group B
4 More Items AddedTestlets Created in
EADMS
Testlets Created in
EADMSTestlets Numbers
Sent to Teachers
Teachers Admin
Testlets
Scores Captured in
EADMS
Administration
Monitored in EADMS
Results Harvested
from EADMS
CHALLENGES (INFO FLOW)
Transfer of directions & information difficult
Expert & Elementary Coordinator teamed
Used Google Drive/Docs to facilitate information flow between teachers, Coordinator, & Expert
Continuous monitoring
CHALLENGES (RESISTANCE)
Tardy Execution & Apathy
Daily Monitoring via EADMS by Coordinator
Progressive Chain of Command feedback
CHALLENGES (TOUGH ITEMS)
Difficulty of Writing Claim 2, 3, and 4 items
Claim 1 Concept & Procedures, 57%, DOK 1-2
Claim 2 Problem Solving, 6%, DOK 2-4
Claim 3 Communicating Reasoning, 23%, DOK 2- 4
Claim 4 Modeling & Data Analysis, 9%, DOK 2-4
CHALLENGES (REPORTING)
Data Retrieval
20 Candidate Items
49 Unique Testlets
Problem aligning results in reporting system for the aggregate of Testlets
Reprograming of reports necessary
CHALLENGES (ANALYSIS)
Data Analysis
Delay in calculating performance via SA (Residual Gain Scores)
Aligning data for IRT Analysis of the Grand Testlet
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
T-1
T-2
T-3
ANCILLARY RESULTS (EQ ITEMS)
ANCILLARY RESULTS (ALIGNMENT)
% of Teachers with ≥ 80% Alignment
1st Cycle
85% Group A (Control)
39% Group B (Treatment) before Expert
5th Cycle
81% Group A (Control)
65% Group B (Treatment) before Expert
EXAMPLE CRITICAL ITEM
ANCILLARY RESULTS (CRITICAL ITEM TYPE)
% of Teachers choosing Critical Item Type
1st Cycle
8% Group A (Control)
4% Group B (Treatment) before Expert
5th Cycle
31% Group A (Control)
12% Group B (Treatment) before Expert
TEACHER SURVEY
1. More training on using data Formatively
2. More training on building effective Testlet
3. Did NOT like separate Testlets for each Teacher at a sight—want Testlets same at sight to facilitate grade level group analysis
TEACHER SURVEY
4. Liked easy access to what they judged to be SA compliant items
5. Liked the concept of more control to teachers
NEXT STEPS
Continue DIFF through 16-17 with school-level Testlets
Develop analytic protocol
Add targeted professional development in:
Building effective Testlets
Formative use of assessment data
Process hard data from experiment
Present at 2017 NCSA
Q & A
CONTACT US
Michael R. McCormick [email protected]
Jennifer M. Doskocil [email protected]
Sandy Sanford [email protected]
Pete Goldschmidt [email protected]