![Page 1: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/1.jpg)
DYNAMIC INSTRUCTIONAL FORMATIVE FEEDBACK
(DIFF)PILOT STUDY
Val Verde Unified School District
Michael R. McCormick
Superintendent
June 20, 2016
![Page 2: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/2.jpg)
CAST OF CHARACTERS
Michael R. McCormick, Superintendent
Jennifer M. Doskocil, Elementary Coordinator
Sandy Sanford EdD, Assessment Consultant
Pete Goldschmidt PhD, Research Consultant
![Page 3: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/3.jpg)
VAL VERDE USD
Perris, Riverside County, CA
19,841 students
74% Hispanic
15% African American
6% White
82% Poverty
22% English Learners (30% at K)
13% Special Ed
![Page 4: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/4.jpg)
OUR NEED
Transition to a Formative approach to assessment as opposed to a Competition approach
Transition to an individual Teacher-Driven approach to assessment as opposed to a District-Dictated approach
![Page 5: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/5.jpg)
John Hattie
![Page 6: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/6.jpg)
James Popham
![Page 7: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/7.jpg)
Rick Stiggins
![Page 8: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/8.jpg)
“Accountability is something that is left when
responsibility has been subtracted.”Pasi Sahlberg, 2011
![Page 9: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/9.jpg)
OUR BRAND OF “FORMATIVE”
Both...
Short-Cycle “Pure” Formative Assessments, plus...
Medium-Cycle Assessments (≈Monthly) treated, analyzed, & with results employed “Formatively”
![Page 10: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/10.jpg)
PILOT RESEARCH QUESTIONS
Will teachers given full authority to choose all assessment items (Control Group) OR teachers choosing half the assessment items with an expert choosing the other half (Treatment Group) build assessments that align better to Smarter Balanced specifications with respect to mix of DOK, Item Types, Task Models, & Claims?
Which Group will perform better on the SA?
![Page 11: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/11.jpg)
TARGETED PROFESSIONAL DEVELOPMENT
3 hours training on the value of
DOK mix
Item Type mix
Task Model mix
Claim mix (Math)
Continual emphasis via Instructional Coaches
Reminders & emphasis via Candidate Item Lists with each Assessment Cycle
![Page 12: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/12.jpg)
EXPERIMENT (SCOPE)
5th Grade Mathematics—all students
Medium Cycle Assessments (≈Monthly)
5 Assessment Cycles
2-3 Content Standards per Testlet
Testlets (one per cycle; always 8 Items)
To be used Formatively
![Page 13: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/13.jpg)
EXPERIMENT (ITEM ALIGNMENT)
All Items created by Assessment Consultant (aka “Expert”)
Each item created to exactly match appropriate Smarter Balanced Item Specification
Mix (Item Type, Task Model, DOK, & Claim) as close as possible to SA variety
![Page 14: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/14.jpg)
![Page 15: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/15.jpg)
EXPERIMENT (CANDIDATE LIST)
Expert builds Candidate List of 20 items in advance of each Assessment Cycle. Items are authored in EADMS
2-3 Focus Content Standards per Cycle
Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment
Cover sheet (Group A or B specific) describing process with the following sheets showing each item with indication of standard, item type, Task Model, DOK, and Claim/Target
![Page 16: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/16.jpg)
![Page 17: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/17.jpg)
EXPERIMENT (GROUPS)
12 Elementary Schools (randomly parsed)
6 in Control Group (A)—26 teachers (23 net)
Each teacher chooses 8 items from Candidate List
Expert builds Testlets
6 in Treatment Group (B)—26 teachers (26 net)
Each teacher chooses 4 items from Candidate List
Expert chooses 4 items (to match SA specs)
Expert builds Testlets
![Page 18: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/18.jpg)
EXPERIMENT (ADMINISTRATION)
Candidate Item List
Published Teachers Study List
Group AChoose 8 Items
Group BChoose 4 Item
Group B
4 More Items AddedTestlets Created in
EADMS
Testlets Created in
EADMSTestlets Numbers
Sent to Teachers
Teachers Admin
Testlets
Scores Captured in
EADMS
Administration
Monitored in EADMS
Results Harvested
from EADMS
![Page 19: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/19.jpg)
CHALLENGES (INFO FLOW)
Transfer of directions & information difficult
Expert & Elementary Coordinator teamed
Used Google Drive/Docs to facilitate information flow between teachers, Coordinator, & Expert
Continuous monitoring
![Page 20: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/20.jpg)
CHALLENGES (RESISTANCE)
Tardy Execution & Apathy
Daily Monitoring via EADMS by Coordinator
Progressive Chain of Command feedback
![Page 21: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/21.jpg)
CHALLENGES (TOUGH ITEMS)
Difficulty of Writing Claim 2, 3, and 4 items
Claim 1 Concept & Procedures, 57%, DOK 1-2
Claim 2 Problem Solving, 6%, DOK 2-4
Claim 3 Communicating Reasoning, 23%, DOK 2- 4
Claim 4 Modeling & Data Analysis, 9%, DOK 2-4
![Page 22: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/22.jpg)
CHALLENGES (REPORTING)
Data Retrieval
20 Candidate Items
49 Unique Testlets
Problem aligning results in reporting system for the aggregate of Testlets
Reprograming of reports necessary
![Page 23: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/23.jpg)
CHALLENGES (ANALYSIS)
Data Analysis
Delay in calculating performance via SA (Residual Gain Scores)
Aligning data for IRT Analysis of the Grand Testlet
![Page 24: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/24.jpg)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
T-1
T-2
T-3
![Page 25: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/25.jpg)
ANCILLARY RESULTS (EQ ITEMS)
![Page 26: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/26.jpg)
ANCILLARY RESULTS (ALIGNMENT)
% of Teachers with ≥ 80% Alignment
1st Cycle
85% Group A (Control)
39% Group B (Treatment) before Expert
5th Cycle
81% Group A (Control)
65% Group B (Treatment) before Expert
![Page 27: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/27.jpg)
EXAMPLE CRITICAL ITEM
![Page 28: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/28.jpg)
ANCILLARY RESULTS (CRITICAL ITEM TYPE)
% of Teachers choosing Critical Item Type
1st Cycle
8% Group A (Control)
4% Group B (Treatment) before Expert
5th Cycle
31% Group A (Control)
12% Group B (Treatment) before Expert
![Page 29: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/29.jpg)
TEACHER SURVEY
1. More training on using data Formatively
2. More training on building effective Testlet
3. Did NOT like separate Testlets for each Teacher at a sight—want Testlets same at sight to facilitate grade level group analysis
![Page 30: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/30.jpg)
TEACHER SURVEY
4. Liked easy access to what they judged to be SA compliant items
5. Liked the concept of more control to teachers
![Page 31: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/31.jpg)
NEXT STEPS
Continue DIFF through 16-17 with school-level Testlets
Develop analytic protocol
Add targeted professional development in:
Building effective Testlets
Formative use of assessment data
Process hard data from experiment
Present at 2017 NCSA
![Page 32: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/32.jpg)
Q & A
![Page 33: Dynamic Instructional Formative Feedback (DIFF) Pilot Study...Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment Cover sheet (Group A or B specific) describing](https://reader030.vdocuments.net/reader030/viewer/2022040904/5e786f1d928eec2a8827b8cd/html5/thumbnails/33.jpg)
CONTACT US
Michael R. McCormick [email protected]
Jennifer M. Doskocil [email protected]
Sandy Sanford [email protected]
Pete Goldschmidt [email protected]