presented to: ccsso 2012 national conference on student assessment june 27, 2012
DESCRIPTION
Quality Assurance in a Chaotic World. An External Perspective. Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012. The External Perspective. HumRRO serves as National Assessment of Educational Progress (NAEP) Quality Assurance contractor - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/1.jpg)
Presented to:CCSSO 2012 National Conference on Student Assessment
June 27, 2012
Quality Assurance in a Chaotic WorldAn External Perspective
![Page 2: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/2.jpg)
The External Perspective
HumRRO serves as – National Assessment of Educational Progress (NAEP) Quality
Assurance contractor
– Independent evaluator of California High School Exit Examination (CAHSEE)
– Psychometric second-checker for several states
In these roles, we stand outside operational assessments and provide feedback, warnings, guidance to federal and state clients
This presentation focuses on NAEP.
2
![Page 3: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/3.jpg)
NAEP Quality Assurance
Several years ago, the National Center for Education Statistics (NCES) contracted with a single vendor for NAEP. This vendor, in turn, managed subcontracts.
NCES now contracts directly with several agencies for:– Item Development
– Web/Technology Development, Operations and Maintenance
– Materials Distribution, Processing and Scoring
– Sampling and Data Collection
– Design, Analysis, and Reporting
– NAEP State Service Center
– Alliance
– Quality Assurance
3
![Page 4: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/4.jpg)
NAEP QA: Definitions
Quality Control (QC): integrated steps embedded within a process that check, verify, and document that data, analysis, and results are accurate.
Quality Assurance (QA): an external set of steps that
verify the QC process is effective, complete, and efficient.
4
![Page 5: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/5.jpg)
Categories of QA Activities
Participation in planning and monitoring meetings Observation of operational meetings (NAGB, DAC, NVS) Review of contractor QC plans Observation of operational processes Special studies
5
![Page 6: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/6.jpg)
Participation in planning and monitoring meetings
Annual Design Summit Annual Quality Summit Periodic contractor meetings Inter-contractor quality teams
6
![Page 7: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/7.jpg)
Observation of operational meetings
National Assessment Governing Board (NAGB) Design and Analysis Committee (DAC) NAEP Validity Studies Panel (NVS)
7
![Page 8: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/8.jpg)
Review of contractor QC plans
Annual advance review of QC plans Critical components include:
– Completeness
– Ongoing self-monitoring
• Recording relevant data
• Monitoring of trends
– Continuous process improvement
• Feedback loops
Importance of closing the loop with operational contractors
8
![Page 9: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/9.jpg)
Observation of operational processes
“Site visits” in four phases:– Item development
– Test administration
– Processing and scoring
– Analysis and reporting
Brief site visit reports after each event– Includes recommendations & commendations that are logged centrally
Annual summary report for each phase– Reflects upon overall patterns
9
![Page 10: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/10.jpg)
Special studies
Quality Assurance Technical Panel (QATP) Small number of studies with a quality focus, such as:
– Use of trend papers in scoring
– Investigation of “interesting” analytic results in advance of public release
– Investigation of alternative approaches to full population estimates
– Evaluation of alternative approaches to NAEP-TIMSS linkage
– Evaluating improvements to processes fir development and review of math items
10
![Page 11: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/11.jpg)
Examples of Specific Contributions
Improved contractor QC plans (e.g., specific decision criteria and follow-up actions)
Improved scoring process Improved training of field staff Increased self-monitoring by operational contractors Independent confirmation of a variety of findings before
public release Specific suggestions for following up on NAGB questions
and issues
11
![Page 12: Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012](https://reader036.vdocuments.net/reader036/viewer/2022062723/56813f5c550346895daa2c02/html5/thumbnails/12.jpg)
Benefits of an External QA Contractor
Another set of eyes Unbiased perspective Free from some of the time and workload stresses of
operational contractors
12