reciprocal accountability: using local and state assessment data to produce annual accountability...
TRANSCRIPT
Reciprocal accountability: Using local and state assessment data to produce annual
accountability determinations
Scott MarionNational Center for the Improvement of Educational Assessment
CCSSO: National Conference on Student AssessmentSan Diego, CA
June 22-24, 2015
2Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Beginning to Address Challenges of Assessment of CBE
3Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comprehensive Assessment Systems
• When done well, competency-based assessment systems can be instantiations of comprehensive assessment systems– Multiple types of assessments, e.g., summative, interim, unit-
based, formative
– Multiple loci of control—classroom, school, district, and state• What is the role of local curriculum (not standards) in CBE-based
assessment design?
• In many (most?) cases, comprehensive assessment systems require state and local cooperation
• PACE’s reciprocal system requires tight coherence among the state, collaborative, and local assessment systems
4Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Key Technical Challenges/Requirements for PACE
1. Reporting achievement (status)– In the language of USED, “annual determinations”
2. Calculating and reporting “growth”3. Defining, evaluating, and monitoring
comparability4. Ensuring and enhancing equity
• Focus today on producing comparable, annual determinations…
5Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comparability
• A lot of energy focused on comparability over the past five years (e.g., PARCC-Smarter Balanced)
• Psychometricians worry a lot about the differences among things like equating, concordance, projection…– Congressional staff now worried about differences among “equal,”
“equivalent,” and “comparable”
• The U.S. has a particular obsession with “interchangeability,” but we always need to ask ourselves comparable for what
• We also need to learn from other countries and some enlightened measurement specialists in the US
6Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Comparability Challenges in a Reciprocal SystemPACE
Comparable Annual DeterminationsPACE Common Performance Task
District-Level Competency Scores
Competency 1
Local performance assessments
Competency 2
Local performance assessments
Competency 3
Local performance assessments
And so on
Smarter Balanced in three grades
7Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Focus on Student Work
• Ultimately, comparability will be evaluated and achieved through intense interrogation of student work
• Multiple layers of the process– Within school calibration and comparable scoring– Development of common PLDs closely linked to Smarter Balanced– Across district comparability analyses– “Standard setting”
8Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Cross-District Comparability
• Student work samples are used to evaluate the comparability of scoring (and, by inference, expectations) across districts– The goal is not to repeat the inter-rater reliability analyses conducted
at the local levels
• Rather, the plan is to evaluate prototypical and borderline papers from each district to judge the extent to which students are held to common expectations
• For example….
9Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Matching Student Work to PLDs
Beginning Developing Approaching Target On Target Above Target Conceputalization
Equipartitions manipulatives of odd and even numbers (e.g., 27 divided in 3 groups; 24 divided in 4 groups) and explains equal or fair shares
places manipulatives within 100 into arrayed groups of more than two and explains (verbally or in written form) the relationship between repeated addition and multiplication
Self-creates models (e.g., manipulatives or drawings) within 100 into arrayed groups more than two to demonstrate the relationship between addition and multiplication
places manipulatives within 100 into arrayed groups of more than two and explains (verbally or in written form) the relationship between multiplication and division
Models and explains how models can show the relationship between multiplication and division
10Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Standard Setting
• Initial achievement levels need to be established within-district– Unique assessments for determining competency
• Comparability analyses will help calibrate differences in standards (e.g., proficiency) across districts
• Profile approach will be used with the student work generated from the common tasks as anchors– Smarter Balanced results will be used as key benchmark data– Next year, we will have work samples collected throughout the year
11Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Some additional resources
Domaleski, C., Gong, B., Hess, K., Marion, S., Curl, C., Peltzman, A. (2015). Assessment to support competency-based pathways. Washington, DC: Achieve. www.Achieve.org and www.nciea.org
Marion, S.F. & Buckley, K. (in press). Design and implementation considerations of performance-based and authentic assessments for use in accountability systems. In Braun, H. (ed). Meeting the Challenges to Measurement in an Era of Accountability. Washington, DC: NCME.
Marion, S. (2015, Feb). Two sides of the same coin: Competency based education and Student Learning Objectives. Published by Competency Works. http://www.competencyworks.org/resources/two-sides-of-the-same-coin-competency-based-education-and-student-learning-objectives/
Marion, S., & Leather, P. (2015). Assessment and accountability to support meaningful learning. Education Policy Analysis Archives, 23(9). http://dx.doi.org/10.14507/epaa.v23.1984
McCLarty, K. L. & Gaertner, M. (2015). Measuring mastery: Best practices for assessment in competency-based education. Washington, DC: American Enterprise Institute
For more information:
Center for Assessmentwww.nciea.org
12Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Scott [email protected]
13Marion_Reciprocal Accountability_CCSSO_June 24, 2015
Discussion Questions
1. What are the opportunities and challenges associated with implementing reciprocal accountability in your state?– Technical?– Policy?– Practical?
2. What should be the state’s role in a reciprocal system for ensuring that all students are being held to comparable expectations and being provided equitable opportunities to learn?