kice 1 © regents of the university of california revisiting performance assessment 2.0 in 21 st...

KICE 1 © Regents of the University of California

Revisiting Performance

Assessment 2.0 in 21st Century

Accountability

Eva L. BakerNational Center for Research on Evaluation, Standards, & Student TestingGraduate School of Education & Information StudiesUniversity of California, Los Angeles

Korea Institute for Curriculum and EvaluationSeoul, Korea

27 October 2010


Assessment Revisited: Update on U.S. Educational Policy 2.0

• What have been the major issues in U.S. plans for reform, assessments, and accountability?

• What is Race to the Top?

• Where are we now? Performance assessment?

• What practical changes will need to occur?

• Comments and concerns


Educational Reform Update• Continuation and extension of President

Bush’s policies

• Educational performance (NAEP) is flat with some small gap closing; international standing poor

• Belief is that large bureaucracies (districts) and teacher unions cause low growth

• Adopting of policies to centralize some aspects of education and decentralize others

• Competition among States for resources


Race to the Top Provisions• Adopting standards and assessments that prepare

students to succeed in college and the workplace (Common Core of Standards 40+ states)

• Building data systems that measure student growth and success, and inform teachers and principals how to improve instruction

• Recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most

• Value Added Modeling (VAM)

• Turning around their lowest-performing schools


Where Are We Now? Race to the Top

• Competition among states for discretionary education money

• Competition heated because of poor economy and layoffs from schools (40 states have applied)

• Need to get State legislative approval to conform to federal requirements—charter schools; individual and teacher databases; different union relationships

• Note: “good” schools and districts have high test scores, despite evidence of direct practice, cheating, and reduction in curriculum offerings

• Teachers are mostly blamed for failure, and unions are demonized, charter schools are the answer


Additional Efforts

• Standards focus on college- and career-readiness in reading and math (science and history)

• Create pipelines and incentives to put the most effective teachers in high-need schools

• Alternative pathways to teacher and principal certification

• Charter school legislation (State-lifting caps)

• STEM


Race to the Top—Round 1

• 10 billion available

• Tennessee and Delaware won round one

1. Tennessee $500,000,000

2. Delaware $100,000,000


Race to the Top Awards— Round 2

1. Massachusetts $250,000,000

2. New York $700,000,000

3. Hawaii $75,000,000

4. Florida $700,000,000

5. Rhode Island $75,000,000

6. District of Columbia $75,000,000

7. Maryland $250,000,000

8. Georgia $400,000,000

9. North Carolina $400,000,000

10. Ohio $400,000,000


Common Core State Standards (CCSS)

• Mathematics and Literacy, for every grade

• Literacy includes “informative reading and writing,” topics include biological sciences and history

• Standards developed at the initiative of the State education officer and governor

• 35+ states participate, may have additional standards


Quality of Common Core State Standards

• Generally regarded as excellent

• Fewer, Clearer, Higher = Many, Clearer, Higher

• Good illustrative material

• Sequences unverified

• Use of ontologies by CRESST to clarify meaning and internal relationships funded by the Gates Foundation

• Developing formative and curriculum-based assessments and evaluating existing options


Common Core Standards Progression


Formal Representation

1 X 5

© Regents of the University of California


Assessment Funding for RTT• To avoid redundant procurement, States

banded together in three different consortia to bid for resources to design and specify assessments

• Assessments to include formative, for classroom use, interim for prediction within grade, accountability

• Reporting by growth and value-added modeling

• Enables performance assessment as part of the package


Two Consortia Won• Smarter Balance, with a bottom-up teacher

approach that promises the use of computer adaptive testing (45) states ($160M) dollars

• PARCC supported by Achieve, Inc., a Washington think-tank (26 states; $185.87M)

• Period of performance – 4 years, collaboration with universities, commercial groups

• Both groups at this point seem relatively conservative in their views


Financing Assessment

• Total awards for assessment in the $350M area

• Testing companies are anxious, needing to win the RFPs from the States even if they don’t agree with their plans

• Maintenance would go to publishers with States providing oversight

• Who owns standards? Assessments?


Smarter Balanced Consortia• New generation assessment system—deep disciplinary

understanding and higher-order thinking skills demanded by knowledge-based economies

• Relevant to ongoing improvements in instruction and learning

• Useful for all members of the educational enterprise: students, parents, teachers, school administrators, members of the public, and policymakers

• Formative and summative assessments, organized around Common Core Standards, that support high-quality learning and the demands of accountability

• Feasible, innovative assessment and fiscally sustainable

• RFPs to private companies


Smarter Balance• Assessments are integrated into a coherent

learning system of standards, curriculum, assessment, instruction, and teacher development

• Teachers and other instructional experts develop formative and summative assessments of standards—supports are provided to enable thoughtful teaching

• Teachers are integrally involved in the design, development, and scoring of assessment items and tasks

• Teachers’ roles include the construction and review of items/tasks, the definition of scoring guides, selection of student work exemplars, and scoring


21st Century Skills• Assessments include evidence of actual student

performance on challenging tasks that evaluate standards of 21st Century learning

• They emphasize deep knowledge of core concepts within and across the disciplines, problem solving, analysis, synthesis, and critical thinking

• Technology is designed to support assessment and learning systems

• by delivering the assessments; enabling adaptive technologies to better measure student abilities across the full spectrum of student performance

• evaluate growth in learning

• supporting online simulation tasks that test higher-order abilities


Performance Assessment

• Open ended

• Multiple steps

• Synthesis of standards

• More than one content area?

• Team or individual

• Judged by raters or markers


English Language Arts


Technology-Driven

• Semi-automated design with templates

• Just-in-time steps

• Help possible

• Simulations of processes

• Complex problem solving and decision making

• Written products, demos, research


Technical Issues—Again• Few “items”—old psychometrics

• Comparability of tasks, quantitative and qualitative features

• Confounded “difficulty”

• Vertical scaling

• Reporting for accountability, instructional feedback

• Cost

• Renewal


Technical Needs

• Robust and rapid online scoring

• Diagnostic and accountability accuracy

• Instructional sensitivity

• Extensive or inappropriate help

• Memorable tasks need to be replenished

• Cost (back to reusable designs)

• Game frameworks


Additional Concerns• Economics suggest that total amount available

from Federal government will be reduced

• Likely outcome is reduced innovation

• What kinds of validity studies should be conducted?

• CRESST scholars have proposed studies of accuracy of classification, utility of findings for improvement, instructional sensitivity, accommodations, and other fairness approaches—a struggle since there will be few revision resources and no full trial of the system


Research and Development Needs

• Validity approaches

• Modeling

• Sampling

• Verification of sequences of CCS and assessments

• Utility to teachers of different levels of training and experience

• Openness of feedback

• Continuous modification and longitudinal data

• Value-added issues


Speaking Truth to Power

• Not easy unless you are friends

• Policymakers in U.S. have odd mix of research-based ideas and preferences—now almost an overwhelming wave in one direction

• Desire conflicting goals without recognizing technical conflicts

• What should researchers do?


http://www.cse.ucla.edu

Eva L. [email protected]

voicefax

email

kice 1 © regents of the university of california revisiting performance assessment 2.0 in 21 st...

Documents