An Overview of the Race-to-the-Top Assessment Consortia

Joe Willhoft, Executive DirectorSMARTER Balanced Assessment Consortium

NATD Annual Meeting April 8, 2011New Orleans

22Apr. 8, 2011

• Council of Chief State School Officers (CCSSO), National Governors Association (NGA), and Achieve, Inc. agree to partner on a common standards project

• Summer 2009: College and career readiness standards developed in English/Language Arts and Mathematics

• K-12 learning progressions developed leading to college and career readiness in high school

• Multiple rounds of feedback from states, teachers, researchers, higher education, and the general public

• June 2, 2010: Final Common Core State Standards (CCSS) released

33Apr. 8, 2011

Aligned with college and work expectationsFocused and coherentInclude rigorous content and application of

knowledge through high-order skillsBuild upon strengths and lessons of current state

standardsInternationally benchmarked so that all students are

prepared to succeed in our global economy and society

Based on evidence and researchState led – coordinated by NGA Center and CCSSO

44Apr. 8, 2011

• The US Department of Education’s Race-to-the-Top competition awarded extra points to a state’s application if the state provided evidence of adopting, or moving toward adoption of “a” common core of standards

• When adopting the standards, states may not remove standards, but may add to them as long as the added state-specific standards comprise no more than 15% of the total

55Apr. 8, 2011

• As of April 1, 2011, 43 states including the District of Columbia have adopted the Common Core State Standards

• The eight states that have not yet adopted the standards: Alaska, Minnesota, Montana, Nebraska, North

Dakota, Texas, Virginia, Washington

66Apr. 8, 2011

For more information about the CCSS, go to...

www.corestandards.org

77Apr. 8, 2011

The MOSAIC Formative Assessment Consortium• In addition to the points awarded in RTTT

applications for adoption of the Common Core, states get points if they are part of an assessment consortium for formative (not summative) assessments.

• In Oct/Nov 2009, Wisconsin, Nebraska, Missouri, Iowa, and Kansas establish a formative assessment consortium, called MOSAIC.

• About 20-25 states sign MOUs to join the MOSAIC consortium

88Apr. 8, 2011

The SMARTER Summative Assessment Consortium• In fall of 2009 the US Dept of Ed signals it will divert

$360M from RTTT for summative assessments aligned to a common core, and begins planning meetings

• December 2009: Oregon, Idaho, Washington, Utah, Hawaii confer and agree to collaborate on an online, adaptive summative assessment that would include performance items

• The “Summative Multi-state Assessment Resources for Teachers and Educational Researchers” (SMARTER); Web site is launched; weekly planning calls begin

99Apr. 8, 2011

SMARTER and MOSAIC Merge• About 20 states are engaged in both SMARTER

and MOSAIC

• January 2010: MOSAIC states want their formative and interim tools to align to a common summative scale –

• MOSAIC elects to merge with SMARTER and adopt the SMARTER name; MOUs are merged

1010Apr. 8, 2011

SMARTER merges with Balanced Assessment System

• About 20 states (leadership from New England Common Assessment Program and West Virginia) work with Linda Darling-Hammond on a consortium that would include a balance of stand-alone tests and extensive performance events and projects

• March 2010: The Balanced Assessment Consortium and SMARTER join to work on a common proposal as the “SMARTER Balanced Assessment Consortium”

1111Apr. 8, 2011

PARCC has an overlapping history• Fall 2009 conversations among chiefs in Florida,

Massachusetts, and nine or ten other states to develop a common assessment with a design similar to that of Florida, with a level of rigor at least at the level of Massachusetts.

• Early engagement with Achieve, Inc. as a development partner

• Commitment to a “through course” design, with focus on summative results

1212Apr. 8, 2011

• Fall 2009, declares $350M for summative assessments• USED meetings, with invited papers and testimony

about what a multi-state assessment program should look like: • Nov. 12-13 (Boston)• Nov. 17-18 (Atlanta)• Dec. 1-2 (Denver)

• NRC Board on Testing and Assessment:• 2 conferences on new directions for state

assessments: Dec. 10-11 and Apr. 6-7• Through March, no indication on what the assessment

grant will look like

1313Apr. 8, 2011

• April 7: USED release of “Notice Inviting Applications” (NIA) for the Race to the Top Assessment Program

• Applications due in 11 weeks

• No more than two awards up to $150M each for summative ELA/Math assessments in grades 3-8 and high school, with possible supplement in $10M pieces

• No more than one high school course assessments; $30M

• Grants for development only; preparing assessments that can be given in 2014-15

1414Apr. 8, 2011

• “Applicant” is a single state acting on behalf of the consortium; consortium must have at least 15 states with at least 5 governing states

• Two types of membership: Governing state (part of only one consortium, part of decision-making) or advisory (can be in both consortia, less involved in decisions)

• Member states submit an MOU signed by Governor, Chief School Officer, President of State Bd of Ed, Chief Procurement Officer

• Membership condition: States must adopt the consortium’s common core by Dec. 31, 2011 or drop

• Acquire services of a Project Management Partner

1515Apr. 8, 2011

• Assess acquisition of and progress toward “college and career readiness (students able to take credit-bearing courses in English/math upon entry to college)

• Have common, comparable scores across all consortium states

• Provide achievement and growth information that can be used for teacher and principal evaluation and professional development

• Assess all students, except those with “significant cognitive disabilities”

• Be administered online, with timely results• Use multiple measures

Federal Register /Vol. 75, No. 68 / Friday, April 9, 2010 pp. 18171-85

1616Apr. 8, 2011

• 31 states – 17 Governing, 14 Advisory• WestEd was as interim Project Mgt Partner• Substantial foundation support from Sandler,

Hewlett, and Gates foundations for WestEd contract and convening of consortium states; CCSSO provided fiscal management

• Requested $150 million and a $10M supplemental

• Obtained support from higher ed institutions (IHEs) representing > 75% of 2009 college-enrolled seniors

1717Apr. 8, 2011

• Two comprehensive assessment awards to the only two bidders:SMARTER Balanced Assessment Consortium (SBAC)

Partnership for the Assessment of Readiness for College and Careers (PARCC)

• Both awarded their entire request:SBAC ~ $160M PARCC ~$170M

• No award for high school course exams (one unsuccessful bidder)

• Supplemental $16M to SBAC and PARCC; to support states’ implementation of Common Core

1818Apr. 8, 2011

Fiscal Agent: Washington State

17 Governing States 12 Advisory StatesCT, HI, ID, KS, ME, MI, MO, MT, NC, NH, NM, NV, OR, UT, VT, WA, WI, WV

AL, CO, DE, IA, KY, ND, NJ, OH, PA, SC, SD, WY

Membership a/o 3-31-11

19Apr 8, 2011

30 States in the SMARTER Balanced Consortium(a/o March 31, 2011)

20Apr 8, 2011

20

Governing Board State Participating State

25 States in the PARCC Consortium(a/o March 31, 2011)

2121Apr. 8, 2011

Gov Adv Gov A/P Gov ParticConnecticut G Alabama A/P Arizona GHawaii G Colorado A/P Arkansas GIdaho G Delaware A/P California PIowa A Kentucky A/P District of Columbia GKansas G New Jersey A/P Florida GMaine G North Dakota A/P Georgia GMichigan G Ohio A/P Illinois GMissouri G Pennsylvania A/P Indiana GMontana G South Carolina A/P Louisiana GNevada G 0 9 Maryland GHew Hampshire G Massachusetts GNew Mexico G Mississippi PNorth Carolina G New York GOregon G Oklahoma GSouth Dakota A Rhode Island GUtah G Tennessee GVermont G 14 2Washington GWest Virginia GWisconsin GWyoming ATOTAL 18 3

PARCC Only

AlaskaMinnesotaNebraska

TexasVirginia

5

Not in Either

SBAC OnlyIn both

Consortia

Consortia Designs

2323Apr. 8, 2011

An eligible applicant’s application must include...“...(a) An executive summary of the eligible applicant’s proposed

project;

(b) A theory of action that describes in detail the causal relationships between specific actions or strategies in the eligible applicant’s proposed project and its desired outcomes for the proposed project, including improvements in student achievement and college- and career-readiness;

(c) A plan for designing and developing the proposed assessment system;

(d) A plan for research and evaluation of the proposed assessment system;

(e) A plan for implementing the proposed assessment system; and

(f) A project management plan (including a workplan and timeline)...” 

Federal Register; April 9, 2010 (p. 18174)

2424Apr. 8, 2011

• Grounded in a thoughtful, standards-based curriculum and represent an integrated system

• Assessments produce evidence of student performance

• Teachers involved in development and scoring of assessments

• System is state-led with transparent governance

• Assessments are structured to continuously improve teaching and learning

• Useful information on multiple measures that is educative for all stakeholders

• Adhering to established professional standards

2525Apr. 8, 2011

• State policies and practices that support consortium goals• Clear and rigorous content and skill targets for college

and career readiness• Effective communication of policies and standards• Supports and resources for teachers• Technology supports for learning and assessment systems• Rigorous summative assessments• Interim assessments and formative tools providing

actionable information• Teachers engagement in design, scoring, reporting of

assessments• Assessment information designed to improve learning

26Apr. 8, 2011

The SMARTER Balanced Theory of Action

All students leave high

school college and

career ready

Summative adaptive

assessments are benchmarked to

college and career readiness

Technology supports

innovative and

comprehensive

assessments

Technology provides

increased access to learning

State policies and practices

support increased

expectations

Common Core State

Standards specify K-12 expectations

for college and career

readiness

Clear communication of

expectations to stakeholders

Professional

capacity-buildingPD and other

supports for teachers to instruct on the CCSS

Teachers design and

score assessment items and

tasks

Interim assessments are used as progress

checks

Teachers use formative tools and

practices to improve

instruction

2727Apr. 8, 2011

How do we get from here... ...to here?

All students leave high

school college and career ready

Common Core State Standards specify K-

12 expectation

s for college and

career readiness ...and what can an assessment

system do to help?

28Apr. 8, 2011

The SMARTER Balanced Theory of Action

All students leave high

school college and career ready

Adaptive summative assessments

benchmarked to college & career

readinessCommon

Core State Standards specify K-

12 expectation

s for college and

career readiness

2929Apr. 8, 2011

Summative assessments using online computer adaptive technologies• Efficiently provide accurate measurement of all students, across

the spectrum of knowledge and skills, with shorter tests• Incorporate adaptive precision into performance tasks and events• Will assess full range of CCSS in English language arts and

mathematics; will include a variety of item types• Describe both current achievement and growth across time,

showing progress toward college- and career-readiness• Scores can be reliably used for state-to-state comparability, with

standards set against research-based benchmarks • The option of giving the summative tests twice a year.

30Apr. 8, 2011

The SMARTER Balanced Theory of Action

All students leave high

school college and career ready

Adaptive summative assessments

benchmarked to college & career

readinessCommon

Core State Standards specify K-

12 expectation

s for college and

career readiness

Interim assessments

that are flexible and open

3131Apr. 8, 2011

Optional interim assessments• Are aligned to and reported on the same scale as the summative

assessments• Help identify specific needs of each student, so teachers can provide

appropriate, targeted instructional assistance• Incorporate significant involvement of teachers in item and task

design and scoring• Are non-secure and fully accessible for use in instruction and

professional development activities • Provide students and teachers with clear examples of the expected

performance on common standards.

32Apr. 8, 2011

The SMARTER Balanced Theory of Action

All students leave high

school college and career ready

Adaptive summative assessments

benchmarked to college & career

readinessCommon

Core State Standards specify K-

12 expectation

s for college and

career readiness

Teachers can access formative tools and practices

to improve instructio

n

Interim assessments

that are flexible and open

3333Apr. 8, 2011

Web-based formative assessment resources• Online resources on assessment literacy, aligning assessments

to CCSS, and formative assessment guides• Training for local development of item and tasks and design and

use of scoring guides • Support of best practices through online learning modules • Comprehensive information portal, providing:

educator access to information about student progress toward college- and career-readiness

exchange of student performance history across districts and states

34Apr. 8, 2011

The SMARTER Balanced Theory of Action

All students leave high

school college and career ready

Adaptive summative assessments

benchmarked to college & career

readinessCommon

Core State Standards specify K-

12 expectation

s for college and

career readiness

Teachers can accessformative tools and practices

to improve instructio

n

Interim assessments

that are flexible and open

Optional Interim assessment system — no stakes

Summative assessment for accountability

Last 12 weeks of year*

DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.

Scope, sequence, number, and timing of interim assessments locally determined

2 PERFORMANCETASKS Each:

• Reading/Writing• Math

COMPUTERADAPTIVE TESTS w/

Re-take Option

The SMARTER Balanced Design

* Windows may be adjusted based on results from the research agenda and final implementation decisions.

English Language Arts and Mathematics, Grades 3 – 8 and High School

Computer Adaptive Tests and Performance Tasks

INTERIM ASSESSMENT

Computer Adaptive Tests and Performance Tasks

INTERIM ASSESSMENT

(Source: ETS K-12 Center for Assessment and Performance Management)

Through-courseASSESSMENT4

• Speaking• Listening

25%

Through-courseASSESSMENT 1• ELA• Math

50%

Through-courseASSESSMENT 2• ELA• Math

90%

END OF YEARCOMPREHENSIVE

ASSESSMENT

75%

Through-courseASSESSMENT 3• ELA• Math

PARTNERSHIP RESOURCE CENTER: Digital library of released items, formative assessments, model curriculum frameworks, curriculum resources, student and educator tutorials and practice tests, scoring training modules, and professional development materials

Summative assessment for accountability

Required, but not used tor accountability

Apr 19, 2023

English Language Arts and Mathematics, Grades 3 - 11

The PARCC Design

37

Jamal Abedi

UC Davis/CRESST

Randy Bennett

ETS

Derek Briggs

University of Colorado

Greg Cizek

University of North Carolina

David Conley

University of Oregon

Linda Darling-Hammond

Stanford University

Brian Gong

The Center for Assessment

Ed Haertel

Stanford University

Joan Herman

UCLA/CRESST

Jim Pellegrino

University of Illinois, Chicago

W. James Popham

UCLA, Emeritus

Joe Ryan

Arizona State University

Martha Thurlow

University of Minnesota/NCEO

38

Henry BraunBoston College

Bob BrennanUniversity of Iowa

Derek BriggsUniversity of Colorado

Wayne CameraCollege Board

Linda CookRetired, ETS

Ronald HambletonUniversity of Massachusetts

Gerunda HughesHoward University

Huynh HuynhUniversity of South Carolina

Michael Kolen University of Iowa

Suzanne LaneUniversity of Pittsburgh

Robert LuechtUniversity of North Carolina, Greensboro

Jim PellegrinoUniversity of Illinois at Chicago

Barbara PlakeUniversity of Nebraska, Lincoln

Rachel QuenemoenNational Center for Educational Outcomes

Laurie WiseHuman Resources Research Organization (HumRRO)

Challenges

4040Apr. 8, 2011

Multiple testing occasions per year in grades 3-8 and high school• Infrastructure readiness• Student readiness / Opportunity-to-learn issues• Rapidly-changing platform technologies• Software interoperability

Maintaining interoperability across consortia• 2 CCSS consortia; 2 “1%” consortia; 2 ELD

consortia; 2 science consortia(?)• Standardization vs. Innovation

4141Apr. 8, 2011

Development of growth scales• “Progress toward college / career readiness• Within grade or out-of-level testing?• Comparability across consortia

Standard setting for college/career readiness• What to use for empirical criteria and database?

Does college ready = career ready?

Reliable and Feasible Scoring of New Item Types• Reliance on AI scoring

4242Apr. 8, 2011

Who does what in 2014?• RTTA is for development only; consortia end in 2014• Within grade or out-of-level testing?• Comparability across consortia

Research needs: Bridge studies (old-to-new) and Comparability studies (across states)

Development and Use of Common Protocols• Common accommodations• Loss of local control?

4343Apr. 8, 2011

Ensuring that clear communications are well understood• Across multiple states• Down to the school/classroom level

Maintaining coherent and effective governance• Perils of being too top heavy• Risks if too disconnected from leadership

4444Apr. 8, 2011

...the SMARTER Balanced Assessment Consortium can be found online at

www.k12.wa.us/SMARTER