Managing Multi-State Assessment Consortia

Lessons From American Diploma Project Network Assessment Consortium

2AMERICAN DIPLOMA PROJECT NETWORK

Background: ADP Assessment Consortium Consortium Organization and Governance Potential Application Requirements: Demonstrating

Capacity for Success Increasing Odds of Success

3AMERICAN DIPLOMA PROJECT NETWORK

American Diploma Project Assessment Consortium

Initiated in 2005 by 9 states in Achieve-led ADP Network consortium has grown to 15 states

State share broader college- and career-ready policy agenda

Started with Algebra II EOC exam, expanded to include Algebra I

Agreed-upon policy purposes Improve curriculum and instruction and ensure consistency of

content and rigor Enable states to compare results across and within states Indicate readiness for college credit bearing mathematics

Agreement to college-ready cut score set by Achieve

4AMERICAN DIPLOMA PROJECT NETWORK

ADP Assessment Consortium

• Arkansas, Indiana, Kentucky, Maryland, Massachusetts, New Jersey, Ohio, Pennsylvania, Rhode Island

• Ohio acted as “lead state” in unprecedented multi-state procurement arrangement

• Pearson awarded contract in 2007 with addition of Algebra I in 2008

• Since the consortium began, six additional states have joined: Arizona, Florida, Hawaii, Minnesota, North Carolina, and Washington.

More states are expressing interest in the exam and joining the consortium…

In summer 2006, nine states issued RFP for the development of an Algebra II EOC Exam:

ID

AZ

UT

MT

WY

NM

CO

AL

SC

TN

KY

INOH

NC

SD

KS

NE

MN

WI

IA

IL

MO

AR

MS

OK

ND

OR

CA NV

WA

TX

PA

VA

NY

CT

WV MD

NJ

VTNH

MA

DE

RI

HI

GA

FL

ME

MI

LA

AK

DC

5AMERICAN DIPLOMA PROJECT NETWORK

Consortium Organization and Governance

ADP Consortium Member States Ohio – lead state Coordination and Direction Team (CDT) Achieve Center for Assessment – technical advisor Pearson – test developer selected competitively

6AMERICAN DIPLOMA PROJECT NETWORK6

New State Members

Membership is extended to any ADP Network state wanting to join

New member states must: Agree to terms of Partnership Agreement and MOU Agree to terms stated in the contract (but may add language) Participate in all CDT meetings, item development and review meetings Make firm, long range commitments

Contractual Personnel Financial

7AMERICAN DIPLOMA PROJECT NETWORK7

Role of the Lead State: Ohio

Appoint a procurement officer Conduct procurement Provide guidance to the CDT Administer the contract

Manage contract amendments Incorporate new states into the contract

8AMERICAN DIPLOMA PROJECT NETWORK8

The Coordination and Direction Team

Includes assessment directors or other high-ranking policy-making officials from each member state in the consortium

Oversees production and implementation of the Algebra I & II program

Ensures that legal and policy needs of each state are addressed during team deliberations and decision making

9AMERICAN DIPLOMA PROJECT NETWORK9

The CDT and Decision Making

The CDT states have: Multiple representatives One vote

Process of consensus, collaboration, and compromise: Relating to the policies of each state Repeated as necessary Less discussion time needed as process matures

For contractual matters, decisions must be unanimous

10AMERICAN DIPLOMA PROJECT NETWORK

Achieve’s Role

Convener and consensus builder Brought chiefs, state content and assessment directors,

governors & postsecondary leaders to the table Consortium Management

Communications with and among states Consensus building Project manager/oversee development contract Assembled additional expertise (technical, legal)

Content expertise ADP Benchmark standards Model course descriptions

11AMERICAN DIPLOMA PROJECT NETWORK

Achieve’s Role

Promote Postsecondary participation Math faculty at 2- and 4-year institutions System leadership With key national higher education organizations

Define college readiness Conceptually Set achievement levels and cut scores

12AMERICAN DIPLOMA PROJECT NETWORK

Key Lessons for States: Participating in a Consortium Takes Time and Communication

Active involvement of chief state school officer in start up phase is essential, so that state policy objectives and improvement strategy drive assessment design

Essential but difficult to sustain their engagement over time, as project moves to development and implementation phases

Governors and legislatures play critical roles in state testing policy; they must be engaged and informed along the way

The transaction costs for SEA staff are high compared with single-state assessment development

13AMERICAN DIPLOMA PROJECT NETWORK

Key Lessons for States: It’s Not Just About the Test

Developing common summative assessments is difficult Summative assessments not enough – coherent approach

that includes formative and interim assessments, curriculum and instructional materials, professional development is necessary to support instructional improvement

That requires more ambitious effort, and broader and deeper consensus among participating states

Common standards provides critical foundation for such an approach

14AMERICAN DIPLOMA PROJECT NETWORK

Key Lessons for States: Shared Vision is Essential

States must have common standards and must hold firm to a shared vision: of policy objectives of uses and purposes of assessments of assessment system design

Scope of the vision will determine other practices must also be held in common. The more comprehensive and coherent the vision, the more issues states must address in common

Within a consortium, willingness to compromise on behalf of larger vision and shared goals is essential To develop summative tests alone, states must still be willing to compromise on scores of

details (e.g. testing windows, testing modes, test security, test guidelines, calculator policy for mathematics, accommodations, etc.)

Additional compromise will be needed in more ambitious approaches

15AMERICAN DIPLOMA PROJECT NETWORK

Potential Application Requirements: Consortia Should be Asked to Demonstrate:

High degree of consensus around: Policy Objectives

Instructional Improvement Vision of assessment system

Leadership Capacity in each state Support of Governor Active Leadership by and commitment of Chief State

School Officer (not just state testing directors) and Postsecondary leaders (for college-readiness)

Engagement of State Board of Education and Legislature as appropriate in each state

16AMERICAN DIPLOMA PROJECT NETWORK

Potential Application Requirements: Consortia Should be Asked to Demonstrate:

Technical capacity across the consortium, matched to assessment system design

Consortium Governance and Decision-Making Structure

Consortium Procurement Strategy Roles and Responsibilities of Consortium

members and partners, including supporting organizations

17AMERICAN DIPLOMA PROJECT NETWORK

Increasing the Odds of Success Over Time

Applications should include a sustainability plan that addresses:

Anticipated (and unanticipated) leadership turnover, especially for governors and chiefs

Strategies for building and sustaining support among key stakeholders Ongoing communications with teachers, students and parents about

purposes and uses of assessments

Annual assessment costs, especially if: Projected annual costs expected to exceed current costs Continuous development (e.g., new items) is part of design, beyond life of

federal grant

Ongoing commitment of partner organizations through grant period

Managing Multi-State Assessment Consortia

Lessons From American Diploma Project Network Assessment Consortium