Overview of Data Quality Initiative

andReno Institute

Overview of Data Quality Initiative

andReno Institute

Vickie Schray, ChiefProgram Analysis Branch

Division of Vocational & Technical Education

Presentation OutlinePresentation Outline

• DQI Background• Changes for Reno Institute• Negotiation Process for 2000-2001• New Approach for Pilot-Testing• Negotiation Process for Years 3, 4, & 5• CAR and State Accountability Plans• Team Sessions• State Feedback on New Approach

DQI BackgroundDQI Background

Efforts to Date• Core indicator framework• State pilot projects• Regional TA meetings• State plan review process

Need for the DQI• Stakeholder confidence and credibility• Support state Perkins implementation• Managing continuous improvement

DQI BackgroundDQI Background

Goals• Improve student population definitions

• Improve selection and implementation of measures and measurement approaches

• Improve state systems for assuring data quality

Guiding Principles• Federal/state collaboration

• Peer networking

• Innovative use of technology

DQI BackgroundDQI Background

New Orleans DQI—February, 2001• Improve population definitions• Improve data quality for Core Indicators 1-3• Review quality criteria and scoring rubrics• Review peer collaborative resource network

Reno DQI—May, 2001• Improve data quality for special populations• Improve data quality for Core Indicator 4• Review state baseline and performance levels

DVTE Accountability Plan Negotiations—April-June, 2001

Changes for Reno InstituteChanges for Reno Institute

Accuracy of State Information

More Time, Fewer States Per Group

Improvements in Peer Evaluation Resource Guide

More Information on State Systems

Peer Review--Group Forms and Evaluation

Written State Feedback

Negotiation Process for2000-2001

Negotiation Process for2000-2001

Phase I: Negotiating Initial Baseline and Performance Levels

• Round 1-- Questions and Issues• Round 2-- Proposed Modifications and

Levels• Final--Interim Levels (#)

Phase II: Negotiating Final Baseline and Performance Levels

• Verification, updating, providing missing information

New Process:Proposed State Changes

New Process:Proposed State Changes

Setting Baseline Levels• Greater state choice in approaches and

years

Setting Performance Levels• State-by-state negotiation of performance

ceilings--not fixed 90%, state negotiated benchmarks

• State-by-state negotiation of 3-year and annual levels--not fixed improvement rates of 0.5% & 0.25 %

New Approach for Pilot-testing at RenoNew Approach for

Pilot-testing at Reno

Setting Baseline Levels• 2000-2001– most recent year or averaging up

to 3 years• New – greater choice of years and use of

alternative objective, replicable methods

Setting Performance Levels• 2000-2001– fixed 90% ceilings and

improvement rates of 0.5% & 0.25%• New -- State-negotiated performance

excellence benchmarks and flexible improvement rates resulting in comparable overall improvement

Negotiation Processfor Years 3, 4, and 5Negotiation Processfor Years 3, 4, and 5

Round 1: Certification of Accuracy and Completeness of State Accountability Plans

• Timelines: April 19-May 9, 2001

• Results Used for Reno Institute and Starting Negotiation Process-Rounds 2 and 3

Round 2: Verification and Negotiation of Baseline Levels

• Timelines: May 15-June 6, 2001

• Modified Approach from Reno Institute

Negotiation Process for Years 3, 4, and 5

Negotiation Process for Years 3, 4, and 5

Round 3: Verification and Negotiation of 3-Year and Annual Performance Levels

• Timelines: May 10-June 29, 2001• Modified Approach from Reno Institute

CAR and State Accountability Plans

CAR and State Accountability Plans

CAR Database• Annual reporting of actual performance for most

recent program year.• December 2001 (2000-2001) first year for

reporting against agreed-upon performance levels from State Plan

• December 2000 (1999-2000) used for first Report to Congress

State Accountability Plan• Negotiated baseline levels• Negotiated performance levels

Team Session 1:Overview of State Systems

Team Session 1:Overview of State Systems

Definition and Size of Student Populations

• Implications of definitions for performance

Definitions of Measures and Measurement Approaches

• Student populations being addressed

Data Quality Improvement Priorities• Implications for performance levels

Team Session 2:Non-Traditional Participation

and Completion

Team Session 2:Non-Traditional Participation

and Completion

Identifying Non-Traditional Programs• Approach for linking to national/state

occupational data

Measure Construction• Student populations being addressed

Data Quality Improvement Priorities• Student/program coverage--all non-trad

programs• Implications for performance levels

Team Session 3:Special PopulationsTeam Session 3:

Special Populations

Statewide Definitions

Procedures for Identifying Students

Reliability of Classification

Comparable Student Coverage of Special Populations on Performance Measures

Team Session 4:Setting Baseline Performance

Levels

Team Session 4:Setting Baseline Performance

Levels

Selection of Years• Use of most recent data

Method for Calculating Baseline Levels

• Formal, objective and replicable by others

Adjusting to Data Quality Improvements

Team Session 5:Setting 3-Year and Annual

Performance Targets

Team Session 5:Setting 3-Year and Annual

Performance Targets

Determining Performance Excellence Benchmarks

• State approach based on national/state data

Setting 3-Year and Annual Performance Levels

• “Fact-based” decision-making--gap analysis and expected results from strategies

Comparable Improvement• Comparing overall improvement • Justification of lower overall improvement

State Feedback on New Approach

State Feedback on New Approach

Report Out on Peer Review Process and Approach

• Determining Performance Ceilings• Setting Baseline Levels• Setting Performance Targets

Additional State Feedback• Regional Meetings• Peer Collaborative Resource Network

Incorporating Changes into Round 2 and Round 3 Negotiations

Subindicator Baseline Year(s) Method

1S1 1999-2000 Most Recent Year

1S2 1999-2000 Most Recent Year

2S1 1997-1998, 1998-1999, 1999-2000 3-Year Average

3S1 1999-2000 Most Recent Year

4S1 6 Years (1994-2000) Working Group Recommendation

4S2 6 Years (1994-2000) Working Group Recommendation

1P1 1997-1998, 1998-1999, 1999-2000 3-Year Average

1P2 1997-1998, 1998-1999, 1999-2000 3-Year Average

2P1 1997-1998, 1998-1999, 1999-2000 3-Year Average

3P1 1997-1998, 1998-1999, 1999-2000 3-Year Average

4P1 6 Years (1994-2000) Working Group Recommendation

4P2 6 Years (1994-2000) Working Group Recommendation