Overview of Data Quality Initiative
andReno Institute
Overview of Data Quality Initiative
andReno Institute
Vickie Schray, ChiefProgram Analysis Branch
Division of Vocational & Technical Education
Presentation OutlinePresentation Outline
• DQI Background• Changes for Reno Institute• Negotiation Process for 2000-2001• New Approach for Pilot-Testing• Negotiation Process for Years 3, 4, & 5• CAR and State Accountability Plans• Team Sessions• State Feedback on New Approach
DQI BackgroundDQI Background
Efforts to Date• Core indicator framework• State pilot projects• Regional TA meetings• State plan review process
Need for the DQI• Stakeholder confidence and credibility• Support state Perkins implementation• Managing continuous improvement
DQI BackgroundDQI Background
Goals• Improve student population definitions
• Improve selection and implementation of measures and measurement approaches
• Improve state systems for assuring data quality
Guiding Principles• Federal/state collaboration
• Peer networking
• Innovative use of technology
DQI BackgroundDQI Background
New Orleans DQI—February, 2001• Improve population definitions• Improve data quality for Core Indicators 1-3• Review quality criteria and scoring rubrics• Review peer collaborative resource network
Reno DQI—May, 2001• Improve data quality for special populations• Improve data quality for Core Indicator 4• Review state baseline and performance levels
DVTE Accountability Plan Negotiations—April-June, 2001
Changes for Reno InstituteChanges for Reno Institute
Accuracy of State Information
More Time, Fewer States Per Group
Improvements in Peer Evaluation Resource Guide
More Information on State Systems
Peer Review--Group Forms and Evaluation
Written State Feedback
Negotiation Process for2000-2001
Negotiation Process for2000-2001
Phase I: Negotiating Initial Baseline and Performance Levels
• Round 1-- Questions and Issues• Round 2-- Proposed Modifications and
Levels• Final--Interim Levels (#)
Phase II: Negotiating Final Baseline and Performance Levels
• Verification, updating, providing missing information
New Process:Proposed State Changes
New Process:Proposed State Changes
Setting Baseline Levels• Greater state choice in approaches and
years
Setting Performance Levels• State-by-state negotiation of performance
ceilings--not fixed 90%, state negotiated benchmarks
• State-by-state negotiation of 3-year and annual levels--not fixed improvement rates of 0.5% & 0.25 %
New Approach for Pilot-testing at RenoNew Approach for
Pilot-testing at Reno
Setting Baseline Levels• 2000-2001– most recent year or averaging up
to 3 years• New – greater choice of years and use of
alternative objective, replicable methods
Setting Performance Levels• 2000-2001– fixed 90% ceilings and
improvement rates of 0.5% & 0.25%• New -- State-negotiated performance
excellence benchmarks and flexible improvement rates resulting in comparable overall improvement
Negotiation Processfor Years 3, 4, and 5Negotiation Processfor Years 3, 4, and 5
Round 1: Certification of Accuracy and Completeness of State Accountability Plans
• Timelines: April 19-May 9, 2001
• Results Used for Reno Institute and Starting Negotiation Process-Rounds 2 and 3
Round 2: Verification and Negotiation of Baseline Levels
• Timelines: May 15-June 6, 2001
• Modified Approach from Reno Institute
Negotiation Process for Years 3, 4, and 5
Negotiation Process for Years 3, 4, and 5
Round 3: Verification and Negotiation of 3-Year and Annual Performance Levels
• Timelines: May 10-June 29, 2001• Modified Approach from Reno Institute
CAR and State Accountability Plans
CAR and State Accountability Plans
CAR Database• Annual reporting of actual performance for most
recent program year.• December 2001 (2000-2001) first year for
reporting against agreed-upon performance levels from State Plan
• December 2000 (1999-2000) used for first Report to Congress
State Accountability Plan• Negotiated baseline levels• Negotiated performance levels
Team Session 1:Overview of State Systems
Team Session 1:Overview of State Systems
Definition and Size of Student Populations
• Implications of definitions for performance
Definitions of Measures and Measurement Approaches
• Student populations being addressed
Data Quality Improvement Priorities• Implications for performance levels
Team Session 2:Non-Traditional Participation
and Completion
Team Session 2:Non-Traditional Participation
and Completion
Identifying Non-Traditional Programs• Approach for linking to national/state
occupational data
Measure Construction• Student populations being addressed
Data Quality Improvement Priorities• Student/program coverage--all non-trad
programs• Implications for performance levels
Team Session 3:Special PopulationsTeam Session 3:
Special Populations
Statewide Definitions
Procedures for Identifying Students
Reliability of Classification
Comparable Student Coverage of Special Populations on Performance Measures
Team Session 4:Setting Baseline Performance
Levels
Team Session 4:Setting Baseline Performance
Levels
Selection of Years• Use of most recent data
Method for Calculating Baseline Levels
• Formal, objective and replicable by others
Adjusting to Data Quality Improvements
Team Session 5:Setting 3-Year and Annual
Performance Targets
Team Session 5:Setting 3-Year and Annual
Performance Targets
Determining Performance Excellence Benchmarks
• State approach based on national/state data
Setting 3-Year and Annual Performance Levels
• “Fact-based” decision-making--gap analysis and expected results from strategies
Comparable Improvement• Comparing overall improvement • Justification of lower overall improvement
State Feedback on New Approach
State Feedback on New Approach
Report Out on Peer Review Process and Approach
• Determining Performance Ceilings• Setting Baseline Levels• Setting Performance Targets
Additional State Feedback• Regional Meetings• Peer Collaborative Resource Network
Incorporating Changes into Round 2 and Round 3 Negotiations
Subindicator Baseline Year(s) Method
1S1 1999-2000 Most Recent Year
1S2 1999-2000 Most Recent Year
2S1 1997-1998, 1998-1999, 1999-2000 3-Year Average
3S1 1999-2000 Most Recent Year
4S1 6 Years (1994-2000) Working Group Recommendation
4S2 6 Years (1994-2000) Working Group Recommendation
1P1 1997-1998, 1998-1999, 1999-2000 3-Year Average
1P2 1997-1998, 1998-1999, 1999-2000 3-Year Average
2P1 1997-1998, 1998-1999, 1999-2000 3-Year Average
3P1 1997-1998, 1998-1999, 1999-2000 3-Year Average
4P1 6 Years (1994-2000) Working Group Recommendation
4P2 6 Years (1994-2000) Working Group Recommendation