Pouring a Foundation for Program Improvement with

Quality SPP/APR Data

OSEP’s message regarding Indicators 1, 2, 13 and 14 - data collection and improvement strategies

Ruth RyderUSDE/OSEP/MSIP

Updates

• Status of revisions to information collection (Indicator/Measurement Table)– Received recommendations

that Indicator 13 be revised• Longer• Shorter• Tweaked

– In process

Updates

• Status of OSEP’s review of the APR and revised SPP submissions– Opportunity for Clarification– Response Table– Determinations– Letters in early June

OSEP Review Process

• State contacts did initial review

• Division did facilitated review

• Division leadership “triaged” all Status Tables

• Opportunity for Clarification

• Developing Response Tables

Indicators 1 and 2

• Only a few States continued to do the comparison to all youth

• Many States are using 618 State-reported data

• Many States revised their improvement activities, usually adding more specific activities for “out years”

Indicator 1 and 2 Issues

• Some States could not provide 06-07 data (05-06 data were provided)

• Great variations in calculation methodologies (more using cohort)

• Not even close to meeting targets

• Improvement activities – (“kitchen sink” approach or minimalist approach)

Indicator 13

• All States submitted data– We questioned the validity

and reliability of a few States

• State compliance ranged from 4.9% to 100%– About 15 States were below

50% compliance– Many States could not

demonstrate timely correction of previously identified noncompliance

Indicator 13 Issues• What exactly are States

reporting to us?– More than half of the States are

using the NSTTAC checklist or some variation

– Remaining States are using their own checklists and it’s often hard to tell what requirements they are evaluating

• What does timely correction look like for this indicator?

Indicator 14

• With a few exceptions, States were able to give us data

• About 8 States did not provide valid and reliable data– Denominator– Only graduates

Indicator 14 Issues

• What do the reported data represent?– Many States did not

describe the respondent group

– Can’t determine if the respondent group is representative of the population

• Small sample sizes• Improvement activities

focus on data collection

The Challenges: 2007

• From our review of the Feb 2007 submissions we identified patterns of challenges –

– The Basics– Data– Compliance– Improvement

The Successes and Challenges: 2008

• Successes– The Basics – Much better, States

provided the required information, etc.

– Data – Much better, correct measurement, correct year

– Compliance – More accurate data, more evidence of timely correction

– Improvement Activities – Many States revised and/or added

The Successes and Challenges: 2008

• Challenges– The Basics – Keep up the good

work!– Data – Reconciling database

data with monitoring system data, calculation methodologies for 1 and 2

– Compliance – Documenting timely correction, improving performance

– Improvement Activities – Purposeful, linked, sequenced, evidence-based

Improvement Activities: External TA Analysis Categories

• Improve data collection and reporting

• Improve systems administration and monitoring

• Build systems and infrastructures of technical assistance and support

• Provide technical assistance/training/ professional development

(Continued)

• Clarify/examine/develop policies and procedures

• Program development• Collaboration/coordination• Evaluation• Increase/adjust FTE

One State’s Perspective on Making the Grade with the SPP/APR• Attend as many OSEP-funded

TA offerings as possible• Provide accurate and reliable

data and if can’t, explain why and what you’re doing about it

• Analyze data by local programs

• Develop standard headings, stems and data formats to use for all indicators

One State’s Perspective on Making the Grade with the SPP/APR

• Maintain documentation that you:– Identify noncompliance at the

local level– Identify research-based

improvement activities that are a match to the identified problems

– Require and approve corrective action plans with appropriate timelines

– Oversee timelines and require proof of correction (evidence of success)

Examples

What You’re Doing is Working!

• From 1987 to 2003:

– Postsecondary enrollment rose from 15% to 32%

– 4-year college enrollment rose from 1% to 9%

What You’re Doing is Working!

• More academic coursework

• More above-average grades

• More congruency between age and grade level

• More support services

What You’re Doing is Working!

• More students with disabilities are exiting with a standard diploma

• 1996 to 2006, rates rose from 42% to 56%

• Fewer students with disabilities are dropping out

• From 1996-2006, rates declined from 47% to 26%