district management council academic return on investment (a-roi)
TRANSCRIPT
D I S T R I C T M A N AG E M E N T C O U N C I L
ACADEMIC RETURN ON INVESTMENT (A-ROI)
MINDSET
THE WHY OF A-ROI
White Paper: Results-Driven Decision-Making and Academic Return on Investment
• “K-12 public education performance has been stagnant for years despite increased fiscal investments, both at the state and local level”.• “…school districts have ramped up their focus on
measuring academic outcomes, most measures do not promote looking at sustained improvement or recognize the need to differentiate programming..”• “.. It’s hard to know what works, what works best for
different students, how long a program or investment should be in place prior to producing intended outcomes.”
STRATEGIC FRAMEWORK
How does A-ROI align with our score card and strategic
framework?
THREE CRITICAL COMPONENTS
We need:
1. A shift in mindset2. An analytical tool3. Align staff to develop and implement a new
approach
EVOLUTION OF SCHOOL DISTRICT’S MINDSET
Focus on Outcomes Focus on Cost Reduction
Focus on Value
No Child Left Behind forces states to set achievement standards and implement standardized tests
The fiscal crisis strained local tax revenue
Rising pressure on school districts to improve outcomes with reduced funding
Governments created annual progress benchmarks and report cards.
School districts faced a reduction in funds
Simultaneous consideration of cost and outcomes to drive sustained improvement.
School districts instituted multiple strategies to improve outcomes.
Increasing competition for public funds
2000 2008 2014
FOCUS ON OUTCOMES
• The focus on outcomes has resulted in a vast number of metrics being calculated and tracked across school districts in the country.
• Many metrics are not actionable and may mostly be compliance or accountability- related.
TYPES OF METRICS REQUESTED BY KEY STAKEHOLDERS
State
District
School Community
School Leaders
Teachers
CHALLENGES WITH OUTCOME FOCUS
• The data is used in aggregate form and lacks granularity to make real improvements because it assumes most students have similar needs• The data is not tied to instructional approaches
and thus can not be replicated or improved• The data does not show which teachers are
effective with which students• The data does not take into account other
relevant attributes
FOCUS ON COSTS
• Different districts spend very different amounts to serve similar students populations.
• Within the same district, the cost of educating a student may vary based on teaching strategy or program (special ed. vs. Title vs. ADSIS vs. AVID or resource vs. co-teaching vs. inclusion)
Per Pupil Amount
$0 $5,000 $10,000
District ADistrict BDistrict C
CHALLENGES WITH COST APPROACHES
• Costs estimates typically only include teacher costs and leave out cost of administration, support services, etc.
Incomplete
• Costs are calculated by time or by program and do not reflect whether or not students realize the intended outcome.
Not tied to specific
outcomes
• Costs are captured by department or by school, which are not granular enough to show changes in instructional process.
Not captured at specific student
level
DMC’S VALUE-BASED APPROACH
DMC Value-Based Analysis Framework
Student Segmentation
• Educational need• Other relevant
attributes
Student Outcomes
• Mastery
Costs
• Direct and indirect• Duration for attainment
KEY BENEFITS OF A VALUE-BASED APPROACH
Key Benefit1. Focus on value and
effectiveness not absolute results
2. Promote innovation and experimentation
3. Compare results across schools, district, and even regions
RationaleAllows identification of cost-effective programs or interventions that have the greatest impact on student performance.
Frees up resources to invest in other areas and allows the district to test newer strategies.
Allows normalization of outcomes and costs across districts and may allow easy identification and replication.
STUDENT SEGMENTATION
In most districts, segmentation of students takes place based on NCLB categories or funding streams and only classroom teachers have a student by student look.Issues• This does not provide sufficient information for
tailored interventions- instead, all students receive the same treatment.• All thStuden
tMath Score
Current Segmentation
A 59 ESL
B 60 F/R
C 61 Spec. Ed.
All three students receive math intervention
STUDENT OUTCOMES
Potential outcome measures
• Improvement from level F to G in Fountas & Pinnell in 6 months
• 80% score on Algebra 1 final
• Score of 4 on 6th grade writing rubric-persuasive essay
Key Practices
• Monitors progress toward academic goals
• Measures outcomes associated with specific student needs
• Includes both absolute and growth metrics
• Assesses performance frequently.
Good outcome measures are actionable
STUDENT OUTCOMES
StudentsFormative Assessmen
ts
Desired Results
• Student specific
• Individualized need and key attributes
• Allows for timely and targeted interventions
COSTS
Identify Major Steps
and Map Process
Calculate Direct Costs
Calculate Indirect Costs
Compute Total Cost
per student
Complete Cost Analysis
Choose an achievement Benchmark
Create a map that outlines the major steps for student to reach benchmark
Estimate the direct cost of each service
Allocate services that indirectly support a student’s learning
Add direct and indirect costs to obtain annual cost
KEY TAKE-AWAYS
• Schools have many measures of outcomes, but few actionable.• In order to capture actionable outcome measures,
schools need performance data segmented by student need and key attributes.• Capture all costs connected with serving student,
not just the easy ones.• Calculate the cost to achieve an outcome, not just
by the budget year.
USING A VALUE-BASED APPROACH
1. Find or create natural experiments to evaluate programs• Existing programs: Use student-level data when
analyzing programs that already exist and look for natural experiments to compare value.
• Proposed programs: Ensure that the rollout of new programs have measurable outcomes and meaningful comparisons.
2. Give value-based decision-making a seat at the table
3. Formalize the program approval, modify, or abandonment process• Base on data and separate from the budget cycle
A-ROI TIMELINE
Aug Sept Oct Nov Dec Jan Feb Mar
Select target programs
Conduct value-based analyses
Find meaning
Budget Cycle
OUTCOME MEASURES
Monitors progress toward academic goals
Does not measure process or satisfaction metrics
Includes both absolute and growth metrics
Assess performance frequency
Measures outcomes associated with specific student needs
Key Attributes
of Effective Outcome Measures
POSSIBLE PROGRAMS FOR A-ROI
• AVID• ADSIS• LLI