Are there really good schools?: The role of changing demographics within schools

Pete Goldschmidt

&

Motoaiki Hara

CRESST Conference 2004University of California, Los Angeles

Are there really good schools  

= are there excellent schools? 

or 

= are there any good school or are there simply good students grouped together?

  There is a long history of disentangling this question. 

e.g. effective schools researchColeman ReportEducational Production functionsValue Added models

Data: 

- Large urban school district

- 300,000 students (20% sample)

- 510 elementary schools

- 10 years of observations

o     4 years per student

- Panel and cohort dataset.

-        State NRT assessment

o     Mathematics and Reading

 

- Census data by zip code.

Are school means (singular or over time) appropriate indicators of quality? -        do not adjust for inputs.-        artificially attribute all of the variation in student scores to schools.-        Assume equally precise measures of school performance.-        Potential ecological fallacy.-        Potential confounding factors.

Means over time may or may not be correlated. 

-   If correlated – indicates that schools with high scores one year have high scores the next, but does not address whether schools or students are good.

 

if uncorrelated – indicates that mean performance is inconsistent, but does not address whether this is due to changing school quality or changing student composition.

MATH1994

MATH1995

MATH1996

MATH1997

MATH1998

MATH1999

MATH2000

MATH2001

MATH2002

MATH1993

MATH1995

MATH1997

MATH1999

MATH2001

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Correlations Among School Means

0.95-1.00

0.90-0.95

0.85-0.90

0.80-0.85

0.75-0.80

0.70-0.75

Value Added Model: 

Value added estimates preferable to means because they attempt to account for potential confounding factors.

 

Using annual measure of school performance we can correlate annual value added estimates.

 

May or may not be correlated.

MA

TH

EB

I1

MA

TH

EB

I2

MA

TH

EB

I3

MA

TH

EB

I4

MA

TH

EB

I5

MA

TH

EB

I6

MA

TH

EB

I7

MA

TH

EB

I8

MA

TH

EB

I9

MATHEBI0

MATHEBI3

MATHEBI60.000.100.200.300.400.500.600.700.800.901.00

Correlations Among Cross-sectional Value Added Estimates

0.90-1.000.80-0.900.70-0.800.60-0.700.50-0.600.40-0.500.30-0.400.20-0.300.10-0.200.00-0.10

Value Added con’t 

After accounting for student background, school performance much less consistent.

 

Does this indicate that good students, not good schools are identified under current year by year analyses?

School Performance\Quality: 

Given that school performance is related to student background and that there is a concern about the ramifications to “disadvantaged” schools, what is the role of school context?

 

Common value added modeling approach is some form of longitudinal model from which value added estimates are derived.

 Value added estimate of interest is often for growth (i.e difference between actual and expected growth).

 Effect of school context in growth model estimates the extent to which school context mediates growth.

That is, achievement growth might be higher in schools with high average SES than in schools with low average SES.

This is a static view of the role of school context and only addresses between school differences in context.

 

Two Questions:

 1)   1) At what time point should school context be measured in a growth model?

 2) What is the effect of changing demographics within a school?

  Dilemma:

Want to examine effect of changing school context on changing achievement. That is, want to use school demographics as a time varying covariate – but can’t simply add to the between occasion, within-student, part of the growth model because it mixes units within a level.

Solution 

- Use a doubly time nested design.

 -        Two approaches to analyze.

o     Measurement within time

o     Two stage model.

 

Two Stage model:

 Stage 1- cross-sectional Value added model for each year.

VA for each school for each year is net of student characteristics.

 Stage 2- longitudinal model using VA estimates nested within schools.

Each school has (approximately) 10 observations.

Preliminary Results 

-        Expect mean to be 0.

-        Expect mean growth to be 0.

 

 -        What are the effects over time of school context?

o     Generally none.

  Preliminary results indicate that school performance is relatively invariant to changes in school context.

-        Community context effects?

o     Small

   Preliminary results indicate that school performance is relatively invariant to changes in community context.

After accounting for student characteristics, school context effects between schools is generally small. Similarly, after accounting for student characteristics, changes in school context have relatively minor effects on changes on school performance.

If changing school enrollment characteristics have relatively little influence on performance, then likely that changing school structure, policies, and practice may account for low correlations between yearly performance estimates.

The broad issue we address here is whether there really are good schools – that is, they remain top performers based on value added results over a ten year period, or whether top performance is fleeting and more closely linked to the cohort(s) that happen to be passing through the system at a particular time. The results from these queries begin to address the likelihood that schools not meeting AYP will be able to do so at some point in time despite “unfavorable” demographic circumstances.