The Reincarnation of Effective Schools Research: Rethinking the Literature on District Effectiveness

Tina Trujillo

Assistant Professor, UC Berkeley

AERA-sponsored Research Conference Thinking Systemically: Improving Districts Under Pressure

Rochester, NY November 2-4, 2011

Purpose

• Compile methodological and conceptual profiles of DER

• Compare and contrast DER and SER – Notions of success – Purposes of schooling – Contextualized nature of school performance – Theoretical interpretations of student success

Guiding questions • What are the major correlates of effective districts, and how do they

compare to earlier correlates of effective schools? • What are the major methodological designs for studying effective

districts, and how do they compare to earlier designs for studying effective schools?

• What are the major conceptual dimensions of the district

effectiveness research, and how do they compare to the dimensions of the earlier school effectiveness research?

• What are the implications of the district effectiveness research for the study of educational leadership, improvement, and equity?

Data sources

50 primary documents on district effectiveness • 34% peer-reviewed journal articles • 16% books or book chapters • 26% conference papers, lab reports • 20% foundation, think tank, advocacy reports

Bounding the review

• Original results from studies of relationship between district-level policies, routines, behaviors and classroom-level outcomes

• ERIC database, Google Scholar, literature reviews, bibliographies of district-related literature, internet inquiries

Rounds of analysis

Round 1 • Coded methodological features Round 2 • Coded correlates of effectiveness Round 3 • Coded correlates according to major

conceptual themes from critiques of SER

SER: Contributions

• Instructionally effective schools existed for poor, urban children

• Reinvigorated discourse on organizational and instructional characteristics of schools that affect student achievement

• 7+ correlates of school effectiveness

SER: Methodological concerns

• Inadequate sampling • Inadequate comparisons • Lack of longitudinal analyses • Inadequate designs for studying school-

and classroom-level processes • Inadequate measures of effectiveness

SER: Conceptual concerns

• Reductionist view of purposes of education

• Inadequate treatment of socio-political and normative contexts of schooling

• Inadequate theoretical frames

District Effectiveness Research:

Methodological characteristics

Sampling and selection criteria

• 40%: anecdotal evidence, reputation, convenience • 56%: outlier status (unusually high effectiveness) • 58%: snapshot data • 16%: size • Comparisons

– 63% high to high performers – 22% high to low performers – 15% high to average performers or a cross-section

Methods

0%

20%

40%

60%

80%

100%

Case

stud

y

Survey

Mixed metho

ds

Regr

essio

n/Co

rrelat

ion HLM

Other

Ethn

ograph

y

Expe

rimen

t

Designs

* Percentages do not add up to 100% because multiple codes could apply to each study.

Data collection

0%

20%

40%

60%

80%

Interv

iew

Seco

ndar

y tes

t sco

re ana

lysis

Docum

ent a

nalys

is

Obser

vatio

n

Questi

onna

ire

Focu

s gro

up

Uncle

ar

Other

Logs

Techniques

* Percentages do not add up to 100% because multiple codes could apply to each study.

Research Subjects

0%

20%

40%

60%

80%

Distric

t adm

inistr

ator

s

Princ

ipals

Teac

hers

Stud

ents

Pare

nts/Co

mmun

ityOth

er

Uncle

ar

Participants

Measures of effectiveness

0%

20%

40%

60%

80%

100%

Test-based Other

Measures of Effectiveness

District Effectiveness Research:

Conceptual characteristics

Common correlates of effective districts and schools

0%

20%

40%

60%

80%

100%

Clea

r miss

ion or

visio

n

Freq

uent

mon

itorin

g

Instr

uctio

nal le

ader

ship

High ex

pecta

tions

Oppor

tunit

y to lea

rn/T

ime o

n ta

sk

Safe

and or

derly

envir

onmen

t

Home-

scho

ol re

lation

s

Correlates of Effective Schools

Technical correlates

0%

20%

40%

60%

80%

100%

Focu

s on

stude

nt out

comes

Clea

r miss

ion or

visio

n

Freq

uent

mon

itorin

g

Acco

unta

bility

Organ

izatio

nal c

oher

ence

Stan

dard

s alig

nmen

t

Focu

sed, qua

lity PD (t

each

ers)

Plann

ing an

d go

al se

tting

Instr

uctio

nal le

ader

ship

Focu

sed, qua

lity PD (p

rincip

als)

Oppor

tunit

y to l

earn

/Tim

e on ta

sk

Data a

nalys

is

Safe

and or

derly

envir

onmen

t

Theo

ry of a

ction

Most Common Correlates

Socio-political correlates

0%

10%

20%

30%

40%

Home-schoolrelations

Coalition-building,relationship-building

Explicit goals forequity, power, social

justice

Acknowledgement ofsocial, historical

context

Most Common Correlates

Normative correlates

0%

20%

40%

60%

80%

High expectations Collective norm-building

Ideologies or beliefsabout learners,

community

Most Common Correlates

Conceptual dimensions of DER

0%

20%

40%

60%

80%

Technical Normative Socio-political

Conceptual

Under-theorized studies

• 26%: explicit theoretical frameworks or grounded theories

• Widespread use of “lists” • Conventional conception of “school

district”

Implications

• Hyper-rationalized models of district leadership

• Simplified notions about district improvement

• Decontextualized understandings of district capacity (poverty, race, systemic inequities)