michael coelli - university of melbourne

Introduction Environment Estimation Results Conclude

How Principals Affect Schools

Mike Helal1 and Michael Coelli2

1Melbourne Institute of Applied Economic and Social Research, University of Melbourne2Department of Economics, University of Melbourne

Evidence Based Teaching Summit

19 October, 2016


The Research Partnership

This research was conducted as part of a now-completed ResearchPartnership between:

the Victorian Department of Education and Training (DET) and

the Melbourne Institute.

This project maximised the linkage of DET datasets to answerimportant research questions.

The benefits of the Research Partnership for DET included new insights across theearly childhood, schools and skills portfolios to drive evidence-based policy andprograms, greater understanding and utility of DET datasets and support for criticalthinking provided by the Melbourne Institute researchers.

All views expressed in the research paper are those of the authors, and should not beattributed to DET or the Melbourne Institute.


Our Research Questions

1 Do individual school principals affect student test scores?

2 What are some of the potential pathways by which effectiveprincipals (in terms of raising test scores) impact the schoolsthey lead?


Why do Economists study Education?

Education is so important for:

1 Individual outcomes:

– income, health, wealth, satisfaction, everyday skills,

– the next generation.

2 Societal outcomes:

– lower crime, welfare and health expenditure,

– higher tax revenues and civic participation.

3 The overall economy and growth:

– more innovation, faster technological adoption, job spillovers.

It is also a major component of public expenditure and public policy.

Education is also an important industry in Australia.


What do Economists Focus On?

Economists attempt to estimate the causal effect of a number ofschool and other factors on student outcomes, including:

1 Overall school expenditure, class size, early learning.

2 Individual teachers, their training and attributes.

3 Student peers, local environment.

4 Parental resources, individual student background,

5 School or other interventions,

6 School choice, education sector, charter schools.

7 School accountability policies (e.g. My School website).


What Economists Bring to the Table

Economists focus on generating systematic statistical evidence ofcausal effects.

Employing measurable externally valid outcomes:

– standardised test scores, graduation,

– post-school education, income, health.

A variety of techniques are used to deal with the endogeneity ofmany school factors:

– randomisation via experimentation,

– policy changes,

– “naturally” occurring random variation,

– sharp cut-offs in policy rules.


Motivation for this Study of School Principals

As school leaders, principals may be able to significantlyinfluence the learning within their schools.

This may potentially occur without changing the teaching staff.

We wish to identify pathways of effectiveness:

– inform school leader training and recruitment strategies.

More generally in economics:

– learn about effective leadership beyond school environment.


Our Estimation Strategy

Principal effectiveness measured using achievement data onprimary school students in public schools in Victoria, Australia.

We use two methods that isolate the effect of principals from theeffect of schools themselves:

– identified by changes in principals within schools (turnover).

Staff and parent survey responses used to measure specificschool factors.

We identify which school factors (if any) are associated witheffective principals (the pathways).


Related Literature

In the Education literature, principals long recognised as equallyimportant as teachers in improving student outcomes:– Leithwood et al (2004); Leithwood and Jantzi (2005);– Day et al (2009); Seashore et al (2010).

In Economics, studies of teacher effectiveness have boomed:- e.g. Rockoff (2004); Rivkin et al (2005); Chetty et al (2014).

Economics studies of principal effectiveness are more limited:– Branch et al (2012, 2013); Coelli and Green (2012);– Dhuey and Smith (2014a, b); Grissom et al (2015).

Economics studies of potential pathways of principal effectiveness(using turnover) focus on teacher characteristic changes:– Branch et al (2012, 2013);– Dhuey and Smith (2014b); Bohlmark et al (2015).


Victorian Public School System

1 Two thirds of primary students attend public schools.

2 Victorian principals have considerable autonomy (OECD, 2013).

They are responsible for:

(a) managing school budgets,

(b) hiring and allocating staff, and

(c) identifying under-performing staff and managing such staffaccording to government policy.

3 Principals chosen by individual school boards.


Principal Characteristics and Changes

1 Principals very mobile across schools prior to first appointment.

2 Principals moving from one school to another also common:

– approx. 15% of schools have a new principal each year.

3 Principal changes were more likely:

(a) in smaller schools,

(b) in secondary schools,

(c) among older principals,

(d) if the principal had served more years (longer tenure), and

(e) in schools with lower achievement in math / low SES.

4 Majority of principal moves were to larger schools:

– pay is higher (urban, high NESB).


Principal Characteristics

Characteristic Mean Median ProportionFemale 0.472Experience at first principal appointment 23.7 25.0Age at first principal appointment 44.0 44.0Schools worked at prior to first principal job 6.5 6.0Schools worked at as a principal 2.0 2.0

Hiring principals from outside the school was on the rise.

Prior position average trendStaff, same school 30% declineStaff, other school 28% increasePrincipal, same school 4% –Principal, other school 27% some increaseUnknown 11% –

Notes: Descriptive statistics for full sample of 4,665 principals who served inVictorian Government schools between 1997 and 2011.


Measures of Student Achievement

We use the following data on student outcomes when estimatingprincipal effectiveness.

Test scores from 1997-2007 at grades 3 and 5.

Focus on reading and mathematics.

We matched student test scores in grades 3 and 5 using studentnames and school only to construct value-added (“gain”) modelsof achievement (match rate 72%).

Matched students had slightly higher test scores, and were lesslikely to be NESB or ATSI.


Table 5: Summary Statistics - Students

Matched students All StudentsVariable Mean Std Dev Mean Std DevFemale 0.492 0.488Non-English Speaking Background 0.147 0.185Aboriginal or Torres Strait Islander 0.009 0.012Achievement - Year 3

Reading 2.349 0.767 2.308 0.782Writing 2.419 0.663 2.374 0.685Numeracy 2.340 0.696 2.304 0.711Mathematics 2.280 0.638 2.245 0.652

Achievement - Year 5Reading 3.203 0.781 3.161 0.789Writing 3.200 0.759 3.152 0.769Numeracy 3.193 0.745 3.154 0.749Mathematics 3.145 0.668 3.106 0.675

Number of students 264,826 366,293

Notes: Statistics for matched student sample who undertook AIM in grades 3 and 5between 1997 and 2007, and for all students observed in the same years.


Measures of Pathways

We attempt to discern the school factors that effective principalsinfluence (“pathways”).

1 School factors are measured by staff and parental responses toschool surveys.

2 We estimate the effect of school principals on these specificschool factors.

3 We then investigate whether any of these factors are improved byeffective principals – as measured by student achievement.

Each school factor is a composite index based on answers to a set ofquestions.


Table 6: Summary Statistics - School Surveys

Mean Std Dev Min MaxStaff survey

School morale 74.55 13.46 25.00 100Supportive leadership 76.52 13.21 19.29 100Goal congruence 78.50 10.84 42.08 100Professional interaction 78.86 10.01 22.45 100Professional growth 72.45 11.31 26.82 100

Parent surveyGeneral satisfaction 5.78 0.46 4.11 7Quality of teaching 5.63 0.41 4.21 7Academic rigour 5.06 0.34 2.94 6General environment 5.17 0.37 2.78 6Customer responsiveness 5.21 0.37 3.00 6Reporting 5.17 0.32 3.05 6

Notes: Staff surveys - 1,576 schools offering grades 3 and 5 in 2007. Parent surveysfor first two items - 1,586 schools in 2007. Parent surveys for remaining four items -1,589 schools in 2003 (last year available).


Estimating Principal Effectiveness

We employ two methods that both:

isolate the effect of principals on student achievement from theeffect of schools, and

control for observed student, school and peer characteristics.

If we find evidence of variation in principal effectiveness,it implies BOTH:

1 principals can affect student outcomes, AND

2 there is variation in principal “quality”.


Method 1 - Variance Decomposition

Intuition:If individual principals affect student achievement,then cross-cohort variation in student achievement should be higherin schools with multiple principals over the same period.

Benefit of method:– A statistical TEST of whether there is significant principal variation.

Outcome:– a measure of the VARIANCE of principal effectiveness.


Method 2 - Fixed Effects Regressions

Idea:using standard regression,estimate effect of individual school principals on student achievementcontrolling for: school fixed effects, student and peer characteristics.

Outcome:measures of DIFFERENCE in effectiveness of each principal fromthe other principals leading the same school at different times.

Benefit of method:we can use estimated differences in effectiveness to examine whetherthey coincide with differences in responses to staff and parent surveys(school factors).


Identification

Identifying assumptions:

1 Within school changes in unobserved student and principal“quality” are unrelated.

2 Principals are drawn randomly with fixed “quality” θp frompools with common variance σ2

p .

Different schools can, however, draw principals with differentmean “quality”.

3 Unobserved student “quality” is drawn randomly from poolswith common variance across schools.

Schools can also draw students with different mean unobserved“quality”.


Estimation Preliminaries

Individual test scores were first normalised (mean 0, variance 1)within each year, grade and domain.

Prior to implementing the Variance Decomposition Method,individual student test scores were first adjusted for:

1 individual student characteristics- gender, NESB and ATSI indicators, plus interactions,

2 school characteristics- socio-economic status, school NESB proportion, regional orremote location,

3 student peer characteristics- proportions of other students in the same school, grade level,and year that were female, NESB and ATSI.

Grade 5 scores also adjusted for grade 3 scores in specific estimates.


Principal Effectiveness - Variance Decomposition Method

reading mathematicsGrade 3

β̂1 = σ̂2p = Variance 0.0183 0.0247

(s.e.) (0.0128) (0.0167)Standard Deviation 0.135 0.157

Grade 5β̂1 = σ̂2

p = Variance 0.0215** 0.0229*(s.e.) (0.0108) (0.0137)Standard Deviation 0.147 0.151

Value added 3-5β̂1 = σ̂2

p = Variance 0.00775 0.0270*(s.e.) (0.0104) (0.0151)Standard Deviation 0.0880 0.164

Notes: 1,100 schools (approx). *** p < 0.01, ** p < 0.05, * p < 0.1.


Principal Effectiveness - Fixed Effects Regressions

reading mathematicsGrade 3

Standard Deviation of principal FEs 0.291 0.310SD of shrunk FEs 0.144 0.172

Grade 5Standard Deviation of principal FEs 0.259 0.308SD of shrunk FEs 0.098 0.162

Value added 3-5Standard Deviation of principal FEs 0.229 0.302SD of shrunk FEs 0.105 0.191

Notes: Grade 3 - 1,780 principals, Grade 5 and VAM - 1,875 principals.Shrinking these standard deviation measures deals with measurement error bias.


Summary of Findings - Achievement

Principals have sizable effects on student achievement:

– not much smaller than the effect of individual school teachers.

The two estimation methods yield very similar results.

Interpreting the SIZE of these estimates:

– having a principal that is one standard deviation higher in theprincipal quality distribution raises test scores by:

0.09 - 0.16 of a standard deviation in individual student scores,

OR

approximately 0.14 - 0.22 of a year’s worth of learning.


Principal Effects on School Factors

Variance Decomposition Method Fixed Effects MethodFactor Variance s.e. SD Raw SD Shrunk SDStaff survey

School morale 0.220*** (0.0421) 0.469 0.623 0.494Supportive leadership 0.334*** (0.0569) 0.578 0.701 0.568Goal congruence 0.168*** (0.0336) 0.410 0.578 0.452Professional interaction 0.244*** (0.0586) 0.494 0.665 0.519Professional growth 0.145*** (0.0442) 0.381 0.642 0.514

Parent surveyGeneral satisfaction 0.167*** (0.0550) 0.409 0.609 0.455Quality of teaching 0.157*** (0.0299) 0.397 0.498 0.373Academic rigour 0.296*** (0.0526) 0.544 0.607 0.403General environment 0.162*** (0.0592) 0.402 0.584 0.394Customer responsiveness 0.262*** (0.0644) 0.512 0.630 0.462Reporting 0.325*** (0.0923) 0.570 0.612 0.406

Notes: Sample sizes differ across factors, as not all were available over all years.*** p < 0.01, ** p < 0.05, * p < 0.1. SD - standard deviation.


Estimating the Pathways - Staff Survey

Simple regressions of shrunk principal effectiveness on achievementand shrunk principal effect on school factors.

Reading MathematicsSchool Morale 0.0104* 0.0244***

(0.00532) (0.00927)Supportive leadership 0.00291 0.00391

(0.00463) (0.00807)Goal congruence 0.0160*** 0.0416***

(0.00581) (0.0101)Professional interaction 0.00417 0.0236***

(0.00507) (0.00881)Professional growth 0.0213*** 0.0447***

(0.00519) (0.00910)

Notes: Value-added models of achievement used to identify effective principals.*** p < 0.01, ** p < 0.05, * p < 0.1.


Supportive Leadership

1 Staff are able to approach the school’s leaders to discussconcerns and grievances.

2 The school’s leaders don’t really know the problems faced bystaff (reversed).

3 There is support from the leaders in this school.

4 There is good communication between staff and the leaders inthis school.

5 The leaders in this school can be relied upon when things gettough.


Goal Congruence

1 The staff are committed to the school’s goals and values.

2 The goals of this school are not easily understood (reversed).

3 The school has a clearly stated set of objectives and goals.

4 My personal goals are in agreement with the goals of this school.


Professional Interaction

1 I feel accepted by other staff in this school.

2 I have the opportunity to be involved in cooperative work withother members of staff.

3 There is good communication between groups in this school.

4 Staff in this school can rely on their colleagues for support andassistance when needed.

5 Staff frequently discuss and share teaching methods andstrategies with each other.

6 There is good communication between staff in this school.

7 I receive support from my colleagues.


Professional Growth

1 I am encouraged to pursue further professional development.

2 Others in this school take an active interest in my careerdevelopment and professional growth.

3 The professional development planning in this school takes intoaccount my individual needs and interests.

4 There are opportunities in this school for developing new skills.

5 It is not difficult to gain access to in-service courses.


Estimating the Pathways - Parent Survey

Reading MathematicsQuality of teaching 0.00252 0.00991

(0.00702) (0.0121)General satisfaction 0.00390 0.00624

(0.00575) (0.00991)Academic rigour 0.000120 -0.00639

(0.00790) (0.0151)Customer responsiveness -0.0106 -0.0332**

(0.00688) (0.0131)Reporting -0.0100 -0.0243

(0.00783) (0.0149)General environment -0.00743 -0.0212

(0.00809) (0.0154)

Notes: Value-added models of achievement used to identify effective principals.*** p < 0.01, ** p < 0.05, * p < 0.1.


Customer Responsiveness

1 The school takes the concerns I have seriously.

2 This school is managed well.

3 I believe there is effective educational leadership within theschool.

4 I am given the opportunity to be involved in the school’seducational activities.


Concluding Remarks

Principals matter in terms of raising student achievement ANDthere is considerable variation in principal quality in Victoriangovernment primary schools.

Principals have considerable influence on many school factors asrevealed in staff and parental surveys.

Effective principals appear to promote professional developmentand goal congruence in their schools.

Professional interaction also appears to be important in numeratesubjects.


For Further Details

The full research paper is available from the Melbourne Institute:

Helal, Mike† and Michael Coelli‡ (2016)

“How Principals Affect Schools”

Melbourne Institute Working Paper Series Working Paper No. 18/16

http://www.melbourneinstitute.com/downloads/working paper series/wp2016n18.pdf

† Melbourne Institute of Applied Economic and Social Research‡ Department of Economics

Faculty of Business and Economics