equity in learning?

Equity in LEarning?A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS

IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES

The opinions expressed in this publication are those of the contributors and do not necessarily reflect the policies or views of UNICEF.

The designations employed in this publication and the presentation of the material do not imply on the part of UNICEF the expression of any opinion whatsoever concerning the legal status of any country or territory or of its authorities or the delimitations of its frontiers.

Extracts from this publication may be freely reproduced with due acknowledgement using the following reference: UNICEF, 2012. Equity in Learning? A Comparative Analysis of the pisA 2009 Results in Central and Eastern Europe and the Commonwealth of independent states, Geneva: UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS).

For further information and to download this or any other publication, please visit the UNICEF CEE/CIS website at www.unicef.org/ceecis.

All correspondence should be addressed to:

UNICEF Regional Office for CEE/CIS Education Section

Palais des Nations

CH 1211 Geneva 10

Switzerland

Copyright: © 2013 United Nations Children’s Fund (UNICEF)

Editing: Stephen Boyle

Design: www.services-concept.ch

Cover Photo: UNICEF/SWZ/00503

Equity in LEarning?A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS

IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES

3

taBLE OF COntEntS

Table of ConTenTs

boxes, figures and Tables .....................................................................................................................................................................................................................5

abbreviations and acronyms ..........................................................................................................................................................................................................8

acknowledgements ..........................................................................................................................................................................................................................................9

foreword .......................................................................................................................................................................................................................................................................10

executive summary .....................................................................................................................................................................................................................................12

ChapTer 1: InTroduCTIon ....................................................................................................................................................................................20

ChapTer 2: overall performanCe ..............................................................................................................................................28

overview of performance ..................................................................................................................................................................................................................31

absolute disadvantage .............................................................................................................................................................................................................................33

reading literacy ................................................................................................................................................................................................................................................ 34

mathematics literacy ..................................................................................................................................................................................................................................35

science literacy .................................................................................................................................................................................................................................................. 36

performance on different reading competencies ........................................................................................................................................37

Key findings on overall performance ........................................................................................................................................................................... 42

ChapTer 3: InvesTIgaTIng equITy In performanCe .......................................................................44

Within-country disparities in performance ........................................................................................................................................................ 46

gender differences in performance ..................................................................................................................................................................................49

relationship between socio-economic background and reading performance ..........................................51

Immigrant background and reading performance .................................................................................................................................. 54

resilient students ........................................................................................................................................................................................................................................... 56

distribution of resources across schools ..................................................................................................................................................................57

school location ....................................................................................................................................................................................................................................................59

between-school variance in reading performance and in socio-economic background............. 60

Key findings on equity in performance ..................................................................................................................................................................... 63

ChapTer 4: Trends In performanCe over TIme ........................................................................................66

Trends in reading since 2000 ...................................................................................................................................................................................................... 68

Trends in reading since 2000, 2003 and 2006 ....................................................................................................................................................70

Trends in mathematics and science...................................................................................................................................................................................72

Trends in variance in performance ...................................................................................................................................................................................73

Trends in the relationship between socio-economic background and performance ........................74

Equity in LEarning?

4

A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE

AND THE COMMONWEALTH OF INDEPENDENT STATES

The case of poland .........................................................................................................................................................................................................................................75

The case of Turkey .........................................................................................................................................................................................................................................76

The case of Kyrgyzstan ...........................................................................................................................................................................................................................76

Key findings on trends ...........................................................................................................................................................................................................................78

ChapTer 5: sChool- and sysTem-level faCTors

assoCIaTed WITh performanCe ..........................................................................................................................................................80

Country wealth and student performance ............................................................................................................................................................. 82

pre-primary school attendance ............................................................................................................................................................................................... 84

differentiation: school selection and ability grouping ..................................................................................................................... 86

school autonomy ............................................................................................................................................................................................................................................. 89

school choice .........................................................................................................................................................................................................................................................91

accountability ...................................................................................................................................................................................................................................................... 93

school resources .............................................................................................................................................................................................................................................. 93

learning time ....................................................................................................................................................................................................................................................... 94

school climate ..................................................................................................................................................................................................................................................... 95

Key findings on school- and system-level factors ...................................................................................................................................... 97

ChapTer 6: summary ConClusIons and polICy suggesTIons .............................100

Country and system factors associated with student performance ............................................................................102

policy challenges: ensuring quality education for all by tackling low performance

and reducing large disparities ................................................................................................................................................................................................105

mitigating the impact of socio-economic background on performance .............................................................106

Tailoring policy interventions to meet specific country challenges ........................................................................106

achieving high quality with equity in educational outcomes: tradeoffs and realities ................109

specific strategies to tackle low performance in reading literacy ...............................................................................110

overall conclusion ....................................................................................................................................................................................................................................111

referenCes .............................................................................................................................................................................................................................................113

annex 1: overvIeW of performanCe In Cee/CIs CounTrIes ...................................116

annex 2: Tables and fIgures ..................................................................................................................................................................124

5

BOxES, FigurES and taBLES

boxes, fIgures and Tables

boxes

box 1 Country groupings ................................................................................................................................................................................................................... 23

box 2 Concepts of literacy in pIsa 2009 .....................................................................................................................................................................25

box 3 Tasks students can typically do at and below the baseline levels of proficiency ................33

box 4 pIrls and TImss studies in learning achievement .............................................................................................................. 40

fIgures

figure 1 percentage of 15-year-old students scoring below level 2 in reading, pIsa 2009 ..............35

figure 2 percentage of 15-year-old students scoring below level 2 in mathematics,

pIsa 2009 ..................................................................................................................................................................................................................................................................... 36

figure 3 percentage of 15-year-old students scoring below level 2 in science,

pIsa 2009 ......................................................................................................................................................................................................................................................................37

figure 4 performance difference between the pIsa 2009 combined reading

scale and each aspect subscale ..................................................................................................................................................................................................39

figure 5 performance difference between the pIsa 2009 combined reading

scale and each text format subscale ................................................................................................................................................................................. 40

figure 6 mean performance vs. 95th to 5th percentile difference: average science,

reading and mathematics, pIsa 2009 .............................................................................................................................................................................. 48

figure 7 performance difference between girls and boys, pIsa 2009 .........................................................................51

figure 8 score point difference in reading associated with one unit increase in the pIsa

index of economic, social and cultural status and percentage of variance in student

reading performance explained by student socio-economic background ..........................................................53

figure 9 percentage of 15-year-old students scoring below level 2 in reading,

by immigrant background ................................................................................................................................................................................................................55

figure 10 percentage of resilient students among disadvantaged students .......................................................57

figure 11 simple correlation between the school mean socio-economic background

and school resources ................................................................................................................................................................................................................................ 58

figure 12 difference in reading score of students in schools from various locations

from students in rural schools, after accounting for the pIsa index of economic,

social and cultural status ....................................................................................................................................................................................................................59

figure 13 between-school variance in student reading performance

(and in socio-economic background), as a percentage of total variance between

and within schools .......................................................................................................................................................................................................................................61

Equity in LEarning?

6



figure 14 school-level score point difference in reading associated with half a unit

increase in the school mean pIsa index of economic, social and cultural status

(between-school gradient), and student-level score point difference associated

with one unit increase in the student-level socio-economic background index

(within-school gradient) ..................................................................................................................................................................................................................... 63

figure 15 a) observed changes in mean reading performance between 2000 and 2009,

and changes adjusted for socio-demographic differences b) Changes between 2000

and 2009 of percentage of boys and of girls below level 2 ......................................................................................................... 69

figure 16 annual observed changes in mean reading performance between 2000

and 2009, 2003 and 2009 or 2006 and 2009 (largest gap available); changes adjusted for

socio-demographic differences; and linear trends across all available pIsa assessments ...................71

figure 17 annual observed changes in mean mathematics and science performance

between 2003 and 2009 or 2006 and 2009 ................................................................................................................................................................72

figure 18 Change in variance in reading performance between 2000 and 2009

(as a percentage of 2000 variance) and difference in percentage of between-school

variance in reading performance (2009-2000) ..................................................................................................................................................73

figure 19 Change between 2000 and 2009 in the student-level score point difference

in reading performance associated with one unit increase in the pIsa index of

economic, social and cultural status (overall association); in student-level difference

associated with unit increase in student-level esCs (within-school association);

and in school-level difference associated with unit increase in school mean esCs

(between-school association).......................................................................................................................................................................................................74

figure 20 gdp per capita in equivalent us dollars (converted using purchasing

power parities) vs. average mean performance in pIsa reading, mathematics

and science ............................................................................................................................................................................................................................................................... 83

figure 21 percentage of 15-year-old students in pIsa having attended pre-primary

education for more than one year, one year of less, or not at all (based on student

self-reports) ..............................................................................................................................................................................................................................................................85

figure 22 performance difference in reading between students who report having

attended pre-primary school for more than one year and those without pre-primary

school attendance, before and after accounting for the socio-economic background

of students ................................................................................................................................................................................................................................................................. 86

figure 23 how much autonomy individual schools have over: a) resource allocation;

b) curriculum and assessment .................................................................................................................................................................................................. 90

figure 24 difference in performance on the reading scale between public

and private schools ......................................................................................................................................................................................................................................92

figure 25 Comparing countries’ mean performance in reading, pIsa2009 ...................................................126

figure 26 Comparing countries’ mean performance in mathematics, pIsa2009 ..................................127

figure 27 Comparing countries’ mean performance in science, pIsa2009 ....................................................128

figure 28 Comparing countries’ range of performance in reading, pIsa2009 ..........................................129

7

BOxES, FigurES and taBLES

figure 29 Comparing countries’ range of performance in mathematics, pIsa2009 .........................130

figure 30 Comparing countries’ range of performance in science, pIsa2009 ...........................................131

figure 31 strength of the socio-economic gradient and reading performance .......................................133

figure 32 mean performance in pIsa 2009 vs. pIrls 2006 (4th grade) ....................................................................139

figure 33 mean performance in pIsa 2009 vs. TImss 2007 (8th grade) ...................................................................140

figure 34 mean performance in pIsa 2009 vs. TImss 2007 (4th grade) ...................................................................141

Tables

Table 1 Countries participating in pIsa 2009, and whether they participated

in previous years ............................................................................................................................................................................................................................................ 30

Table 2 mean performance in reading, mathematics and science literacy, pIsa 2009 ............................31

Table 3 Within-country disparities in performance: difference between 95th

and 5th percentile in reading, mathematics and science literacy, pIsa 2009 ....................................................47

Table 4 vertical and horizontal differentiation policies of school systems ...................................................... 88

Table 5 mean performance in reading, mathematics and science literacy, pIsa 2009 ..................124

Table 6 PISA index of economic, social and cultural status (ESCS): average among

all students, within-country disparities (difference between 95th and 5th percentile),

interquartile range at the student and school levels, average by pre-primary school

attendance...............................................................................................................................................................................................................................................................132

Table 7 assessment and accountability practices ....................................................................................................................................134

Table 8 resources and learning time ...........................................................................................................................................................................136

Table 9 school climate .........................................................................................................................................................................................................................138

Equity in LEarning?

8



abbrevIaTIons and aCronyms

CEE/CIS Central and Eastern Europe and the Commonwealth of Independent States

ESCS Economic, Social and Cultural Status

EU European Union

GDP Gross Domestic Product

ICT Information and Communication Technology

IEA International Association for the Evaluation of Educational Achievement

OECD Organisation for Economic Cooperation and Development

PIRLS Progress in International Reading Literacy Study

PISA Programme for International Student Assessment

TIMSS Trends in International Mathematics and Science Study

9

aCknOwLEdgEmEntS

aCKnoWledgemenTs

Equity in Learning? A Comparative Analysis of the PISA 2009 Results in Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) was commissioned by the UNICEF Regional Office for CEE/CIS.

Giorgina Brown was the lead author of the paper. She was responsible for all data analysis and discussion in chapters 1 through 5 and for assembling the whole report. Aaron Benavot was responsible for writing chapter 6 on Conclusions and Policy suggestions. Special thanks go to both authors.

Philippe Testot-Ferry was responsible for the overall design, development and coordination of the study. Erin Tanner provided comments and revisions, and Petronilla Murithi provided valuable administrative assistance.

The data came from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA). OECD/PISA bears no responsibility for the way data are used or presented.

Copy-editing was by Stephen Boyle. Design and layout was by www.services-concept.ch

Equity in LEarning?

10



foreWord

As the 2015 target date for achieving the Millennium Development Goals (MDGs) approaches, the numerous debates which are taking place to discuss the post-2015 agenda are increasingly focusing on the issue of the quality of education. With the combined concern of many partners and stakeholders for equity and quality of education, interest has shifted to focus on the results of education processes, in terms of learning outcomes and trends across social groups.

These discussions are particularly relevant for countries of Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) which, since the transition, have been facing a crisis in education quality. Education systems in the region are in fact generating growing inequalities in learning outcomes. Countries spending the least on education show the worst results; disparities in learning outcomes are wide and increasingly stratified by socio-economic status and gender. Reforms to improve the quality and relevance of education have been initiated but have not penetrated into the classrooms, especially in poor and rural areas; outdated curricula and teaching methods prepare students for memorisation of facts rather than application of the skills which are critical for performance in knowledge economies.

Despite the efforts of governments to reform their education systems, about half the adolescents in the region leave basic education without mastering core skills in reading and mathematics, and without the competencies necessary in today’s societies. Adolescents from marginalized communities are far more likely to leave school with low levels of learning achievement, less knowledge, and fewer skills and proficiencies than their peers from better-off communities. Inequities in learning and performance eventually lead to dropping out and exclusion from schooling. This can have important consequences for countries, both in terms of economic performance as well as internal stability.

This publication, Equity in Learning?, offers a foundation for a better understanding of the factors influencing students’ achievement as well as for future reflection and action to improve the outcomes of basic education in the region. It uses sound statistical methods to analyse the most recent data on learning achievement from the OECD’s Programme for International Student Assessment (PISA), from an equity perspective. The study:

1) Presents data on equity gaps in learning outcomes for reading, mathematics and science among 15-year-old students based on the results of PISA 2009.

2) Identifies trends over time and across countries, with particular attention to the link between quality and equity.

3) Proposes policy recommendations that, as evidenced by the data, respond to the challenges in reaching high levels of learning achievement.

Participation in PISA and/or other international assessments demonstrates countries’ strong commitment to monitoring and improving learning outcomes and to being open to understanding their challenges in education quality. This study concentrates on the 13 participating countries which had, in 2009, a Programme of Cooperation in education with UNICEF: Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, Russian Federation1, Serbia and Turkey. For the purpose of comparison, this report also presents results from eight

1 As of 2013, UNICEF and the Russian Federation no longer have a programme of cooperation. However, in this report, data from the Russian Federation is included because at the time of the data analysis UNICEF and the Russian Federation were cooperating in education.

11

FOrEwOrd

countries in the region that became EU member states in 2004: Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia.

This research calls on all countries in the region to:

1) Strengthen and refocus policy efforts to improve equity in the quality and relevance of basic education − education systems must improve teacher quality and student engagement in learning.

2) Tackle low student performance with targeted strategies to reduce disparities in learning.

3) Ensure full and equitable implementation of education policies.

We hope that this report will contribute to a better understanding of students’ academic achievement in the CEE/CIS region. We also hope that it will serve as a basis for triggering national dialogue on the importance of monitoring, assessing and analysing learning outcomes and their trends across social groups, and will inspire initiatives to reduce equity gaps in learning.

Marie-Pierre Poirier Regional Director for CEE/CIS UNICEF

Equity in LEarning?

12



exeCuTIve summary

This UNICEF study examines educational outcomes in the 13 participating countries in the CEE/CIS region which had, in 2009, a Programme of Cooperation in education with UNICEF: Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, the Russian Federation2, Serbia and Turkey. It focuses on new data which have become available since the publication in 2009 of the study Learning Achievement in the CEE/CIS Region, which was commissioned by the UNICEF Regional Office for CEE/CIS. In particular, it examines the most recent results of the Programme for International Student Assessment − a large-scale international assessment conducted by OECD in 74 countries. For the purpose of comparison, results of eight countries that became member states of the European Union (EU) in 2004 – the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia (referred to as the EU8 countries) – are also presented (see country grouping in Box 1).

The last round of PISA was undertaken in 2009 and 2010 in order to assess students’ preparedness for adult life as they near the end of secondary education; it focused on reading literacy, but also provided performance results on mathematics and science literacy. The term literacy is used in PISA to point to its broad approach in measuring knowledge and skills focusing on ability to use these in novel situations.

This report starts by describing overall performance levels in the 21 “study focus countries” (CEE/CIS and EU8 countries) in reading, mathematics and science in PISA 2009. It looks also at average literacy scores on different reading competencies, and at the percentage of students who are at a disadvantage in absolute terms, being below the baseline level of achievement identified by PISA (defined as the common international benchmark − Level 2). The PISA results are compared with two other large ongoing surveys of learning achievement: the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS). In order to investigate equity in performance, the report looks at the extent of overall within-country disparities in achievement levels across the 15-year-old student population, in terms of the size of the gap between top achievers and the lowest achievers (relative disadvantage). It then examines inequality in performance in various sub-national populations (by sex, socio-economic background, immigrant status and school location) and inequality in the distribution of resources across schools. It also looks at the percentage of ‘resilient students’ in each country, who despite their disadvantaged background manage to achieve high levels of performance, and examines the extent to which variability in performance and in socio-economic background is concentrated between (rather than within) schools, resulting in academic and social exclusion.

The report continues with an analysis of trends in overall performance, adjusting for demographic changes, in within-country disparities, and in the relationship between socio-economic background and reading performance. It then looks at school- and system-level factors affecting performance, presenting the context by looking at the level of national wealth available for educational expenditure and its relationship to performance. The factors described relate mainly to the following: pre-primary school attendance; school selection and ability grouping; school autonomy; school choice; accountability; school resources; learning time and school climate. The report concludes with a summary and policy recommendations. The following are some of the main results:


13

ExECutiVE Summary

Overview of performance and absolute disadvantage

In terms of performance averages in the three literacy domains of science, reading and mathematics, there is a clear separation in ranking among the study focus countries between the EU8 countries, at the top (with averages similar to those of OECD countries), and CEE/CIS countries, at the bottom. In reading, an average of 48 per cent of students in the CEE/CIS countries do not reach the baseline level of achievement, compared with 19 per cent among EU8 and OECD countries. In mathematics, the figure rises to 53 per cent of students in CEE/CIS countries failing to reach the baseline level, compared with 21 per cent in EU8 countries, while in science the number of students failing to reach the baseline figure in CEE/CIS countries is 47 per cent, compared with 15 per cent in EU8 countries.

The best-performing country is Estonia, which performs significantly above the OECD average in all three literacy domains. Poland performs above the OECD average in reading and science and at a level with the OECD average in mathematics, while Slovenia is above the OECD average in mathematics and science. Hungary is at a level with the OECD average in all three literacy domains, as is the Czech Republic in mathematics and science, and Slovakia in mathematics. All other country means among the study focus countries are significantly lower than the OECD average. The best-performing CEE/CIS countries are Croatia (in reading), and Croatia together with the Russian Federation (in mathematics and science), followed by Turkey. Croatia is at a level with EU8 countries Czech Republic, Slovakia and Lithuania in reading, and Slovakia, Lithuania and Latvia in science. At the other extreme, Kyrgyzstan is by far the lowest-performing country among all 74 PISA countries, as was the case in the previous PISA survey (in 2006). In Kyrgyzstan, more than 80 per cent do not reach the baseline level of achievement in the three literacy scales; these 15-year-old students may not be capable of the basic tasks that will enable them to participate effectively and productively in life situations. Other CEE/CIS countries where the majority of students do not reach the baseline level of performance are Albania, Georgia and Kazakhstan (in all three literacy scales), Azerbaijan (in reading and science), Moldova (in reading and mathematics) and Montenegro (in mathematics and science).

Looking at performances in different reading competencies, students in the study focus countries seem to be better at obtaining information and understanding the meaning of a text, rather than at reflecting on the implications of its content; the ‘reflect and evaluate’ aspect seems to be the most problematic for several of the low performers in the region. In PISA 2006, which focused on science literacy, we saw that students in the region were better at explaining phenomena scientifically than at identifying scientific issues or using scientific evidence. Comparing the results of different surveys, CEE/CIS and EU8 countries tend to do better, relative to the OECD countries, in TIMSS and PIRLS than in PISA. One common explanation is that PISA assesses the use and application of knowledge and skills in real-life situations, while TIMSS and PIRLS focus more on measuring the mastery of an internationally agreed formal curriculum.

Equity in performance

In terms of equity measures and other characteristics measured by PISA, there is no such separation between EU8 and OECD countries, on the one side, and CEE/CIS countries, on the other, as there is for average performance. In the study focus countries, there are generally fewer disparities in performance than in OECD countries, with EU8 and CEE/CIS countries having on average similar disparities. Differences between countries in average performance are small compared with differences in performance within countries. On average, within-country differences in performance in the study focus countries (represented by the gap between the top achievers, at the 95th percentile, and the lowest achievers, at the 5th percentile) range from a minimum of about the equivalent of

Equity in LEarning?

14



six years of schooling in Azerbaijan3 to nine years in Bulgaria. While within-country differences are large in all countries, they are relatively small in some countries compared with others: as well as Azerbaijan, the three countries Estonia, Latvia and Romania also have relatively narrow gaps between high and low performers over the three literacy scales. The example of Estonia reassures us that high absolute standards of performance are not incompatible with low levels of within-country disparities.

There are a number of performance inequalities in various sub-national populations. In terms of gender differences, girls have significantly higher average scores in reading than boys in all 74 participating countries. The average gender gap in reading achievement is larger in the study focus countries (48 points) than it is in the OECD (39 points). Azerbaijan has the smallest gap and Albania the largest. For the countries that participated in the survey of digital reading, the gender gap is usually smaller than with print reading. In mathematics, boys have an advantage in seven countries (Montenegro, Croatia, Turkey, Serbia, Hungary, Estonia and Azerbaijan) and girls in four countries (Albania, Georgia, Lithuania and Kyrgyzstan). There is no significant difference in the remaining 10 study focus countries. Gender differences in mathematics and science are much smaller than in reading. In science, differences in average scores of boys and girls in the majority of PISA countries are not statistically significant. However, among the study focus countries there is a tendency for girls to perform better than boys, with girls outperforming boys in 14 of the 21 countries. Overall, girls outperform boys by an average of 10 points across the study focus countries (while the average difference in the OECD region is zero).

The socio-economic background of students and schools has generally a strong influence on performance, with most students who perform poorly coming from socio-economically disadvantaged backgrounds. On average in the study focus countries, 13 per cent of differences in reading performance within countries are associated with differences in socio-economic background. A student from a more socio-economically advantaged background (the top one-seventh) scores on average 36 points more than a student with an average background − equivalent to almost one year of schooling. Differences range from 21 points in Azerbaijan to 51 points in Bulgaria.

Nevertheless, despite such divergence there are always some students who despite their disadvantaged background manage to achieve high levels of performance; the OECD defines these as ‘resilient students’. Generally, top-performing countries have also the largest percentage of resilient students. Among the study focus countries, the largest percentage of resilient students is found in Turkey, where 42 per cent of all disadvantaged students (i.e. those from the bottom quarter of the distribution of socio-economic background in their own country) score in the top quarter in reading among students with similar socio-economic background from all countries.

While the quantity of resources tends to be more favourable (i.e. there are lower student-teacher ratios) in schools with lower socio-economic intake, in many countries the more disadvantaged students attend schools with lower quality resources (i.e. they have a lower proportion of full-time teachers with advanced university degrees). In general, students in urban schools perform better than students in rural schools, and students in larger towns perform better than those in smaller towns. Taking into account socio-economic background, there are still more than 80 points difference in reading performance (equivalent to about two years of schooling) between students in city schools and rural schools in Bulgaria, Kyrgyzstan and Hungary.

Whatever the reason, large between-school differences in what students learn are an indication of inequality in the school system. Countries in which students are not separated in different schools

3 This equates to six times the typical gap in OECD countries between the average reading performance of 15-year-old students in two adjacent grades.

15

ExECutiVE Summary

according to socio-economic background also tend to have small between-school differences in performance and high overall performance, as is the case in Estonia and Latvia. On the other hand, the proportion of between-school variance in performance is more than 50 per cent in Romania, Slovenia, Hungary and Turkey, indicating that there is a larger variation in the average performance of schools in these countries than in individual scores within schools4. In most countries, it is the school’s economic, social and cultural status that appears to have a much greater effect on students’ performance than the individual student’s background.

Trends in time

By monitoring changes in educational outcomes over time – which has been possible because PISA is an ongoing triennial survey − it emerges that in some cases overall performances have improved and inequalities reduced. Of course, data for each subject area are comparable only from the point in time when the first in-depth assessment took place. Consequently, results in reading are comparable only from 2000, in mathematics from 2003, and in science from 2006.

Among the eight study focus countries with data on reading since 2000, there was an improvement in Albania, Hungary, Latvia and Poland, a decline in the Czech Republic, and no significant change in Bulgaria, Romania and the Russian Federation. Part of the change can be attributed to changes in sampling methods and the socio-demographic profile of students; by adjusting for such changes, the performance change in the Czech Republic and Hungary becomes non-significant. The percentage performing below the baseline was reduced in this nine-year period by 14 per cent in Albania, 13 per cent in Latvia, and eight per cent in Poland. Because Latvia and Poland raised the performance of their lowest-achieving students while maintaining the performance level among the highest-achieving students, variation in reading performance decreased too. The largest decrease in variance was seen in Latvia, with a 39 per cent decrease − making Latvia one of the countries with the least within-country disparity − followed by Romania (with a 22 per cent decrease) and Poland (20 per cent decrease). In Romania, there was a decline in the performance of high-achievers, with no change for low-achievers. The share of between-school variance in reading performance remained fairly similar in most countries except in Poland, which had been one of the study focus countries with the largest share of between-school variance in 2000 (62 per cent). By 2009, it had the smallest share of all (19 per cent). This change has been attributed to the institutional reform as a result of which 15-year-old students are no longer separated into different types of schools. The relationship between socio-economic background and reading performance diminished in the Czech Republic, Albania and Latvia, while it increased in Romania. The between-school association of socio-economic intake and performance at the school level decreased most in Latvia and Poland, and it also fell in the Russian Federation.

For countries that did not participate in PISA 2000, reading performance has improved in Serbia and Turkey since 2003, and in Kyrgyzstan and Montenegro since 2006, while in Slovenia there has been a significant decline since 2006. In mathematics, performance has since 2003 increased in Turkey and declined in the Czech Republic. Of countries that did not participate in 2003, Kyrgyzstan and Romania have increased their mean mathematics performance since 2006, while in Lithuania performance has declined. In science, Turkey’s and Poland’s mean performance has increased since 2003, while in the Czech Republic, Montenegro and Slovenia it has declined.

4 The performance variation that can be attributed to differences in student results in different schools (as by comparing school averages) is larger than the performance variation that can be attributed to differences in the performance of students within schools (that cannot be attributed to differences between schools). So, students within a given school tend to have similar performance, compared with students in different schools.

Equity in LEarning?

16



School- and system-level factors

High-performing countries which have a weak relationship between socio-economic background and performance (the OECD’s main indicator of equity) share a number of commonalities. The OECD describes them as: a high value is placed on education; there are clear and ambitious standards; there is an emphasis on ensuring the quality of teachers and principals; and high-quality learning is provided consistently to every student. One study focus country, Estonia, is included in the small group of eight school systems deemed to be successful, among the 65 PISA 2009 countries, for a performance above the OECD average and a lower-than-average impact of socio-economic background.

Unlike in the richer OECD countries, where the relationship between national income and educational performance is considerably weaker, in the study focus countries there is a close association between student performance and GDP per capita. It seems that money matters below a certain threshold, and indeed the lowest performers in PISA are all countries with lower levels of GDP per capita. Nevertheless, despite certain study focus countries lacking the economic resources to provide sufficient educational opportunities, and the fact that all study focus countries have GDP per capita rates well below the OECD mean, some countries in the region perform significantly better in PISA than the OECD mean. The implication is that once there are enough resources for the basics, it is the way those resources are used that affect educational performance.

Some systems are comprehensive, with all 15-year-olds following the same programme, as in Estonia, Latvia and Poland, while others are stratified, with a selection of students streamed into different programmes or schools. Among the eight school systems with above-average performance and below-average socio-economic inequalities in PISA, including Estonia, none show high levels of student differentiation. Early differentiation and selection contributes to between-school inequalities and academic exclusion. Also, in countries where more students repeat grades, or where it is more common to transfer weak or disruptive students out of a school, overall results tend to be worse and socio-economic differences in performance wider.

Another important feature of school organisation is the degree of autonomy that schools have in taking decisions on various matters. Overall, across the range of tasks involving resource allocation, principals in the EU8 countries report more autonomy (with 59 per cent reporting sole responsibility of schools) than in OECD countries (45 per cent), with the lowest degree of autonomy reported in the CEE/CIS countries (37 per cent). In general, there is more joint responsibility between schools and higher authorities reported in curriculum and assessment than in resource allocation. The highest levels of autonomy are again found in the EU8 countries (with on average 66 per cent of principals reporting sole responsibility of schools, 27 per cent joint responsibility and seven per cent reporting that responsibility is solely that of higher authorities). This degree of autonomy was higher than the OECD average (where 60 per cent of principals report sole responsibility of schools, 24 per cent report joint responsibility and 16 per cent report responsibility is that of higher authorities only). CEE/CIS countries report the lowest levels of autonomy (36 per cent sole responsibility of schools, 23 per cent joint responsibility and 41 per cent higher authorities only). In PISA, countries that grant greater autonomy to schools to design curricula, decide courses, establish student assessment policies, and determine course content and textbooks used tend to show better performance in reading than those that do not, while there is no correlation at the country level between performance and autonomy in resource allocation.

The only type of resource that PISA shows to be correlated with student performance is the level of teachers’ salaries relative to national income. The OECD observes that raising teacher quality is

17

ExECutiVE Summary

more effective in improving student outcomes than is creating smaller classes, noting that systems prioritising teachers’ pay over smaller classes tend to achieve higher levels of performance. Within countries, socio-economically advantaged schools tend to have more educational resources and tend to perform better, suggesting the need for a more equitable distribution of resources across schools. However, after accounting for socio-economic background, resources do not seem to make a difference.

Among all PISA countries, principals in Turkey had the highest perception of problems with instruction due to teacher shortages, followed by Kyrgyzstan and Kazakhstan in CEE/CIS. Kyrgyzstan, followed by Turkey, was the most likely to report that instruction in schools is hindered by a lack of adequate material resources. Learning is disrupted the most by both student and teacher behaviour in Turkey, according to principals, to a degree far above that of any other PISA country.

Schools with better disciplinary climates, more positive behaviours among teachers and better teacher-student relations tend to achieve higher scores in reading. Most of this effect is connected to socio-economic background, since more disciplined classes are generally attended by students with advantaged backgrounds (who tend to perform better and may reinforce a climate conducive to learning). But even controlling for socio-economic background, part of the performance advantage remains. The challenge is to weaken this association between background and climate, partly by changing the social mix of students in some schools. Generally, a large proportion of differences are accounted for by socio-economic factors, if not solely then at least jointly with other factors. Nevertheless, there are still factors related to school characteristics which can be affected by public policy that seem to make a difference, both to absolute levels of performance as well as to disparities across the system.

If disadvantage becomes established at an early age, given the importance of home background, then attempts to mitigate such disadvantage need to begin before a child even starts compulsory school. Even when socio-economic background is kept constant, students who have attended pre-primary education for more than a year tend to show higher performance in reading than those who have not, in all the study focus countries except Estonia, Latvia and Croatia. The average difference is 18 points, while the largest difference is observed in Kyrgyzstan where it corresponds to more than one year of schooling (47 points).

Summary conclusions and policy suggestions

At present, CEE/CIS countries confront a number of pressing policy challenges in education. First and foremost, there is an acute need to improve the provision of quality education for all students. Among other things, this means establishing school systems that improve teacher quality and effectiveness and enhance student learning experiences and outcomes. Second, countries need to tackle low student performance and find ways to reduce disparities in the knowledge, skills and proficiencies which students obtain after completing a compulsory school cycle. A third challenge involves the loose linkages between educational reforms and policy intentions, on the one hand, and school realities and classroom dynamics, on the other. In many countries the actual implementation of educational policies and intentions in local schools is uneven and partial. Finally, it is vital for CEE/CIS countries to create learning environments which move beyond the rote memorization of facts and teacher-dominated pedagogy, and focus instead on the application of knowledge and skills to new situations, with an emphasis on innovation, creativity, problem-solving and open dialogue. These policy challenges are widespread throughout the region, and are especially pronounced in the school systems serving poor and rural communities.

This report makes a convincing case for an essential assertion: public policies in education matter. They have made, and will continue to make, a real difference in the lives of children and youth, and

Equity in LEarning?

18



in the prosperity and social solidarity of countries and communities in the CEE/CIS region. For public policies to foster quality and equity in education and to provide effective learning environments in schools and classrooms, they must be based on the collection and examination of rigorous evidence from different sources. The PISA assessment programme, which systematically explores country-specific and cross-country relationships in student performance in reading, mathematics and science, provides a strong platform to examine and debate the efficacy of different policy options in education. For many countries in the CEE/CIS region, well designed, methodologically rigorous and transparent studies of student achievement and learning deficits, such as PISA, have been far too rare in the past.

In addition, there are several other vital messages that emerge from this report:

(1) It is imperative that CEE/CIS countries strengthen and refocus their policy efforts to improve the quality, equity and relevance of basic education. Access to quality basic education is a basic right for each and every child in CEE/CIS. The results and analyses of the 2006 and 2009 PISA assessments provide concrete evidence of which policy options and reforms are more or less likely to bring about concrete changes towards this goal. This report has paid particular attention to policies that improve the performance of low-achieving students and schools; that expand access to pre-primary education, especially for disadvantaged and minority children; that help all students learn to read, and read for enjoyment; and that focus on ways to improve teacher quality and effectiveness.

(2) Quality education is not simply a matter of securing adequate inputs to schooling, such as sufficient schools, laboratories, textbooks, computers, instructional time and trained teachers. Quality education is also a matter, and increasingly so in the light of globalization, of the quality of knowledge, competencies, skills and attitudes that children take away from their school experiences. Thus, there is a well-defined need for a shift in how quality education is conceived, measured and monitored in the CEE/CIS region. All policy stakeholders should be involved in supporting student learning and monitoring learning outcomes. Especially useful in this regard are current international assessments such as PISA, PIRLS and TIMMS. Other assessments (regional, national or sub-national) which are appropriate, relevant, transparent and methodologically rigorous would also be useful.

(3) Most CEE/CIS countries have made great strides in increasing enrolment and attendance rates in basic education (up to, and sometimes including, upper-secondary education). However, these achievements mask deep-rooted inequalities in the quality of education provided and the benefits – for personal development, work opportunities and lifelong learning – to young people who have completed their formal schooling. The monitoring of learning deficits and disparities through international assessments is a powerful tool for addressing equity challenges in education and society, especially in the CEE/CIS region, where such inequalities are rampant.

19

ExECutiVE Summary

ChapTer 1

InTroduCTIon

© UNICEF

22

Equity in LEarning?A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE


ChapTer 1: InTroduCTIon

The transition states of Central and Eastern Europe and the Commonwealth of Independent States, which are moving from Soviet education systems to more modern and nationally relevant systems, are confronted with major obstacles to achieving successful outcomes for school quality and learning. Education systems in the region are also confronted with inequalities in learning outcomes, commonly stratified by socio-economic status. Since the publication in 2009 of the study Learning Achievement in the CEE/CIS Region, which was commissioned by the UNICEF Regional Office for CEE/CIS, new data on educational outcomes in the region have become available. In particular, the most recent results of the Programme for International Student Assessment − a large-scale international assessment conducted by the OECD − provide an excellent opportunity for analysing not only the quality and relevance of basic education in this region, but also equity issues, including new countries and trends over time. This study is based mainly on results as published by the OECD in the several volumes on PISA 2009 Results5.

PISA is an ongoing triennial survey launched by the OECD in order to assess students’ ‘preparedness for adult life’ as they near the end of secondary education, and evaluate ‘the quality, equity and efficiency of school systems’6. PISA measures the performance of 15-year-old students in three core competencies: reading, mathematics and science, and has thus far been administered four times, in 2000, 2003, 2006 and 2009. In each year, PISA measured students’ overall performance in the three competencies and conducted an in-depth investigation of students’ skills in one of the three subjects. In the last run (2009) the main focus was on reading, as it was in PISA’s first edition, in 2000; PISA 2003 focused on mathematics, and PISA 2006, on science.

In total, 65 countries participated in PISA 2009. Ten additional participants administered the same assessments in 2010 (within the PISA 2009 plus project)7. This study focuses primarily on the 13 participating countries with which UNICEF had a Programme of Cooperation in education in 2009, referred to in the report as “CEE/CIS countries”: Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, the Russian Federation8, Serbia and Turkey. Compared with the last PISA edition (2006), four more CEE/CIS countries participated in the 2009 survey9. Kazakhstan participated for the first time in 2009. Albania did not participate in 2003 and 2006, but did participate in PISA 2000. Georgia and Moldova participated for the first time in 2010 within the PISA 2009 plus project10. For the purpose of comparison, results of the eight EU countries that became member states of the EU in 2004 – the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia – are also presented (“EU8 countries”). Together, the CEE/CIS and EU8 countries are referred to as “study focus countries”. The OECD average is used as a benchmark. Since the last round of PISA, four new countries have joined the OECD: Chile, Estonia, Israel and Slovenia (see Box 1 for the complete list).

5 OECD 2010 vols.I-V. For more information on the PISA survey please refer to this publication (e.g. examples of test questions are included) or to http://www.pisa.oecd.org/, where also the microdata can be downloaded.

6 OECD 2010 vol. I, p.3.

7 This makes a total of 74 participating countries, since the data for Dubai (2009) is included in the United Arab Emirates (2009 plus).


9 The results of PISA 2006 for the CEE/CIS countries were discussed in UNICEF 2009.

10 Results for the PISA 2009 plus participants are contained in Walker 2011, which came out as this report was being finalised. Limited data is available in Walker 2011, compared with the main PISA 2009 volumes published by OECD, which explains why Georgia and Moldova are not always present in the tables and graphs of this report.

INTRODUCTION

23

CHAPTER 1

box 1 Country groupings (as of 2009)

OECD Countries (34) : Australia, Austria, Belgium, Canada, Chile, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, lceland, Ireland, Israel, Italy, Japan, Republic of Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovakia, Slovenia, Spain, Sweden, Switzerland, Turkey, United Kingdom, United States.

EU8 Countries (8) : Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia, Slovenia.

CEE/CIS Countries (as of 2009) (13) : Albania, Azerbaijan, Bulgaria, Croatia, Kazakhstan, Kyrgyzstan, Montenegro, Romania, Russian Federation, Serbia, Turkey, plus Georgia and Moldova (in PISA Plus 2009).

Study Focus Countries (21) – EU 8 countries and CEE/CIS countries : Albania, Azerbaijan, Bulgaria, Croatia, Czech Republic, Estonia, Georgia (in PISA 2009 Plus), Hungary, Kazakhstan, Kyrgyzstan, Latvia, Lithuania, Moldova (in PISA 2009 Plus), Montenegro, Poland, Romania, Russian Federation, Serbia, Turkey, Slovakia, Slovenia.

There are many critiques of international achievement surveys such as PISA. Comparing education systems is difficult due to differences in size, structure, intended curricula and overall educational goals. Some students may be better prepared and more experienced in responding to such tests and will do better than those who are confronted with them for the first time, independent of their reading skills11. We know nothing from PISA on how different countries assign more or less importance to teaching other subjects, such as history, philosophy, geography, arts or citizenship, or how successful students are with such subjects. A survey with a different focus may change the relative standing of a country’s education system both in terms of the level and distribution of student performance. Some discuss the appropriateness of standardized tests across a wide array of cultures, languages and curricula as they minimize ethnic, linguistic and religious subcultures within and across countries. Others argue that without longitudinal performance data on the same sample of students (which would adjust for achievement prior to attending a stage of schooling), it is impossible to make valid inferences about the effects of school and system policies that are distinguishable from the influences of social background, economic circumstances and cultural context12. In some of the poorer countries, a large proportion of students have already left or dropped out of school by age 15 and therefore are not tested, thus giving a distorted picture of the efficiency of the education system in such countries. Questionnaire responses of students and school principals may be misleading given cross-cultural differences in how individuals respond, and in how responses are affected by social desirability or political interests13. We know little about the teachers (and what we know is based on the views of students and principals), who are thought to have a significant impact on student learning.

Mindful of such issues, we will make the best use of the information available, considering that PISA is the result of a rigorous collaborative effort involving consultations between a wide range of countries and a large group of international experts and institutions14, which developed conceptual frameworks, assessment instruments, sampling frame, quality assurance procedures and reliable coding techniques − with a well-documented methodology and an internationally monitored sampling and data-collection process.

11 See Hirtt 2010.

12 Goldstein 2004.

13 OECD 2010 vol. IV, p.29-30.

14 For example, the Australian Council for Educational Research (ACER); the Dutch National Institute for Educational Measurement (CITO) and University of Twente; the German Institute for International Educational Research (DIPF); the Japanese National Institute for Educational Policy Research (NIER); WESTAT, in the United States; and Capstan Linguistic Quality Control and Analysis of Educational Systems and Practices Unit (aSPe), in Belgium.

24



In 2009, nationally representative samples were drawn totalling around 470,000 students across 65 countries. These students were randomly selected to participate in PISA in order to represent about 26 million 15-year-olds. PISA 2009 plus involved testing around 46,000 students across 10 countries, representing a total of about 1.38 million 15-year-olds. Students spent two hours on a test with about half of the questions requiring them to construct their own answers, while the other half were multiple-choice items. They also answered a half-hour questionnaire about themselves, focusing on their personal background, classroom experiences, learning habits, attitudes and reading practices. Their principals answered a questionnaire about their schools, including demographic characteristics and the quality of the learning environment at school.

The total assessment time was six-and-a-half hours (54 per cent on reading, 23 per cent on mathematics and 23 per cent on science), with each student taking a subset of the PISA questions, involving different combinations of 13 test booklets. Within each selected school, 35 students were tested (a maximum of three students with the same booklet), usually by a test administrator trained and employed by the PISA National Centre15. To ensure consistency in the coding process, some questions were coded by four different coders, who independently attributed a score to the answers following an internationally agreed-on coding guide.

Participating countries contributed questions that were reviewed, piloted and refined, with an expert group recommending the final selection based on their technical quality as well as cultural appropriateness16. In PISA 2009, in order to test whether questions were generally relevant across different cultural and curricular contexts, countries were asked to identify those items from the PISA tests which it considered most relevant and appropriate for testing 15 year-olds’ preparedness for adult life. Results showed that the proportion of questions answered correctly by students did not depend significantly on whether countries were only scored on their preferred questions or on all17. Students participating in PISA 2000 have been followed up in order to confirm the relevance of the PISA results with respect to real-life situations, and high-performing students in PISA were found to reach generally higher levels of educational attainment and success in the labour market than low achievers in PISA18.

But what does PISA intend to measure? The term literacy is used in PISA to point to its broad approach in measuring knowledge and skills focusing on the ability to use these in novel situations. Box 2 describes the three concepts of literacy used in PISA for measuring reading, mathematics and science performance.

In PISA 2009, more testing time was devoted to reading literacy, which could consequently be analysed in more detail. A redeveloped and expanded framework for assessing literacy was used in PISA 2009 to ensure that the definition of reading literacy was adequately covered in the tests and reflected ways in which reading had changed since 200019. It included paper as well as digital media, although the main reporting of results focused on print reading. Digital texts were divided into authored texts, in which the reader was receptive, and message-based text, in which the reader could change the text and communicate. Digital reading requires an extended use of skills, such as non-sequential reading, use of multiple texts and critical thinking. In the 2009 survey, reading in a digital format was introduced. Twenty countries undertook a computer-based assessment to analyse

15 OECD 2010 vol. I, pp. 24 and 26.

16 OECD 2010 vol. I, p. 45.

17 OECD 2010 vol. I, p. 36.

18 OECD 2010 vol. I, p. 32.

19 OECD 2010 vol. I, p.38.

INTRODUCTION

25

CHAPTER 1

how well students read digital texts, among which were Hungary and Poland. An analysis of how students access and integrate what they read was added to the standard survey, having acquired relevance given the prevalence of digital reading.

box 2 Concepts of literacy in pIsa 2009

Reading literacy: The capacity of an individual to understand, use, reflect on and engage with written texts in order to achieve his/her goals, to develop his/her knowledge and potential, and to participate in society. In addition to decoding and literal comprehension, reading literacy also involves interpretation and reflection, and the ability to use reading to fulfill one’s goals in life. PISA focuses on reading to learn rather than learning to read. Therefore, students are not assessed on the most basic reading skills.

Mathematical literacy: The capacity of an individual to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals in recognizing the role that mathematics plays in the world and in making well-founded judgments and decisions that constructive, engaged and reflective citizens would require. Mathematical literacy is related to wider, functional use of mathematics; engagement includes the ability to recognize and formulate mathematical problems in various situations.

Scientific literacy: The extent to which an individual:

• Possesses scientific knowledge and uses that knowledge to identify questions, acquire new knowledge, explain scientific phenomena and draw evidence-based conclusions about science-related issues.

• Understands the characteristic features of science as a form of human knowledge and enquiry.

• Shows awareness of how science and technology shape our material, intellectual and cultural environments.

• Engages in science-related issues and with the ideas of science, as a reflective citizen.

Scientific literacy requires an understanding of scientific concepts, as well as the ability to apply a scientific perspective and to think scientifically about evidence.

Source: OECD 2010, vol.I Figure I.1.2.

Texts are also categorized according to the format they are presented in (continuous, in sentences; non-continuous, in lists; mixed; or multiple texts, brought together from one or more source) and type of text. This refers to the rhetorical structure of the text and includes description (typically answering ‘what’), narration (‘when’), exposition (‘how’), argumentation (‘why’), direction (providing instructions), and transaction (exchanging information). There are three aspects to the reader’s purpose and approach to the text: to ‘access and retrieve’ information; to ‘integrate and interpret’ what they read; and to ‘reflect and evaluate’ on the text − standing back from it and relating it to their own experiences. From the author’s point of view, there are four intended uses: personal, to satisfy one’s own interests; public, relating to wider society; educational, used in instruction; and occupational, related to work.

The metric for the overall reading score is based on a mean for OECD countries of 500 in PISA 2000 and a standard deviation of 100. To help interpret what student scores mean, the scale is divided into seven levels which correspond to ascending difficulty of tasks (Levels 1b, 1a, and 2 to 6), and these levels are described in terms of the reading skills and proficiency needed to complete the tasks at each level. Two levels have been newly added in PISA 2009: Level 6 describes very high

26



levels of reading proficiency and Level 1b includes easier tasks than previously included in PISA, in order to know more about what kinds of tasks students with low reading proficiency are capable of (Level 1 has been relabelled 1a)20. Another way to help interpret student scores is by looking at what is the typical gap in performance scores between students in two adjacent school grades. “For the 32 OECD countries in which a sizeable number of 15-year-olds in the PISA sample were enrolled in at least two different grade levels, the difference between students in the two grades implies that one school year corresponds to an average of 39 points on the PISA reading scale”21. The mathematics mean score was set at 500 for OECD countries in 2003, when mathematics was the focus of the PISA survey, and the mean science score was set at 500 in 2006 (for the 30 OECD countries [the figure falls to 498 when the four new OECD countries are included]). Proficiency levels 1 to 6 were maintained for both scales22.

This report is structured into six chapters. Following the introductory chapter 1, Chapter 2 describes overall performance levels in the CEE/CIS countries in reading, mathematics and science in PISA 2009. It looks at average literacy scores on the three literacy scales, as well as on different reading competencies, and at the percentage of students who are at a disadvantage in absolute terms, being below the baseline level of achievement identified by PISA. The PISA results are compared with two other large ongoing surveys of learning achievement: TIMSS and PIRLS.

Chapter 3 investigates equity in performance. Firstly, it looks at the extent of overall within-country disparities in achievement levels across the 15-year-old student population, in terms of the size of the gap between top achievers and the lowest achievers (relative disadvantage), and examines the relationship between overall performance and within-country disparities. It then examines inequality in performance in various sub-national populations (by sex, socio-economic background, immigrant status and school location) and inequality in the distribution of resources across schools. It also looks at the percentage of resilient students in each country, who despite their disadvantaged background manage to achieve high levels of performance. Finally, it examines the extent to which variability in performance and in socio-economic background is concentrated between (rather than within) schools, resulting in academic and social exclusion.

Chapter 4 looks at trends in performance over time, since the first full assessment took place for each literacy scale. It examines trends in reading performance of boys and girls since the first PISA assessment in 2000, adjusted for demographic changes. Trends in mathematics performance are examined since 2003, and in science performance since 2006. Changes in within-country disparities are examined by looking at trends in variance in reading performance between 2000 and 2009, and at trends in the relationship between socio-economic background and reading performance in the same period. Two case studies are presented of countries showing significant improvements over time (Poland and Turkey), as well as that of the lowest-performing country (Kyrgyzstan).

Chapter 5 looks at school- and system-level factors affecting performance. It presents the context by looking at the level of country wealth available for educational expenditure and its relationship to performance. It then examines various school factors that might affect performance outcomes. The factors described relate mainly to pre-primary-school attendance, school selection and ability grouping, school autonomy, school choice, accountability, school resources, learning time and school climate.

20 PISA modified and enhanced only the way reading literacy was assessed, it being the main focus of study and major assessment area of 2009.

21 OECD 2010 vol.II, p.27.

22 OECD 2010 vol.I, p.130, 147.

INTRODUCTION

27

CHAPTER 1

Chapter 6 presents summary conclusions and policy suggestions. It begins by summarising key findings about the relationships between student performance in the 2009 PISA assessment and selected country and system characteristics. It then discusses relevant policy interventions and programmes for CEE/CIS countries seeking to increase student performance and reduce learning disparities. A particular emphasis is placed on strategies to enhance reading literacy, which is the focus of the 2009 PISA assessment. It concludes by calling on CEE/CIS countries to: 1) implement reforms that meaningfully improve the quality and relevance of basic education; 2) conduct appropriate, relevant and transparent assessments, such as PISA, PIRLS and TIMMS, which provide a solid basis for monitoring and improving teaching effectiveness and learning outcomes; and 3) institute measures that reduce educational disparities and inequalities so that all children in the region can benefit from educational opportunities, realise their potential, and lead capable and dignified lives by fully participating in society and the economy.

UNICEF/SWZK00730/GIACOMO PIROZZI

ChapTer 2

overall performanCe

Overview of Performance ................................................................................................................................31

Absolute disadvantage ........................................................................................................................................33

Reading literacy .............................................................................................................................................................34

Mathematics literacy ..............................................................................................................................................35

Science literacy ..............................................................................................................................................................36

Performance on different reading competencies ............................................................37

Key findings on overall performance ................................................................................................42

30



ChapTer 2: overall performanCe

Table 1 Countries participating in pIsa 2009, and whether they participated in previous years.

OECD countriesPISA

2000

PISA

2003

PISA

2006

PISA

2009Other countries

PISA

2000

PISA

2003

PISA

2006

PISA

2009

Australia x x x x Albania 2001 x

Austria x x x n.c Argentina 2001 x x

Belgium x x x x Azerbaijan r.s. x

Canada x x x x Brazil x x x x

Chile 2001 x x Bulgaria 2001 x x

Czech Republic x x x x Chinese Taipei x x

Denmark x x x x Colombia x x

Estonia x x Costa Rica 2010

Finland x x x x Croatia x x

France x x x x Dubai (UAE) x

Germany x x x x Georgia 2010

Greece x x x x Himachal Pradesh-India 2010

Hungary x x x x Hong Kong-China 2001 x x x

Iceland x x x x Indonesia 2001 x x x

Ireland x x x x Jordan x x

Israel 2001 x x Kazakhstan x

Italy x x x x Kyrgyzstan x x

Japan x x x x Latvia x x x x

Republic of Korea x x x x Liechtenstein x x x x

Luxembourg n.c. x x x Lithuania x x

Mexico x x x x Macao-China x x x

Netherlands n.c. x x x Malaysia 2010

New Zealand x x x x Malta 2010

Norway x x x x Mauritius 2010

Poland x x x x Miranda-Venezuela 2010

Portugal x x x x Moldova 2010

Slovakia x x x Montenegro x x

Slovenia x x Panama x

Spain x x x x Peru 2001 x

Sweden x x x x Qatar x x

Switzerland x x x x Romania 2002 x x

Turkey x x x Russian Federation x x x x

United Kingdom n.c. n.c x x Serbia x x x

United States x x m.s. x Shanghai-China x

Singapore x

Tamil Nadu-India 2010

Thailand 2001 x x x

Trinidad and Tobago x

Tunisia x x x

United Arab Emirates 2010

Uruguay x x x

x: indicates data available and comparable. n.c.: indicates data available but not comparable. 2001 or 2002: indicates country took the PISA 2000 test that year, i.e. one or two years later than most. m.s.: Reading data for United States in 2006 not available. r.s.: Mathematics data for Azerbaijan in 2006 not comparable. CEE/CIS countries are shown in red; EU8 countries are shown in green italic (here and in subsequent tables and figures). Source: OECD 2010 vol.V Table A5.1 and p. 30, Walker 2011.

OVERALL PERFORMANCE

31

CHAPTER 2

overview of performance

Average performance data in science, reading and mathematics for the study focus countries that participated in PISA 2009, as shown in Table 2, reveals a clear separation in the ranking of the three literacy domains between the EU8 countries, at the top (with averages similar to that of the OECD), and the CEE/CIS countries, at the bottom. Estonia performs significantly above the OECD average in all three literacy domains (at a level with Germany in all three23). Except for Croatia, and for Turkey in one case, all EU8 countries have higher rankings than the CEE/CIS countries.

Table 2 mean performance in reading, mathematics and science literacy, pIsa 2009.

Reading Mathematics Science mean

Estonia 501 512 528 514

Poland 500 495 508 501

Slovenia 483 501 512 499

Hungary 494 490 503 496

Czech Republic 478 493 500 490

Slovakia 477 497 490 488

Latvia 484 482 494 487

Lithuania 468 477 491 479

Croatia 476 460 486 474

Russian Federation 459 468 478 469

Turkey 464 445 454 455

Serbia 442 442 443 442

Bulgaria 429 428 439 432

Romania 424 427 428 427

Montenegro 408 403 401 404

Moldova 388 397 413 399

Kazakhstan 390 405 400 399

Azerbaijan 362 431 373 389

Albania 385 377 391 384

Georgia 374 379 373 375

Kyrgyzstan 314 331 330 325

Study focus countries mean 438 445 449 449

OECD mean 493 496 501 497

Study focus countries-OECD -55 -51 -52 -48

EU8-OECD -7 -3 2 -3

CEE/CIS-OECD -84 -81 -85 -84

Countries are ranked in order of average performance on the three literacy scales. Values in the last three rows refer to the difference between mean scores of country groupings. Source: OECD 2010 vol.1 Table I.A, Walker 2011 Table B2.1, B3.1, B3.3.

23 Mean scores are not statistically significantly different from Estonia’s score (OECD 2010 Fig. I.2.15, I.3.10, I.3.21). Among OECD countries, Estonia scores statistically significantly below only Finland, Japan and the Republic of Korea in science; it scores below Canada, New Zealand and Australia in reading; and also below Switzerland, Canada, Netherlands and New Zealand in mathematics. See Table 5 in Annex 2 for mean results of all countries participating in PISA 2009.

EU8 mean (494)

Study focus countries mean

CEE/CIS mean (413)

OECD mean

32



Poland performs above the OECD average in reading and science and at a level with the OECD average in mathematics, while Slovenia is above the OECD in mathematics and science. At a level with the OECD average are Hungary in all three literacy domains, the Czech Republic in mathematics and science, and Slovakia in mathematics. All other country means among the study focus countries are significantly lower than the OECD average.

The best-performing CEE/CIS country in reading is Croatia, followed by Turkey and the Russian Federation. In mathematics and science, Croatia and the Russian Federation score above other CEE/CIS countries. Croatia is at a level with some EU8 countries: the Czech Republic, Slovakia and Lithuania in reading, and Slovakia, Lithuania and Latvia in science. At the other extreme, Kyrgyzstan is by far the lowest performing country among all 74 PISA participants (together with Himachal Pradesh – India); the gap separating the mean performance of Kyrgyzstan with that of the top-performer in PISA, Shanghai-China, is equivalent to more than six years of schooling (i.e. six times the typical gap in the OECD between 15-year-old students in two adjacent grades).

It is important to note that not all differences are statistically significant, since PISA is a sample survey and is subject to sampling errors. A confidence interval is applied around the sample mean within which there is a 95 per cent certainty that the population mean will occur. Figures 25 to 27 in Annex 2 show the pairs of countries that have mean scores which are not sufficiently different from each other to be distinguished with confidence. These differences should therefore not be taken into account.

Another important matter to consider when looking at the results for each country is that only 15-year-olds in school took the test. In Turkey, Azerbaijan24 and Albania, more than one third of children in this age group are out of school, and it is likely that their literacy levels and socio-economic backgrounds are lower relative to those students in school25. This causes both an overestimation of average performances in the country, and an underestimation of within-country differences and the association of background with performance. It is worthy of note that Turkey, even though it is one of the countries with the highest percentage of 15-year-olds not participating in PISA, has the lowest mean socio-economic background and the largest range of socio-economic backgrounds among PISA students26 in the region. If more 15-year-olds were at school, Turkey’s education system would have to deal with an even lower average socio-economic background of its student intake, and a larger range.

Furthermore, in different school systems 15-year-olds have gone through different years of formal schooling. For example, on average the majority of 15-year-olds taking the PISA test in the OECD were in the tenth grade, while in the study focus countries the majority were in the ninth grade (there were more tenth graders only in Turkey, Slovakia and Slovenia)27.

24 The data on mathematics performance of Azerbaijan had not been included in the UNICEF 2009 publication reviewing the results of PISA 2006 because of some grave inconsistencies (one of the lowest performing countries in reading and science, it had less students at the lowest levels of achievement in mathematics than the top performers, and following queries about this the OECD did acknowledge problems with the data). This year the data is more consistent and has been included, but there is still a large difference between the mathematics results and the reading and science results. This difference is larger than in any of the other 64 countries participating in PISA.

25 The share of the 15-year-old population not covered by the PISA test is 43 per cent in Turkey and Azerbaijan, 39 per cent in Albania, 33 per cent in Kyrgyzstan, 28 per cent in Bulgaria, 23 per cent in the Russian Federation, 22 per cent in Lithuania, 19 per cent in Latvia, 17 per cent in Serbia, and is 13 per cent or less in the other study focus countries. The OECD average is 13 per cent (Coverage index 3, Table A2.1, OECD vol.I).

26 OECD 2010 vol.II, Figure II.3.2.

27 OECD 2010 vol.I, Table A2. 4a.

OVERALL PERFORMANCE

33

CHAPTER 2

absolute disadvantage

In this section we examine the proportion of students in each country who have not mastered the basic skills deemed necessary to progress in education and function in society. These students are at a disadvantage in absolute terms. Measuring absolute disadvantage, defined as the percentage of students having a low level of achievement relative to a common international benchmark, is another way of assessing a country’s educational performance28. This may be more meaningful than looking at average scores, which merely indicate the position of each country’s education system relative to others. Countries in which a large proportion of students fail to reach a minimum level of competence give cause for concern: can these young citizens fully participate in society and in the labour market with such weak competence, and how might that impact the country’s future productivity and competitiveness?

Each proficiency level identified by PISA is defined by benchmark skills and abilities. Level 2 is identified as the baseline level of proficiency, suggesting that students who achieve this level are capable of the basic tasks that will enable them to participate effectively and productively in life situations related to each of the PISA literacy domains. Those who do not reach this level have not mastered the basic skills tested and may, according to the OECD, also have serious difficulties in benefiting from further educational and learning opportunities throughout life. For example, the Canadian Youth in Transition Survey has shown that students scoring below reading literacy Level 2 risk poorer post-secondary participation or lower labour market outcomes at age 19 and 2129. Box 3 shows which tasks students can typically do at and below the baseline levels of proficiency in reading, mathematics and science.

28 Mean performance is closely related to the share of poorly performing students i.e. the percentage of those who do not reach the baseline level of achievement.

29 OECD 2010 vol.I, p.52.

box 3 Tasks students can typically do at and below the baseline levels of proficiency

reading

Level 2 (baseline level): Some tasks at this level require the reader to locate one or more pieces of information, which may need to be inferred and may need to meet several conditions. Others require recognising the main idea in a text, understanding relationships, or construing meaning within a limited part of the text when the information is not prominent and the reader must make low level inferences. Tasks at this level may involve comparisons or contrasts based on a single feature in the text. Typical reflective tasks at this level require readers to make a comparison or several connections between the text and outside knowledge, by drawing on personal experience and attitudes.

Level 1a: Students can locate one or more independent pieces of explicitly stated information; recognise the main theme or author’s purpose in a text about a familiar topic; make a simple connection between information in the text and common, everyday knowledge. Typically, the required information in the text is prominent and there is little, if any, competing information. The reader is explicitly directed to consider relevant factors in the task and in the text.

Level 1b: Students can locate a single piece of explicitly stated information in a prominent position in a short, syntactically simple text with a familiar context and text type, such as a narrative or a simple list. The text typically provides support to the reader, such as repetition of information, pictures or familiar symbols. There is minimal competing information. The reader may need to make simple connections between adjacent pieces of information.

34



mathematics

Level 2 (baseline level): Students can interpret and recognise situations in contexts that require no more than direct inference. They can extract relevant information from a single source and make use of a single representational mode. Students at this level can employ basic algorithms, formulae, procedures, or conventions. They are capable of direct reasoning and literal interpretations of the results.

Level 1: Students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow immediately from the given stimuli.

Science

Level 2 (baseline level): Students have adequate scientific knowledge to provide possible explanations in familiar contexts or draw conclusions based on simple investigations. They are capable of direct reasoning and making literal interpretations of the results of scientific inquiry or technological problem solving.

Level 1: Students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow explicitly from given evidence.

Source: OECD 2010, vol.I Figures I.2.12, I.3.8, I.3.19.

reading literacy

On average in reading, 48 per cent of students in the CEE/CIS countries do not reach the baseline level of achievement, compared with 19 per cent among the EU8 and OECD countries30. Figure 1 shows that the percentage of students not reaching Level 2 proficiency in reading in the EU8 countries ranges from 13 per cent in Estonia to 24 per cent in Lithuania. Among the CEE/CIS countries, only Croatia is within this range, with just 22 per cent of students below Level 2 in reading. Turkey (25 per cent) and the Russian Federation (27 per cent) are close behind. In Bulgaria, Romania, Serbia and Montenegro between a third and a half of students do not reach the baseline level. A majority of students in Kyrgyzstan (83 per cent), Azerbaijan (73 per cent), Georgia (62 per cent), Kazakhstan (59 per cent), Moldova and Albania (57 per cent) do not reach this level. Kyrgyzstan has the biggest problems: it is the only country where the majority do not even reach Level 1a (59 per cent), and 30 per cent of students do not manage to reach even the lowest level of measured performance (Level 1b). At most, only one per cent of students in EU8 countries and a maximum of 14 per cent (Georgia) in other CEE/CIS countries are at such a low level.

30 Averages in this report are always un-weighted, corresponding to the arithmetic mean of the country estimates (i.e. the size of the education system in each country is not taken into account).

OVERALL PERFORMANCE

35

CHAPTER 2

figure 1 percentage of 15-year-old students scoring below level 2 in reading, pIsa 2009.

Slovakia 22

0 20 40 60 80 100

KyrgyzstanAzerbaijan

GeorgiaKazakhstan

MoldovaAlbania

Montenegro

BulgariaRomania

SerbiaRussian Federation

TurkeyLithuania

Czech Republic

Percentage below Level 2 in reading

� Below Level 1b

� Level 1b� Level 1a

8473

6359

5757

5048

4141

3328

2525

23Croatia 22

SloveniaEU8 MEAN

OECD MEAN

PolandEstonia

1921

19

Hungary 18Latvia 18

1513

CEE/CIS MEAN

Source: OECD 2010 vol.1 Table I.2.1, Walker 2011 Table B2.2.

mathematics literacy

On average in mathematics, 53 per cent of students in CEE/CIS countries do not reach the baseline level of achievement, compared with 21 per cent in EU8 countries and 22 per cent in the OECD. Among EU8 countries, there are between 13 per cent of students in Estonia and 26 per cent in Lithuania failing to attain Level 2 (Figure 2), but these figures rise in the CEE/CIS countries to 29-33 per cent in the Russian Federation and Croatia; 41-42 per cent in Serbia and Turkey; and 45-47 per cent in Azerbaijan, Romania and Bulgaria; while the majority are below the baseline for mathematics in Montenegro and Kazakhstan (56-59 per cent), Moldova, Albania and Georgia (61-69 per cent), and lastly Kyrgyzstan (87 per cent). Again, it is only in Kyrgyzstan where the majority of students (65 per cent) do not reach even the lowest level of measured performance (Level 1).

36



figure 2 percentage of 15-year-old students scoring below level 2 in mathematics, pIsa 2009.

0 20 40 60 80 100

KyrgyzstanGeorgiaAlbania

MoldovaKazakhstan

MontenegroCEE/CIS MEAN

BulgariaRomania

AzerbaijanTurkeySerbia

CroatiaRussian Federation

LithuaniaLatvia

Percentage below Level 2 in mathematics

� Below Level 1

� Level 1

8768

6761

6059

534747

4542

4133

2926

2322Czech Republic

HungaryOECD MEAN

SlovakiaEU8 MEAN

Estonia

2222

2121

Poland 20Slovenia 20

13

Source: OECD 2010 vol.1 Table I.3.1, Walker 2011 Table B3.2.

science literacy

On average in science, 47 per cent of students in CEE/CIS countries do not reach the baseline level of achievement, compared with 15 per cent in EU8 countries and 18 per cent in the OECD. In the EU8 countries, between 8 per cent in Estonia and 19 per cent in Slovakia of students fail to reach Level 2 (see Figure 3). Among CEE/CIS countries, Croatia is within this range (18 per cent), followed by the Russian Federation (22 per cent), Turkey (30 per cent), Serbia (34 per cent), and Bulgaria, Romania and Moldova (39-47 per cent). In the rest of the CEE/CIS countries, a majority of students do not reach the baseline of achievement. This includes Montenegro, Kazakhstan and Albania (54-57 per cent), Georgia and Azerbaijan (66-70 per cent), and Kyrgyzstan (82 per cent). In Kyrgyzstan, the majority do not reach even the lowest level of measured performance (53 per cent do not reach Level 1).

OVERALL PERFORMANCE

37

CHAPTER 2

figure 3 percentage of 15-year-old students scoring below level 2 in science, pIsa 2009.

0 20 40 60 80 100

Kyrgyzstan

AzerbaijanGeorgiaAlbania

KazakhstanMontenegro

RomaniaBulgaria

SerbiaTurkey

Russian FederationSlovakia

CroatiaOECD MEAN

Percentage below Level 2 in science

� Below Level 1

� Level 1

8270

6657

5553

CEE/CIS MEAN 4747Moldova

4139

3430

221919

1818Czech Republic

LithuaniaEU8 MEAN

SloveniaLatvia

Poland

Hungary

Estonia

1517

151414

138

Source: OECD 2010 vol.1 Table I.3.4 Walker 2011 Table B3.4.

performance on different reading competencies

There is sufficient detail in PISA 2009, which focused on reading literacy, to analyse the strengths and weaknesses of different reading competencies in each of the countries surveyed. About one quarter of the PISA questions on reading were assigned to the ‘access and retrieve’ subscale (involving skills associated with finding, selecting and collecting information), around one half to the ‘integrate and interpret’ subscale (processing what is read to make sense of a text), and the remaining questions to the ‘reflect and evaluate’ subscale (engaging with a text while drawing on information, ideas or values external to it).

The ‘reflect and evaluate’ aspect seems to be the most problematic for several of the low performers in the region. It is also the aspect which is generally the most problematic for the study focus countries, with all but five countries performing most poorly in it relative to the other aspects. Azerbaijan, the Czech Republic, Kazakhstan, Montenegro and the Russian Federation each score more than 15 points less in this aspect compared with their overall scores. It is interesting to note that while there are 11 study focus countries with a negative performance of at least minus nine in the ‘reflect and evaluate’ aspect compared with their overall scores, there are no other countries in PISA with such large negative differences in this aspect. Students in this region seem to be better at obtaining information and understanding the meaning of a text, rather than at reflecting on the implications of its content. They “appear to be less accustomed to critically evaluating and reflecting upon what they read, and more accustomed to using texts to find and analyse information” 31.

31 OECD 2010 vol.I, p.71.

38



Kyrgyzstan does less well also on the ‘access and retrieve’ scale (-15), while the largest positive differences on this scale are Croatia (+16) and Slovakia (+13). Kyrgyzstan, Montenegro and Azerbaijan are the countries which do best on the ‘integrate and interpret’ scale, with scores of 12-13 points more than on the combined reading scale.

In PISA 2006, which focused on science literacy, there was sufficient detail to analyse the strengths and weaknesses of different scientific competencies in each country32. Students received scores for their capacity in identifying scientific issues, explaining phenomena scientifically and using scientific evidence. The results mirrored those we see now for reading: in general, students in the study focus countries were better at explaining phenomena scientifically than at identifying scientific issues or using scientific evidence. One possible explanation in 2006 was that countries in the region were still using outdated teaching techniques, with a focus on the delivery, accumulation and reproduction of facts. Value is placed in these systems on the memorization of test-related contents, within a curriculum that is itself test-driven.

The same conclusions were reached in comparing the results of PISA to those of other international surveys of achievement. CEE/CIS countries tend to do better, relative to the OECD countries, in both TIMSS and PIRLS relative to PISA. One common explanation of this is that PISA assesses the use and application of knowledge and skills in real-life situations, while TIMSS and PIRLS focus more on measuring the mastery of an internationally agreed formal curriculum (see Box 4).

In PISA 2009, scores were also analysed separately for different forms of reading materials. A little under two-thirds of the questions were in continuous prose format (complete sentences and paragraphs), traditionally associated with reading in the classroom, while a little under one-third were non-continuous format (comprising one or more lists, i.e. tables, graphs, maps, forms or diagrams), considered more important in the sciences and adult life. Five per cent were in mixed format. It is noteworthy that the five countries with the largest negative performance difference of non-continuous texts with respect to the combined reading scale (Montenegro, Azerbaijan, Albania, Kazakhstan and Kyrgyzstan) are those with the lowest mean performance overall, and where the majority do not reach the baseline level of achievement.

For 20 countries which undertook a computer-based assessment, among which were Hungary and Poland, it was possible to analyse how well students read digital texts and compare this with the general reading scale, which was based only on printed texts. While in most countries results in digital and print reading were quite similar33, in Hungary and Poland students performed significantly worse in digital reading than in print reading (respectively 26 and 37 performance points less on average − the largest negative differences except for Colombia)34. While both countries scored at the average in print reading, of the 16 OECD countries participating in the computer-based survey, they were significantly below average in digital reading (in terms of a composite index including both print and digital reading). In terms of absolute disadvantage, there were about 10 per cent more students scoring below the baseline in digital reading compared with print reading in both countries. The percentage scoring below the baseline in digital reading in Hungary was 27 per cent, while in print reading it was only 18 per cent; in Poland, 26 per cent scored below the baseline in digital

32 UNICEF 2009, pp.34-5.

33 The correlation between digital and print reading performance is 0.83. In fact, this is the same correlation as between print reading and mathematics performance (average for the 16 OECD countries participating in the computer-based assessment), while the correlation between print reading and science performance is 0.88. In comparison, the correlation of digital reading with mathematics and science is 0.76 and 0.79, respectively (OECD 2011, p.74).

34 OECD 2011, p.77.

OVERALL PERFORMANCE

39

CHAPTER 2

reading, while the figure fell to 15 per cent in print reading35. Whether this would have been a general pattern in the region is a matter of conjecture.

figure 4 performance difference between the pIsa 2009 combined reading scale and each aspect subscale.

Difference from combined reading scale

� Reflect and evaluate� Integrate and interpret� Access and retrieve

-30 -20 -10 0 10 20

Turkey

Latvia

Romania

Estonia

Poland

Croatia

Hungary

Lithuania

Albania

Slovakia

Bulgaria

Serbia

Slovenia

Kyrgyzstan

Czech Republic

Kazakhstan

Russian Federation

Montenegro

Azerbaijan -27

-250

0

0

0

0

0

-19

-18

-16

-14-15

-13

-12

-12

-12

-9-5-5

-5

-5-3

-3

-1

-2

-8

-53

8

8

222

2

16

72

8

8

134

7

73

66

13

19

76

97

13

12

Countries are ranked by the performance difference between the combined reading scale and the ‘reflect and evaluate’ aspect subscale. Source: OECD 2010 vol.I, Figure I.2.28.

35 OECD 2011, p.76.

40



figure 5 performance difference between the pIsa 2009 combined reading scale and each text format subscale.

-25 -20 -15 -10 -5 0 5 10 15

Estonia

Latvia

Romania

Turkey

Czech Republic

Croatia

Serbia

Poland

Slovakia

Lithuania

Slovenia

Hungary

Russian Federation

Bulgaria

Montenegro

Azerbaijan

Albania

Kazakhstan

Kyrgyzstan

Difference from combined reading scale

� Non-continuous texts� Continuous texts

-215

8

7

0

4

4

1

1

2

2

2

2

2

2

3

11

1

3

-20

-18

-11

-10

-8

-7

-7

-7

-6

-6

-5

-4

-4

-4

-3

0

0

-1

-4

Countries are ranked by the performance difference between the combined reading scale and the ‘non-continuous’ text format subscale. Source: OECD 2010 vol.I, Figure I.2.37.

box 4 pIrls and TImss studies in learning achievement

PISA is not the only international survey of student learning achievement. Two other large ongoing surveys are the Trends in International Mathematics and Science Study, which has been conducted every four years since 1995, and the Progress in International Reading Literacy Study, conducted every five years since 2001. PIRLS 2006 assessed the reading achievement of students in the fourth grade (aged on average about 10 years) across 40 countries, while TIMSS 2007 assessed student achievement in mathematics and science in the eighth grade (aged about 14 years), as well as the fourth grade, across 59 countries. Both surveys also collected information from students, teachers and school principals on home and school factors, while PIRLS also collected information from parents on activities used to foster early literacy skills. The surveys were directed by the TIMSS & PIRLS International Study Center at Boston College, United States, under the auspices of the International Association for the Evaluation of Educational Achievement (IEA), which has been carrying out international studies of school achievement, attitudes and curricula since 1959.

OVERALL PERFORMANCE

41

CHAPTER 2

TIMSS and PIRLS differ from PISA in a number of dimensions that can be expected to contribute to results differences between them. The differences involve the following: type of achievement assessed and degree of detail involved; age of target population (PISA measures the achievement of children of a given age, while TIMSS and PIRLS cover children in a given grade level); sample design and response rates; form of test (e.g. proportion of multiple choice and open-ended items); and the item-response model applied to the data (i.e. how respondents’ answers are summarized into a single score). While TIMSS and PIRLS focus more on measuring the mastery of a common set of agreed-upon curricular themes and topics, PISA is intended to measure broader skills, trying to look at how students would be able to use what they have learned in real-life situations. But both try to measure the performance of education systems so that these can be improved, so we do expect some similarities in results.*

Twelve of our study focus countries participated in PIRLS 2006: Bulgaria, Georgia, Hungary, Latvia, Lithuania, Moldova, Poland, Romania, the Russian Federation, Slovakia, Slovenia and the former Yugoslav Republic of Macedonia. All except the former Yugoslav Republic of Macedonia also participated in PISA 2009. Their results, relative to other countries, are generally better than they are in PISA, especially for the Russian Federation and Bulgaria (see Figure 32 in Annex 2). Indeed, the Russian Federation comes out on top (at a level with Hong Kong and Singapore) in PIRLS. Hungary and Bulgaria also had higher levels of achievement than the majority of other participants. Most of the highest-achieving countries in PIRLS 2006 showed significant improvement since 2001, including the Russian Federation, Hungary, Slovenia and Slovakia. The PIRLS report highlights the effects of the structural changes that both the Russian Federation and Slovenia underwent during this period through which students now receive one more year of primary schooling.

Sixteen of our study focus countries took part in TIMSS 2007: the Czech Republic, Georgia, Hungary, Lithuania, the Russian Federation, Slovenia, and Turkey (participating both at the fourth and eighth grades), Kazakhstan, Latvia, Slovakia (fourth grade only), Bulgaria, Romania and Serbia (eighth grade only), as well as three CEE/CIS countries which have not participated in PISA: Armenia, Bosnia and Herzegovina, and Ukraine. The top performers in both mathematics and science were Asian countries (Singapore, Chinese Taipei, Hong Kong, Japan and Republic of Korea). Our study focus countries did better generally in TIMSS than in PISA, particularly in the case of Kazakhstan (see Figure 33 and Figure 34 in the Annex). An exception was Turkey, which participated in the survey of eighth-grade students and performed less well than it did in PISA. In both mathematics and science, the Russian Federation did very well (in both grades), as did Kazakhstan (which participated only in the survey of fourth-grade students). In science, study focus countries that performed well were Hungary (in both grades), Latvia (which participated only in the survey of fourth-grade students), and the Czech Republic and Slovenia (eighth grade only).

In PIRLS, as in PISA, girls outperformed boys in reading. However, in mathematics and science the picture was different: in TIMSS, at the fourth grade, the differences in achievement between boys and girls were negligible in approximately half the countries in both mathematics and science. In the remaining countries, girls achieved higher scores than boys in about half and vice versa in the other half. At the eighth grade, the differences in achievement between boys and girls were negligible in about one third of the countries. Girls achieved higher scores than boys in the majority of the remaining countries, especially in mathematics. Among our study focus countries, girls scored higher in mathematics in Kazakhstan and the Russian Federation (in the fourth grade), and in Bulgaria, Romania and Lithuania (in the eighth grade); girls scored

42



Key findings on overall performance

In terms of performance averages in the three literacy domains of science, reading and mathematics, we have seen that there is a clear separation in ranking among the study focus countries that participated in PISA 2009 between the EU8 countries, at the top (with averages similar to those of OECD countries), and CEE/CIS countries, at the bottom. Above the OECD average are Estonia (in all three literacy domains), Poland (in reading and science) and Slovenia (in mathematics and science). At a level with the OECD average are Hungary (in all three literacy domains), the Czech Republic (in mathematics and science) and Slovakia (in mathematics). All other study focus countries score significantly lower than the OECD average.

The best-performing CEE/CIS country in reading is Croatia, and Croatia together with the Russian Federation in mathematics and science. Turkey follows. At the other extreme, Kyrgyzstan is the lowest-performing country among all 74 PISA countries. In Kyrgyzstan, more than 80 per cent do not reach the baseline level of achievement in the three literacy scales. Other CEE/CIS countries where the majority of students do not reach this level are Albania, Georgia and Kazakhstan (in all three literacy scales), Azerbaijan (in reading and science), Moldova (in reading and mathematics) and Montenegro (in mathematics and science).

On average in reading, 48 per cent of students in CEE/CIS countries do not reach the baseline level of achievement, compared with 19 per cent among the EU8 and the OECD countries. In mathematics, 53 per cent of students in CEE/CIS countries do not reach the baseline level, compared with 21 per cent in EU8 countries. In science, 47 per cent of students in CEE/CIS countries do not reach the baseline level, compared with 15 per cent in EU8 countries.

higher in science in Bulgaria and Romania (in the eighth grade). In mathematics, boys did better in the Czech Republic, Slovakia and Slovenia (in the fourth grade); while in science, boys did better in the Czech Republic and Slovakia (in the fourth grade), and in the Czech Republic and Hungary (in the eighth grade).

Looking at the results in PIRLS and TIMSS of the CEE/CIS countries that did not participate in PISA, we can see that The Former Yugoslav Republic of Macedonia is the worst performing CEE/CIS country in PIRLS, although it fared better than some other countries participating, such as Qatar, Indonesia and Trinidad and Tobago. In the eighth-grade TIMSS survey, Armenia does very well in mathematics, with a score significantly below only those of the Russian Federation and Hungary, among our study focus countries participating in TIMSS. Ukraine and Bosnia and Herzegovina are at a level with Bulgaria and Romania in mathematics, while in science Ukraine does better than them both and also outperforms Serbia, at a level with Armenia. In the fourth-grade TIMSS survey, Ukraine outperforms only one country, Georgia. Armenia does well in mathematics, above the Czech Republic, but is at a level with Ukraine in science. (See also Exhibit 1.2 in Martin et al. 2008, Mullis et al. 2007 and 2008).

*The correlation between average PISA and TIMSS 8th-grade scores is 0.90; PISA and TIMSS 4th-grade scores is 0.80; PISA and PIRLS scores is 0.77 (for the 27, 26, and 31 countries, respectively, with data on both surveys). In these same countries, the two-way correlations between reading, mathematics and science PISA scores are all above 0.95.

OVERALL PERFORMANCE

43

CHAPTER 2

Looking at performances in different reading competencies, students in the study focus countries seem to be better at obtaining information and understanding the meaning of a text, rather than at reflecting on the implications of its content: the ‘reflect and evaluate’ aspect seems to be the most problematic for several of the low performers in the region. In PISA 2006, which focused on science literacy, it emerged that students in this region were better at explaining phenomena scientifically than at identifying scientific issues or using scientific evidence. Comparing results on different surveys, CEE/CIS countries tend to do better, relative to the OECD countries, in TIMSS and PIRLS than in PISA. One common explanation of this is that PISA assesses the use and application of knowledge and skills in real-life situations, while TIMSS and PIRLS focus more on measuring the mastery of an internationally agreed formal curriculum.

© UNICEF

ChapTer 3

InvesTIgaTIng equITy In performanCe

Within-country disparities in performance ..............................................................................46

Gender differences in performance ...................................................................................................49

Relationship between socio-economic background and reading performance .................................................................................................................................51

Immigrant background and reading performance ..........................................................54

Resilient students .......................................................................................................................................................56

Distribution of resources across schools ....................................................................................57

School location ..............................................................................................................................................................59

Between-school variance in reading performance and in socio-economic background ....................................................................................................60

Key findings on equity in performance ..........................................................................................63

46



ChapTer 3: InvesTIgaTIng equITy In performanCe

Average performance levels mask significant variation in performance within countries. Where within-country differences in performance are large, there is concern when education is considered in terms of its function of furthering equality of opportunity and social cohesion.

To give an overview of the extent of inequality in performance within countries, we will look at the performance distribution in terms of the size of the gap between top achievers and the lowest achievers, or relative disadvantage. Later, we will examine inequality in performance in various sub-national populations (by sex, socio-economic background, immigrant status and community size) and inequality in the distribution of resources across schools. The impact of the family and socio-economic background of students and schools on learning outcomes is what is used by the OECD as a measure of equity in the distribution of learning opportunities36.

Within-country disparities in performance

Differences between countries are small compared with differences in performance within countries. Across the OECD, just 11 per cent of variation in reading performance can be attributed to differences among countries, compared with 35 per cent among schools and 55 per cent among individual students. Across all PISA 2009 countries, 25 per cent of performance variation is attributed to differences among countries, 30 per cent among schools and 45 per cent among students37.

Table 3 shows the gap between the top achievers (95th percentile) and the lowest achievers (5th percentile) for each country in each of the three literacy domains38: a range in performance excluding five per cent of students at the top and at the bottom of the performance distribution. This range is very large in all countries: to give some perspective, the average range of performance within an average study focus country is more than seven times the average progression in scores from one year to the next. But within-country differences are less in some countries than in others, and range from about the equivalent of six years of schooling in Azerbaijan to nine years in Bulgaria. In comparison, the difference between the average performance of the top-performing study focus country, Estonia, and the bottom-performer, Kyrgyzstan, is less than five times the average progression between two years of schooling.

Estonia, Azerbaijan, Latvia and Romania have relatively narrow gaps between high and low performers over the three literacy scales. Across all PISA countries, only Indonesia has less difference between low and high performers than Azerbaijan, on average over the three scales. OECD points out that the variance in student performance in countries with very low average performance may be underestimated, since it is difficult to distinguish between low and very low levels of performance39. At the opposite end, Bulgaria has a much wider distribution of performance than all other study focus countries, indicating that there are considerable inequalities in what students learn within Bulgaria’s education system. In PISA 2009, only Qatar, Israel, and Trinidad and Tobago had larger disparities. In some countries, large differences in performances can be explained by different ethnic or language groups within the country, while in others it may be connected to characteristics of the education system, such as tracking and selection at an early age. In the study focus countries, there are generally fewer disparities in performance than in OECD

36 OECD 2010 vol.II, p.26.

37 OECD 2010 vol.II, p.37.

38 The 95th percentile is the score not reached by 95 per cent of students (and reached only by the highest-scoring 5 per cent of students). The 5th percentile is the score not reached by the lowest-scoring 5 per cent of students (but reached by the other 95 per cent of students).

39 OECD 2010 vol.II, p.50.

INVESTIGATING EQUITY IN PERFORMANCE

47

CHAPTER 3

countries, with only Albania, Bulgaria, the Czech Republic, Slovakia and Slovenia having larger disparities than the OECD mean (averaging across the literacy domains), and only Bulgaria having substantially higher. The EU8 (at 290) and CEE/CIS countries (289) have on average similar score-point disparities.

Table 3 Within-country disparities in performance: difference between 95th and 5th percentile in reading, mathematics and science literacy, pIsa 2009.

Reading Mathematics Science Mean

Azerbaijan 251 207 245 234

Latvia 262 259 254 258

Romania 293 260 257 270

Estonia 274 265 277 272

Turkey 270 310 265 282

Serbia 274 298 277 283

Croatia 284 292 276 284

Lithuania 283 290 280 284

Moldova 293 281 283 286

Kazakhstan 301 272 286 287

Poland 293 290 286 290

Montenegro 304 279 286 290


Hungary 300 303 288 297

Kyrgyzstan 328 269 299 299

Georgia 321 281 297 300

Slovakia 297 311 308 306

Slovenia 297 314 306 306

Albania 326 300 291 306


Bulgaria 368 324 344 346

Study focus countries mean

296 285 287 289

OECD mean 305 300 308 304

Study focus countries-OECD

-9 -15 -21 -15

EU8-OECD -16 -8 -18 -14

CEE/CIS-OECD -4 -19 -23 -15

Countries are ranked in order of average range in performance (95th to 5th percentile) on the three literacy scales. Values in the last three rows refer to the difference between mean scores of country groupings. Source: OECD 2010 vol.1 Tables I.2.3, I.3.3, I.3.6, Walker 2011 Table B2.1, B3.1, B3.3.

It should be remembered that PISA is a sample survey and, due to sampling error, not all differences between countries are statistically significant. In fact, percentile differences have larger standard errors than means, so that the confidence intervals applied around the sample estimates (within

OECD mean

Study focus country mean

48



which there is a 95 per cent certainty that the real difference in the population will occur) are rather large40. Many of the differences between countries are therefore not statistically significant. Consequently, there is no precise ranking between countries, and the position of each country should not be analysed in detail41. This is one reason why we do not rely on only one indicator of inequality.

Is there a relationship between overall performance and within-country disparities? What we would like to see is that there is no trade-off between high performance and low disparities. Concentrating on raising performance standards will not mean that this should be at the price of large disparities, nor that a focus on reducing within-country differences will have the effect of depressing the average. Looking at Figure 6, we can see that, in general, overall achievement and within-country disparities are not associated. For example, looking at two among the lowest-performing countries, we find that Azerbaijan has small disparities while Albania has relatively large. Looking at two countries with a similarly high level of performance, we find that Latvia has small disparities while the Czech Republic has large. What is important for educational policy is that high absolute standards of performance are not incompatible with low levels of within-country disparities; indeed, this is the ideal policy goal: high performance and high equity (or low inequality).

figure 6 mean performance vs. 95th to 5th percentile difference: average science, reading and mathematics, pIsa 2009.

Estonia

Poland

Slovenia Hungary

Czech Republic Slovakia

Latvia

Lithuania Croatia

Russian Federation

Turkey

Serbia

Bulgaria

Romania

Montenegro Kazakhstan

Azerbaijan

Albania Kyrgyzstan

Moldova

Georgia

200

250

300

350

300 350 400 450 500 550

ave

rag

e 95

th-5

th p

erce

nti

le d

iffer

ence

average mean performance

EU8 countries are indicated with a green square, while CEE/CIS countries are indicated with a red lozenge. OECD average corresponds to values of Slovenia. Source: OECD 2010 vol.1 Tables I.2.3, I.3.3, I.3.6 , Walker 2011 Table B2.1, B3.1, B3.3.

40 Furthermore, looking at the percentile differences, a less robust pattern emerges across the different tests and sources available, such as the TIMSS and PIRLS surveys, compared with mean outcomes. Results on within-country differences in performance can be influenced by the choice of item-response model used to summarize the answers of students into a single score (see Brown et al. 2007). But even within a single survey such as PISA, the percentile differences in reading, mathematics and science literacy do not correlate so strongly between each other as do mean results across the three literacy scales.

41 Figure 28 to Figure 30 in Annex 2 show whether differences between countries are significant or not.


49

CHAPTER 3

gender differences in performance

One of the United Nations Millennium Development Goals is to reduce gender disparities in access to education, with a concern for the disadvantages faced by females. Among the CEE/CIS countries in PISA, only Turkey had a problem with this target until recently42. However, despite this apparent equity of access it is necessary to examine whether there are any gender disparities in achievement once students are in school. The concern about underachievement in education is not only for females, but also for males, in particular in reading achievement.

In reading, girls have significantly higher average scores than boys in all 74 participating countries, and the gender gap has widened in the last 10 years. The gender gap in reading achievement is larger in the CEE/CIS region than it is in the OECD. In the OECD, girls outperformed boys by 39 points, on average – equivalent to one school year’s difference – while among the study focus countries there is an overall gender gap of 48 points. However, the gender gaps exist to different extents within the region. The smallest difference is in Azerbaijan (24 points), with the gap in the other study focus countries ranging from 38 points in Hungary to 62 points in Albania (see Figure 7).

In many countries there are more than double the number of boys not reaching the baseline level of performance than there are girls. For example, in Croatia that means that while there are 13 per cent of girls not reaching the baseline level of achievement, there are 31 per cent of boys not reaching that same level. In many countries, tackling underperformance in reading may mean targeting boys. The OECD suggests that differences in how boys and girls approach learning and how engaged they are in reading account for most of the gap in reading performance. For instance, it found that almost 70 per cent of the difference in reading performance between different genders was the indirect result of disparities in how much boys and girls reported enjoying reading and knowing about effective strategies to summarise complex information in their reading43. It is especially among boys that enjoyment of reading has deteriorated through the years44. Generally, the advantage in favour of girls is less in non-continuous texts, and this may be associated with the fact that girls tend to favour reading longer texts while boys prefer comics and newspapers. The advantage in favour of girls is also less on the ‘integrate and interpret’ subscale, while it is more on the ‘reflect and evaluate’ subscale.

For 20 countries which undertook a computer-based assessment, among which were Hungary and Poland, it was possible to analyse how well girls and boys read digital texts and compare this with the general reading scale, which was based only on printed texts. In all countries, while girls also outperformed boys in digital reading, the difference was less marked than in print reading: girls in the OECD scored on average 24 points more than boys in digital reading, compared with a difference of 39 points in print reading45. According to the OECD, “these results suggest that it might be possible to harness boys’ relatively strong performance in digital reading and use it to improve their overall proficiency as readers”46.

Gender differences in mathematics and science are much smaller than in reading. In mathematics, among the 65 countries participating in PISA 2009, there are 35 countries in which boys have an advantage and five countries in which girls have an advantage. The average difference in favour of

42 UNICEF 2007, p. 88.

43 OECD 2010 vol.III, p.13, p.88.

44 OECD 2010 vol.V, p.14.

45 OECD 2011, p.78-80.

46 OECD 2011, p.86.

50



boys in the OECD is 12 points, while it is only three points in the study focus countries. There is an advantage for boys in Montenegro, Croatia, Turkey, Serbia, Hungary, Estonia and Azerbaijan (seven countries) and an advantage for girls in Albania, Lithuania and Kyrgyzstan (three countries). There is no significant difference in the remaining 11 study focus countries (see Figure 7). This is different from the results of the TIMSS survey (see Box 4), where girls more often outperformed boys, and this may be explained by a different focus of the mathematics tests47.

In science, in the majority of countries participating in PISA − and especially among OECD countries − differences in average scores for boys and girls are not statistically significant. Among the study focus countries, there is a tendency for girls to perform better than boys, with girls outperforming boys in 14 of the 21 countries and by an average of 10 points; in the OECD, the average difference between girls and boys is zero. In 2006, when science was the main assessment focus, it was possible to analyze gender differences in various aspects of science literacy. It was found that girls scored higher in identifying scientific issues and, generally, in using scientific evidence, while boys tended to be better at explaining phenomena scientifically48.

However, the OECD recognises that attention needs to be given to the fact that “the limited gender differences in science performance have not been reflected in equal choices to study science: on average, nearly twice as many males as females in OECD countries are graduating with science degrees”49, including mathematics and computing. Although progress has been made in addressing the traditional disadvantage of females in scientific subjects at the end of compulsory schooling, much still needs to be done in order to make science and mathematics attractive choices for both females and males in tertiary education.

47 PISA 2003 included a one-off assessment of student problem-solving skills, closely linked to mathematics but requiring little or no curriculum content. There were no significant gender differences in the average problem-solving performance in any of the eight study focus countries participating in PISA 2003, and in only a few of all participating countries did a statistically significant difference exist (in one country in favour of boys, and in five countries in favour of girls). According to the PISA report: ‘The result may be viewed as an indication that in many countries there are no strong overall disadvantages for male students or female students as learners, but merely gender-specific strengths or preferences for certain subjects’ (OECD 2004, p. 109).

48 See UNICEF 2009, pp.51-2.

49 OECD 2007, p. 61.


51

CHAPTER 3

figure 7 performance difference between girls and boys, pIsa 2009.

Azerbaijan

Hungary

Serbia

Estonia

Turkey

Russian Federation

Czech Republic

Romania

Croatia

Slovakia

Kazakhstan

Poland

Latvia

Montenegro

Moldova

Slovenia

Kyrgyzstan

Lithuania

Georgia

Bulgaria

Albania

� Science� Mathematics� Reading

Difference in mean (boys - girls)

-70 -60 -50 -40 -30 -20 -10 0 10 20

-7-24 8

-38 12

-1-44 9

-12-43 11

-3-45 2

-55-48

-103-43

-911-51

-13-51

-63-50

2-7

-47

12-13

-53

3-14

-45

-112-39

Girls dobetter thanboys

Boys dobetter thangirls

1-14

-55

-20-61 -4

-19-61 -3

-17-59 -6

-22-53 -6

-29-62 -11

-9-43 -1

Countries are ranked by the sum of differences between boys and girls on the three scales. (Typically differences up to ±5 are not significant).

relationship between socio-economic background and reading performance

How much socio-economic background affects educational performance is a key indicator of equity, in terms of equal distribution of educational opportunities. Socio-economic background in PISA is measured though an index including highest occupational status and level of education of the father or mother, as well as the number of home possessions as a proxy for wealth50. The OECD uses the term ‘socio-economic gradient’ to refer to the relationship between socio-economic background and

50 Home possessions include a desk, a quiet place to study, own room, own calculator, educational software, internet access, works of art, books including classical literature, poetry books, books to help with school work, dictionary, and dishwasher, VCR/DVD player, cellular phones, televisions, computers, cars. The index was constructed so that the OECD average was 0 and about two-thirds of students in OECD countries were between the values of -1 and 1 (1 was the standard deviation). See OECD 2010 vol.II, p.29. It should be taken into account that the measures of socio-economic background included in the PISA index are weaker approximations of socio-economic status in some countries than in others, which could result in an apparently weaker relationship with performance (OECD 2010 vol.II, p.64).

52



performance, summarized by a plotted line showing the predicted reading score based on the index of economic, social and cultural status of each student51.

The socio-economic background of students and schools has generally a strong influence on performance: most students who perform poorly are from socio-economically disadvantaged backgrounds. The OECD suggests that highly educated parents may invest more of their time educating their children, while parents with prestigious occupations may become role models for their children, and wealthy families may provide more educational resources. Furthermore, stress levels, parent-child interaction and language quality and richness may differ according to the socio-economic level of the family52, each of which can affect student performance. Although the relationship between socio-economic background and performance is present in all countries, the strength of the association varies across schooling systems, with some managing to mitigate existing inequalities more than others. A relatively weak relationship between the socio-economic background of students and their performance indicates an equitable distribution of educational opportunities in a school system.

On average in the study focus countries, a student from a more socio-economically advantaged background (top one-seventh) scores 36 points more in reading than a student with an average background, which we have already seen is equivalent to almost one year of schooling53. The average difference, referred to as the ‘slope’ of the gradient, generally holds constant for each increase of one standard deviation in socio-economic background, since the relationship tends to be roughly linear54. This means that in most countries there is no particular level of socio-economic status below which performance declines sharply. However, in Kyrgyzstan the performance advantage increases at higher levels of socio-economic background, while in Slovakia, Hungary and Poland the advantage is less at higher levels of socio-economic background. Differences range from 21 points in Azerbaijan to 51 points in Bulgaria (see the red dots in Figure 8). There is a close relationship between such average point differences by socio-economic background and the overall within-country differences in performance we saw earlier.

Nevertheless, for any group of students with similar backgrounds there is a considerable range of performance. How large this range is defined by the strength of the relationship. The proportion of variation in student performance accounted for by socio-economic background is in fact referred to as the ‘strength’ of the gradient. The higher this proportion, the better the performance of a student can be predicted from his or her socio-economic and cultural status. On average in the study focus countries, 13 per cent of differences in reading performance within countries are associated with differences in socio-economic background (the figure is 14 per cent in the OECD). The smallest strength in the relationship between background and performance can be found in a low-performing country, Azerbaijan (seven per cent), as well as in the top-performing country in the region, Estonia (eight per cent). Estonia, as well as Latvia albeit to a lesser extent, is an example of a country with a low proportion of students who do not reach the baseline level of achievement, an overall small range in performance, relatively small differences by socio-economic background, and weak relationship between socio-economic background and performance.

The strength of the gradient is highest in Hungary, where it reaches 26 per cent, followed by Bulgaria (20 per cent) and Turkey (19 per cent − see the blue bars in Figure 8). Hungary has, therefore, a relatively

51 The results on the reading test are used as indicators of performance, since reading was the focus of PISA 2009: similar results are obtained taking mathematics or science performance.

52 OECD 2010 vol.II, p.30, 50.

53 Such analyses are limited to reading only since reading was the focus of the PISA 2009 assessment and “previous analyses have shown that the relationship between the socio-economic background of students and schools and learning outcomes generally does not vary markedly across the subject areas of reading, mathematics and science that are measured by PISA.” (OECD 2010 vol.II, p.26).

54 See OECD 2010 vol.II, p.14, 54, 57.


53

CHAPTER 3

large difference in mean score between advantaged and disadvantaged students, and at the same time has relatively few disadvantaged students escaping from this rule. In Turkey, although the performance difference associated with a unit increase in the socio-economic background index is quite small (29 points, as in Estonia and Latvia), the relationship between background and performance is relatively strong (19 per cent of variance explained by such an index), so fewer disadvantaged students escape from the rule and perform at the level of more advantaged students. Furthermore, Turkey is the country with the largest range in the socio-economic background index in the CEE/CIS region, indicating large socio-economic disparities among households, so the socio-economic gradient will have a larger impact in Turkey than in other countries with similar slopes. It has also to deal with the lowest mean socio-economic background: more than half of all 15-year-old students in Turkey have a socio-economic background which is below that of the least-advantaged 15 per cent of students in OECD countries55.

The OECD points out that countries with steep socio-economic gradients and above-average strength of relationship between socio-economic background and learning outcomes may benefit from targeting disadvantaged children through a specialized curriculum, additional instructional resources or economic assistance for these students, while for those countries with weaker gradients and less variation in student performance more universal policies will likely be more appropriate56.

figure 8 score point difference in reading associated with one unit increase in the pIsa index of economic, social and cultural status and percentage of variance in student reading performance explained by student socio-economic background.

0

10

20

30

40

50

60

0

10

20

30

40

50

60

Bu

lgar

ia

Hu

nga

ry

Cze

ch R

epu

blic

Slo

vaki

a

Kyr

gyz

stan

Slo

ven

ia

Pola

nd

OE

CD

ME

AN

Kaz

akh

stan

Ru

ssia

nFe

der

atio

n

Ro

man

ia

Lith

uan

ia

Cro

atia

Alb

ania

Mo

nte

neg

ro

Turk

ey

Latv

ia

Est

on

ia

Ser

bia

Aze

rbai

jan

Percentage of variance in performance explained by students' socio-economic background

Sco

re p

oin

t d

iffer

ence

Perc

enta

ge

of

vari

ance

in s

tud

ent

per

form

ance

Difference in score associated with a unit increase in socio-economic background index

7%10% 8% 10% 10% 11% 11% 11% 12% 12%

26%

20%

14% 14% 14% 15% 14% 15% 15%

19%21 27

29 29 2931 31 32 33

3637 38 38 39 39

40 41

4648

51

Source: OECD 2010 Figure II.1.4, Table II.3.3. Countries to the left of the chart have a more equitable distribution of learning outcomes by socio-economic background than countries to the right of the chart.

55 OECD 2010 vol.II, p.62.

56 OECD 2010 vol.II, pp.104-5.

54



If we look at the percentage of variance in reading performance explained by various aspects of family background − that is, not only the socio-economic background defined on the basis of parent education, occupation and family possessions, but also including family structure (whether from single-parent family), immigrant status and language spoken at home57 − on average a further eight per cent of variance in reading performance is explained by these variables. Such family characteristics generally affect educational performance results jointly, being often associated with each other. For example, on average among the study focus countries (the OECD average is very similar), these selected factors together explain 22 per cent of variance in reading performance. Looking at unique contributions, and keeping the other factors constant, 4.2 per cent of performance variance is explained by cultural possessions and books at home; 2.8 per cent by parents’ highest level of education and occupational status; 1.1 per cent by home educational resources; 0.5 per cent by wealth; 0.3 per cent by immigrant status and language spoken at home; and 0.1 per cent on family structure on its own. The remaining 13 per cent of variance is explained by more than one of the factors impacting on reading performance jointly.

Immigrant background and reading performance

Although we have seen that immigrant status explains on average only 0.3 per cent of variation in performance, it is nevertheless an equity issue if students with an immigrant background do not have the same learning opportunities as other students.

In PISA, native students (those born or with at least one parent born in the country of assessment) are distinguished from students with an immigrant background, who include first-generation students (themselves and parents foreign-born) and second-generation students (born in the country of assessment but with foreign-born parents). First- and second-generation students are distinguished since the former may have to face additional difficulties in entering a new education system, or learning a new language, and their performance will be partly affected by their experiences prior to entering the country, with some of the differences possibly due to different backgrounds across immigrant cohorts 58. Furthermore, it is noted that countries may differ in terms of the national origins and socio-economic, educational and linguistic backgrounds of their immigrant populations, affecting the results.

While in the OECD 10 per cent of students have an immigrant background, the figure falls to four per cent in the study focus countries. Only Croatia, the Russian Federation and Kazakhstan have a percentage of students with an immigrant background which is above the OECD average (11-12 per cent)59. In the OECD, first-generation students score on average 52 points fewer than native students, while second-generation students fare relatively better, although they still have an average score that is 33 points below that of native students. Part of the performance advantage of native students is explained by the fact that often students with an immigrant background are also socio-economically disadvantaged (the average score difference between native and immigrant background students is reduced from 43 to 27 points after accounting for socio-economic background) – although among the study focus countries this is not necessarily the case. In fact in the region there is more often no statistically significant correlation between the indexes of socio-economic status and immigrant background, except a slightly negative one in Slovenia, Croatia, Kazakhstan, the Russian Federation and Serbia60. In about half of the study focus countries with data (Slovenia, Estonia, Russian Federation, Croatia, Lithuania, Czech

57 OECD 2010 vol.II, p.44.

58 OECD 2010 vol.II, p.66.

59 OECD 2010, table II.4.1.



55

CHAPTER 3

Republic and Latvia), students without an immigrant background perform better in PISA, while it is the opposite in Hungary, Azerbaijan, Kyrgyzstan, Kazakhstan, Serbia and Montenegro, where it is actually students with an immigrant background who perform better61.

One difference that is apparent between students in the study focus countries with immigrant backgrounds and those in the OECD is that in the study focus countries a higher percentage of such students speak the language of assessment at home. Nevertheless, even accounting for differences in language spoken at home, part of the variation in performance levels remains. The OECD points out that students with an immigrant background tend to attend schools with a more disadvantaged socio-economic intake (although this is not true in Estonia and the Czech Republic), but with comparable student-teacher ratios and other school characteristics to those attended by students without an immigrant background62.

Looking at absolute levels of educational disadvantage by immigrant status on average in the OECD, we see that 17 per cent of native students do not reach the baseline in reading, while the figures are 27 per cent for second-generation and 36 per cent for first-generation students. On the other hand, on average among the eight study focus countries for which we have separate data for the three groups, there are as many native students performing below the baseline as second-generation students (31 per cent), while the difference between these and first-generation students (38 per cent of whom perform below the baseline) is smaller than in the OECD.

figure 9 percentage of 15-year-old students scoring below level 2 in reading, by immigrant background.

Native students

First-generation studentsSecond-generation students

Perc

enta

ge

bel

ow

Lev

el 2

in r

ead

ing

0

10

20

30

40

50

60

70

80

90

100

Kyr

gyz

stan

Aze

rbai

jan

Kaz

akh

stan

Mo

nte

neg

ro

Ser

bia

Ru

ssia

nFe

der

atio

n

Lith

uan

ia

Cze

ch R

epu

blic

Cro

atia

Slo

ven

ia

Hu

nga

ry

Latv

ia

Est

on

ia

1217

1719 22

23 23 2533

49

59

72

83

Countries are ranked by percentage of native students (without an immigrant background) who score below Level 2. Estonia, Latvia, Lithuania, Kyrgyzstan and Azerbaijan have too few first-generation students for their performance to be reported separately. Source: OECD 2010 Table II.4.2.

In Figure 9, the bars correspond to the percentage of native students scoring below the baseline in reading (and since they represent the vast majority, the percentage corresponds roughly to that of all students). The largest differences between the performances of students in the three groups can be found in Slovenia. In fact, first-generation students in Slovenia are more than twice as likely to

61 In Poland, Slovakia, Turkey, Bulgaria, Romania, Albania, there are too few students with immigrant backgrounds (less than 1 per cent) to analyse their data separately. Student performance is analysed only for countries where there are at least 30 students (in this case with immigrant background) dispersed in five different schools who are compared with other students (OECD 2010 vol.V p.84).

62 OECD 2010 vol.II, p.79.

56



score in the bottom quarter of the reading performance distribution63, while for all other study focus countries the likelihood is lower than the OECD average of 1.9.

resilient students64

We have seen that student socio-economic background has, generally, a strong influence on performance, and that most students who perform poorly are from socio-economically disadvantaged backgrounds. However, there are always some students who despite their disadvantaged background manage to achieve high levels of performance – more than predicted by the socio-economic gradient line. The OECD defines these as resilient students, or internationally successful disadvantaged students65, indicating by such terms those students who come from the bottom quarter of the distribution of socio-economic background in their own country while scoring in the top quarter in reading among students with similar socio-economic background from all countries. That is, the actual performance of a disadvantaged student is compared with the performance predicted by the average relationship among students from similar socio-economic background across countries66.

As could be expected, top-performing countries generally also have the largest percentage of resilient students. On average in the EU8, 27 per cent of disadvantaged students are resilient (the figure is 31 per cent in the OECD), while among CEE/CIS countries it is half this amount (14 per cent). One notable exception is Turkey, which has the highest percentage of resilient students among the study focus countries (42 per cent of all disadvantaged students in the country – see Figure 10), while being in the middle of the ranking table of performance in the region, as well as having a relatively strong relationship between background and performance67. Also among the CEE/CIS countries, Croatia does quite well with respect to the percentage of resilient students (27 per cent).

The OECD published a report on resilient students, based on the PISA 2006 survey, which focused on science68. Although the report used performance in science as a measure of student achievement, it notes that the vast majority of students who are resilient with respect to science are also resilient in at least one if not both of the other domains.

The report identifies two factors that appear to be associated with successful academic performance among disadvantaged students. The first is the extent to which disadvantaged students adopt positive approaches to learning. Resilient students are more motivated to learn, more engaged with science and most of all have greater self-confidence in their ability to learn science than their disadvantaged low-achieving peers (although in part it might be that they are more self-confident because they perform well). The second factor associated with performance is the amount of time spent in regular science lessons. On average, across OECD countries disadvantaged students spent 20 per cent less time learning science at school than their more advantaged peers. Taking more science courses benefits disadvantaged students more than it does their more advantaged peers. On the other hand, school characteristics measured by PISA did not appear to play a major role in promoting science performance among disadvantaged students.


64 For more on resilient students, and what factors are associated with the academic achievement of resilient students, see also OECD 2011a (based on PISA 2006 data). Findings imply that resilient students generally have more positive learning approaches and spend more time in regular lessons at school than other disadvantaged students.

65 OECD 2010 vol.II, p.13, 62, 64.

66 Defined in this way, on the basis of whether they perform among the best of all students of all countries combined, rather than among the highest performing of their own country, the results are partly connected to the overall level of achievement of the country (the correlation between percentage of resilient students and overall performance is 0.79).

67 It must be kept in mind that Turkey is the country with the highest rate of out-of-school children at age 15, so probably the most disadvantaged students in the country do not enter the statistics at all.

68 OECD 2011a.


57

CHAPTER 3

figure 10 percentage of resilient students among disadvantaged students.

0

5

10

15

20

25

30

35

40

45

Kyr

gyz

stan

Aze

rbai

jan

Kaz

akh

stan

Mo

nte

neg

ro

Bu

lgar

ia

Ro

man

ia

Alb

ania

Ser

bia

Ru

ssia

nFe

der

atio

n

Lith

uan

ia

Slo

vaki

a

Cze

chR

epu

blic

Slo

ven

ia

Hu

nga

ry

Cro

atia

Latv

ia

Est

on

ia

Pola

nd

Turk

ey

Perc

enta

ge

of

resi

lien

t st

ud

ents 42%

37%34%

31%

27% 26% 25%

21% 21% 20% 19%

15%

11% 10% 9% 8%6%

5%1%

Resilient students come from the bottom quarter of the distribution of socio-economic background in their own country, while scoring in the top quarter in reading among students from all countries with similar socio-economic backgrounds. The percentage is over this bottom 25 per cent of students (‘disadvantaged’) in the distribution of socio-economic background. The percentage of resilient students over all 15- years-old students is therefore the percentage value above divided by four (e.g. it is 10.5 per cent in Turkey). Source: OECD 2010 Figure II.1.4, Table II.3.3.

Consequently, the report suggests that policies aimed at fostering positive approaches to learning among disadvantaged students (who generally have lower levels of motivation) could help facilitate resilience. At the same time, it points out that unless policies aimed at promoting greater motivation and positive attitudes to science learning are directed specifically to disadvantaged students, although they will still result in absolute improvements in science achievement, the policies may contribute to widening existing inequalities in performance across social groups, since in some countries the performance increase associated with motivation would be smaller for disadvantaged students than for their more advantaged peers69.

distribution of resources across schools

Coupled with the concern for equality in learning outcomes within a country is the concern for an equitable distribution of resources across schools. In other words, it is important that a school’s resources should not be related to the average socio-economic background of its students. The main resource elements within schools are teachers. In fact, in most countries disadvantaged schools have either a better teacher/student ratio (as is the case in around half of the OECD countries – and an even higher proportion among the study focus countries) or else there is no difference. Figure 11 shows the strength of the association between school mean socio-economic background and school resources: positive values indicate that the higher the socio-economic intake of a school, the better the resources, while negative values indicate that resources are more favourable in schools with lower socio-economic intake. Turkey and Slovenia (together with Israel and the United States among the OECD countries) are the only study focus countries where socio-economically disadvantaged schools70 tend also to be deprived in terms of the quantity of basic resources (i.e. there are larger student/teacher ratios): as can be seen from Figure 11, the correlation between the two variables, even if not particularly strong, is positive in only these two countries. In the Czech Republic, Hungary, Poland and Romania

69 OECD 2011a, pp.72, 78.

70 A school’s socio-economic background is the average socio-economic index of students attending that school.

58



there is no significant correlation between the two variables. In all other study focus countries the correlation is negative, meaning that the quantity of resources tends to be more favourable (i.e. there are lower student/teacher ratios) in schools with lower socio-economic intake, especially in Estonia and Kazakhstan. It is possible to hypothesize that this is the result of an explicit policy objective of moderating disadvantage71. However, there may be other factors influencing both variables.

Figure 11 shows also that the correlation between the school mean socio-economic background and the percentage of teachers with university degrees among all full-time teachers is generally positive, albeit not very strong. In other words, in many countries the more disadvantaged students attend schools that have a lower proportion of full-time teachers with advanced university degrees (i.e. they have lower quality resources). This is especially the case in Slovenia and Azerbaijan, and is contrary to the equitable distribution of resources. Consequently, it could exacerbate the disadvantage faced by such students. However, it is not the case (i.e. there is no significant correlation between intake and percentage of highly qualified teachers) in Romania, Hungary, Serbia, Turkey, Estonia and Poland, while in Slovakia it is in fact disadvantaged schools which tend to have a higher percentage of highly qualified teachers.

figure 11 simple correlation between the school mean socio-economic background and school resources.

Slovenia

Azerbaijan

Albania

Montenegro

Czech Republic

Kyrgyzstan

Kazakhstan

Russian Federation

Croatia

Latvia

Lithuania

Bulgaria

Romania

Hungary

Serbia

Turkey

Estonia

Poland

Slovakia

Correlation with school socio-economic background

-0.60 -0.40 -0.20 0.00 0.20 0.40

0.00

0.60

0.25

-0.23

-0.15

-0.33

-0.08

-0.27

-0.44

-0.29

-0.32

-0.38

-0.21

-0.21

0.02

-0.02

-0.11

0.26

-0.43-0.01

0.55

0.44

0.38

0.38

0.37

0.35

0.34

0.31

0.28

0.19

0.19

0.17

0.11

0.07

0.06

0.04

0.00

-0.05

-0.21

� Student/teacher ratio� Percentage of teachers with university-level degree among all full-time teachers

Countries are ranked by the correlation of the percentage of teachers with university-level degree. Lighter shaded bars indicate a non-significant correlation. The correlation with the student/teacher ratio was multiplied by -1. University-level degrees refer to ISCED 5A level degrees. Source: OECD 2010, Table II.2.2, Figure II.2.3.

71 OECD 2010 vol.II, p.43.


59

CHAPTER 3

school location

Inequalities across different types of communities within countries are a cause for concern. In general, students in urban schools perform better than students in rural schools, and students in larger towns perform better than those in smaller towns. This holds true, although to a lesser extent and not in all countries, even after accounting for differences in socio-economic background72. For example, in the study focus countries there is an average difference in reading performance of 67 points between students in city schools and those in rural schools. Even after the effect of socio-economic background has been taken into account, the average difference is still 40 points (equivalent of one year of schooling). Furthermore, in some countries in the region the gap is far higher: as can be seen in Figure 12, after taking into account socio-economic background there are still more than 80 points difference (equivalent to two years of schooling) between city and rural schools in Bulgaria, Kyrgyzstan and Hungary − with Bulgaria and Hungary already having the highest percentage of variance in student performance due to students’ socio-economic background. This is the largest difference among all countries participating in PISA.

figure 12 differences in reading scores of students in schools from various locations compared with students from rural schools, after accounting for the pIsa index of economic, social and cultural status.

Dif

fere

nce

in r

ead

ing

sco

re (

vs. r

ura

l sch

oo

ls)

city schools small town schools

-60

-40

-20

0

20

40

60

80

100

larger town schools

Ser

bia

Pola

nd

Mo

nte

neg

ro

Ru

ssia

n F

eder

atio

n

Lith

uan

ia

Est

on

ia

Aze

rbai

jan

Latv

ia

Cro

atia

Kaz

akh

stan

Slo

ven

ia

Cze

ch R

epu

blic

Ro

man

ia

Alb

ania

Slo

vaki

a

Turk

ey

Hu

nga

ry

Kyr

gyz

stan

Bu

lgar

ia

94 9486

70

54

49

46 43 43

36 29

2120

20

1816

159

-9

Countries are ranked by difference in performance of city vs. rural schools. Lighter shades indicate a non-significant correlation. Rural schools are in areas with fewer than 3,000 inhabitants, small towns between 3,000 and 15,000, larger towns between 15,000 and 100,000, cities more than 100,000 inhabitants. Source: OECD, PISA 2009 database, Table II.2.6.

72 These results are different from those of PISA 2006, where students in schools in smaller communities generally had lower mean science performance than those in larger communities only if differences in socio-economic context were not accounted for. Controlling for socio-economic context, the association seemed to be in the opposite direction (looking at the relationship in schools of all PISA countries together). Accounting also for a series of school and system factors, only in the Russian Federation and Kyrgyzstan did the original relationship remain (see UNICEF 2009, pp.64-66).

60



The pattern is similar among OECD countries, although far less pronounced (40 points in the first case, falling to 23 points once the effect of socio-economic background has been taken into account).

While it may be the case that urban schools provide more educational resources to children, including libraries, it is difficult to generalize because differences between communities are manifold and vary across countries. In Kyrgyzstan, for example, schools which provide instruction in Russian (which tend to have access to better textbooks, and have students of higher socio-economic background) are mostly in cities, and generally have much better results than schools which provide instruction in Kyrgyz (which are mostly in villages) and Uzbek.

between-school variance in reading performance and in socio-economic background

In this section we will examine the pattern of within-country differences in each country, and how much of the overall variation in performance is attributable to differences between schools.

The PISA 2009 report contains statistics that deconstruct the total variance into two components: ‘between-school’ and ‘within-school’, distinguishing between the variance that is attributable to differences in results attained by students in different schools, and that attributable to the range of student results within schools. The proportion of variance in student performance between schools gives us an indication of the extent to which there is separation between low and high achievers in different schools (‘academic exclusion’). Large variations between schools (which indicates less variation among students within schools, i.e. students with similar abilities are being grouped together) may be due to the ways in which students are allocated to schools (through institutional differentiation, selection, tracking or streaming). However, there may be variation between schools even in countries with comprehensive systems that are not explicitly selective. For example, variation can be a result of geographical segregation according to socio-economic or cultural background, school choices made by families or differences in the quality of instruction (which in itself is difficult to measure)73. Whatever the reason, large between-school differences in what students learn are an indication of inequality in the school system.

A different matter, but often connected to between-school variance in performance, is whether students are separated in different schools according to socio-economic background (‘social exclusion’), rather than according to performance. The OECD points out that countries with low social exclusion tend also to have low academic exclusion, as well as high mean performance. Indeed, Estonia and Latvia − two countries which have few overall within-country differences and few differences by socio-economic background − have a low proportion of between-school variance in socio-economic background (indicated by red dots in Figure 13) and in performance (indicated by blue bars in Figure 13). This shows that providing similar learning opportunities across schools is compatible with high overall achievement. At the opposite end, Hungary and Turkey show both large social and academic exclusion (i.e. students with similar socio-economic background and performance attend the same schools). These are countries with early institutional differentiation, where students are separated into different school types at 11 years of age.

When the overall situation across all study focus countries is considered, it emerges that between-school difference in performance varies greatly. Overall, there is less social exclusion (with 29 per cent of variance occurring between schools) than academic exclusion (41 per cent of variance), and less between-school variance in socio-economic background than in student performance – in other words, there are more differences between schools in terms of their average performance outcomes

73 OECD 2010 vol.II, p.84.


61

CHAPTER 3

than there are in their socio-economic intakes. In fact, the within-school component of variance in socio-economic background is greater than the between-school component in all countries, while for performance this is not the case (the proportion of between-school variance in performance is more than 50 per cent in Romania, Slovenia, Hungary and Turkey, indicating that there is a larger variation between the average performance of schools in these countries, compared with the variation in individual scores). Hungary and Turkey have the largest between-school variance in student performance among all PISA participants.

The OECD indicates how such data can be used to design policies to improve performance and equity: for example, depending on the extent to which low performance is concentrated by school, it could be more relevant to target low-performing schools (if between-school variance in performance is high) than low-performing students within schools (if between-school variance in performance is low)74.

figure 13 between-school variance in student reading performance (and in socio-economic background), as a percentage of total variance between and within schools75.

Between-school variance in student performance Between-school variance in socio-economic background

Perc

enta

ge

of

bet

wee

n-s

cho

ol v

aria

nce

0

10

20

30

40

50

60

70

80

Turk

ey

Hu

nga

ry

Slo

ven

ia

Ro

man

ia

Bu

lgar

ia

Cze

chR

epu

blic

Ser

bia

Cro

atia

Aze

rbai

jan

Slo

vaki

a

Kaz

akh

stan

Mo

nte

neg

ro

Kyr

gyz

stan

Alb

ania

Lith

uan

ia

Ru

ssia

nFe

der

atio

n

Est

on

ia

Latv

ia

Pola

nd

19% 21%

22%25% 26%

31%

36% 36% 36%40% 42%

47%49% 49% 50% 51%

57%

67% 67%

27% 25%

18%

28%26%

32%28%

23%

28% 28%

23% 23% 23% 25%

42%

35%

25%

37%

46%

Socio-economic background measured with PISA index of social, economic and cultural status. Countries are ranked by proportion of variance in student performance that lies between schools. Source: OECD 2010, Table II.5.1 and II.5.2. Between-school variance in student performance is 22 per cent in Georgia and 29 per cent in Moldova.

Furthermore, the association between performance and socio-economic background − the gradients discussed earlier − can be separated into a within-school component, describing how a student’s socio-economic background is related to performance within a common school environment, and a between-school component, describing how a school’s average level of performance is related to the average socio-economic background of its student intake.

74 OECD 2010 vol. II, p.104.

75 The component of variance that lies between schools is also referred to as intra-class correlation. It ranges from zero (same distribution for every school – completely desegregated system) to 1 (or 100 per cent) in which students within schools have a similar performance/background but school averages vary. Results are influenced by differences between countries in how schools are defined and organized and sampling units chosen (e.g. in some countries schools were defined as administrative units, even if spanning several geographically separate institutions, in others as physical school buildings) (OECD 2010 vol. II, p.99).

62



In most countries, the school’s economic, social and cultural status appears to have a much greater effect on students’ performance than the individual student’s background (see Figure 14). This means that a student attending a school with a socio-economically advantaged intake will tend to have substantially higher scores than a student with the same socio-economic background but in a school with a more disadvantaged intake. Conversely, two students with different socio-economic backgrounds attending the same school will have a smaller gap in performance. The between-school gradient is very large in the Czech Republic, where for example two students with similar backgrounds, one attending a school with an average background and the other attending a school with an advantaged background (represented by half a unit increase in school mean socio-economic background), could be expected to have a performance gap of 62 points (corresponding to more than one year of schooling). The average such gap in the study focus countries is 28 points. On the other hand, for two students attending the same school, one student having an average background and the other having an advantaged background (socio-economic background differing by one standard deviation), the predicted performance gap would be just 14 points (the average in the study focus countries is 13 points)76. At the opposite end, Poland and Latvia, which we have seen have also low absolute disadvantage and between-school variance, show small differences in school performance according to school socio-economic intake (as does Azerbaijan).

Consequently, it seems that the general socio-economic background of a student’s peers counts more than his or her own background in predicting performance 77. Unfortunately, the data available does not help us understand why there is such a relationship between school intake and performance. Students from more advantaged backgrounds may have access to better learning environments, either by choosing better schools or actively creating better schooling conditions; effects may be due to peer interactions, student selection, but also to differing levels of disciplinary problems, teacher morale or other characteristics that may vary in schools according to socio-economic intake, thus reinforcing the advantage of better-off students. Also, parents of students attending socio-economically advantaged schools may be more engaged in the student’s learning at home, whatever their own socio-economic level78.

76 The difference between 25th and 75th percentile of the PISA index of economic, social and cultural status is on average 1.3 among students, while it is half this amount among schools (0.65), both for the OECD and the study focus countries. Therefore we take a socio-economically advantaged student to be a standard deviation above the average, while a socio-economically advantaged school to be half a standard deviation above the average. It is also of interest to look at the interquartile ranges for each country (Table 6 in Annex 2). For example, the Czech Republic has a small range in school (and student) socio-economic background, which has to be kept in mind when noting the big differences in scores according to background.

77 Another way to grasp the importance of the school’s socio-economic intake over a student’s own socio-economic background is by comparing the percentage of between-school variance in performance explained by the school socio-economic intake, which is on average 55 per cent in the study focus countries, with the percentage of within-school variance in student performance explained by the individual student’s socio-economic background, which is on average only 3 per cent. Within-school social differences account for at most 4 per cent of student-level performance variation in the study focus countries, except in Poland where it is 10 per cent. On the other hand at the school level, except for Azerbaijan (13 per cent), the strength of the relationship ranges from 37 per cent in Kazakhstan and Romania to 67-70 per cent in the Czech Republic, Turkey and Montenegro. (OECD 2010 vol. II, Table II.5.2.)

78 OECD 2010 vol. II, p. 92. Given existing data limitations, there is no basis for concluding that the effect is causal: it describes an association observed in the distribution of school performance.


63

CHAPTER 3

figure 14 school-level score point difference in reading associated with half a unit increase in the school mean pIsa index of economic, social and cultural status (between-school gradient), and student-level score point difference associated with one unit increase in the student-level socio-economic background index (within-school gradient).

Azerbaijan

Poland

Latvia

Russian Federation

Albania

Romania

Estonia

Lithuania

Kazakhstan

Serbia

Turkey

Kyrgyzstan

Montenegro

Croatia

Slovakia

Hungary

Slovenia

Bulgaria

Czech Republic

0 10 20 30 40 50 60 70

8

31

19

21

13

10

16

16

19

6

8

16

11

10

17

7

2

11

14

13

15

15

19

19

20

20

21

25

26

30

31

33

35

36

38

39

41

62

� Between-school gradient� Within-school gradient

Countries are ranked by between-school gradient. Source: OECD 2010, Table II.5.2.

Key findings on equity in performance

In the study focus countries, there are generally fewer disparities in performance than in OECD countries. Differences between countries in average performance are small compared with differences in performance within countries. On average, within-country differences in performance in the study focus countries (the gap between the top achievers, at the 95th percentile, and the lowest achievers, at the 5th percentile) range from a minimum of about the equivalent of six years of schooling in Azerbaijan (six times the typical gap in the OECD between the average reading performance of 15-year-old students in two adjacent grades) to nine years in Bulgaria. While within-school differences are large in all countries, in some countries they are relatively small compared

64



with others: following Azerbaijan, the three countries Estonia, Latvia and Romania also have relatively narrow gaps between high and low performers over the three literacy scales. The example of Estonia reassures us that high absolute standards of performance are not incompatible with low levels of within-country disparities.

When attention turns to performance inequalities in various sub-national populations, it emerges that in terms of gender differences girls have significantly higher average scores in reading than boys in all 74 participating countries. The average gender gap in reading achievement is larger among the study focus countries (48 points) than it is in the OECD (39 points). The smallest gap is in Azerbaijan and the largest is in Albania. For the countries that participated in the survey of digital reading, the gender gap is usually smaller than in print reading.

Gender differences in mathematics and science are much smaller than in reading. In mathematics, there is an advantage for boys in seven study focus countries (Montenegro, Croatia, Turkey, Serbia, Hungary, Estonia and Azerbaijan), an advantage for girls in three countries (Albania, Lithuania and Kyrgyzstan), and there is no significant difference in the remaining 11 study focus countries (while boys do better in the majority of PISA countries). In science, differences in the average scores of boys and girls are not statistically significant in the majority of PISA countries. However, among the study focus countries there is a tendency for girls to perform better than boys, with girls outperforming boys in 14 of the 21 countries. Indeed, girls outperform boys by an average of 10 points among study focus countries (while the average difference in the OECD is zero).

The socio-economic background of students and schools has generally a strong influence on performance: most students who perform poorly are from socio-economically disadvantaged backgrounds. A student from a more socio-economically advantaged background (i.e. top one-seventh) scores on average 36 points more than a student with an average background, which as we have seen is equivalent to almost one year of schooling. Differences range from 21 points in Azerbaijan to 51 points in Bulgaria. However, in most countries it is the school’s average economic, social and cultural status that appears to have a much greater effect on students’ performance than the individual student’s background.

Nevertheless, as we have seen there are always some students who despite their disadvantaged background manage to achieve high levels of performance – the resilient students. Among the study focus countries, the largest percentage of resilient students is found in Turkey, where 42 per cent of all disadvantaged students (those from the bottom quarter of the distribution of socio-economic background in their own country) score in the top quarter in reading among students with similar socio-economic backgrounds from all countries.

In many countries, socio-economically disadvantaged students have also the added disadvantage of attending schools with lower quality resources (i.e. with a lower proportion of full-time teachers with advanced university degrees). However, although the quality of resources tends to be lower the quantity of resources tends to be more favourable (i.e. there are lower student-teacher ratios) in schools with lower socio-economic intake.

In general, students in urban schools perform better than those in rural schools, and students in larger towns perform better than those in smaller towns. Taking into account socio-economic background, there are still more than 80 points difference in reading performance (equivalent to about two years of schooling) between students in city schools and rural schools in Bulgaria, Kyrgyzstan and Hungary.

Whatever the reason, large between-school differences in what students learn are an indication of


65

CHAPTER 3

inequality in the school system. Performance differences between schools are highest in Romania, Slovenia, Hungary and Turkey: the proportion of between-school variance in performance is more than 50 per cent in these countries (where students within a given school tend to have similar performance levels, compared with students in different schools). It is generally those countries in which students are not separated into different schools according to socio-economic background which tend to have small between-school differences in performance and high overall performance (such as Estonia and Latvia).

© UNICEF

ChapTer 4

Trends In performanCe over TIme

Trends in reading since 2000 .......................................................................................................................68

Trends in reading since 2000, 2003 and 2006 ........................................................................70

Trends in mathematics and science ...................................................................................................72

Trends in variance in performance ......................................................................................................73

Trends in the relationship between socio-economic background and performance ....................................................................................................................74

The case of Poland.....................................................................................................................................................75

The case of Turkey .....................................................................................................................................................76

The case of Kyrgyzstan ........................................................................................................................................76

Key findings on trends ..........................................................................................................................................78

68



ChapTer 4: Trends In performanCe over TIme

PISA, being an ongoing triennial survey, makes it possible to monitor changes in educational outcomes over time. Trends across PISA assessments show that, rather than being inevitable or fixed, educational results can be improved by, for example, reducing the percentage of poorly performing students or the inequalities within countries.

For each subject, data is comparable only from the point when the first in-depth assessment took place. PISA 2000 focused on reading and provided a performance scale which was the baseline for future assessments. Consequently, comparability in reading stretches across nine years and covers all four assessments. In mathematics, where the focus and full assessment was in 2003, comparability covers six years and three assessments, while in science comparability covers only two assessments, starting in 200679. It is noteworthy that comparisons over time for mathematics and science are more limited than in reading since there have not yet been two full assessments in these subjects80. Trends have been annualised (by dividing the change in performance by the number of years between two assessments) to make them comparable across the three subject areas, but greater variability in reading trends is expected, as the longer period for reading provides more opportunities to reflect changes in education systems81.

Some trend data is available for all study focus countries except for Kazakhstan, Georgia and Moldova, which participated for the first time in 2009 or 2010. Comparable results for 2000 and 2009 are available for eight countries: Albania (which did not participate in 2003 or 2006), Bulgaria and Romania (which did not participate in 2003), the Russian Federation, the Czech Republic, Hungary, Latvia and Poland. Serbia, Turkey and Slovakia joined PISA in 2003, while Azerbaijan, Croatia, Kyrgyzstan, Montenegro, Estonia, Lithuania and Slovenia first participated in 2006.

Trends in reading since 2000

The average reading performance in the 26 OECD countries with comparable data remained similar between 2000 and 2009, despite most countries having increased their investment in education in recent years: between 2000 and 2007 expenditure per primary and secondary student increased on average in the OECD by around 25 per cent in real terms82. For the five EU8 countries in the OECD with data (Estonia, Poland, Hungary, Czech Republic and Slovakia), the expenditure increase was 64 per cent.

Nevertheless, among the 38 PISA countries with comparable data, 13 did show an improvement in reading performance. Among the study focus countries, there was an improvement in Albania, Hungary, Latvia and Poland, although there was a decline in the Czech Republic, which now scores below the OECD average. Among all PISA countries with comparable data, only three registered a decline in performance. There was no significant change observed in Bulgaria, Romania and the Russian Federation (see the blue bars in Figure 15a).

79 OECD averages include different countries depending on availability of data for the year compared. For comparing reading performance in 2000 and 2009, the OECD benchmark includes 26 countries (out of 34 current members), but 23 countries for comparisons over all four PISA assessments (including 2003 and 2006). For comparing mathematics between 2003 and 2009, 28 countries are included in the OECD average. For science, between 2006 and 2009, 33 countries are in the average (OECD 2010 vol.V, p.136).

80 To be able to compare performance over time there are some common assessment items used in each survey: given their limited number, the risk in measurement errors is higher and the confidence band for comparisons over time defining the statistical significance of changes is wider than for single-year data (OECD 2010 vol.V, p.26).

81 OECD 2010 vol.V, p.28.

82 OECD 2010 vol.V, p.38. Data is not available for non-OECD countries.

Trends in Performance over Time

69

CHAPTER 4

While the average change in reading performance among the eight study focus countries with data in both years was 10 points, it was just one point (a non-significant change) among the 26 OECD countries. Albania’s performance increased most among the study focus countries with data, although still remaining among the lowest-scoring countries in 2009 (only Chile and Peru had a larger increase among all PISA participants). Poland, which had performed below the OECD average in 2000, scored above the OECD average (and above the Czech Republic) in 2009, while Latvia surpassed the Russian Federation.

Part of the change can be attributed to changes in sampling methods and socio-demographic profile of students: a decline in performance may not necessarily be associated with a decline in the quality of educational services, but rather with a more challenging socio-economic context83. In fact, by adjusting for sampling differences in age and gender distribution, the decline in the performance of the Czech Republic becomes non-significant. Furthermore, by adjusting for socio-economic background, immigrant status and language spoken at home, predicting performance changes that would have occurred if the composition of the student population in 2000 had been similar to the one in 2009, the performance improvement of Hungary also becomes non-significant. On the other hand, with such adjustments the increase in performance for Albania and Latvia appears even larger (see the red dots in Figure 15a), given that in 2009 the student population had a more disadvantaged background than in 2000.

figure 15 a) observed changes in mean reading performance between 2000 and 2009, and changes adjusted for socio-demographic differences. b) Changes between 2000 and 2009 of percentage of boys and girls below level 2.

-20 -20%

-15%

-10%

-5%

0%

5%

10%

20

30

40

50

60

-10

10

0

Alb

aniaS

core

po

int

chan

ge

in r

ead

ing

per

form

ance

Ch

ang

e in

per

cen

tag

e b

elo

w L

evel

2

Latv

ia

Po

lan

d

Hu

ng

ary

Bu

lgar

ia

Ru

ssia

nFe

der

atio

n

Ro

man

ia

Cze

chR

epu

blic

Alb

ania

Latv

ia

Po

lan

d

Hu

ng

ary

Bu

lgar

ia

Ru

ssia

nFe

der

atio

n

Ro

man

ia

Cze

chR

epu

blic

� 2009-2000 observed change� 2009-2000 change adjusted for demographic differences

� Boys� Girls

36

26

21

14

-1 -2 -3

-13

48

32

23

1113

-12-10 -11

-12

-14

-8 -8

-4

2 1

7 7

-17

-11

-7

-1 -1

-8

3

Bars and indicators have a lighter shade of colour where changes are not significant. Countries are ranked by observed change. Source: OECD 2010 vol.V Tables V.2.1 and V.2.7 (a), Tables V.2.5 and V.2.6 (b).

83 OECD 2010 vol.V, p.28.

a) b)

70



Looking at how absolute disadvantage has changed, we see that the percentage performing below the baseline was reduced by 14 points in Albania: from 70 per cent of students in 2000 to 56 per cent in 2009. Latvia reduced the proportion below the baseline by 13 per cent, Poland by eight per cent and Hungary by five per cent: the percentage of low performers dropped where average performance improved. On the other hand, in the Czech Republic the percentage of students below the baseline increased by six per cent84.

Latvia and Poland raised the performance of their lowest-achieving students (at the 10th and 25th percentiles) while maintaining the performance level among the highest-achieving students (at the 90th and 75th percentiles). In other words, in both countries variation in performance decreased and learning outcomes became more equitable. Variation also decreased in Hungary, although there was no significant increase in performance at the 10th percentile. Albania showed improvements among students at all proficiency levels. In Romania, the performance of highest-achieving students declined, while that of the lowest-achieving remained the same. In Bulgaria, the Czech Republic and the Russian Federation the performance of the highest and lowest-achieving students remained unchanged85.

The gender gap in reading in favour of girls increased from 2000 by seven points on average in the OECD, reaching 39 points in 2009, and it did not fall in any PISA country. Among the eight study focus countries with data in both years, the gender gap increased on average by 10 per cent. But a statistically significant increase occurred only in Romania, where the gender gap widened by a substantial amount (29 points), rising from 14 to 43 points86. On average in the OECD, while the percentage of boys below the baseline remained constant (at 24 per cent), among girls it diminished by two percentage points (to 12 per cent). In the eight study focus countries, it diminished by two per cent among boys and by six per cent among girls. In the Czech Republic, the increase in students below the baseline is significant only for boys (by seven per cent). In Romania, where total disadvantage had remained stable, we see an increase for boys and a decrease for girls (see Figure 15b).

Trends in reading since 2000, 2003 and 2006

In order to identify trend indications apparent for the countries that joined PISA after the first round of assessment in 2000, it is necessary to annualise changes. By doing so makes it possible to compare changes in the performance of countries that did not participate in all PISA assessments. The annualised changes are calculated by dividing the performance difference by the number of years between two assessments, using data from 2009 and from the first available assessment for each country. As can be seen from the blue bars in Figure 16, Kyrgyzstan and Montenegro show large improvements (since 2006), as do Serbia and Turkey (since 2003)87. Slovenia, on the other hand, shows a significant decline, while no significant change is apparent in Azerbaijan, Estonia, Croatia or Lithuania (over three years), nor in Slovakia (over six years).

However, changes between different assessments for individual countries need to be compared with caution. For instance, if Bulgaria, Romania and the Russian Federation had not participated in PISA 2000 (from which we saw no change), the following different set of results would have emerged: there would have been a significant increase in the Russian Federation since 2003, and in Bulgaria and Romania since 2006; while a decline would have been apparent in the Czech Republic since

84 OECD 2010 vol.V, Table V.2.2.



87 Please note these are annualised changes, and as such appear quite small. For example, since the last survey three years earlier Kyrgyzstan improved by 29 points, and Serbia improved by 30 points since 2003.


71

CHAPTER 4

2000, and Latvia and Poland (despite an actual increase since 2000) would have shown no significant change since 200388.

The OECD combines reading performance in the four assessments across time into a single trend indicator using linear regression, in order to increase the precision of the results with respect to the use of only two assessments, taking into account that survey results can vary due to sampling and measurement errors (and through hypothesizing linearity of changes). Linear trends can include all countries which participated in at least two assessments, and all can be compared with each other by calculating an annual increase. Compared with annualised observed changes between two assessments, linear trends show slightly smaller improvements in Latvia, and render non-significant Hungary’s observed improvement and Slovenia’s observed decline in performance (see the green triangles in Figure 16).

Adjusting for changes in mean socio-demographic circumstances of countries participating since 2003 or 2006, predicting performance changes that would have occurred if the composition of the student population in earlier assessments had been similar to the one in 2009, we would have seen a slightly smaller improvement in Kyrgyzstan, Serbia and Montenegro (see the red dots in Figure 16), a larger improvement in Turkey, while in Slovakia we would have seen a significant increase in performance (rather than no significant change as was observed).

figure 16 annual observed changes in mean reading performance between 2000 and 2009, 2003 and 2009 or 2006 and 2009 (largest gap available); changes adjusted for socio-demographic differences; and linear trends across all available pIsa assessments89.

-6

6

8

10

12

-4

4

-2

2

0

Kyr

gyz

stan

Mo

nte

neg

ro

Ser

bia

Alb

ania

Turk

ey

Latv

ia

Aze

rbai

jan

Po

lan

d

Hu

ng

ary

Slo

vaki

a

Est

on

ia

Bu

lgar

ia

Ru

ssia

nFe

der

atio

n

Ro

man

ia

Cro

atia

Lith

uan

ia

Cze

ch R

epu

blic

Slo

ven

ia

Sco

re p

oin

t ch

ang

ein

rea

din

g p

erfo

rman

ce

Annualised observed change

Annualised change adjusted for demographic differences

Linear trend9.8

5.2 5.0 4.5 3.92.9 2.9 2.4

1.6 1.40.1 -0.2 -0.3 -0.5

-0.5 -0.5-1.5

-3.8

Bars and indicators have a lighter shade where changes are not significant. Countries are ranked by annualised observed change. Source: OECD 2010 vol.V Table V.2.8.


89 Linear trends are estimated using linear regression applied to data from all PISA cycles (two, three or four assessments depending on country participation). Annualised changes are calculated by dividing the performance difference by the number of years between two assessments (using 2009 and the first available assessment for each country). The results reflect the average score change associated with one calendar year.

72



Trends in mathematics and science

Trend data for mathematics is available only since 2003, when the first full assessment took place, while in science such data is available only since 2006. Between 2003 and 2009, the largest variance in mean mathematics performance among the study focus countries was recorded by Turkey and the Czech Republic, with mean mathematics performance increasing by 22 points in the former, and declining by 24 points in the latter. The decrease meant that the Czech Republic passed from above to below the OECD average (and in the process posted the largest decline among the 39 PISA countries with data in both years). There was no significant change in Hungary, Latvia, Poland, the Russian Federation, Serbia or Slovakia. In Turkey, there was a reduction of 10 per cent in the number below the baseline, while in the Czech Republic there was an increase of six per cent90.

A mixed picture also emerges in the performances in 2006 and 2009 among the countries which did not participate in 2003. Both Kyrgyzstan and Romania increased their mean performances, by 21 points and 12 points respectively, while Lithuania’s performance declined by 10 points91. Figure 17 shows the annualised changes, to allow comparisons to be made between all countries (the blue bar represents mathematics, and the red bar, science).

figure 17 annual observed changes in mean mathematics and science performance between 2003 and 2009 or 2006 and 2009.

-6

6

8

10

12

-4

4

-2

2

0

Turk

ey

Po

lan

d

Ro

man

ia

Kyr

gyz

stan

Ser

bia

Bu

lgar

ia

Latv

ia

Lith

uan

ia

Slo

vaki

a

Ru

ssia

nFe

der

atio

n

Hu

ng

ary

Est

on

ia

Cro

atia

Slo

ven

ia

Aze

rbai

jan

Mo

nte

neg

ro

Cze

chR

epu

blic

An

nu

alis

ed s

core

po

int

chan

ge

Mathematics Science

3.7

0.8

4.1

6.9

0.9

4.9

-0.2

-3.3

-0.3 -0.1

0

-0.8

-2.4 -2.4

-1.0

1.1

-3.9

10

3.4 3.32.5 2.4

1.71.4 1.2

0.6

-0.4-1.2

-2.3-3.1

-3.5-4.1

Bars and indicators have a lighter shade where changes are not significant. Countries are ranked by annualised change in science. Science annualised change is over three years for all countries. Mathematics annualised change is calculated over six years for Czech Republic, Hungary, Latvia, Poland, Russian Federation, Serbia, Slovakia and Turkey and three years for the rest. Source: OECD 2010 vol.V Figure V.1.2.

In science, Turkey’s mean performance has increased considerably since 2006, by 30 points, and Poland’s performance has increased by 10 points. Declines have been evident in the Czech Republic,

90 OECD 2010 vol.V, Tables V.3.1 and V.3.2.



73

CHAPTER 4

(12 points), Montenegro (11 points) and Slovenia (seven points)92. In terms of the percentage of students scoring below Level 2, there has been a substantial decrease in Turkey, the largest among all PISA countries, of 17 per cent: the percentage below the baseline in science passed from 47 to 30 per cent. There have been smaller decreases in Kyrgyzstan, Poland, Serbia and Lithuania (three-four per cent), while in Montenegro there has been a small increase (three per cent).93

Trends in variance in performance

Variance in reading performance, an indicator of the extent of within-country disparity, decreased by three per cent in the 26 OECD countries with data for both 2000 and 200994, and by nine per cent in the eight study focus countries with data. The largest decrease was in Latvia, where there was a 39 per cent fall, making it one of the countries with the least within-country disparity, followed by Romania (with a decrease of 22 per cent) and Poland (with a decrease of 20 per cent) (represented by the blue bars in Figure 18). The results in Latvia and Poland were achieved through an increase in the performance of low-achievers (and the stable results for high-achievers), which also brought about an increase in mean performance, while in Romania there was a decline in the performance of high-achievers, with no change for low-achievers. The only country with an increase in total variance, from already high levels, was Bulgaria (with a 24 per cent increase). It is now the country with the largest within-country disparities among the study focus countries95.

figure 18 Change in variance in reading performance between 2000 and 2009 (as a percentage of 2000 variance), and difference in percentage of between-school variance in reading performance (2009-2000).

-50%

-40%

-30%

30%

-20%

20%

-10%

10%

0%

Ch

ang

e b

etw

een

20

00

and

20

09

Percentage change in total varianceDifference in percentage of between-school variance

Latv

ia

Ro

man

ia

Po

lan

d

Cze

chR

epu

blic

Hu

ng

ary

Ru

ssia

nFe

der

atio

n

Alb

ania

Bu

lgar

ia

-39

-22 -20

-8 -8-5

1

24

-9

5

-44

-4

4

-13-9

-3

Countries are ranked by percentage change in total variance. Source: OECD 2010 vol.V Figure V.4.1.



94 Analysis of trends in equity are done only using data on reading between 2000 and 2009, i.e. in two full assessments.

95 In Bulgaria, there was a decrease in the performance of lowest achievers (at the 10th and 25th percentiles) and an increase in the performance of highest achievers (at the 75th and 90th percentiles), which although on their own were not major changes did contribute to the significant increase in total variance. See OECD 2010, Table V.2.3.

74



The share of between-school variance in reading performance (over the total between-school plus within-school variance) remained similar between 2000 and 2009 for most of the 37 PISA countries with data. The only differences of more than 10 per cent among PISA countries were in two study focus countries: the Russian Federation, with a decrease in the share of between-school variance of 13 per cent − starting from an already low share; and Poland, the only country with a substantial change – a decrease of 44 per cent (represented by red bars in Figure 18). Poland had been one of the study focus countries with the largest share of between-school variance in 2000 (62 per cent), but by 2009 it had the smallest share of all (19 per cent). In other words, the average performance results of different schools in the country are now quite similar (while earlier there had been high- and low-achieving schools). This change has been attributed to the institutional reform as a result of which 15-year-old students are in a comprehensive system where they are no longer separated into different types of schools (see section on The case of Poland).

Trends in the relationship between socio-economic background and performance

The socio-economic background of students, measured through the PISA mean index of economic, social and cultural status, remained largely similar across the PISA assessments96. Among all PISA countries with data, Albania (with a decrease of 0.30) and Bulgaria (with a decrease of 0.23) posted the largest decline in mean socio-economic background between 2000 and 2009, while the largest increase was in Romania (+0.32) and the Russian Federation (+0.31)97.

figure 19 Change in the student-level score-point difference in reading performance associated with one unit increase in the pIsa index of economic, social and cultural status (esCs) between 2000 and 2009 (overall association); in student-level difference associated with single unit increase in student-level esCs (within-school association); and in school-level difference associated with single unit increase in school mean esCs (between-school association).

-80

-60

-40

40

-20

20

0

Ch

ang

e in

sco

re p

oin

t d

iffe

ren

ce

Change in overall associationChange in within-school association

Change in between-school association

Ro

man

ia

Ru

ssia

nFe

der

atio

n

Po

lan

d

Hu

ng

ary

Bu

lgar

ia

Alb

ania

Latv

ia

Cze

chR

epu

blic

111

-2 -4 -4-10 -11 -11

106

31

0 0

-4 -3-7-7

-40

-51

-10

-18 -15

-68

6

Countries are ranked by change in overall association of ESCS (PISA index of economic, social and cultural status) and performance. Statistically significant changes have data labels. Source: OECD 2010 vol.V Figure V.4.3.

96 OECD 2010 vol.V, p.78.



75

CHAPTER 4

On average in the OECD, the relationship between socio-economic background and reading performance remained unchanged between 2000 and 2009. Among the study focus countries, the association diminished in the Czech Republic, Albania and Latvia, while it increased in Romania (which had the largest increase among all PISA countries with data), as can be seen from the blue bars in Figure 19. The between-school association of socio-economic intake and performance at the school level decreased on average in the OECD by seven points, and by 25 points among the eight study focus countries with data, diminishing significantly in the Russian Federation and, above all, in Latvia and Poland, which showed the largest decrease among all PISA countries with data (as represented by the green bars in Figure 19). But in Poland, and to a lesser extent in the Russian Federation, the within-school association of background and performance increased, so that the overall association of background and performance remained unchanged. This suggests that Poland’s school reform had the effect of distributing students from different backgrounds more evenly across schools, but made no difference to the overall association of background and performance98.

In terms of the relationship between immigrant background and performance, it is noteworthy that there was no significant change between 2000 and 2009 in respect of the difference in points between students with and those without an immigrant background in the four study focus countries with enough immigrant students to be able to calculate a difference (i.e. the Czech Republic, Hungary, Latvia and the Russian Federation). The percentage of students with an immigrant background remained broadly similar in all eight countries, except in Latvia and the Russian Federation. Latvia had the largest fall in the number of students with an immigrant background among all PISA countries, with the total decreasing by 18 percentage points (to four per cent of 15-year-old students), while the Russian Federation was one of the countries with the largest increase in the number of immigrant students, rising from eight per cent to 12 per cent of the total student population.

Of the six study focus countries with data99, the only country with significant change between 2000 and 2009 in the difference in reading scores between students whose language spoken at home was the same and those whose language spoken at home was different from the language of assessment was Romania. It passed from having no significant difference between the two groups to having a relatively large difference in favour of students speaking the language of assessment at home (a change of 81 points). The percentage of students who spoke a different language at home to the one in which the assessment was administered increased by six per cent in Bulgaria.

The case of poland100

Poland is often taken as an example of how performance can improve and inequalities can be reduced with educational reform. In 2000, Poland’s performance level in reading was well below the OECD average: over 23 per cent of students had not reached the baseline level of achievement and there were large disparities in performance between students attending various types of secondary schools (typically, the results of students in vocational schools were much lower than those in general academic schools). The Polish Ministry of Education had, prior to these results, in 1998, presented the outline of a reform agenda which planned structural changes to create a new type of school, the lower-secondary gymnasium, with the same general education for all students. The previous structure comprised eight years of primary school followed by secondary general and vocational schools, while as part of the new structure primary education was reduced to six years, followed by the new lower-secondary gymnasium, and then by general or vocational upper-secondary schools.

98 OECD 2010 vol.V, p.80.

99 Romania, Albania, Bulgaria, Czech Republic, Latvia, Russian Federation.

100 From OECD 2010 vol.V, pp.33-5. See also UNICEF 2009, pp.93-5.

76



General education for all was thus extended by one year, and each of the three stages of education now ends with standardised national examinations (which were introduced in 2002 – before this date 15-year-old students did not have experience of standardized testing). A core curriculum was adopted, leaving extensive autonomy to schools to create their own curriculum within a general national framework. After 2000, school funds were transferred to local governments using a per-pupil formula. The reform also introduced a new system of teacher development and evaluation.

The largest improvement in performance was observed in PISA 2003, right after the reform. Students assessed had started primary school in the former system but attended the new lower-secondary gymnasia, all with the same educational curriculum. Results from PISA 2009 suggest that lowest-performing students benefited most from the reform, as we saw earlier. Students from former vocational schools were given a chance to acquire more general skills 101. Overall performance improved, while at the same time within-country disparities were reduced.

The case of Turkey102

Turkey joined PISA in 2003, with a performance far below the OECD average. However, between 2003 and 2009 Turkey showed improvement in mathematics and reading performance, and improvement in science performance between 2006 and 2009, with the largest reduction among PISA countries in the percentage of students below the baseline level.

Turkey implemented several programmes to improve the quality of education. One of these was the Basic Education Programme, first implemented in 1997/98, which made compulsory an eight-year primary education system. As part of the programme, class sizes were reduced and computer laboratories and a foreign language were made available to all. The first students emerged from this system in 2003. Attendance increased from 85 per cent to nearly 100 per cent. At the pre-primary level, attendance increased from 10 to 25 per cent.

New curricula were implemented in 2006/07, starting from the sixth grade, and a new science curriculum was applied in 2008/09, in the ninth grade. It is noteworthy that PISA 2009 students had been taught for one year under this new curriculum. The new curricula and teaching methods emphasise ‘student-centred learning’, giving students a more active role than before, when memorising information had been the dominant approach.

Other programmes included the Girls to Schools Now campaign, which was started in 2003 to ensure that all girls attended primary school. Since 2003, textbooks have been provided free-of-charge. In 2004, a project aimed at teaching students democratic skills was started. Schools were obliged to develop targets and strategic plans to reach them, private investments were used, and new arrangements to train teachers were implemented.

The case of Kyrgyzstan103

We have seen that Kyrgyzstan has shown improvements since 2006 in reading and mathematics, but it is still by far the lowest-performing country among all PISA countries. Significant resources and efforts are invested in education, but with by far the lowest GDP per capita of all participating countries in PISA and a very large school-age population the challenges faced are manifold. It should

101 But once vocational school options are available again, at age 16, performance declines for students in vocational tracks. See OECD 2011c.

102 From OECD 2010 vol.V, pp.70-1.

103 From OECD 2010b.


77

CHAPTER 4

be noted that the country has had 20 years of transition from the Soviet model and suffered an economic collapse after independence.

The Government of Kyrgyzstan attaches great importance to education, as is demonstrated by an increase in educational expenditure from 3.9 per cent of GDP in 2001 to 6.5 per cent in 2007104, and has invited the OECD and the World Bank to conduct a review of Kyrgyz education policies in order to help understand its unsatisfactory performance in PISA 2006. A lengthy report was published following this review which pointed to a number of policy areas in need of attention in order to improve the quality of Kyrgyz education. The areas needing attention included: the curriculum, textbooks, teaching materials, modes of pupil assessment, teacher education, governance and funding arrangements, data, higher education and vocational education and training.

The review points to a gap between policy aspirations (as identified, for example, in the recent document on Education Development Strategy 2011-2020 )105 and the achievement of major reform, which is a long-term process. Despite attempts at reform, the curriculum is still largely as it was in Soviet times. There is an overload of subjects and curriculum content (and a lot of time devoted to languages), with little time for practical, creative or integrated learning, or for students to develop and express their ideas. Teaching methods are teacher-centred and directive. As the OECD identified: “The conceptual framework is narrowly subject-based and academically oriented, and offers limited choice to the students. The textbooks and learning materials are inadequate to support the curriculum, are in short supply and, where available, are often out-of-date106.” Much of teachers’ attention is given to the small percentage of gifted high-ability students coached for success at the Olympiad competitions, which are perceived as the ultimate indicator of quality at schools, with insufficient attention to the needs of the average pupils and the low-achievers.

Assessment tends to focus on the memorization and reproduction of facts rather than on how well pupils apply, analyse and understand the material. Exam questions of the national exit examinations at grades nine and 11 are generally known and published in advance. Students are hence never faced with an exam question they have not seen before, or with a task that requires them to apply their knowledge in a different way, which is one reason underlying the low results in PISA. The review team suggests ending this practice so that teachers and learners stop rehearsing answers to a narrow range of questions, and concentrate instead on competences required in applying what they know to new questions107.

Another problem is the inadequate recruitment of high-quality candidates into teaching and the retention of good teachers. This is due, in part, to the fact that teacher salaries amount to only about 60 per cent of the average wage, and to the low social status of teaching as a career. It tends to be students who cannot gain access to higher-entry courses who end up teaching. They typically have little motivation and are quick to take other opportunities if they present themselves. Many drop out of teacher training and even do not take up the profession after obtaining teacher education qualifications108. The OECD and World Bank review team suggests gradually raising the pupil-teacher ratio from the current 15 (which is generous by international standards) to 20 pupils per teacher. This will facilitate pay rises for teachers.

104 OECD 2010b, p.50.

105 OECD 2010b, p.356.

106 OECD 2010b, p.25.

107 OECD 2010b, p.187-8.

108 OECD 2010b, p.297-8.

78



The depth of the problem is further highlighted by a UNICEF CEE/CIS case study on teacher shortage, which showed that in Kyrgyzstan only 17 per cent of students who enrol in teacher education end up teaching. The situation is compounded by the fact that admission requirements for entrance to teacher-education studies are very low, and by the Government allocating a disproportionately large number of scholarships to university students in pedagogical specializations: 36 per cent of all teacher-education students receive government scholarships. It is assumed that a great number of students enrol in teacher-education studies simply because funding is available, after they are turned down by other degree programmes. Indeed, during the five-year teacher-education diploma programme 37 per cent of those enrolled either abandoned their studies or switched to another programme. Of those who completed their studies with a teacher-education specialization, only 27 per cent chose to become teachers. There is further attrition in the first few years of employment: three years into working as a teacher, only 11 per cent of those who graduate from teacher education remain in their posts109.

Teacher shortage is a grave problem in Kyrgyzstan. To an extent it is masked at the school level by practices that help to fill vacancies but are detrimental to the quality of education, such as the hiring of retired teachers, correspondence teachers (students who work part-time as teachers) and substitute teachers (non-specialists, teaching subjects for which they were not trained or without pedagogical training), or else by other practices, such as by teachers teaching more than the statutory teaching load110.

Key findings on trends

Among the eight study focus countries with data since 2000 in reading, there has been an improvement in Albania, Hungary, Latvia and Poland, a decline in the Czech Republic, and no significant change in Bulgaria, Romania and the Russian Federation. Part of the change can be attributed to changes in sampling methods and socio-demographic profile of students. When such changes are adjusted for, the performance change in the Czech Republic and Hungary becomes non-significant. The percentage performing below the baseline was reduced in this nine-year period by 14 per cent in Albania, 13 per cent in Latvia and eight per cent in Poland. Both Latvia and Poland raised the performance of their lowest-achieving students, while maintaining the performance level among the highest-achieving students. Consequently, variation in reading performance decreased too. The largest decrease in variance was seen in Latvia, with a 39 per cent decrease (making Latvia one of the countries with the least within-country disparity), followed by Romania (-22 per cent) and Poland (-20 per cent). In Romania, there was a decline in the performance of high-achievers, with no change for low-achievers.

The share of between-school variance in reading performance remained fairly similar in most countries, except Poland. Although it had been one of the study focus countries with the largest share of between-school variance in 2000 (62 per cent), by 2009 it had the smallest share of all (19 per cent). The change is attributed to the institutional reform as a result of which 15-year-old students are no longer separated into different types of schools. The relationship between socio-economic background and reading performance diminished in the Czech Republic, Albania and Latvia, while it increased in Romania.

For countries that did not participate in PISA 2000, improvements in reading performance are noted in Serbia and Turkey since 2003, and in Kyrgyzstan and Montenegro since 2006. Slovenia shows a significant decline since 2006. Since 2003 in mathematics, performance has increased in Turkey

109 UNICEF 2011, p.41.

110 UNICEF 2011, p.90.


79

CHAPTER 4

and declined in the Czech Republic. In countries which had not participated in 2003, Kyrgyzstan and Romania have increased their mean mathematics performance since 2006, while in Lithuania performance has declined. In science since 2006, Turkey’s and Poland’s mean performance has increased, while it has declined in the Czech Republic, Montenegro and Slovenia.

© UNICEF

ChapTer 5

sChool- and sysTem-level faCTors assoCIaTed WITh performanCe

Country wealth and student performance .................................................................................82

Pre-primary school attendance ................................................................................................................84

Differentiation: school selection and ability grouping..............................................86

School autonomy ........................................................................................................................................................89

School choice .................................................................................................................................................................91

Accountability ..................................................................................................................................................................93

School resources .........................................................................................................................................................93

Learning time ....................................................................................................................................................................94

School climate .................................................................................................................................................................95

Key findings on school- and system-level factors ...........................................................97

82



ChapTer 5: sChool- and sysTem-level faCTors assoCIaTed WITh performanCe

High-performing countries which have a weak relationship between socio-economic background and performance (the OECD’s main indicator of equity) share a number of common features. The OECD describes them as: a high value is placed on education; there are clear and ambitious standards; there are steps to ensure the quality of teachers and principals; and high-quality learning is provided consistently to every student111. One of the study focus countries, Estonia, is included in the small group of eight with school systems that are deemed to be successful, among the 74 PISA countries, for a performance above the OECD average and a lower-than-average impact of socio-economic background (Poland also has an above-average performance, but is only average when it comes to the impact of socio-economic background)112.

By using the information collected via questionnaires completed by school principals and students, the OECD examines which factors are associated with performance and equity, with and without controlling for socio-economic background113, both at the country/system level and also within each country, separately at the school level. It should be noted that this does not mean that such factors are necessarily causing the level of performance observed. Sometimes, such inferences may be misleading since relationships are not simple and direct, and the association between two variables may be determined by a third, unknown, variable, while on other occasions various characteristics may be interrelated. Again, we should remember that the performance of a child is not only linked to the present school attended, but also to schools attended previously as well as to learning out of school. As the OECD puts it: “The fact that such characteristics are more likely to be found among successful school systems does not mean that they are necessary or sufficient for success” 114.

Using a multi-level model, which tries to define what proportion of total variation in scores can be attributed to the individual student level, to the school level, or to the country level, the OECD observes that 25 per cent of performance variation in reading is attributed to performance differences among PISA countries115. The analysis of the characteristics of successful education systems tries to explain this performance variation. It emerges that within countries, on average 60 per cent of the variation is attributed to differences in the performance of individual students within schools, while 40 per cent of the performance variation is attributed to differences between schools. More than half of such between-school variation (23 per cent) lies in the schools’ socio-economic intake: it is the remaining 17 per cent of within-country differences, on average, which we would like to be able to explain by looking at differences in the characteristics of schools according to performance.

Country wealth and student performance

Before going on to analyse how the organization of schooling may have a part in shaping performance outcomes, we will first examine how country wealth is related to performance. Clearly, a sufficient

111 OECD 2010 vol.I, p.4 and vol.IV, p.27.

112 These countries are those with dark lozenge in the top right quadrant of Figure 31 in Annex 2.

113 Some differences in performance are attributed to both school characteristics (or policies) and socio-economic background, so taking into account socio-economic background may underestimate the strength of a relationship between such characteristics and performance. But not taking it into account, while it may paint a more realistic picture of the schools parents choose for their children, may overstate the relationship. Differences in performance attributed jointly to school policies and socio-economic background give an indication of the extent to which school policies are inequitably distributed according to socio-economic background. See OECD 2010 vol.IV, p.34, p.39.

114 OECD 2010 vol.IV, p.29.

115 OECD 2010 vol.IV, p.26. Among the smaller group of OECD countries it is 11 per cent. In mathematics, the figure is 31 per cent among all PISA countries, and 14 per cent among the OECD countries; in science, the figure is 28 per cent among all PISA countries, and 13 per cent among the OECD. Within countries, the average proportions of between- and within-school variance are similar for all three performance scales and for the OECD and all PISA countries.

School- and SyStem-level FactorS aSSociated with PerFormance

83

CHAPTER 5

level of expenditure is needed to cover elements such as basic infrastructure investment, as well as expenses related to teachers and instructional materials. In most cases, the level of educational expenditure is dependent on national prosperity and income. However, as many have noted, spending alone is by no means sufficient to produce high levels of student performance.

It is important to identify, as a basic context variable, how important national income is in the educational performance of countries. The picture which emerges is not consistent. As Figure 20 indicates, among the study focus countries GDP per capita is in fact closely associated with student performance, whereas there is a much weaker relationship between national income and educational performance among the richer OECD countries116. Furthermore, while the EU8 countries have higher levels of GDP per capita than CEE/CIS countries, except for Croatia, and tend to perform better in the PISA assessment, some study focus countries have a significantly higher performance in PISA than the OECD mean despite having GDP per capitas well below the OECD mean117. Even Shanghai-China, the top-performing education system, has a GDP per capita well below the OECD mean.

figure 20 gdp per capita in equivalent us dollars (converted using purchasing power parities) vs. average mean performance in pIsa reading, mathematics and science.

Estonia Poland Slovenia

Hungary Czech Republic Slovakia Latvia Lithuania Croatia

Russian Federation

Turkey Serbia

Bulgaria Romania

Montenegro Kazakhstan

Azerbaijan

Kyrgyzstan

Albania Georgia

Moldova

Luxembourg

Shanghai-China

Qatar

300

350

400

450

500

550

600

0 20'000 40'000 60'000 80'000

Ave

rag

e m

ean

per

form

ance

GDP per capita (in US$ converted using PPPs)

EU8 CEE/CIS countriesother OECD other PISA

Source OECD 2010, Table I.A and Table IV.3.21c, except GDP per capita for Albania, Canada, Chile, Georgia, Moldova and Qatar from IMF, World Economic Outlook Database, September 2011 and GDP per capita for Shanghai-China from China NBS. GDP per capita refers to 2007 except Shanghai-China 2008.

116 Among the 21 study focus countries, the correlation between performance and GDP per capita is 0.90, while among the 27 OECD countries (excluding the study focus countries) it is only 0.28.

117 Most notably, the top-performer of the region, Estonia, which spends less per student than all other OECD countries except Turkey, Mexico, Chile, Slovakia and Poland. (Cumulative expenditure by educational institutions per student aged 6 to 15 in equivalent US$ converted using PPPs, OECD 2010 vol.IV, Table IV.3.21b).

84



It seems that money matters below a certain threshold, and indeed the lowest-performers in PISA are all countries with lower levels of GDP per capita. It could be the case that some of the study focus countries have insufficient economic resources to provide adequate educational opportunities. For instance Kyrgyzstan, which has by far the lowest GDP per capita of all participating countries, has also by far the lowest mean PISA performance. It is a different situation in the OECD, however, where educational expenditure has been increasing without the desired effects on student performance levels and equity. The results suggest that once there are sufficient resources for the basics, the way they are used start to grow in importance. Indeed, the OECD points out that in OECD countries GDP per capita accounts for only six per cent of the differences in average student performance. “The other 94 per cent reflect the potential for public policy to make a difference118.” Elsewhere, the effect of GDP per capita is more significant, and rises to 30 per cent among all PISA countries119.

pre-primary school attendance

Aside from educational spending, what is it that makes a difference in student outcomes? Does earlier entrance in the school system, before the beginning of compulsory education, help reduce educational inequalities and increase performance in later years? If disadvantage becomes established at an early age, given the importance of home background, attempts to mitigate such disadvantage need to begin even before a child starts compulsory school120.

On average across the OECD and among the EU8 countries, 72 per cent of students report having attended more than one year of pre-primary education. On the other hand, in the CEE/CIS countries only 44 per cent of students have attended more than one year of pre-primary school, while 36 per cent have attended none at all (compared with just eight per cent in the OECD). Even among CEE/CIS countries, there are still large differences: in Romania 88 per cent of students have attended more than one year of pre-primary education, while in Turkey 72 per cent have not attended any pre-primary education at all, which is the highest percentage among all PISA countries121. Furthermore, in Azerbaijan, Kyrgyzstan and Kazakhstan the majority of students did not attend any pre-primary education (see Figure 21).

In all countries, students who have attended pre-primary education for more than one year have a higher average socio-economic background than those who have not (see the last three columns of Table 6 in Annex 2). But even keeping socio-economic background constant, students who have attended pre-primary education for more than one year tend to show higher performance in reading than those who have not, in all study focus countries except Estonia, Latvia and Croatia. The advantage for those having attended pre-primary school is generally less among the study focus countries, with an average of 18 points difference in performance after accounting for the socio-economic background of students, than among OECD countries, where the average difference is 33 points122. The largest difference among the study focus countries is observed in Kyrgyzstan, where the difference corresponds to more than one year of schooling (47 points), even keeping socio-economic background constant (see Figure 22). In most PISA 2009 countries (56 out of the 65 countries), socio-economically disadvantaged and advantaged students benefit equally from pre-

118 OECD 2010 vol.I, p.36. But it should be kept in mind that the trend line is based on a small number of countries.

119 OECD 2010 vol.IV, p.26.

120 See UNICEF 2002, pp. 22-24.

121 The percentage would probably be even higher in the total population of 15-year-olds, considering the low school attendance rate at age 15 in Turkey.

122 Before accounting for background, the difference is on average 37 points in the study focus countries and 54 points in the OECD.


85

CHAPTER 5

primary attendance, with similar differences in performance. Among the study focus countries, the exceptions are Lithuania, where there is greater benefit for disadvantaged students (who score on average six points more than students with a higher socio-economic background123 who attended pre-primary school), and Romania, where there is greater benefit for advantaged students (who score on average 19 points more).

figure 21 percentage of 15-year-old students in pIsa having attended pre-primary education for more than one year, one year of less, or not at all (based on student self-reports).

0 20 40 60 80 100

Pre-primary attendance (% of students)

None

1 year or less

More than 1 year

HungaryRomania

Czech RepublicSlovakiaEstonia

BulgariaSlovenia

Russian FederationLatvia

AlbaniaCroatia

LithuaniaPoland

MontenegroSerbia

KazakhstanKyrgyzstanAzerbaijan

Turkey 72

4

69

63

58

13

36

2

38

27

25

21

21

17

11

10

5

5

4

1

20

15

15

18

50

22

48

12

21

23

13

11

14

15

10

12

9

8

8

17

19

27

37

42

50

51

52

53

66

67

68

74

80

83

87

88

94

Countries are ranked by percentage of students having attended more than 1 year of pre-primary education. Source: OECD 2010, vol.II Table II.5.5.

The OECD suggests that such variance in the relationship between performance and pre-primary attendance could be due to differences in the quality of pre-primary education124. It notes that among OECD countries, the relationship between pre-primary attendance and performance tends to be greater in systems with longer duration, smaller pupil-teacher ratios and higher public expenditure per child in pre-primary education.

123 Students with a one unit higher PISA index of economic, social and cultural status than the OECD average. See OECD 2010 vol.II, p.98 and Table II.5.7.

124 OECD 2010, vol.II p.98.

86



figure 22 performance difference in reading between students who report having attended pre-primary school for more than one year and those without pre-primary school attendance, before and after accounting for the socio-economic background of students.

Kyrgyzstan

Slovakia

Hungary

Poland

Kazakhstan

Romania

Czech Republic

Bulgaria

Albania

Azerbaijan

Russian Federation

Lithuania

Turkey

Serbia

Montenegro

Slovenia

Croatia

Latvia

Estonia

-10 0 10 30 40

Difference in performance

50 60 807020

52

145

297

269

3010

24

1131

1529

1628

1832

1942

2532

2545

2546

3051

3645

3860

4772

1158

-2

� After accounting for socio-economic background� Before accounting for

socio-economic background

Countries are ranked by performance difference after accounting for socio-economic background. Source: OECD 2010, vol.II Figure II.5.9, Table II.5.5. Lighter-shaded bars indicate that difference are not significant.

differentiation: school selection and ability grouping

What elements in the organization of primary and secondary education affect performance? Some systems try to create homogeneous learning environments with the idea that this will help the learning process. The question is: does it work? The OECD distinguishes ‘vertical differentiation’ (including differentiation in grade and education levels, age of entry into school system and grade repetition) and ‘horizontal differentiation’ (including programmes of study, age of selection, school admission and transferral policies, and ability grouping).

Concerning vertical differentiation, the large majority of students generally attend the modal (most common) grade level. However, in some countries 15-year-olds are dispersed across different grades and education levels. In Azerbaijan, Albania and the Czech Republic, for instance, about half of all students attend a different grade from the modal grade and are spread out across lower and upper


87

CHAPTER 5

secondary education125. In the study focus countries, most students start school at the age of seven, except in the Czech Republic and Slovakia where most start at six. On average in the study focus countries, 70 per cent of students report starting school at the age of seven or above (29 per cent at age six), while in the OECD it is more common to start school at age six, with 52 per cent starting at this age (29 per cent at seven or above and 19 per cent – the English-speaking countries – at five or below). Most students are at most one year younger or older than the statutory age of entry. Age of entry into primary school is not associated with performance − rather, as we have seen, it is participation in pre-primary education which makes a difference.

School systems also vary in the degree of horizontal differentiation. Some systems are comprehensive, with all 15-year-olds following the same programme, as in Estonia, Latvia and Poland, while others are stratified, with a selection of students streamed into different programmes, typically academic and/or vocational. On average (both in the OECD and in the study focus countries) streaming takes place at the age of 14, and it takes place as early as age 11 in the Czech Republic, Hungary, Slovakia and Turkey (see the sixth column in Table 4). Also, some school systems are composed of highly selective schools, and are more likely to have homogeneous student populations. On average in the CEE/CIS countries, 59 per cent of students are admitted to schools on the basis of academic records or recommendations of feeder schools, while the figure is 39 per cent in the EU8 and 36 per cent in the OECD. At least three-quarters of all students are selected in this way in Bulgaria, Croatia, Hungary and Serbia, while the number is less than a quarter in Lithuania, Poland and the Russian Federation (see the seventh column in Table 4). Early differentiation and selection will contribute to between-school inequalities and academic exclusion.

Furthermore, in some systems schools can differentiate students within schools through ability groupings, or can transfer students out of schools because of low achievement, special learning needs or behavioural problems (see the final two columns in Table 4). Transferring students is common in Bulgaria, Kyrgyzstan, Romania and Turkey, where more than a third of students were in schools where the principal reported that students would ‘very likely’ be transferred to another school because of such reasons. Ability groupings in all subjects are particularly common in Kazakhstan and the Russian Federation, where more than a third of students were in schools where the principals reported ability grouping for all subjects. Where there are other forms of differentiation, ability groupings may be less relevant. Typically, it is common to have ability groupings for at least some subjects in most countries (68 per cent in the OECD, 61 per cent in the EU8 and 65 per cent in CEE/CIS countries). In CEE/CIS countries, 25 per cent reported ability groupings in all subjects, while only eight per cent reported them in the EU8 and 18 per cent in the OECD.

Among the eight school systems with above-average performance and below-average socio-economic inequality, including Estonia, none show high levels of student differentiation. In countries where more students repeat grades, or where it is more common to transfer weak or disruptive students out of a school, overall results tend to be worse and socio-economic differences in performance wider. The OECD suggests this may be because in such systems there may be less of an incentive to commit to helping lower-achieving students improve126. In school systems with low rates of student transfers, school principals tend to report that schools have more responsibility for curriculum and assessment127. Differences in performance by socio-economic background also tend to be wider in systems where 15-year-olds are divided into more tracks based on their ability, and more so the younger the age of first tracking. On the other hand, such tracking policies seem to make no difference to overall results.

125 OECD 2010 vol.IV, p.64 and Table IV.3.1.

126 OECD 2010 vol.IV, p.13, p.37.

127 OECD 2010 vol.IV, p.38.

88



Table 4 vertical and horizontal differentiation policies of school systems.

Vertical differentiation Horizontal differentiationA

vera

ge

age

of

entr

y in

to

pri

mar

y sc

ho

ol (

stu

den

t se

lf-

rep

ort

)

Per

cen

tag

e o

f st

ud

ents

rep

ort

-in

g t

o h

ave

rep

eate

d a

gra

de

Per

cen

tag

e o

f st

ud

ents

at

mo

dal

gra

de

Nu

mb

er o

f sc

ho

ol t

ypes

or

dis

-ti

nct

ed

uca

tio

nal

pro

gra

mm

es

avai

lab

le t

o 1

5-y

ear-

old

s

Firs

t ag

e o

f se

lect

ion

in t

he

edu

cati

on

sys

tem

Per

cen

tag

e o

f se

lect

ive

sch

oo

ls1

Per

cen

tag

e o

f sc

ho

ols

th

at

tran

sfer

stu

den

ts f

or

vari

ou

s re

aso

ns2

Per

cen

tag

e o

f sc

ho

ols

wit

h

abili

ty g

rou

pin

g f

or

all s

ub

-je

cts3

Estonia 6.9 6 72 1 15 30 10 12

Poland 7.0 5 94 1 16 17 8 4

Slovenia 6.7 1 91 3 14 29 22 5

Hungary 6.8 11 67 3 11 87 14 3

Czech Republic 6.4 4 49 5 11 53 22 7

Slovakia 6.3 4 57 5 11 63 30 7

Latvia 6.8 11 79 1 16 25 15 10

Lithuania 6.8 4 81 2 14-15 13 7 15

Croatia 6.7 3 78 5 14-15 94 18 21

Russian Federation

6.7 3 60 3 15 23 14 38

Turkey 6.9 13 67 3 11 43 35 28

Serbia 6.9 2 96 m m 84 30 17

Bulgaria 6.9 6 89 3 13 76 34 19

Romania 6.9 4 89 3 14 57 40 22

Montenegro 6.7 2 83 3 15 59 7 29

Kazakhstan 6.6 2 73 m m 36 13 36

Azerbaijan 6.6 2 49 2 15 64 15 24

Albania 6.6 5 51 m m 53 17 27

Kyrgyzstan 6.8 4 71 4 14-15 61 38 19

OECD average 6.1 13 74 2.5 14 36 18 13

Study focus countries average

6.7 5 73 2.9 14 51 21 18

EU8 average 6.7 6 74 2.6 14 39 16 8

CEE/CIS average

6.8 4 73 3.3 14 59 24 25

Countries are ranked by average performance in reading, mathematics and science. Source OECD 2010 vol.IV Tables IV.3.1, IV.3.2a, IV.3.2b, IV.3.3a, IV.3.4. 1Percentage of students in schools where principals reported ‘students’ records of academic performance’ or ‘recommendations of feeder schools’ are always considered for student admittance. 2Percentage of students in schools where the principal reported that a student in national modal grade for 15-year-olds in the school would be ‘very likely’ transferred to another school because of one of the following reasons: ‘low academic achievement’, ‘behavioural problems’ or ‘special learning needs’. 3Percentage of students in schools where the principal reported ability grouping for all subjects within school. Number of school types and first age of selection is based on a PISA system-level data collection. ‘m’ indicates the information is missing.


89

CHAPTER 5

school autonomy

Another important feature of school organisation is the degree of autonomy that schools have in taking decisions on various matters. PISA asked school principals whether teachers, the principal, the school’s governing board, the regional or local education authorities or the national education authority had 1) considerable responsibility for allocation of resources to schools (by appointing and dismissing teachers, establishing teachers’ starting salaries and raises, and formulating and allocating school budgets) and 2) responsibility for the curriculum and instructional assessment within the school (by establishing student assessment policies, choosing textbooks, and determining courses offered and their content)128.

Considering the first group of tasks, principals report having the greatest autonomy in Bulgaria, the Czech Republic and Hungary, with around 80 per cent or more of principals reporting that their schools had sole responsibility in matters relating to resource allocation (based on the average across the seven resource tasks – see Figure 23a). The least autonomy is reported in Romania and Turkey, where around 80 per cent reported that responsibility was solely of the regional or national education authority. In terms of individual tasks, in most countries few schools have a major influence on teachers’ salaries. At least 95 per cent of students in Albania, Croatia, Romania and Turkey are in schools whose principals reported that only regional and/or national education authorities had considerable responsibility over teachers’ starting salaries and salary increases. On average in the study focus countries, it is the responsibility of only regional and/or national education authorities to establish teachers’ starting salaries in 69 per cent of schools, and to determine salary increases in 66 per cent of schools.

However, school principals and/or teachers have generally more responsibility in selecting and dismissing teachers and deciding on budget allocations. In Bulgaria, Estonia, Hungary and Latvia, for instance, more than 80 per cent of students are in schools where only principals and/or schools have considerable responsibility for such tasks129. On average in the study focus countries, 75 per cent of schools have sole responsibility for selecting and dismissing teachers, and 58 per cent for deciding on budget allocations within schools. Formulating the school budget is the task with the most mixed results and joint responsibility is common, averaging 27 per cent in the study focus countries. Overall, across all resources allocation tasks, principals in the EU8 countries report on average more autonomy (with 59 per cent reporting sole responsibility of schools) than they do in the OECD (where the figure is 45 per cent). The least autonomy is found in the CEE/CIS countries (37 per cent).

Considering curriculum and assessment tasks, principals report having the greatest autonomy in making decisions about such matters in the Czech Republic, where an average of 88 per cent of students are in schools whose principals report that only school principals and/or teachers had a considerable responsibility for establishing student assessment policies, choosing textbooks, and determining courses offered and their content (see Figure 23b). The countries where schools report the least autonomy for curriculum and assessment tasks are Serbia, Montenegro and Turkey, where an average of around 20 per cent of schools report sole responsibility for such matters. On average in the study focus countries, schools have more autonomy over the choice of textbooks (with 63 per cent of schools reporting sole responsibility) and in assessment policies (57 per cent), while

128 Results need to be interpreted with caution, as school principals may have interpreted the questions differently given that decision-making arrangements vary widely across countries. For example, the question on responsibility for formulating the school budget might have been related by some school principals to the regular school budget, while others without involvement in the regular budget may have related it to supplementary budgets, such as contributions from parents. Some variations within countries can be partly explained by regional differences, particularly in federal education systems. See OECD 2010 vol.IV, pp.68-72.

129 OECD 2010 vol.IV, pp.68-9 and Figure IV.3.3a.

90



the picture is more mixed when it comes to courses offered and their content (with 36/39 per cent of schools reporting sole responsibility, 29/25 per cent reporting joint responsibility, and 35/36 per cent reporting that sole responsibility is that of only regional or national authorities). Overall, there is more joint responsibility over curriculum and assessment than there is over resources. The highest levels of autonomy are again found in the EU8 countries (where on average 66 per cent of schools report sole responsibility, 27 per cent report joint responsibility and seven per cent report that responsibility is that of only higher authorities), where the averages are higher than those of the OECD (where the equivalent figures are 60 per cent only schools, 24 per cent joint and 16 per cent only higher authorities). CEE/CIS countries have the lowest levels of autonomy (36 per cent only schools, 23 per cent joint and 41 per cent only higher authorities).

In PISA, countries that grant greater autonomy to schools to design curricula, decide course offerings, determine course content and textbooks used, as well as establish student assessment policies, tend to show better performance in reading than those that do not, while there is no correlation at the country level between performance and autonomy in resource allocation130. Within countries, the picture is mixed. In some there is a positive relationship between curricular autonomy and student performance, including Lithuania, while schools with greater autonomy show poorer performance in others, including Bulgaria.

figure 23 how much autonomy individual schools have over:

a) resource allocation

10 30 50 70 90

Bulgaria

0 20 40 60 80 100

Czech RepublicHungarySlovakia

LatviaEstonia

SloveniaRussian Federation

LithuaniaCroatia

MontenegroSerbia

Percent of students in schools by locus of responsibility for resources

� Only principals and/or teachers� Both� Only regional and/or national education authority

916192934373939434547515156606478

KazakhstanPoland

KyrgyzstanAzerbaijan

AlbaniaTurkey

Romania

8478

8078736158385039424141322823251516

45

116

810

824

1121

1513

131721

2115

226

1117

130 OECD 2010 vol. IV, p.46.


91

CHAPTER 5

b) curriculum and assessment

10 30 50 70 90

Czech Republic

0 20 40 60 80 100

PolandLithuaniaHungary

EstoniaKyrgyzstan

SlovakiaRussian Federation

AlbaniaRomaniaSlovenia

Latvia

Percent of students in schools by locus of responsibilityfor curricula/assessment

� Only principals and/or teachers� Both� Only regional and/or national education authority

1818202526334244455153555859697172

AzerbaijanBulgaria

CroatiaKazakhstan

TurkeyMontenegro

Serbia

8879

43496056384551178183617

332

2116

17

3933

211936

227

3947

3111

29399

291821

1114

The percentage of students in schools whose principals reported that only ‘principals and/or teachers’, only ‘regional and/or national education authority’, or both ‘principals and/or teachers’ and ‘regional and/or national education authority’ have a considerable responsibility for a) allocation of resources to schools (average of the following tasks: appointing and dismissing teachers, establishing teachers’ starting salaries and raises, formulating and allocating school budgets) - Source: OECD 2010 Figure IV3.3a; and b) the curriculum and instructional assessment within the school (average of the following tasks: establishing student assessment policies, choosing textbooks, determining courses offered and their content) - Source : OECD 2010 Figure IV.3.3b.

One explanation presented by the OECD for these different relationships is that better-performing schools may be granted more autonomy in certain countries, while in others more autonomous schools may cater to lower-performing students who did not obtain access to more prestigious public programmes131. Schools with greater autonomy in resource allocation tend to do better only when they account for results by posting achievement data publicly – otherwise they tend to do worse132.

school choice

In some countries, parents and students can choose the school attended even within the public sector. Where this is the case, schools compete for students and sometimes for funding. In others, the school is assigned on the basis of geographical area of residence. On average in the study focus countries, 74 per cent of students attend schools that compete with at least one other school for enrolment (ranging between 37 per cent in Montenegro and 93 per cent in Slovakia), which is comparable with the OECD average133. At the system level, there is no performance difference associated with more competition between schools for students134. Within individual countries, schools competing for students tend to have higher performance, but generally this is accounted

131 OECD 2010 vol. IV, p.41.

132 OECD 2010 vol. IV, p.14.

133 OECD 2010 vol.IV, p.72 and Table IV.3.8a.

134 OECD 2010 vol. IV, pp. 14, 46.

92



for by socio-economic background (only in Turkey does the relationship with performance remain positive even after accounting for background)135.

A questionnaire on school choice was completed by parents in 14 countries (including Croatia, Hungary and Lithuania). It emerged that, on average, socio-economically disadvantaged parents consider that they have more limited choices of schools for their children because of financial constraints. This may indicate that more freedom in the choice of school can increase socio-economic segregation of schools and favour more socio-economically advantaged students 136.

One element of school choice is the existence of private schools. While on average 15 per cent of schools are privately managed137 in the OECD countries, the figure is only four per cent in the EU8 and two per cent in CEE/CIS countries. Hungary (12 per cent), Albania (11 per cent) and Slovakia (nine per cent) are the only study focus countries where more than three per cent of schools are private. And while there are 10 study focus countries where less than two per cent of students attend privately managed schools, there are only four other countries among all 65 PISA 2009 countries with such a small percentage.

By looking at Figure 24, of the nine study focus countries with enough private schools for a comparison to be made (i.e. countries with more than two per cent privately managed schools), we can see that while private schools generally have higher performances, this can be explained by the fact that private schools tend to have intakes of higher socio-economic background: performance differences in reading are smaller when taking into account student socio-economic background − and even smaller, or even not significant, when taking into account the average socio-economic background of the school intake as well (except in Hungary, where the difference actually favours public schools).

figure 24 difference in performance on the reading scale between public and private schools.

Diff

eren

ce in

per

form

ance

Public – Private

Accounting for studentsocio-economic background

-140

-120

-100

-80

-11 -15-24

-36-50 -57

-65-80

-121

-60

private schools do better

public schools do better

-40

-20

0

20

40

Kyr

gyz

stan

Slo

ven

ia

Alb

ania

Pola

nd

Kaz

akh

stan

Cze

ch R

epu

blic

Slo

vaki

a

Hu

nga

ry

Est

on

ia

Accounting for student andschool socio-economicbackground

Countries are ranked by difference in performance between public and private schools (without taking account of socio-economic background). Socio-economic background is measured using the PISA index of economic, social and cultural status. Lighter-shaded bars and symbols indicate that the difference is not significant. Source OECD 2010 Table IV.3.9.

135 OECD 2010 vol. IV, pp. 42-4. In Kazakhstan, after accounting for socio-economic background, private schools tend to have lower performance.

136 See OECD 2010 vol.IV, p.42.

137 In PISA, schools are considered to be private if they are independently managed and operated, while they can be publicly or privately funded.


93

CHAPTER 5

accountability

To evaluate student learning, standards-based external examinations at the secondary level are universal among all study focus countries, except in Romania and Serbia (where they exist in some parts of the system but not all – covering 78 per cent and 26 per cent, respectively – due to variations between different educational programmes or regions). Furthermore, data is missing for Albania and Kazakhstan (see the second column in Table 7 in Annex 2)138. In the OECD countries, there are nine countries in which standards-based external examinations are completely absent, four countries where they exist in parts of the system, and 20 countries where they are universal. Apart from such external exams at the system level, schools can use standardised tests to be able to compare the results of their students. On average in the study focus countries, 81 per cent of students attend schools where standardised tests are used at least once a year, according to principals (the figure is 76 per cent in the OECD). The lowest percentages can be found in Slovenia (24 per cent), Serbia (46 per cent) and Montenegro (59 per cent), but otherwise they are commonly used throughout the region (see the third column in Table 7 in Annex 2)139.

The study focus countries tend to use achievement data for accountability purposes more than the average OECD country does. Hence, the wide majority of schools in the study focus countries, except Slovenia (38 per cent) and Croatia (45 per cent), use achievement data to evaluate teacher performance: on average 81 per cent of students in the study focus countries attend schools that use achievement data in this way, compared with 45 per cent in the OECD (see the 19th column in Table 7 in Annex 2). Furthermore, the study focus countries demonstrate relatively frequent use of achievement data for benchmarking and information purposes, while most of them − except Hungary, Montenegro, Slovenia and Turkey − also show frequent use of achievement data for decisions which affect schools, as can be seen from Table 7 in Annex 2.

Countries that use standards-based external examinations tend to have better performance results than those that do not. However, there is no clear relationship between performance and the various uses of assessment for accountability purposes140. In three countries, Kyrgyzstan, Romania and Turkey, schools whose principals report that student achievement data are posted publicly perform better than schools which do not post data in such a way. This is the case even after accounting for socio-economic background141.

school resources

The OECD rightly points out that what matters for student achievement is not necessarily the availability of resources so much as the quality of such resources and the quality of their use. However, while an adequate physical infrastructure and up-to-date textbooks do not guarantee good learning outcomes, the absence of such resources is likely to have an adverse effect on learning (as we have seen with GDP per capita)142. Unfortunately, there is no direct measure of teacher quality, which may be the one thing to have the most impact on student learning. Implying that paying teachers more will improve their quality, OECD observes that raising teacher quality is more effective at improving student outcomes than is creating smaller classes, noting that systems prioritising

138 Standards-based exams assess student performance relative to an external standard (which defines what they are expected to know or able to do), not relative to other students in the class or school. See OECD 2010 vol.IV, pp.46, 75 and Table IV.3.11.

139 OECD 2010 vol. IV, p.75, Table IV.3.10.

140 OECD 2010 vol. IV, p.78.

141 OECD 2010 vol.I VIV, p.46. But see Goldstein and Leckie, 2008, on why such posting of results may be misleading.

142 OECD 2010 vol.IV, p.83, with reference to Gamoran, Secada and Marrett, 2000.

94



teachers’ pay over smaller classes tend to achieve higher levels of performance. In fact, the only type of resource that PISA shows to be correlated with student performance is the level of teachers’ salaries relative to national income143. Within countries, socio-economically advantaged schools tend to have more educational resources and tend to perform better, suggesting the need for a more equitable distribution of resources across schools. However, after accounting for socio-economic background, resources do not seem to make a difference.

School principals were asked whether they thought instruction in their school was hindered by a lack of qualified teachers and staff in key areas. Among all PISA countries, principals in Turkey had the highest perception of problems with instruction due to teacher shortages, measuring two standard deviations above the OECD average and at least one standard deviation, approximately, above that of any other country (with a wide variation between principals’ perceptions within Turkey). The other countries which had a higher perception of problems with instruction due to teacher shortages than the OECD average were Kyrgyzstan (0.92), Kazakhstan (0.47), and to a lesser extent the Russian Federation (0.13) (see column two in Table 8 in Annex 2). At the other extreme, the fewest problems reported with teacher shortages were found in Poland (-0.78), Romania (-0.74) and Slovenia (-0.72), with little variation among principals’ replies. It must be kept in mind, however, that principals may have different expectations and benchmarks to determine whether there is a lack of qualified teachers144. Across all PISA countries, there is no correlation between such indexes of teacher shortage and, for example, expenditure per student or GDP per capita.

Another index relates to the hindrance of instruction due to lack of material resources (see column three in Table 8 in Annex 2)145. Among the study focus countries, Slovenia is the least likely to report that instruction in schools is hindered by a lack of adequate material resources (-0.48). Of course, it has the region’s highest GDP per capita and spends by far the most on education (more, indeed, than the OECD average). At the opposite end are Kyrgyzstan (1.72), which is also the poorest country in the region (with GDP per capita which is less than a third that of Albania, the second-poorest country), as well as Turkey (1.35).

As can be seen from columns four to seven in Table 8 in Annex 2, most schools in the study focus countries have a library (94 per cent on average, based on principals’ reports). On average, some 37 per cent of principals report a shortage of library materials, while 79 per cent of students report borrowing books for schoolwork and 66 per cent for pleasure (all higher percentages compared with the OECD mean).

learning time

Self-reports on their typical use of time per week show that, on average, students in the study focus countries devote more time to science (3 hours 35 minutes) than to mathematics or language of instruction (about three hours each). The comparable figures for OECD countries are 3 hours 35 minutes each for mathematics and language of instruction, and 3 hours 20 minutes for science. Students in Croatia and Montenegro spend the least amount of time on these three subjects (less than eight hours a week), while those in Azerbaijan and the Russian Federation spend the most time (more than 11 and a half hours) (see columns eight to 11 in Table 8 in Annex 2).

143 But such data is available for few study focus countries. Other types of school systems’ resources observed were expenditure per student, class size, time in regular lessons, after-school lessons for enrichment and for remedial purposes, and extra-curricular activities.

144 OECD 2010 vol.IV, p.82 and Table IV.3.20.

145 School principals were asked to report on the extent to which the school’s capacity to provide instruction was hindered by the shortage or inadequacy of resources for instruction such as science laboratory equipment, textbooks, computers, internet connection, computer software, library and audio-visual materials. OECD 2010 vol.IV, pp.83-4.


95

CHAPTER 5

Both enrichment and remedial lessons after school are on average more common in the study focus countries (respectively 44 per cent and 36 per cent of students attend such lessons for at least one of the three subjects) than they are in the OECD (respectively 28 per cent and 26 per cent). Around two-thirds or more of students in Azerbaijan and Kazakhstan attend enrichment lessons, and around two-thirds or more of students in Kazakhstan and the Russian Federation attend remedial lessons (see columns 12 and 13 in Table 8 in Annex 2). Kazakhstan, apart from being the country in PISA with the highest percentage attending after-school lessons, also has a wide availability of extra-curricular activities offered by schools, which may include academic activities as well as sports, arts and culture146. It is in fact the country in PISA with the highest extra-curricular activities index (measuring 1.3 – well over one standard deviation above the OECD average). The average index of the study focus countries (0.54) is also higher than that of the OECD (0.17) (see the final column of Table 8)147.

school climate

Students in the study focus countries, as in the OECD, are generally satisfied with the quality of teacher-student relations. However, here too caution is required when comparing results, since students and principals may not necessarily apply the same criteria in assessing the school climate in different countries, or indeed within countries148. In fact, if we look at the three indexes of school climate based on students’ reports, most of the variability in responses lies between students within the same school, rather than between schools (although it must be kept in mind that not all students in a school are in the same class). For example, on average both in the study focus countries and OECD only six per cent of variance in the index of teacher-student relations, 15 per cent of variance in the index of disciplinary climate and seven per cent of variance in the index of teachers’ stimulation of students’ reading engagement lies between schools. This indicates that students in the same schools have different perceptions of school climate (for comparison, on average in the study focus countries the between-school variance in reading performance was 41 per cent and in socio-economic background it was 29 per cent)149.

Students are generally satisfied with the quality of teacher-student relations, with 86 per cent in the study focus countries agreeing with the statement: ‘I get along well with most of my teachers’ − although responses might have been biased because students were compiling a questionnaire on their teachers while at school (social desirability may affect the responses of students in some countries more than in others). The only exceptions to such generally positive teacher-student relations were Slovenia and Poland, where only 30 and 35 per cent of students, respectively, agreed with the statement: ‘most of my teachers are interested in my well-being’. Among all PISA participants, students in Albania and Azerbaijan reported the most positive relationships with their teachers.

146 School principals were asked to report whether the following extra-curricular activities are offered by the school: a band, orchestra or choir; school plays or musicals; a school yearbook, newspaper or magazine; volunteering or service activities; a book club; debating club/activities; a school club or competition for foreign language, mathematics or science; an academic club; arts club/activities; sports team/activities; lectures or seminars; collaboration with local libraries; collaboration with local newspapers.

147 OECD 2010 vol.IV, pp.80-1 and Tables IV.3.16a, IV.3.17a and IV.3.19.

148 OECD 2010 vol.IV, p.88.

149 OECD 2010 vol.IV, Tables and Figures IV.4.1-3. The index of teacher-student relations is based on students’ agreement with the statements: ‘I get along well with most of my teachers’; ‘Most of my teachers are interested in my well-being‘; ‘Most of my teachers really listen to what I have to say‘; ‘If I need extra help, I will receive it from my teachers‘; and ‘Most of my teachers treat me fairly. The index of disciplinary climate is based on students reporting on the frequency of the following: ‘Students don’t listen to what the teacher says’; ‘There is noise and disorder’; ‘The teacher has to wait a long time for the students to quieten down’; ‘Students cannot work well’; and ‘Students don’t start working for a long time after the lesson begins’. The index of teachers’ stimulation of students’ reading engagement is based on students reporting on the frequency of the following: ‘The teacher asks students to explain the meaning of a text’; ‘The teacher asks questions that challenge students to get a better understanding of a text’; ‘The teacher gives students enough time to think about their answers’; ‘The teacher recommends a book or author to read’; ‘The teacher encourages students to express their opinion about a text’; ‘The teacher helps students relate the stories they read to their lives’; and ‘The teacher shows students how the information in texts builds on what they already know’.

96



In general, the views of student-teacher relations, disciplinary climate and especially of teachers’ stimulation of students’ reading engagement are more positive in the CEE/CIS countries than they are in the EU8 or OECD countries (see Table 9 in Annex 2). Students’ views of how conducive classrooms are to learning (disciplinary climate) are most positive in Kazakhstan, Albania and Azerbaijan. Overall in the study focus countries, the majority of students (78 per cent on average) report that ‘never, hardly ever or only in some lessons’ is there noise and disorder. In terms of teachers’ stimulation of students’ reading engagement, students in Kazakhstan and the Russian Federation report teachers offering the most stimulation, with 78-79 per cent of students reporting that teachers recommend a book or author to read in most or all lessons150. It is noticeable that in terms of teachers stimulating reading engagement the top seven countries (across all 65 PISA 2009 countries) are all CEE/CIS countries.

In terms of how student and teacher behaviour affects students’ learning151 (see Table 9 in Annex 2), principals report that learning is disrupted by such behaviour the most in Turkey, where the values of these two indexes − which are highly correlated − are far above those of any other PISA country. In Turkey, principals report poor student-teacher relations hindering learning in 75 per cent of cases, while in the OECD the average is only 12 per cent (as it is in the study focus countries, except Turkey). It is noteworthy that students, on their side, reported positive relationships with their teachers, and that in general there are no correlations between students’ and principals’ views of student-teacher relations. At the opposite end, school principals in Albania report that the school climate is generally positive, with very little student or teacher behaviour hindering learning. The only item that is identified by a majority of principals across the study focus countries as hindering student learning is student absenteeism (with a rate of 61 per cent).

Finally, parental expectations, which are characterised by the pressure on schools to achieve high academic standards, are highest in Albania, the Czech Republic and Slovenia (with figures between 25 and 27 per cent), and lowest in Montenegro and Serbia (with figures between three and six per cent).

Schools with better disciplinary climates, more positive behaviour among teachers and better teacher-student relations tend to achieve higher scores in reading. In part this is connected to socio-economic background, since more disciplined classes are generally attended by students with advantaged backgrounds (who tend to perform better and may reinforce a climate conducive to learning), but even controlling for socio-economic background, part of the performance advantage remains152. A better disciplinary climate is positively related to performance in 12 of the study focus countries153, even independently of school socio-economic intake. On the other hand, school principals’ perceptions of parents’ pressure to raise academic standards and achievement is related to higher student performance mostly before socio-economic background is taken into account. That said, even after taking socio-economic background into account it is still positively related in Bulgaria, Kazakhstan, Latvia and Lithuania.

150 Kazakhstan is also the PISA country with the highest percentage of students who read for enjoyment (OECD 2010, Figure III.2.3).

151 The index of student-related factors affecting school climate is based on principals’ reports on whether student learning was hindered by ‘Student absenteeism’; ‘Disruption of classes by students’; ‘Students skipping classes’; ‘Students lacking respect for teachers’; ‘Student use of alcohol or illegal drugs’; or ‘Students intimidating or bullying other students’. The index of teacher-related factors affecting school climate is based on principals’ reports on whether student learning was hindered by ‘Teachers’ low expectations of students’; ‘Poor student-teacher relations’; ‘Teachers not meeting individual students’ needs’; ‘Teacher absenteeism’; ‘Staff resisting change’; ‘Teachers being too strict with students’; or ‘Students not being encouraged to achieve their full potential’.

152 OECD 2010 vol.IV, p.14, pp.54, 58.

153 Azerbaijan, Croatia, Czech Republic, Kazakhstan, Kyrgyzstan, Latvia, Lithuania, Poland, Romania, Russian Federation, Slovakia, Slovenia.


97

CHAPTER 5

In sum, according to OECD calculations, by combining all such school factors identified in this chapter as being associated with performance, as well as factors involving socio-economic background, it is possible to account for 88 per cent of the observed differences in performance across schools in all PISA countries.

Although this seems to be an extremely large percentage, it does not mean that such factors are the cause of performance differences. Neither does it mean that if any of the variables were changed there would follow a comparable change in performance across all countries154. Furthermore, as we have seen, a large number of differences are accounted for by socio-economic factors, if not solely then at least jointly with other factors. But what is clear is that despite such mitigating variables, there remain factors related to school characteristics which can be affected by public policy that seem to make a difference, both in terms of absolute levels of performance as well as in disparities across the system. This has been demonstrated by the positive effects the change to a more comprehensive education system brought about in Poland.

In fact, systems with high performance and low disparity tend to be comprehensive, while in selective systems, the earlier the selection into tracks the greater the impact of socio-economic background on performance. Furthermore, results tend to be worse and socio-economic differences in performance wider in countries where more students repeat grades or are transferred out of schools. There are other factors associated with higher performance, including the existence of external examinations, higher teacher salaries, and a large degree of school autonomy over what is taught and how students are assessed. Within schools, better teacher-student relations and a better disciplinary climate are associated with high performance. Most of this effect is linked to the better school climate enjoyed by students from advantaged socio-economic backgrounds, and the challenge is to weaken this association between background and climate, in part by changing the social mix of students in some schools, as proposed by the OECD155.

Key findings on school- and system-level factors

One of the study focus countries, Estonia, is included in the small group of eight school systems deemed to be successful, among the 65 PISA 2009 countries, for a performance above the OECD average and a lower-than-average impact of socio-economic background. As mentioned earlier, the common features of such successful countries are described by the OECD as: a high value is placed on education; there are clear and ambitious standards; there are steps to ensure the quality of teachers and principals; and high-quality learning is provided consistently to every student.

Some systems are comprehensive, with all 15-year-olds following the same programme, as in Estonia, Latvia and Poland, while others are stratified, with a selection of students streamed into different programmes or schools. Among the eight school systems with above-average performance and below-average socio-economic inequalities in PISA, including Estonia, none show high levels of student differentiation. Early differentiation and selection contributes to between-school inequalities and academic exclusion. Also, in countries where more students repeat grades, or where it is more common to transfer weak or disruptive students out of a school, overall results tend to be worse and socio-economic differences in performance wider.

Apart from these characteristics, it is important to identify which variables are associated with student performance at the system level. As we have seen, among the study focus countries GDP per capita is closely associated with student performance. However, the relationship between national

154 OECD 2010 vol.IV, pp.103-7.

155 OECD 2010 vol.IV, pp.107.

98



income and educational performance is considerably weaker among the richer OECD countries. It seems that money matters below a certain threshold, and indeed the lowest-performers in PISA are all countries with lower levels of GDP per capita: some of the study focus countries may have fewer economic resources than needed to provide sufficient educational opportunities. On the other hand, although all study focus countries have GDP per capita which is well below the OECD mean, some of them have a significantly higher performance in PISA than the OECD mean. Once there is enough for the basics, the way it is used may start to count more.

One resource that PISA shows us is correlated with student performance is the level of teachers’ salaries relative to national income. The OECD observes that raising teacher quality is more effective in improving student outcomes than is creating smaller classes, noting that systems prioritising teachers’ pay over smaller classes tend to achieve higher levels of performance. Within countries, socio-economically advantaged schools tend to have more educational resources and tend to perform better, suggesting the need for a more equitable distribution of resources across schools. But after accounting for socio-economic background, resources do not seem to make a difference. It should be noted that principals in Turkey had the highest perception of problems with instruction due to teacher shortages among all PISA countries, followed by Kyrgyzstan and Kazakhstan among the study focus countries.

An important feature of school organisation is the degree of autonomy that schools have in taking decisions on various matters. In PISA, countries that grant greater autonomy to schools to design curricula, decide course offerings, establish student assessment policies, determine course content and textbooks used, tend to show better performance in reading than those that do not, while there is no correlation at the country level between performance and autonomy in resource allocation. Overall, across the resource allocation tasks, principals in the EU8 countries tend to report more autonomy (with 59 per cent suggesting resource allocation is the sole responsibility of schools) than in the OECD (where the figure is 45 per cent). The least autonomy is found in the CEE/CIS countries (37 per cent). Overall, there is more joint responsibility reported in curriculum and assessment than there is in resources. The highest levels of autonomy are again found in the EU8 countries (with on average 66 per cent reporting schools have sole responsibility, 27 per cent reporting joint responsibility and seven per cent reporting responsibility is that of only higher authorities), where figures are higher than the OECD average (60 per cent schools, 24 per cent joint and 16 per cent only higher authorities). CEE/CIS countries report the lowest levels of autonomy (36 per cent schools, 23 per cent joint and 41 per cent only higher authorities).

Schools with better disciplinary climates, more positive behaviours among teachers and better teacher-student relations tend to achieve higher scores in reading. Most of this effect is connected to socio-economic background, since more disciplined classes are generally attended by students with advantaged backgrounds (who tend to perform better and may reinforce a climate conducive to learning). But even controlling for socio-economic background, part of the performance advantage remains. The challenge is to weaken this association between background and climate, in part by changing the social mix of students in some schools.

Generally, a large number of differences are accounted for by socio-economic factors − if not solely then at least jointly with other factors. But there are still factors related to school characteristics which can be affected by public policy that seem to make a difference, both in terms of absolute levels of performance as well as in disparities across the system.


99

CHAPTER 5

If disadvantage becomes established at an early age, given the importance of home background, attempts to mitigate such disadvantage need to begin even before a child starts compulsory school. Even keeping socio-economic background constant, in all of the study focus countries except Estonia, Latvia and Croatia, students who have attended pre-primary education for more than a year tend to demonstrate higher performance in reading than those who have not. The average difference is 18 points, while the largest difference is observed in Kyrgyzstan, where it corresponds to more than one year of schooling (47 points).

Country and system factors associated with student performance ................................................................................................................................................................... 102

Policy challenges: ensuring quality education for all by tackling low performance and reducing large disparities .......................................................... 105

Mitigating the impact of socio-economic background on performance ......................................................................................................................................................... 106

Tailoring policy interventions to meet specific country challenges ................................................................................................................................................ 106

Achieving high quality with equity in educational outcomes: tradeoffs and realities ....................................................................................................................................... 109

Specific strategies to tackle low performance in reading literacy ....................................................................................................................................................................................110

Overall conclusion .................................................................................................................................................. 111

102



ChapTer 6: summary ConClusIons and polICy suggesTIons

CEE/CIS countries are continuing their transition to more nationally relevant and Western-oriented education systems. In the past many former socialist societies had fairly developed education systems, characterized by relatively high enrolment and literacy rates156. Nevertheless, differences in family background, political networks and geographical residence often translated into inequalities in access to quality education, especially at the upper-secondary and tertiary levels, and subsequent social mobility157. Following the political transformations of 1989, CEE/CIS countries made a concerted effort to integrate into the world economy by restructuring their political systems and labour markets. As part of this transition process, many educational reforms have been instituted involving such factors as compulsory schooling, educational goals, curricular policies, teacher-training programmes and the content of textbooks.

CEE/CIS countries are continuing to confront a number of pressing policy challenges in education. First and foremost, there is an acute need to improve the provision of quality education for all students. Among other things, this means establishing school systems that improve teacher quality and effectiveness and enhance student learning experiences and outcomes. Secondly, countries need to tackle low student performance and find ways to reduce disparities in the knowledge, skills and proficiencies which students obtain after completing a compulsory school cycle. A third challenge involves the loose linkages between educational reforms and policy intentions, on the one hand, and school realities and classroom dynamics, on the other: in many countries, the actual implementation of educational policies and intentions in local schools is uneven and partial. Finally, it is vital for CEE/CIS countries to create learning environments which move beyond the rote memorization of facts and teacher-dominated pedagogy, and focus instead on the application of knowledge and skills to new situations, with an emphasis on innovation, creativity, problem-solving and open dialogue. These policy challenges are widespread throughout the region, and are especially pronounced in the school systems serving poor and rural communities.

Drawing on evidence reported in the PISA assessments, this chapter addresses directly these policy challenges. It begins by summarising key findings about the relationships between student performance in the 2009 PISA assessment and selected country and system characteristics. It then discusses relevant policy interventions and programmes for CEE/CIS countries seeking to increase student performance and reduce learning disparities. A particular emphasis is placed on strategies to enhance reading literacy, which is the focus of the 2009 PISA assessment. It concludes by calling on CEE/CIS countries to: 1) implement reforms that meaningfully improve the quality and relevance of basic education; 2) conduct appropriate, relevant and transparent assessments, such as PISA, PIRLS and TIMMS, which provide a solid basis for monitoring and improving teaching effectiveness and learning outcomes; and 3) institute measures that reduce educational disparities and inequalities so that all children in the region can benefit from educational opportunities, realise their potential, and lead capable and dignified lives by fully participating in society and the economy.

Country and system factors associated with student performance

1. Country wealth and student performance

Evidence reported in previous chapters shows that GDP per capita is strongly associated with average student performance among the 21 study focus countries. Nevertheless, countries of similar

156 Silova 2009; 2011.

157 Bukodi and Robert 2007; Kraaykamp and Nieuwbeerta 2000; Mateju and Strakova 2005; Prokic-Breuer 2011.

Summary ConCluSionS and PoliCy SuggeStionS

103

CHAPTER 6

prosperity are found to foster rather different levels of student performance in PISA and, above a certain threshold, the relationship between country wealth and performance does not hold. These cross-country comparisons, though lacking policy specificity, underscore a key policy message: well designed and implemented social policies as well as sustained national efforts can allow lower income countries to raise literacy and numeracy proficiencies and produce superior learning outcomes.

2. Pre-primary school attendance

The linguistic, cognitive and social skills that children develop in early childhood, especially in pre-primary schools, are the foundation for subsequent learning in primary and secondary education. This link is especially pertinent for children living in poor households or under vulnerable or marginalized conditions. Participation in early childhood education can boost their performance in school and increase their opportunities later in life. The more time children spend in pre-primary programmes, the better their subsequent performance in school. These conclusions are based on evidence from the 2009 PISA assessment, which shows that in 58 of 65 participating countries 15-year-old students who had attended at least one year of pre-primary education achieved higher performance levels than students who had not, even after accounting for socio-economic background. In some countries, the net benefit was equivalent to one year of formal schooling. Overall, the 2009 PISA assessment underscores the long-term significance of expanding access to pre-primary education, especially to students from disadvantaged and marginalized backgrounds.

3. Institutional differentiation and its impact on performance

In recent decades, many European and Asian countries have made the transition from more differentiated school systems to more homogeneous ones, especially at the lower-secondary level, as enrolment rates increase and access to students from diverse social backgrounds is extended. Previous studies of PISA results have indicated that less differentiated education systems are associated with higher learning outcomes. In this report, we have seen that PISA distinguishes between two kinds of differentiation: vertical differentiation (for example, separating students into different grades and educational levels, policies defining the age of entry into schools and conditions for grade repetition) and horizontal differentiation (for example, differences in the programmes of study offered, the age at which children are selected into different programmes, the prevalence of ability grouping and streaming, and school admission and transferral policies). Horizontal differentiation and early student selection have been shown, both in PISA-based studies and in other research, to contribute to greater between-school inequalities in performance and more academic exclusion.

Among the eight school systems with above-average performance and below-average socio-economic inequalities, including Estonia among the study focus countries, none show high levels of horizontal differentiation. In countries where more students repeat grades, or where weak and disruptive students are more likely to be transferred out of a school, overall results tend to be lower and socio-economic differences in performance wider. The PISA reports suggest that this may be due to a weakening of incentives and policy commitments by all stakeholders to improve the performance of lower-achieving students.

4. School autonomy and accountability

The PISA assessment queried school directors and principals concerning the extent of autonomy they have over various kinds of school matters. Results from PISA 2009 indicate that countries that grant greater autonomy to schools to design curricula, decide course offerings, determine course content and textbooks used, and establish student assessment policies tend to achieve better performance

104



levels in reading than those that do not. However, there is no correlation at the country level between performance and autonomy in resource allocation.

Countries that use standards-based external examinations tend to have better performance results than those that do not, but there is no clear relationship between performance and the use of standardized tests. In three countries – Kyrgyzstan, Romania and Turkey – schools whose principals report that student achievement data are posted publicly perform better than schools that do not post these results, even after accounting for a school’s socio-economic composition.

5. School resources: teacher quality and learning time

The PISA assessment has no direct measure of teacher quality, which a large body of research considers to be a critical factor affecting student learning. A recent UNICEF (2011) report on teachers in the CEE/CIS region draws attention to several on-going issues affecting teacher quality: 1) the shortage of qualified teachers for particular subjects, grades and districts within CEE/CIS countries, especially in schools serving minority language communities; 2) a tendency for teacher shortages to be ‘covered up’ at the school level by the hiring of retired teachers, correspondence teachers and substitute teachers; and 3) conditions created by the stavka system in which the statutory teaching load is very low, thereby compelling teachers to take on additional teaching hours, work as private tutors, solicit fees from parents for various services and/or take up a job outside of school. These conditions place serious constraints on teacher effectiveness158.

The PISA reports do examine the policy issue of whether raising teacher salaries to improve teacher quality is more effective than reducing class size. The reports note that education systems prioritising teachers’ pay, but not those creating smaller classes, tend to achieve higher performance levels in PISA. In fact, one of the few types of school resources shown to be correlated with student performance is the level of teachers’ salaries relative to national income. The aforementioned UNICEF report notes that existing salary structures in many CEE/CIS countries, such as Croatia, are organized around low base salaries to which various supplements are allocated: for notebook checking, being a home-room teacher and offering extracurricular classes and the like. Croatia, for example, has difficulty attracting young people to the teaching profession because of poor salaries, and teachers usually take up additional jobs in order to earn some extra money on account of these low salaries159. This pattern creates considerable inequalities among teachers who are the same age, qualification and rank, especially between teachers who work in small or rural schools and their peers who teach in larger urban or semi-urban schools. There is an acute need to develop policies that mitigate such inequalities and improve teacher pay so as to improve teacher motivation, commitment and effectiveness.

In several CEE/CIS countries, teacher shortages are perceived to be a salient problem − to a much greater extent than in OECD countries. This is especially the case in Kyrgyzstan (with a standard deviation of 0.92), where monthly salaries had been well below the average monthly wage, in fact below the poverty line, until wages were increased by 30 per cent in 2004. Nevertheless, salary levels remain very low160. Teacher shortages are prevalent in Kyrgyzstan’s rural areas: a 2003 study by UNICEF found that, on average, half of the classes needed in core subjects were not conducted. Teacher shortages are also a problem for Kazakhstan (0.47), and to a lesser extent the Russian Federation (0.13). As previously noted, the deeper challenge is the shortage of qualified teachers for particular subjects, grades, schools and districts. Recruiting more students in university-based

158 UNICEF (2011), pp. 92-96.

159 UNESCO-International Bureau of Education, 2011b.

160 UNESCO-International Bureau of Education, 2011c.


105

CHAPTER 6

teacher education programmes is less of an issue than is retaining and finding employment for those who have completed their teaching qualification161. It is clear that improving the percentage of qualified teachers who enter and remain in the teaching profession remains an important policy issue in much of the region.

Finally, the PISA assessment provided limited support to the argument that the quantity of instructional time is significantly associated with student performance. More detailed evidence is needed to examine the impact not only of the quantity, but also the quality, of instructional time – in other words, how instructional time is actually used in different schools and classrooms in the region, and how this affects learning outcomes and disparities.

policy challenges: ensuring quality education for all by tackling low performance and reducing large disparities

Getting all children to complete a full cycle of basic education and providing access to upper-secondary education is a fundamental policy aim in all CEE/CIS countries. While recognizing the importance of this policy, it is equally clear that universalizing access to basic education represents an indispensable means to ensuring that good quality education is available for all students. The ultimate goal – in fact a basic human right – is for all young people to emerge from lower-secondary schools having mastered core skills and competencies in areas such as language (both official and international languages), mathematics and science. By doing so, these graduates become capable of grappling with the more demanding contents taught in upper-secondary and tertiary-level institutions and, more importantly, are better prepared to fully participate in the social, economic and political life of their countries. A recent follow-up study of PISA participants lends credence to this notion: it indicates that levels of reading literacy at age 15 are a strong predictor of participation in post-secondary education and expected future earnings after students enter the labour market162.

Quality education, and its provision, is also about equity and fairness. There are significant disparities within CEE/CIS countries in access to quality education. The unequal distribution of learning experiences and associated outcomes often falls along familiar fault lines: by gender, socio-economic background, poverty, rural-urban residence, peripheral location and disability as well as by immigrant status and ethnicity. Not only do learning disparities vary among these socio-demographic populations, but they also vary substantially between schools in the same community, and even between classrooms in the same school. Parents, policy-makers and often children themselves perceive such learning disparities as evidence of an unfair and inequitable school system. Beyond these perceptions of inequity, the prevalence of large numbers of low-performing students has the potential of exacerbating disparities in opportunity, reinforcing poverty, and perpetuating structural obstacles that marginalise disadvantaged groups from one generation to the next. Ineffective and unequal learning experiences undermine the provision of good-quality education and exacerbate existing social and economic inequalities. In the long term they can have profound implications for a nation’s prosperity and the sense of social solidarity.

It is clear, then, that improving and equalising access to good-quality education represents a clear policy challenge for CEE/CIS countries, with potentially broad societal ramifications. Addressing the educational needs of diverse student populations, sometimes through targeted policies, will help reduce the numbers of low performers and narrow within-country disparities in acquired proficiencies. This, in turn, can enhance the opportunity prospects of young people regardless of

161 UNICEF (2011).

162 OECD (2010) Vol. I, p. 32.

106



their social background and residence. The on-going challenge for national authorities, community officials and local school leaders is to develop effective strategies to ensure that all young people acquire critical knowledge and skills, with which they can realise their potential, lead capable and dignified lives, and participate fully in society. It is therefore vital that CEE/CIS governments carefully consider the design, scope and depth of student learning experiences, and formulate policies and practices drawing from the best existing pool of evidence.

mitigating the impact of socio-economic background on performance

Home background has a strong and significant impact on student performance. This relationship, which has been reported in dozens of national and cross-national studies over many decades, is also reported, not surprisingly, in the PISA 2009 assessment. Students from more privileged home settings are often better prepared when entering school, and are able to mobilize various resources (for example, linguistic, economic and social), which contribute to their success while in school. Not only do they enter school at an advantage, they (and their parents) learn how to successfully navigate the tasks that schools require of them. In addition, students from more privileged families often attend higher-quality schools in which they are more likely to encounter teachers who create stimulating learning environments. PISA evidence indicates, for instance, that more advantaged students tend to be found in schools with a higher share of full-time teachers with advanced university degrees. For all these reasons, and others, schools are thought to amplify over time (or across grades and educational levels) the early grade disparities between students who come from more-advantaged and less-advantaged home settings. While education systems may aim to ‘level the playing field’, the obstacles to do so are numerous and not easily overcome.

Having said that, the impact of home background on student performance is neither set in stone nor constant across countries. This is certainly apparent in the case of the PISA assessment. In some education systems, the impact of the socio-economic index, which PISA employs to capture home background differences, explains a good deal of the variation in student performance. In other systems, the impact of this index is more muted and has less explanatory value. Among CEE/CIS countries, differences in performance between a student from a more socio-economically advantaged background and a student from an ‘average’ background range from 21 points in Azerbaijan to 51 points in Bulgaria (39 points is equivalent to about one year of schooling in the region). To the extent that CEE/CIS countries wish to strengthen equity principles and the sense of fairness, then they should implement policies (see below) that minimize the impact of home background on student performance (i.e. more like Azerbaijan and less like Bulgaria).

As previously noted, students attend schools that differ in relation to the distribution of students’ socio-economic background. In instances where schools are composed of a disproportionate number of more- or less-advantaged students, then the home background variable can have a second, ‘school-level’, effect on performance. In other words, the average socio-economic composition of a school can influence student performance above and beyond the impact of an individual’s personal family background. Indeed, the PISA 2009 results indicate that when schools are ranked according to the average socio-economic level of students upon intake, the school’s impact on performance is greater than the impact of an individual student’s socio-economic background. This means that there is a need to consider school-oriented policies, in addition to those that address the needs of particular groups of marginalized students.

Tailoring policy interventions to meet specific country challenges

This section describes the kinds of policy strategies which are available to policy-makers who wish to moderate the impact of socio-economic background on performance, and create conditions for


107

CHAPTER 6

all students, especially low-performing ones, to realize their potential. The PISA 2009 assessment discusses five categories of policy options163, some of which specifically target low-performing schools or groups of low-performing students, while others are more universal in their intended scope (see below). The efficacy of each policy intervention is determined, in part, by certain conditions obtained in each national setting. The main conditions include, for example, the relative impact of home background on student performance, the composition of schools along key socio-economic and demographic dimensions, and the extent of vertical and horizontal differentiation in the education system.

The five major categories of educational policy interventions are, according to the OECD:

1. Targeting low-performing schools or low-performing students within schools. Policies in this category seek to improve student performance, specifically among students whose levels of achievement are concentrated at the lower end of the performance scale. They include, for example, the provision of remedial classes, an increase in instructional hours or an accelerated learning programme targeting low-performing students, based on their previous academic performance. Early prevention programmes targeting students who are at risk of academic failure (or grade repetition) in primary or secondary schools, or changes to grade repetition policies, if they are directed to low-performing schools and students, are also policies included in this category. The aim of these targeted policies is to raise the performance levels of low-performing students to that of their peers, regardless of their socio-economic background, and reduce performance disparities within schools.

2. Targeting disadvantaged students with additional instructional resources. Policies in this category involve school programmes or curricular initiatives whose aim is to overcome the impact of a distinct ‘disadvantage’ among students – usually based on their socio-economic or socio-demographic background. An example of policy in this category is the Head Start pre-school programme in the United States, which targeted students from poor families. Also included are policies targeting students from other disadvantaged groups such as immigrant children or those belonging to ethnic, religious or linguistic minorities. Bulgaria is a case in point. It needs to target its population of Roma children. According to national figures, UNESCO reports: “School enrolment rates are extremely low for Roma children compared with their non-Roma peers (Roma comprise 10.4 per cent of the country’s total population). Only 47 per cent of Roma children enrol in school and fewer of these children complete primary education. At the secondary level, only 12 per cent of Roma aged 16-19 years old are enrolled, compared with 81 per cent of the national population. Graduation statistics are even more discouraging: only seven per cent of Roma children complete secondary education164”. In general, policies in this category, which increase instructional resources or expand access to specialized programmes for students from a particularly disadvantaged group, are pertinent to countries with strong socio-economic gradients in student performance or gradients based on other social status criteria.

3. Targeting disadvantaged students with additional economical resources. Policies here seek mainly to improve the economic conditions of the households in which students live and, by doing so, improve the conditions for learning and subsequent student performance. Examples include programmes that provide, at little or no cost, meals (breakfast and/or lunch), transportation, textbooks and/or uniforms to students from low-income families. Transfer payments to poor families who ensure the consistent attendance of their children in school is another example of

163 OECD (2010) Vol. II., pp. 104-105.

164 UNESCO-International Bureau of Education, 2011a.

108



such a targeted policy. Compensatory policies targeting schools serving large proportions of low-income students may fit under this category, depending on whether the funds are used to reduce family outlays for school-related expenses. For example, developing targeted polices to reduce significant disparities in pre-school attendance in Kyrgyzstan, which are based on socio-economic status and rural/urban residence, would benefit students from disadvantaged groups and improve educational equity165. In Bulgaria, a policy of partially subsidizing kindergartens, and having low-income families pay lower fees, would also be a step in the right direction166.

4. Raising standards for all students. Universal policies that do not target a particular group or school, but which entail a general system-wide reform to improve student performance, belong to this policy category. Examples range from system-wide changes to school governance and finance decision-making, to examination (such as school leaving exams) or assessment policies, or to the number of hours in which schools are open. Fundamental changes to the subjects offered, the contents of textbooks or syllabus guidelines, the teaching methods used, and the availability of ICT and other curriculum-related reforms are all examples of policy options in this category. Croatia’s 2005 Education Sector Development Plan and school reform project, known as the Croatian National Education Standards, which introduced a new primary school curriculum and system based on competences and learning outcomes, illustrates a universal policy167. These policies are most relevant to countries with relatively weak socio-economic gradients and less variation in student performance.

5. Making education inclusive by bringing marginalized students into mainstream schools and classrooms. Policies in this category promote inclusive education. In some cases, they undo exclusionary practices such as the existence of special education programmes or the provision of separate schools for children with various disabilities. Additional examples of inclusive policies include the redrawing of catchment areas to include marginalized students, the detracking of secondary schools, the mainstreaming of students with disabilities in regular classes, and the bussing of students from low-income neighbourhoods to special ‘magnet’ schools. This category of policies is especially pertinent in countries seeking to reduce notable school-level disparities in student performance.

The PISA 2009 reports provide detailed data to help decision-makers and analysts tailor policy options to different national settings. The detailed data are graphically displayed for each country in a series of figures, which plot average reading performance and the socio-economic composition of the student population for each school sampled. The resulting figures were provided for nine of the 13 CEE/CIS countries and seven of the EU8 countries168. Each dot in these figures represents one school, with the size of the dot proportionate to the number of 15-year-olds enrolled in that school. Each country figure shows the extent to which school differences in performance (whether high or low) vary substantially along socio-economic lines, without indicating the reasons or factors that precipitated this variation. Each figure also displays three gradient lines to illustrate the relationship between: 1) socio-economic background and student performance; 2) socio-economic background and student performance within schools ; and 3) socio-economic background and student performance between schools.

165 UNESCO-International Bureau of Education, 2011c.

166 UNESCO-International Bureau of Education, 2011a.

167 UNESCO-International Bureau of Education, 2011b.

168 See Figures II.C to II.O in OECD (2010) vol. II, pp. 106-120 and Figure 4.12 in Walker (2011), p.80.


109

CHAPTER 6

These figures make it possible for policy analysts to identify which of the five categories of policy options (discussed above) are most relevant in their national context. For example, Kyrgyzstan and several other countries in the region are characterized by students with very low average performance and extremely disadvantaged socio-economic backgrounds, and a disproportionate number of low-performing schools. This means that targeted policies associated with the first three categories have greater relevance under these conditions than do universal policies mentioned in the fourth category. In countries like Bulgaria, Hungary and the Czech Republic, where the socio-economic gradient is relatively steep, policies in the second, third and possibly fifth categories, which target disadvantaged students or schools with large numbers of disadvantaged students, have greater relevance. The main point is clear: countries should carefully consider appropriate policy options in light of the specific educational challenges existing in each national setting.

achieving high quality with equity in educational outcomes: tradeoffs and realities

Many decision-makers in education assume that there is a basic trade-off between excellence and equity, between policies that seek to improve the provision of quality education and the attainment of high achievement levels, on the one hand, and the reduction of student disparities, on the other. Can quality education and high performance levels be achieved without threatening equity? Does the PISA assessment point to certain countries that have adjudicated or bridged this presumed trade-off?

First, by examining changes in average performance levels across PISA assessments (2000, 2003, 2006 and 2009), it is clear that significant progress has been achieved in some countries from one assessment to the next, and in some cases across the entire decade. These trends underscore an important point: student performance levels are neither inevitable nor fixed. Countries have achieved significant progress in learning outcomes over relatively short periods of time. Among the study focus countries, for example, we have seen that there was a marked improvement in reading literacy (since 2000) in Albania, Hungary, Latvia and Poland, while there was a decline in the Czech Republic. Over shorter time spans, reading levels in Serbia and Turkey noticeably improved (since 2003), as did levels in Kyrgyzstan and Montenegro (since 2006). Slovenia, on the other hand, shows a significant decline.

In some countries, improvements in reading were driven by concerted country efforts to raise scores at the bottom end of the distribution, thereby reducing the percentage of poorly performing students and improving educational equity. This pattern was especially apparent in Albania, Poland and Latvia—in each instance, the percentage of low-performing students (i.e., those below Level 2) was significantly reduced.

The PISA 2009 assessment also makes it possible to identify countries that focus on high performance levels in all subject domains (excellence), while concurrently reducing the gap between high- and low-performing students (equity). This melding of excellence and equity is examined empirically in PISA in all three literacy areas: reading, mathematics and science. In operational terms it refers to countries whose students achieve an overall performance above the OECD average and a lower-than-average impact of socio-economic background on performance.

As reported in earlier chapters, a small group of eight of the total 65 PISA 2009 participating systems were deemed to be most successful since they met the criteria for both high quality and low inequity. Estonia is the only study focus country to be included in this group (another study focus country, Poland, has above-average performance but only an average impact of socio-economic background). The PISA assessment points to certain commonalities among this select group of countries. These include, for example:

110



• a high value placed on education;

• an articulation of clear and ambitious standards by educational leaders;

• an emphasis on ensuring the quality of its teachers and principals; and

• a consistent provision of high-quality learning conditions to every student.

Articulating these lofty standards in official documents or policy declarations is one matter. Developing sustainable strategies and concrete means to translate these notions into everyday school realities is quite another. Deeply embedded codes of conduct, some based on political expediency, are not easily transformed. Finding a consensus among different stakeholders, such as ministry officials, teacher unions, parents, and others, as to what teacher quality means can involve a long, drawn-out process. In short, while ‘successful’ systems may appear to share common features nurturing educational excellence and equity, it is not obvious how these features came into being, how long they have existed, or which stakeholders and interest groups worked on their behalf.

specific strategies to tackle low performance in reading literacy

One of the strengths of the PISA 2009 assessment is its focus on student performance in reading literacy. Becoming a proficient reader and integrating positive reading practices in everyday life, both in school and upon graduation, are important goals in and of themselves. Strong reading skills have also been shown to have spill-over effects on achievement in other curricular areas such as mathematics and science. Given the critical importance of reading literacy, what specific strategies should countries consider, based on the evidence emerging from the 2009 assessment, to ensure that all students attain proficiency in basic reading skills?

Learning to read involves mastering a series of cognitive and meta-cognitive information-processing strategies, which, if successful, result in a sustained practice of reading as well as continued motivation and dedication to read for pleasure. Research has consistently shown that student engagement in reading and the development of effective learning strategies enable students to become proficient readers, and can pave a path to their remaining lifelong learners.

Analysis of the PISA results tends to confirm these research findings. In all countries that participated in the PISA 2009 assessment, reading performance is strongest among students who develop effective learning strategies and who read a variety of materials for their own enjoyment. When students read every day, they build up their reading proficiency through practice, and tend to feel more confident and engaged in reading. The more students enjoy reading and read for enjoyment, both printed and online materials, the higher their reading proficiency. In addition, reading fiction appears to have the strongest association with reading performance: students who read fiction regularly score about half a proficiency level above the average. In short, schools that enable students to read widely, particularly books of fiction, are implementing strategies associated with high student performance in reading. Countries that build supportive learning environments to encourage these reading practices will see gains in student performance levels.

The PISA assessment also indicates that gender and socio-economic status are significantly associated with reading proficiency. Boys and, to some extent, socio-economically disadvantaged students tend to be less engaged in reading than girls and socio-economically advantaged students. The former are less likely to read for enjoyment on a daily basis, and they are less likely to read a variety of printed materials, especially works of fiction. Differences in levels of engagement in reading and approaches to learning account for about one-third of socio-economic differences in reading performance, and over two-thirds of gender differences.


111

CHAPTER 6

Thus, countries should develop and approve guidelines, syllabi and textbooks which nurture classroom strategies motivating disengaged readers, regardless of their background. Specific strategies include, for example:

• beginning with easy and interesting texts, such as those found in magazines and newspapers, and then gradually introducing more complex reading texts;

• providing a large supply of diverse texts and books, especially those considered favourites among students, to stimulate students’ interest in reading;

• structuring activities such as book clubs, or supervised after-school sessions allowing students access to online materials.

Reading-promoting strategies should be based on careful consideration of student differences in reading preferences as well as their current reading abilities. When effective, these strategies sustain reader motivation and perseverance.

Research has also shown that students who take responsibility for their own learning – in other words, who employ ‘control strategies’ in which they set their own learning goals and check their own progress – are able to learn more effectively than those who do not169. However, giving all students more autonomy will not necessarily lead to better overall results. Weaker students, in particular, need to be taught how to employ effective learning strategies before being given greater autonomy to choose their own learning styles.

The PISA assessment indicates that an awareness of effective learning strategies is closely associated with proficiency in reading. Reading frequently, while important, is not enough: students who read a lot, but who do not understand how to learn effectively, perform worse in reading than students who read less, but understand what effective learning entails. This confirms findings emerging from previous research: while the enjoyment of reading is a necessary step towards becoming a better reader, it is not sufficient if it does not go hand-in-hand with a good understanding of how to use reading to learn effectively.

overall conclusion

This report makes a convincing case for an essential assertion: public policies in education matter. They have made, and will continue to make, a real difference in the lives of children and youth, and in the prosperity and social solidarity of countries and communities in the CEE/CIS region. For public policies to foster quality and equity in education, and to provide effective learning environments in schools and classrooms, they must be based on the collection and examination of rigorous evidence from different sources. The PISA assessment programme, which systematically explores country-specific and cross-country relationships in student performance in reading, mathematics and science, provides a strong platform to examine and debate the efficacy of different policy options in education. For many countries in the CEE/CIS region, well designed, methodologically rigorous and transparent studies of student achievement and learning deficits, such as PISA, have been far too rare in the past.

In addition, there are several other vital messages that emerge from this report:

1) It is imperative that CEE/CIS countries strengthen and refocus their policy efforts to improve the quality, equity and relevance of basic education. Access to quality basic education is a basic right for each and every child. The results and analyses of the 2006 and 2009 PISA assessments

169 OECD (2010) vol. III, p. 98

112



provide concrete evidence of which policy options and reforms are more or less likely to bring about concrete changes towards this goal. This report has paid particular attention to policies that improve the performance of low-achieving students and schools; that expand access to pre-primary education, especially for disadvantaged and minority children; that help all students learn to read, and read for enjoyment; and that focus on ways to improve teacher quality and effectiveness.

2) Quality education is not simply a matter of securing adequate inputs to schooling, such as sufficient schools, laboratories, textbooks, computers, instructional time and trained teachers. Quality education is also a matter, and increasingly so in the light of globalization, of the quality of knowledge, competencies, skills and attitudes that children take away from their school experiences. Thus, there is a well-defined need for a shift in how quality education is conceived, measured and monitored in the CEE/CIS region. All policy stakeholders should be involved in supporting student learning and monitoring learning outcomes. Especially useful in this regard are current international assessments such as PISA, PIRLS and TIMMS. Other assessments (regional, national or sub-national), which are appropriate, relevant, transparent and methodologically rigorous, would also be useful.

3) Most CEE/CIS countries have made great strides in increasing enrolment and attendance rates in basic education (up to, and sometimes including, upper-secondary education). However, these achievements mask deep-rooted inequalities in the quality of education provided and the benefits – for personal development, work opportunities and lifelong learning – to young people who have completed their formal schooling. The monitoring of learning deficits and disparities through international assessments is a powerful tool for addressing equity challenges in education and society, especially in the CEE/CIS region where such inequalities are rampant.

113

rEFErEnCES

referenCes

Brown, G., J. Micklewright, S.V. Schepf, and R. Waldmann (2007), ‘International surveys of educational achievement: How robust are the findings?’ Journal of the Royal Statistical Society, Series A, vol. 170, part 3, pp. 623–46.

Bukodi, Erzsebet and P. Robert (2007), Occupational mobility in Europe. Project Report. European Foundation for the Improvement of Living and Working Conditions, Dublin. Available online at http://www.eurofound.europa.eu/pubdocs/2007/11/en/1/ef0711en.pdf.

Gamoran, A., W. Secada and C. Marrett (2000), ’The Organizational Context of Teaching and Learning: Changing Theoretical Perspectives’, in M. Hallinan (ed.), Handbook of the Sociology of Education, Springer, New York City, New York.

Goldstein, H. (2004), ‘The Education World Cup: International comparisons of student achievement’, Cadmo 12 (2):63-70.

Goldstein, H., and G. Leckie (2008), ‘School league tables: what can they really tell us?’ Significance 5 (2):67-69.

Hirtt, N. (2010), ‘PISA ? C’est toujours la catastrophe ! - Edition 2009 de l’enquête de l’OCDE sur l’école’, in L’ecole democratique, jeudi 9 décembre 2010 (Aped, Bruxelles).

Kraaykamp, G. and P. Nieuwbeerta (2000), ‘Parental Background and Lifestyle Differentiation in Eastern Europe: Social, Political, and Cultural Intergenerational Transmission in Five Former Socialist Societies’, Social Science Research, 29(1), pp. 92 - 122.

Martin, M.O., Mullis, I.V.S., & Foy, P. (with Olson, J.F., Erberber, E., Preuschoff, C., & Galia, J.) (2008), TIMSS 2007 International Science Report: Findings from IEA’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

Matěju, P. and J. Straková (2005), ‘The Role of the Family and the School in the Reproduction of Educational Inequalities in the Post-Communist Czech Republic’, British Journal of Sociology of Education, 26(1), pp. 17 - 40.

Mullis I.V.S., Michael O. Martin, Ann M. Kennedy, and Pierre Foy (2007), PIRLS 2006 International Report: IEA’s Progress in International Reading Literacy Study in Primary School in 40 Countries, Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

Mullis, I.V.S., Martin, M.O., & Foy, P. (with Olson, J.F., Preuschoff, C., Erberber, E., Arora, A., & Galia, J.) (2008), TIMSS 2007 International Mathematics Report: Findings from IEA’s Trends in International Mathematics and Science Study at the Fourth and Eighth Grades, Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

OECD and UNESCO Institute for Statistics (2003), Literacy Skills for the World of Tomorrow: Further results from PISA 2000. Paris: OECD.

OECD (2004), Learning for Tomorrow’s World: First Results from PISA 2003, Paris: OECD.

OECD (2007), PISA 2006: Science Competencies for Tomorrow’s World: Vol. 1 (Analysis) and Vol. 2 (Data). Paris: OECD.

OECD (2009), PISA 2009 Assessment Framework - Key Competencies in Reading, Mathematics and Science.

Equity in LEarning?

114



OECD (2010), PISA 2009 Results: What Students Know and Can Do – Student Performance in Reading, Mathematics and Science (Volume 1).

OECD (2010), PISA 2009 Results: Overcoming Social Background - Equity in Learning Opportunities and Outcomes (Volume 2).

OECD (2010), PISA 2009 Results: Learning to Learn - Student Engagement, Strategies and Practices (Volume 3).

OECD (2010), PISA 2009 Results: What makes a School Successful - Resources, Policies and Practices (Volume 4).

OECD (2010), PISA 2009 Results: Learning Trends – Changes in Student Performance since 2000 (Volume 5).

OECD (2010b), Reviews of National Policies for Education: Kyrgyz Republic 2010 - Lessons from PISA, OECD Publishing.

OECD (2011), PISA 2009 Results: Students On Line – Digital Technologies and Performance (Volume 6).

OECD (2011a), Against the Odds: Disadvantaged Students Who Succeed in School, OECD Publishing.

OECD (2011b), ‘Poland: Secondary education reform’ in Lessons from PISA for the United States – Strong Performers and Successful Reformers in Education, OECD Publishing.

OECD (2011c), ‘The Impact of the 1999 Education Reform in Poland’, OECD Education Working Papers, No. 49.

Prokic-Breuer, Tijana (2011), ‘The relative importance of social and cultural capital for educational performance: Eastern versus Western Europe’, (Wissenschaftszentrum Berlin für Sozialforschung. Discussion papers SP 1, 2011-403), Berlin, 55 S.

Silova, Iveta (2009), ‘Varieties of educational transformation: The post-socialist states of Central/Southeastern Europe and the former Soviet Union’, Pp. 295-320 in R. Cowen and A. Kazamias (eds.), International Handbook of Comparative Education, Netherlands: Springer Publishers.

Silova, Iveta (2011), ‘Education and post-socialist transformations in Central Asia: Exploring margins and marginalities’. Pp. 1-23 in I. Silova (ed.) Globalization on the margins: Education and post-socialist transformations in Central Asia. Charlotte, NC: Information Age Publishing.

UNESCO-International Bureau of Education, 2011a, ‘Bulgaria’, in World Data on Education, VII edition, 2010/2011. http://www.ibe.unesco.org/fileadmin/user_upload/Publications/WDE/2010/pdf-versions/Bulgaria.pdf

UNESCO-International Bureau of Education, 2011b, ‘Croatia’, in World Data on Education, VII edition, 2010/2011. http://www.ibe.unesco.org/fileadmin/user_upload/Publications/WDE/2010/pdf-versions/Croatia.pdf

UNESCO-International Bureau of Education, 2011c, ‘Kyrgyz Republic’, in World Data on Education, VII edition, 2010/2011. http://www.ibe.unesco.org/fileadmin/user_upload/Publications/WDE/2010/pdf-versions/Kyrgyzstan.pdf

UNICEF (2002), A league table of educational disadvantage in rich nations, Innocenti Report Card No.4, UNICEF Innocenti Research Centre, Florence.

UNICEF (2007), Education for Some More than Others? A regional study on education in Central and Eastern Europe and the commonwealth of independent states (CEE/CIS), Geneva: UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States.

UNICEF (2009), Learning Achievement in the CEE/CIS Region: A comparative Analysis of the PISA 2006

115

rEFErEnCES

Results, Geneva: UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States.

UNICEF (2011), Teachers: A Regional Study on Recruitment, Development and Salaries of Teachers in the CEE/CIS Region, Geneva: UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS).

Walker M. (2011), PISA 2009 Plus Results: Performance of 15-year-olds in reading, mathematics and science for 10 additional participants, ACER Press.

116



annex 1: overvIeW of performanCe In Cee/CIs CounTrIes

We look in some detail, below, at the position of the CEE/CIS countries individually, in terms of average performance, and also in terms of equity measures and other characteristics measured by PISA. CEE/CIS countries have generally lower average but fewer disparities in performance than OECD countries.

albania

Among the CEE/CIS countries, Albania’s average performance is ranked towards the bottom. Both in reading and in science, Albania is at a level with Kazakhstan, although it is above Azerbaijan, Georgia and Kyrgyzstan. In mathematics, Albania ranks above only one country, Kyrgyzstan, and is at a level with Georgia. As many as 57 per cent of 15-year-old students do not reach the baseline level of achievement in reading and in science, a figure which rises to 68 per cent in mathematics.

When reading performance is examined separately for different text formats, Albania, together with Kazakhstan and Kyrgyzstan, is one of the countries with the largest negative performance differences for non-continuous texts (comprising one or more list, i.e. tables, graphs, maps, forms and diagrams), with respect to the combined reading scale, scoring 18 points less than for the overall score, which includes questions in continuous prose format.

In terms of within-country disparities, Albania has relatively large gaps between high and low performers, and is the only CEE/CIS country − along with Bulgaria − on average above the OECD mean. It has among the largest gender differences in performance in all three literacy domains, always in favour of girls, among all countries participating in PISA. Differences are larger in reading (62 points) than in science (29), and are smallest in mathematics (11).

Albania did not participate in PISA 2003 or 2006, but did participate in PISA 2000 (in the repeat survey in 2001). Between 2000 and 2009, Albania’s reading performance increased most among CEE/CIS countries (only Chile and Peru had a larger increase among all PISA participants), with data showing improvements among students at all proficiency levels. The percentage performing below the baseline was reduced by 14 points: from 70 per cent of students in 2000 to 56 per cent in 2009. Among all PISA countries with data, Albania shows the largest decline in the PISA mean index of economic, social and cultural status between 2000 and 2009 (-0.30). The association between socio-economic background and reading performance also diminished in that period.

In Albania, as well as in Azerbaijan, about half of all students attend a different grade than the modal grade, and are spread out across lower and upper-secondary education. Students in Albania, as well as in Azerbaijan, report the most positive relationships with their teachers among all PISA participants, and students’ views on how conducive classrooms are to learning (disciplinary climate) are generally positive.

In Albania, more than one third of 15-year-old children are out of school, and we can imagine that their literacy levels and socio-economic backgrounds are poorer than of those in school. The share of the 15-year-old population not covered by the PISA test in Albania is 39 per cent. This causes both an overestimation of average performances in the country, and also an underestimation of within-country differences and the association of background with performance.

ANNEX 1

117

azerbaijan

Azerbaijan’s overall performance is also ranked towards the bottom of the table of CEE/CIS countries. In reading and science, Azerbaijan’s average is above that of only Kyrgyzstan (and is at a level with Georgia in science). In mathematics, on the other hand, Azerbaijan scores on average at a level with Bulgaria and Romania, and above Kazakhstan, Montenegro, Albania, Georgia and Kyrgyzstan. A substantial share of students in Azerbaijan do not reach the baseline level of achievement in reading (73 per cent) and in science (70 per cent), while is the figure falls to 45 per cent in mathematics.

In terms of performance in different reading competencies, Azerbaijan has one of the largest differences between the ‘reflect and evaluate’ aspect, where it does most poorly (with 27 points less than on the overall score), and the other aspects: ‘access and retrieve’ and ‘integrate and interpret’.

Azerbaijan has the smallest within-country difference among all the study focus countries in terms of the gap between its top achievers (95th percentile) and lowest achievers (5th percentile), although the difference is still the equivalent of six times the average progression in scores from one year to the next. On average over the three scales, only Indonesia among all PISA countries has a smaller difference between its lowest and highest performers than Azerbaijan. Azerbaijan has the smallest gender difference in reading performance (in favour of girls, as in all countries). In science, there is a smaller advantage for girls, and in mathematics for boys. Azerbaijan has also the lowest difference in average scores between students with average and advantaged socio-economic backgrounds (21 points), and the weakest link in the relationship between background and performance (with only seven per cent of the proportion of variation in student performance accounted for by socio-economic background). Among the study focus countries, Azerbaijan also shows the smallest difference in school performance according to schools’ socio-economic intake.

Azerbaijan participated in PISA 2006 and since that date there has been no significant change in the country’s performance in reading and science. It should be noted that the Azerbaijan data on mathematics performance in 2006 was not included in UNICEF 2009 : it was not considered comparable because of some grave inconsistencies (for instance, despite being one of the lowest-performing countries in reading and science, Azerbaijan had fewer students at the lowest levels of achievement in mathematics than the top performers in PISA. Following queries about this, the OECD acknowledged problems with the data). In PISA 2009, the mathematics data for Azerbaijan was found to be more consistent and has now been included, although even after the largest decline among PISA countries there is still a large difference between the mathematics results and the reading and science results; this difference is larger than in any of the other 64 countries participating in PISA 2009.

In Azerbaijan, as well as in Albania, about half of all students attend a different grade than the modal grade, and are spread out across lower and upper-secondary education. Students in both Azerbaijan and Albania report the most positive relationships with their teachers, among all PISA participants, and students’ views of how conducive classrooms are to learning (disciplinary climate) are generally positive.

In Azerbaijan, more than one third of 15-year-old children are out of school, and we can imagine that their literacy levels and socio-economic backgrounds are poorer than of those in school. The share of the 15-year-old population not covered by the PISA test in Azerbaijan is 43 per cent. This causes both an overestimation of average performances in the country, and also an underestimation of within-country differences and the association of background with performance. In Azerbaijan, a high percentage of students participating in PISA have not attended any pre-primary education (69 per cent).

118



Bulgaria

Bulgaria’s average performance in PISA is above the average for CEE/CIS countries but below the study focus countries average. Bulgaria’s average performance in reading and science is at a level with Romania and Serbia, and above Montenegro, Kazakhstan, Albania, Georgia, Azerbaijan and Kyrgyzstan. In mathematics, it is at a level with Azerbaijan and Romania and above Montenegro, Kazakhstan, Albania, Georgia and Kyrgyzstan. In Bulgaria, 41 per cent of 15-year-old students do not reach the baseline level of achievement in reading. The figures are 47 per cent in mathematics and 39 per cent in science.

In the PIRLS international survey of achievement, Bulgaria does better than in PISA, with higher achievement than the majority of other participants.

Bulgaria has the largest within-country differences among all the study focus countries in terms of the gap between its top achievers (95th percentile) and lowest achievers (5th percentile), equivalent to nine times the average progression in scores from one year to the next. In PISA, only Qatar, Israel, and Trinidad and Tobago have larger disparities. It has one of the largest gender difference in reading performance among all countries participating in PISA, together with Albania and Georgia (in favour of girls, as in all countries), and girls also tend to do better in science. Bulgaria has also the largest difference in average scores between students with average and advantaged socio-economic backgrounds (51 points), and the relationship between background and performance is strong (with 20 per cent of the proportion of variation in student performance accounted for by socio-economic background). Among the CEE/CIS countries, Bulgaria also shows the largest differences in school performance according to schools’ socio-economic intake (among the study focus countries, only that of the Czech Republic is larger).

Bulgaria, along with Kyrgyzstan, has the largest difference by school location among all PISA countries: even after taking into account socio-economic background, the mean reading performance in city schools is 94 points higher (equivalent to more than two years of schooling) than the figure in rural schools.

Bulgaria participated in PISA 2000 (the repeat survey in 2001) and in PISA 2006, but not in PISA 2003. Between 2000 and 2009, there was no significant change in average reading performance, nor has there been any change in mathematics or science since 2006. However, between 2000 and 2009 there was an increase in within-country disparities: the total variance in reading performance increased by 24 per cent, from already high levels. Among all PISA countries with data, Bulgaria (at -0.23) together with Albania (-0.30) show the largest decline in the PISA mean index of economic, social and cultural status between 2000 and 2009.

Bulgaria has the greatest autonomy among the study focus countries in matters relating to resource allocation, with 84 per cent of principals reporting that their schools had sole responsibility for resource allocation (over tasks such as appointing and dismissing teachers, establishing teachers’ starting salaries and increments, and formulating and allocating school budgets).

Croatia

Croatia is, along with the Russian Federation, the best-performing CEE/CIS country in mathematics and science. It is at a level with some EU8 countries, including the Czech Republic, Slovakia and Lithuania, in reading; and Slovakia, Lithuania and Latvia, in science. In terms of absolute disadvantage, we see that just 22 per cent of students in Croatia are below the benchmark level in reading, 33 per cent in mathematics and 18 per cent in science. Boys have a higher average mathematics performance compared with girls (with girls having a higher average in reading, as in all countries).

Croatia joined PISA in 2006. Since 2006 no significant change is noted in Croatia’s performance in reading, mathematics or science.

ANNEX 1

119

georgia

Georgia took part in the PISA 2009 plus project, administering the PISA 2009 assessment in 2010.

Among the CEE/CIS countries, Georgia’s average performance is towards the bottom of the table: in reading, it scores above only Azerbaijan and Kyrgyzstan; in mathematics, above Kyrgyzstan and at a level with Albania; and in science, above only Kyrgyzstan, and at a level with Azerbaijan. As many as 62 per cent of 15-year-old students do not reach the baseline level of achievement in reading, 69 per cent in mathematics and 66 per cent in science.

Georgia, together with Albania and Bulgaria, has one of the largest gender differences in reading performance among all countries participating in PISA (in favour of girls, as in all countries). Girls also tend to do better in science.

In Georgia, the between-school variance is only 22 per cent of the total variance. In other words, there is relatively little variance in performance associated with schools, students with different abilities being relatively evenly spread across the school population. However, Georgia does have a high school-level exclusion rate in its sample. In fact, only 88 per cent of its nationally desired population was covered in the survey because it excluded all schools where Azerbaijani or Armenian was the language of instruction, as well as all schools within Abkhazia or South Ossetia.

kazakhstan

Kazakhstan participated for the first time in PISA in 2009. Among the CEE/CIS countries, Kazakhstan’s average performance is relatively low. In reading, it is at a level with Albania and above Azerbaijan, Georgia and Kyrgyzstan. In mathematics, it is at a level with Montenegro and above Albania, Georgia and Kyrgyzstan. In science, it is at a level with both Albania and Montenegro and above Azerbaijan, Georgia and Kyrgyzstan. In Kazakhstan, 59 per cent of 15-year-old students do not reach the baseline level of achievement in reading and mathematics; in science, the figure is 55 per cent.

In the ongoing TIMSS international survey of achievement of fourth grade students, Kazakhstan does better relative to other countries than it does in PISA.

When reading performance is examined separately for different text formats, Kazakhstan, together with Albania and Kyrgyzstan, is one of the countries with the largest negative performance differences on non-continuous texts (comprising one or more lists, i.e. tables, graphs, maps, forms and diagrams), with respect to the combined reading scale, scoring 20 points less than on the overall score, which includes questions in continuous prose format.

Kazakhstan has the highest percentage of students in PISA attending after-school lessons for at least one of the three subjects (attendance of both enrichment and remedial lessons is at around 80 per cent), and has the highest extra-curricular activities index (which includes academic activities as well as sports, arts and culture). However, a high percentage of students have not attended any pre-primary education (58 per cent).

kyrgyzstan

Kyrgyzstan is by far the lowest-performing country among all 65 countries participating in PISA 2009: the gap separating the mean performance of Kyrgyzstan with that of the top-performer in PISA, Shanghai-China, is equivalent to more than six years of schooling (six times the typical gap in the OECD between 15-year-old students in two adjacent grades). Kyrgyzstan has also by far the lowest GDP per capita of all participating countries.

120



The vast majority of students in Kyrgyzstan (83 per cent) do not reach the baseline level of achievement in reading. Kyrgyzstan is the only country where the majority do not even reach an even lower level, Level 1a (59 per cent), and 30 per cent of students do not manage even to reach the lowest level of measured performance (Level 1b). Only one per cent of students in EU8 countries are at this lower level, while the next largest proportion in the CEE/CIS countries is Albania, with 11 per cent. In mathematics, 87 per cent do not reach the baseline level, and it is only in Kyrgyzstan where the majority of students (65 per cent) do not even reach the lowest level of measured performance (Level 1). In science, 82 per cent do not reach the baseline level and 53 per cent do not even reach the lowest level of measured performance. Girls have a higher average mathematics performance compared with boys (which is also the case in reading, as in all countries).

When reading performance is examined separately for different text formats, Kyrgyzstan, together with Albania and Kazakhstan, is one of the countries with the largest negative performance differences on non-continuous texts (comprising one or more lists, i.e. tables, graphs, maps, forms and diagrams), with respect to the combined reading scale, scoring 21 points less than on the overall score, which includes questions in continuous prose format.

Kyrgyzstan, along with Bulgaria, has the largest difference by school location among all PISA countries: even after taking into account socio-economic background, the mean reading performance in city schools is 94 points higher (equivalent to more than two years of schooling) than the figure in rural schools.

In Kyrgyzstan, 63 per cent of students have not attended any pre-primary education. Students who have attended pre-primary education for more than a year generally have a higher performance than those who have not. Kyrgyzstan has the largest such performance advantage among CEE/CIS countries, corresponding to more than one year of schooling (47 points) even when socio-economic background is kept constant.

Kyrgyzstan joined PISA in 2006. The country’s mean reading and mathematics performance has improved since 2006, while science performance has remained unchanged.

See also the section on The case of Kyrgyzstan.

moldova

Moldova took part in the PISA 2009 plus project, administering the PISA 2009 assessment in 2010.

Moldova has a very low GDP per capita, as measured in US dollars using purchasing power parity, which is above that of only Kyrgyzstan. However, it does manage to obtain better results in PISA than several CEE/CIS countries, especially in science. In reading, Moldova scores above Georgia, Azerbaijan and Kyrgyzstan, at a level with Albania and Kazakhstan. In mathematics, it scores above Georgia, Albania and Kyrgyzstan, at a level with Montenegro and Kazakhstan. In science, it scores above Georgia, Azerbaijan, Kyrgyzstan, Albania, Montenegro and Kazakhstan. In Moldova, 57 per cent of 15-year-old students do not reach the baseline level of achievement in reading; the figures are 61 per cent in mathematics and 47 per cent in science. Girls outperform boys in both reading and science.

montenegro

Among the CEE/CIS countries, Montenegro’s average performance is quite low. In reading, its average score is above Kazakhstan, Albania, Azerbaijan, Georgia and Kyrgyzstan; in mathematics, it is at a level with Kazakhstan and above Albania, Georgia and Kyrgyzstan; while in science, it is at a level with Kazakhstan and above Albania, Azerbaijan, Georgia and Kyrgyzstan. In Montenegro, 50 per

ANNEX 1

121

cent of 15-year-old students do not reach the baseline level of achievement in reading, 58 per cent in mathematics and 54 per cent in science. Boys have a higher average mathematics performance compared with girls (with girls having a higher average in reading, as in all countries).

In terms of performance in different reading competencies, Montenegro has one of the largest differences between the ‘reflect and evaluate’ aspect, where it does most poorly (with 25 points less than on the overall score), and the other aspects: ‘access and retrieve’ and ‘integrate and interpret’.

Montenegro joined PISA in 2006. Since 2006, the country’s mean reading performance has improved, while there has been no change in mathematics and a slight decline in science.

romania

Romania’s average performance in PISA is above that of the average CEE/CIS country, but is below the study focus countries’ average. In reading and science, it is at a level with Bulgaria and above Montenegro, Kazakhstan, Albania, Azerbaijan, Georgia and Kyrgyzstan. In mathematics, it is at a level with Bulgaria and Azerbaijan, and above Montenegro, Kazakhstan, Albania, Georgia and Kyrgyzstan. In Romania, 40 per cent of 15-year-old students do not reach the baseline level of achievement in reading; the figures are 47 per cent in mathematics and 41 per cent in science.

Romania participated in PISA 2000 (the repeat survey in 2002) and in PISA 2006, but not in PISA 2003. Between 2002 and 2009, there was no significant change in average reading performance. The performance of highest-achieving students declined, while that of lowest-achieving students remained the same. In fact variance in reading performance, an indicator of the extent of within-country disparities, decreased by 22 per cent. Romania has relatively narrow gaps between high and low performers over the three literacy scales.

Among the eight study focus countries with data, only in Romania was there a statistically significant change in the gender gap, which widened by a substantial amount, passing from 14 to 43 points in favour of girls. While the total percentage below the baseline level of performance has remained stable, there was an increase for boys and a decrease for girls.

In mathematics and science, comparable data is available for Romania since 2006: there was a slight increase in mean mathematics performance while there was no significant change in mean science performance.

Among all PISA countries with data, Romania (at +0.32), as well as the Russian Federation (+0.31), show the largest increases in the PISA mean index of economic, social and cultural status between 2000 and 2009. In Romania, the association between socio-economic background and reading performance also increased in that period (the largest increase among all PISA countries with data).

In matters relating to resource allocation, school principals in Romania, along with Turkey, reported the least autonomy in the region, with around 80 per cent of responsibility attributed solely to the regional/national education authority.

russian Federation

In mathematics and science, the Russian Federation is, along with Croatia, the best-performing CEE/CIS country. In reading, it is also one of the best-performing CEE/CIS country, at a level with Turkey and second only to Croatia. In the Russian Federation, 27 per cent of 15-year-old students do not reach the baseline level of achievement in reading; the figures are 29 per cent in mathematics and 22 per cent in science.

122



In two other large ongoing international surveys of achievement, TIMSS and PIRLS, the Russian Federation does better than in PISA: in PIRLS it comes out on top, at a level with Hong Kong and Singapore, and above all EU8 participating countries.

The Russian Federation joined PISA at the very start and has participated in all four rounds of data collection since 2000. Between 2000 and 2009, there was no significant change in average reading performance. There was also no significant change in mathematics, since 2003, nor in science, since 2006 (the first years with comparable data).

Among all PISA countries with data, the Russian Federation (at +0.31) as well as Romania (+0.32) show the largest increases in the PISA mean index of economic, social and cultural status between 2000 and 2009. In the Russian Federation, the between-school association of socio-economic intake and performance at the school level diminished in the same period, but the within-school association of background and performance increased slightly so that the overall association of background and performance remained unchanged.

Serbia

Serbia’s average performance in PISA is above that of the average CEE/CIS country but below the study focus countries’ average. In reading and science, it is at a level with Bulgaria, and below Croatia, Turkey and the Russian Federation. In mathematics, it is at a level with Turkey, and below Croatia and the Russian Federation. In Serbia, 33 per cent of 15-year-old students do not reach the baseline level of achievement in reading; the figures are 41 per cent in mathematics and 34 per cent in science. Boys have a higher average mathematics performance compared with girls (with girls having a higher average in reading, as in all countries).

Serbia participated in PISA 2003 and 2006. The country’s mean reading performance has improved since 2003, while there has been no significant change in mathematics, or science, since 2006.

Serbia is the only country, among the study focus countries with data, which does not generally make use of standards-based external examinations at the secondary level to evaluate student learning (they exist in about a quarter of the system). Standardised tests are also not much used as assessment practice: only 46 per cent of principals report that they are used in their schools at least once a year – the lowest percentage among the study focus countries, except Slovenia.

turkey

Turkey’s average performance in PISA rates relatively highly compared with other CEE/CIS countries; it is just above the study focus countries’ average. In reading, it scores on average below only Croatia, among the CEE/CIS countries, while it is at a level with the Russian Federation and one EU8 country: Lithuania. In mathematics and science, it scores below Croatia and the Russian Federation, and is at a level with Serbia in mathematics. In Turkey, 25 per cent of 15-year-olds in school do not reach the baseline level of achievement in reading; the figures are 42 per cent in mathematics and 30 per cent in science. Boys have a higher average mathematics performance than girls (with girls having a higher average in reading, as in all countries).

Turkey participated in TIMSS eighth grade survey, and − unlike most CEE/CIS countries − did worse compared with its relative position in PISA.

In terms of equity measures, the difference in reading performance associated with a unit increase in the socio-economic background index is quite small in Turkey (29 points), although the relationship between background and performance is relatively strong (with 19 per cent of variance

ANNEX 1

123

in performance explained by such an index), so fewer disadvantaged students escape the rule and perform at the level of more advantaged students. Furthermore, Turkey has the largest range on the socio-economic background index in the CEE/CIS region, indicating large socio-economic disparities among households, so the ‘socio-economic gradient’ (referring to the relationship between socio-economic background and performance) will have a larger impact in Turkey than in other countries with similar ‘slopes’. It has also to deal with the lowest mean socio-economic background: more than half of all 15-year-old students in Turkey have a socio-economic background which is below that of the least-advantaged 15 per cent of students in OECD countries.

Turkey has the highest percentage of resilient students, or internationally successful disadvantaged students, among all study focus countries (42 per cent of all disadvantaged students in the country).

There is significant social and academic exclusion in Turkey (students with similar socio-economic background and performance attending the same schools). Together with Hungary, it has the largest between-school variance in student performance among all PISA participants: 67 per cent of the total between- and within-school variance lies between schools, which indicates that there is a large variation in average performance between schools. Students are separated into different school types early (at 11 years of age). Turkey and Slovenia (together with Israel and the United States among OECD countries) are the only study focus countries where socio-economically disadvantaged schools are also deprived in terms of quantity of basic resources (larger student-teacher ratios).

Turkey participated in PISA 2003 and 2006. The country’s mean reading and mathematics performance has improved since 2003. In mathematics, Turkey saw a reduction of 10 per cent in the percentage below the baseline. In science, Turkey’s mean performance has increased considerably since 2006 (the first year of comparable data). Turkey showed the largest decrease in the percentage of students scoring below the baseline in science among all PISA countries: from 47 per cent in 2006 to 30 per cent in 2009.

In matters relating to resource allocation, school principals in Turkey, along with Romania, reported the least autonomy among the study focus countries, with around 80 per cent reporting that responsibility attributed solely to the regional/national education authority.

Principals in Turkey had the highest perception of problems with instruction due to teacher shortages among all PISA countries, with reports indicating two standard deviations above the OECD average and about one standard deviation or more above that of any other country (with a wide variation between the principals’ perceptions within Turkey). Far more than any other PISA country, principals in Turkey have the perception that learning is disrupted by student and by teacher behaviour.

In Turkey, more than one third of 15-year-old children are out of school, and we can imagine their literacy levels and socio-economic backgrounds to be poorer than of those in school. The share of the 15-year-old population not covered by the PISA test is 43 per cent. This causes both an overestimation of average performances in the country, and also an underestimation of within-country differences and the association of background with performance. It is worthy of note that Turkey, even though it is one of the countries with the highest percentage of 15-year-olds not participating in PISA, has the lowest mean socio-economic background and the largest range of socio-economic backgrounds among PISA students in the region. In Turkey, 72 per cent of 15-year-old students have not attended any pre-primary education, which is the highest percentage among all PISA countries (see also the section on The case of Turkey).

124



annex 2: Tables and fIgures

Table 5 mean performance in reading, mathematics and science literacy, pIsa 2009.

reading mathematics science mean

Shanghai-China 556 600 575 577

Hong Kong-China 533 555 549 546

Finland 536 541 554 543

Singapore 526 562 542 543

Republic of Korea 539 546 538 541

Japan 520 529 539 529

Canada 524 527 529 527

New Zealand 521 519 532 524

Chinese Taipei 495 543 520 520

Australia 515 514 527 519

Netherlands 508 526 522 519

Liechtenstein 499 536 520 518

Switzerland 501 534 517 517

Estonia 501 512 528 514

Germany 497 513 520 510

Belgium 506 515 507 509

Macao-China 487 525 511 508

Poland 500 495 508 501

Iceland 500 507 496 501

Norway 503 498 500 500

United Kingdom 494 492 514 500

Denmark 495 503 499 499

Slovenia 483 501 512 499

Ireland 496 487 508 497

France 496 497 498 497

United States 500 487 502 496

Hungary 494 490 503 496

Sweden 497 494 495 496


Portugal 489 487 493 490

Slovakia 477 497 490 488

Austria 470 496 494 487

Latvia 484 482 494 487

Italy 486 483 489 486

Spain 481 483 488 484

Luxembourg 472 489 484 482

Lithuania 468 477 491 479

Croatia 476 460 486 474

Greece 483 466 470 473

ANNEX 2

125

reading mathematics science mean


Dubai (UAE) 459 453 466 459

Israel 474 447 455 459

Malta+ 442 463 461 455

Turkey 464 445 454 455

Serbia 442 442 443 442

Chile 449 421 447 439

Bulgaria 429 428 439 432

United Arab Emirates+ 431 421 438 430

Costa Rica+ 443 409 430 427

Uruguay 426 427 427 427

Romania 424 427 428 427

Thailand 421 419 425 422

Mexico 425 419 416 420

Mauritius+ 407 420 417 415

Miranda-Venezuela†+ 422 397 422 414

Trinidad and Tobago 416 414 410 414

Malaysia+ 414 404 422 413

Montenegro 408 403 401 404

Jordan 405 387 415 402

Brazil 412 386 405 401

Moldova+ 388 397 413 399

Colombia 413 381 402 399

Kazakhstan 390 405 400 399

Argentina 398 388 401 396

Tunisia 404 371 401 392

Azerbaijan 362 431 373 389

Indonesia 402 371 383 385

Albania 385 377 391 384

Georgia+ 374 379 373 375

Qatar 372 368 379 373

Panama 371 360 376 369

Peru 370 365 369 368

Tamil Nadu-India‡+ 337 351 348 345

Himachal Pradesh-India‡+ 317 338 325 327

Kyrgyzstan 314 331 330 325

Countries are ranked in order of average performance on the three literacy scales. Values in the last three rows refer to the difference between average scores of country groupings. + Participated in the PISA Plus project in 2010. † School response rate did not meet PISA standards. ‡ Student sampling did not meet PISA standards. Source: OECD 2010 vol.1 Table I.A, Walker 2011 Table B2.1, B3.1, B3.3.

Statistically significantly above the OECD average

Not statistically significantly different from the OECD average

Statistically significantly below the OECD average

126



figure 25 Comparing countries’ mean performance in reading, pIsa 2009.

A dot on grey background indicates that there is no statistically significant difference in mean performance between countries in corresponding row and column, while all other differences are significant.

Readingmean

501 Estonia500 Poland494 Hungary484 Latvia483 Slovenia478 Czech Republic477 Slovakia476 Croatia468 Lithuania464 Turkey459 Russian Federation442 Serbia429 Bulgaria424 Romania408 Montenegro390 Kazakhstan388 Moldova385 Albania374 Georgia362 Azerbaijan314 Kyrgyzstan

Est

on

iaP

ola

nd

Hu

ng

ary

Lat

via

Slo

ven

iaC

zech

Rep

ub

lic

Slo

vaki

aC

roat

iaLi

thu

ania

Turk

eyR

uss

ian

Fed

erat

ion

Ser

bia

Bu

lgar

iaR

om

ania

Mo

nte

neg

roK

azak

hst

anM

old

ova

Alb

ania

Geo

rgia

Aze

rbai

jan

Kyr

gy

zsta

n

Countries are ranked by mean performance. Where there is a line separating two adjacent countries (listed in the second column), the difference between their mean performance is significant.

Source: OECD 2010 vol.1 Figure I.2.15, Walker 2011 Table B2.1.

ANNEX 2

127

figure 26 Comparing countries’ mean performance in mathematics, pIsa 2009.


Mathsmean

512 Estonia501 Slovenia497 Slovakia495 Poland493 Czech Republic490 Hungary482 Latvia477 Lithuania468 Russian Federation460 Croatia445 Turkey442 Serbia431 Azerbaijan428 Bulgaria427 Romania405 Kazakhstan403 Montenegro397 Moldova379 Georgia377 Albania331 Kyrgyzstan

Est

on

iaS

love

nia

Slo

vaki

aP

ola

nd

Cze

ch R

epu

bli

cH

un

gar

yL

atvi

aLi

thu

ania

Ru

ssia

n F

eder

atio

nC

roat

iaTu

rkey

Ser

bia

Aze

rbai

jan

Bu

lgar

iaR

om

ania

Kaz

akh

stan

Mo

nte

neg

roM

old

ova

Geo

rgia

Alb

ania

Kyr

gy

zsta

n



128



figure 27 Comparing countries’ mean performance in science, pIsa2009.


Sciencemean

528 Estonia512 Slovenia508 Poland503 Hungary500 Czech Republic494 Latvia491 Lithuania490 Slovakia486 Croatia478 Russian Federation454 Turkey443 Serbia439 Bulgaria428 Romania413 Moldova401 Montenegro400 Kazakhstan391 Albania373 Azerbaijan373 Georgia330 Kyrgyzstan

Est

on

iaS

love

nia

Po

lan

dH

un

gar

yC

zech

Rep

ub

lic

Lat

via

Lith

uan

iaS

lova

kia

Cro

atia

Ru

ssia

n F

eder

atio

nTu

rkey

Ser

bia

Bu

lgar

iaR

om

ania

Mo

ldo

vaM

on

ten

egro

Kaz

akh

stan

Alb

ania

Aze

rbai

jan

Geo

rgia

Kyr

gy

zsta

n



ANNEX 2

129

figure 28 Comparing countries’ range of performance in reading, pIsa 2009.

95th-5thpercentilein reading

Bu

lgar

ia368 Bulgaria328 Kyrgyzstan326 Albania305 OECD average304 Montenegro302 Czech Republic301 Kazakhstan300 Hungary298 Russian Federation297 Slovakia297 Slovenia293 Poland293 Romania284 Croatia283 Lithuania274 Estonia274 Serbia270 Turkey262 Latvia251 Azerbaijan

Kyr

gy

zsta

nA

lban

iaO

EC

D a

vera

ge

Mo

nte

neg

roC

zech

Rep

ub

lic

Kaz

akh

stan

Hu

ng

ary

Ru

ssia

n F

eder

atio

nS

lova

kia

Slo

ven

iaP

ola

nd

Ro

man

iaC

roat

iaLi

thu

ania

Est

on

iaS

erb

iaTu

rkey

Lat

via

Aze

rbai

jan

not statistically significant difference

Difference between 95th and 5th percentile (of country in row) significantly smallerthan comparison country (column)

Difference between 95th and 5th percentile (of country in row) significantly largerthan comparison country (column)

Countries are ranked by range in performance (95th-5th percentile).

130



figure 29 Comparing countries’ range of performance in mathematics, pIsa 2009.

95th-5thpercentile

in maths

Bu

lgar

ia314 Slovenia324 Bulgaria

311 Slovakia310 Turkey308 Czech Republic303 Hungary300 OECD average300 Albania298 Serbia292 Croatia290 Poland290 Lithuania280 Russian Federation279 Montenegro272 Kazakhstan269 Kyrgyzstan265 Estonia260 Romania259 Latvia207 Azerbaijan

not statistically significant differenceQ95-Q5 (of country in row) significantly smaller than comparison country (column)Q95-Q5 (of country in row) significantly larger than comparison country (column)

Slo

ven

iaS

lova

kia

Turk

eyC

zech

Rep

ub

lic

Hu

ng

ary

OE

CD

ave

rag

eA

lban

iaS

erb

iaC

roat

iaP

ola

nd

Lith

uan

iaR

uss

ian

Fed

erat

ion

Mo

nte

neg

roK

azak

hst

anK

yrg

yzs

tan

Est

on

iaR

om

ania

Lat

via

Aze

rbai

jan


ANNEX 2

131

figure 30 Comparing countries’ range of performance in science, pIsa 2009.

95th-5thpercentilein science

Bu

lgar

iaC

zech

Rep

ub

lic

Slo

vaki

aO

EC

D a

vera

ge

Slo

ven

iaK

yrg

yzs

tan

Ru

ssia

n F

eder

atio

nA

lban

iaH

un

gar

yK

azak

hst

anM

on

ten

egro

Po

lan

dLi

thu

ania

Est

on

iaS

erb

iaC

roat

iaTu

rkey

Ro

man

iaL

atvi

aA

zerb

aija

n

344 Bulgaria318 Czech Republic308 Slovakia308 OECD average306 Slovenia299 Kyrgyzstan297 Russian Federation291 Albania288 Hungary286 Kazakhstan286 Montenegro286 Poland280 Lithuania277 Estonia277 Serbia276 Croatia265 Turkey257 Romania254 Latvia245 Azerbaijan

not statistically significant differenceQ95-Q5 (of country in row) significantly smaller than comparison country (column)Q95-Q5 (of country in row) significantly larger than comparison country (column)


132



Table 6 pIsa index of economic, social and cultural status: average among all students, within-country disparities (difference between 95th and 5th percentile), interquartile range at the student and school levels, average by pre-primary school attendance.

Average students’

ESCSD

iffe

ren

ce b

etw

een

th

e 9

5th

an

d 5

th p

er-

cen

tile

s o

n t

he

ES

CS

Inte

rqu

arti

le r

ang

e o

f th

e d

istr

ibu

tio

n*

of

the

stu

den

t-le

vel E

SC

S

Inte

rqu

arti

le r

ang

e o

f th

e d

istr

ibu

tio

n o

f sc

ho

ol m

ean

ES

CS

Average ESCS of students with pre-primary school

attendance of:

None1 year or less

More than 1 year

Turkey -1.16 4.02 1.78 0.94 -1.53 -0.37 0.21

Albania -0.95 3.44 1.45 0.75 -1.27 -0.95 -0.78

Kyrgyzstan -0.65 3.02 1.39 0.65 -0.86 -0.48 -0.10

Azerbaijan -0.64 3.18 1.48 0.88 -0.80 -0.35 -0.20

Kazakhstan -0.51 2.66 1.20 0.55 -0.75 -0.28 -0.11

Romania -0.34 2.93 1.12 0.64 -0.85 -0.52 -0.30

Poland -0.28 2.86 1.12 0.54 -0.52 -0.59 0.03

Montenegro -0.24 3.09 1.34 0.72 -0.64 -0.21 0.11

Russian Federation -0.21 2.51 1.25 0.57 -0.50 -0.27 -0.11

Hungary -0.20 3.14 1.34 0.85 -0.35 -0.44 -0.18

Croatia -0.18 3.04 1.18 0.57 -0.63 -0.38 0.14

Latvia -0.13 2.75 1.38 0.61 -0.39 -0.31 -0.00

Bulgaria -0.11 3.08 1.38 0.72 -0.40 -0.16 -0.05

Slovakia -0.09 2.70 1.05 0.58 -0.56 -0.20 -0.04

Czech Republic -0.09 2.30 0.96 0.45 -0.22 -0.16 -0.05

Lithuania -0.05 2.99 1.58 0.67 -0.40 -0.01 0.22

Serbia 0.07 3.17 1.35 0.58 -0.25 0.03 0.26

Slovenia 0.07 2.78 1.36 0.70 -0.30 -0.06 0.21

Estonia 0.15 2.53 1.22 0.50 -0.06 -0.07 0.21

Study focus country mean

-0.29 2.96 1.31 0.66 -0.59 -0.30 -0.03

OECD mean 0.00 2.92 1.29 0.65 -0.41 -0.13 0.13

EU8 mean -0.08 2.76 1.25 0.61 -0.35 -0.23 0.05

CEE/CIS mean -0.45 3.10 1.36 0.69 -0.77 -0.36 -0.09

* Difference between the 75th and 25th percentiles.

Countries are ranked by average student’s ESCS. Source: OECD 2010 Figure II.3.2, Tables II.3.2, II.5.2, II.5.5.

ANNEX 2

133

figure 31 strength of the socio-economic gradient and reading performance.

ArgentinaColombia

LuxembourgAustria

Dubaï (UAE)

Mexico

Thailand

Brazil

FranceDenmark

Spain

Israel

Greece

IrelandSwitzerlandNetherlands

Australia

Korea Finland

Hong Kong-China

IcelandLiechtenstein

United KingdomMacao-China

CanadaJapan

Norway

Chinese Taipei

Trinidad and Tobago

TunisiaIndonesia

Jordan

Qatar

Italy

Belgium

GermanyUnited States

Singapore

Shanghai-China

SwedenNew Zealand

Peru

Bulgaria

Turkey

Uruguay

Hungary

Chile

Lithuania

Romania

Albania

Kazakhstan

Panama

Kyrgyzstan

Poland

Czech RepublicSlovakia

Slovenia

Russian Federation

LatviaCroatia

Montenegro

Estonia

Azerbaijan

Kyrgyzstan

Serbia

300

350

400

450

500

550

600

5 01015

Percentage of variance in performance explained by the PISA index of economic,social and cultural status (r-squared x 100)

202530

Mea

n s

core

Above-average reading performanceAbove-average impact of socio-economic background

Above-average reading performanceBelow-average impact of socio-economic background

Below-average reading performanceAbove-average impact of socio-economic background

Below-average reading performanceBelow-average impact of socio-economic background

OE

CD

aver

age

OECD average

Strength of the relationship between performance and socio-economic background abovethe OECD average impact

Strength of the relationship between performance and socio-economic background not statisticallysignificantly different from the OECD average impact

Strength of the relationship between performance and socio-economic background belowthe OECD average impact

This Figure is taken from OECD 2010 vol.II p.58 (Figure II.3.3). Labels for EU8 countries have been coloured red and labels for CEE/CIS countries have been coloured green.

134



Table 7 assessment and accountability practices.

Countries are ranked by country’s average performance on the three literacy scales. Source: OECD 2010 Tables IV.3.11, IV.3.10, IV.3.12, IV.3.13, IV.3.14.

Exi

sten

ce o

f st

and

ard

s-b

ase

d e

xte

rnal

exa

min

ati

on

s (P

ISA

sys

tem

-lev

el d

ata

co

lle

ctio

n)

Percentage of students in schools where the principal reported… Percentage of students in schools where the principal reported…

the following assessment practices are used at least once

a year:

assessments of students in national modal grade for 15-year-olds are used for the following purposes:

the following uses of achievement data (accountability purposes):

the school provides information to parents on student

performance:

the methods used to monitor the practice of teacher for language

of instruction are:S

tan

dar

dis

ed

te

sts

Teac

her

-dev

elo

pe

d t

est

s

Teac

her

s’ j

ud

gm

enta

l ra

tin

gs

Stu

den

ts p

ort

foli

os

Stu

den

t as

sig

nm

ents

/pro

ject

s/h

om

ewo

rk

To in

form

par

ents

ab

ou

t th

eir

child

’s p

rog

ress

To m

ake

de

cisi

on

s ab

ou

t st

ud

ents

’ re

ten

tio

n

or

pro

mo

tio

n

To g

rou

p s

tud

ents

fo

r in

stru

ctio

nal

pu

rpo

ses

To c

om

par

e th

e sc

ho

ol t

o <

dis

tric

t o

r n

ati

on

al>

per

form

ance

To m

on

ito

r th

e sc

ho

ol’s

pro

gre

ss f

rom

yea

r to

yea

r

To m

ake

jud

gem

ents

ab

ou

t te

ach

ers’

e

ffe

ctiv

ene

ss

To i

den

tify

asp

ect

s o

f in

stru

ctio

n o

r th

e cu

rric

ulu

m t

ha

t co

uld

be

imp

rove

d

To c

om

par

e th

e sc

ho

ol w

ith

oth

er s

cho

ols

To c

om

par

e th

e sc

ho

ol w

ith

oth

er s

cho

ols

or

wit

h n

ati

on

al/r

eg

ion

al p

erfo

rman

ce

Po

ste

d p

ub

licl

y

Use

d i

n e

valu

ati

on

of

the

pri

nci

pal

’s

per

form

ance

Use

d i

n e

valu

ati

on

of

teac

her

s’

per

form

ance

Use

d i

n d

eci

sio

ns

abo

ut

inst

ruct

ion

al

reso

urc

e al

loca

tio

n t

o t

he

sch

oo

l

Trac

ked

ove

r ti

me

by

an a

dm

inis

tra

tive

au

tho

rity

Rel

ati

ve t

o o

ther

stu

den

ts i

n t

he

sam

e sc

ho

ol

Rel

ati

ve t

o n

ati

on

al o

r re

gio

nal

b

ench

mar

ks

As

a g

rou

p r

ela

tive

to

stu

den

ts i

n t

he

sam

e g

rad

e in

oth

er s

cho

ols

Rel

ati

ve t

o n

ati

on

al o

r re

gio

nal

ben

chm

arks

o

r as

a g

rou

p r

ela

tive

to

stu

den

ts i

n t

he

sam

e g

rad

e in

oth

er s

cho

ols

Test

s o

r as

sess

men

ts o

f st

ud

ent

ach

ieve

men

t

Teac

her

pe

er r

evie

w (

of

less

on

pla

ns,

as

sess

men

t in

stru

men

ts, l

ess

on

s)

Pri

nci

pal

or

sen

ior

staf

f o

bse

rva

tio

ns

of

less

on

s

Ob

serv

ati

on

of

clas

ses

by

insp

ect

ors

or

oth

er p

erso

ns

exte

rnal

to

th

e sc

ho

ol

Estonia 100% 83 100 100 73 100 98 82 26 67 85 72 78 61 74 32 43 74 16 87 27 52 25 58 27 52 25 58

Poland 100% 97 100 97 71 100 99 98 33 57 95 79 92 57 63 53 80 86 14 90 59 74 49 77 59 74 49 77

Slovenia 100% 24 99 100 54 99 98 96 24 53 92 40 75 43 62 36 73 38 15 69 16 41 4 43 16 41 4 43

Hungary 100% 76 98 99 62 99 97 83 52 69 87 60 65 61 73 33 56 82 13 50 62 40 21 45 62 40 21 45

Czech Republic 100% 89 99 94 56 99 97 89 40 65 89 60 84 62 72 31 54 79 4 56 62 61 32 63 62 61 32 63

Slovakia 100% 94 100 100 62 100 100 97 47 52 86 80 86 68 73 63 51 73 11 86 61 54 27 58 61 54 27 58

Latvia 100% 96 100 99 90 97 99 94 34 92 98 92 98 82 92 25 32 81 38 51 22 11 18 21 22 11 18 21

Lithuania 100% 85 100 99 61 99 99 94 55 55 95 71 81 48 62 25 38 73 15 66 41 a 19 19 41 a 19 19

Croatia 100% 68 100 97 44 100 99 89 59 72 94 55 82 66 81 22 19 45 13 84 63 a 29 29 63 a 29 29

Russian Federation

100% 84 100 90 88 99 100 96 52 83 99 98 100 95 98 76 82 97 65 99 69 77 51 84 69 77 51 84

Turkey 100% 72 99 96 85 99 94 71 73 73 84 71 55 72 83 50 46 72 31 77 91 72 70 81 91 72 70 81

Serbia 26% 46 100 99 37 99 95 84 61 35 96 63 79 57 63 56 50 78 19 84 93 27 36 47 93 27 36 47

Bulgaria 100% 92 100 97 48 99 100 79 42 84 90 92 70 78 86 33 58 87 30 78 88 65 56 69 88 65 56 69

Romania 78% 82 100 96 99 98 99 89 66 87 97 85 92 83 92 62 89 92 66 75 72 71 48 81 72 71 48 81

Montenegro 100% 59 100 100 48 100 40 12 21 38 55 56 57 45 47 76 82 96 24 98 87 35 27 39 87 35 27 39

Kazakhstan m 95 100 97 92 99 99 96 56 93 99 99 99 89 95 83 77 98 83 99 94 74 73 85 94 74 73 85

Azerbaijan 100% 98 99 91 71 96 97 96 81 82 89 94 89 85 93 86 91 96 68 81 99 92 83 95 99 92 83 95

Albania m 95 98 45 69 97 97 88 68 78 100 90 89 77 88 34 86 97 79 81 86 42 40 51 86 42 40 51

Kyrgyzstan 100% 98 99 92 92 98 94 78 63 84 98 99 90 85 93 66 86 96 85 98 91 84 63 90 91 84 63 90

OECD average 66% 76 98 94 77 99 98 78 50 54 77 47 77 46 59 37 36 45 33 66 47 47 23 52 47 47 23 52

Study focus country average

94% 81 100 94 69 99 95 85 50 69 91 77 82 69 79 50 63 81 36 79 67 57 41 60 67 57 41 60

EU8 average 100% 80 99 98 66 99 99 92 39 64 91 69 82 60 71 37 53 73 16 69 44 48 24 48 44 48 24 48

CEE/CIS average 89% 81 100 91 70 98 92 80 58 73 91 82 82 76 84 58 70 87 51 87 85 64 52 68 85 64 52 68

OECD-Study focus countries

-28% -5 -1 0 8 0 3 -7 0 -16 -14 -29 -5 -23 -20 -13 -27 -36 -4 -13 -21 -10 -17 -8 -21 -10 -17 -8

ANNEX 2

135

Exi

sten

ce o

f st

and

ard

s-b

ase

d e

xte

rnal

exa

min

ati

on

s (P

ISA

sys

tem

-lev

el d

ata

co

lle

ctio

n)

Percentage of students in schools where the principal reported… Percentage of students in schools where the principal reported…

the following assessment practices are used at least once

a year:

assessments of students in national modal grade for 15-year-olds are used for the following purposes:

the following uses of achievement data (accountability purposes):

the school provides information to parents on student

performance:

the methods used to monitor the practice of teacher for language

of instruction are:

Sta

nd

ard

ise

d t

est

s

Teac

her

-dev

elo

pe

d t

est

s

Teac

her

s’ j

ud

gm

enta

l ra

tin

gs

Stu

den

ts p

ort

foli

os

Stu

den

t as

sig

nm

ents

/pro

ject

s/h

om

ewo

rk

To in

form

par

ents

ab

ou

t th

eir

child

’s p

rog

ress

To m

ake

de

cisi

on

s ab

ou

t st

ud

ents

’ re

ten

tio

n

or

pro

mo

tio

n

To g

rou

p s

tud

ents

fo

r in

stru

ctio

nal

pu

rpo

ses

To c

om

par

e th

e sc

ho

ol t

o <

dis

tric

t o

r n

ati

on

al>

per

form

ance

To m

on

ito

r th

e sc

ho

ol’s

pro

gre

ss f

rom

yea

r to

yea

r

To m

ake

jud

gem

ents

ab

ou

t te

ach

ers’

e

ffe

ctiv

ene

ss

To i

den

tify

asp

ect

s o

f in

stru

ctio

n o

r th

e cu

rric

ulu

m t

ha

t co

uld

be

imp

rove

d

To c

om

par

e th

e sc

ho

ol w

ith

oth

er s

cho

ols

To c

om

par

e th

e sc

ho

ol w

ith

oth

er s

cho

ols

or

wit

h n

ati

on

al/r

eg

ion

al p

erfo

rman

ce

Po

ste

d p

ub

licl

y

Use

d i

n e

valu

ati

on

of

the

pri

nci

pal

’s

per

form

ance

Use

d i

n e

valu

ati

on

of

teac

her

s’

per

form

ance

Use

d i

n d

eci

sio

ns

abo

ut

inst

ruct

ion

al

reso

urc

e al

loca

tio

n t

o t

he

sch

oo

l

Trac

ked

ove

r ti

me

by

an a

dm

inis

tra

tive

au

tho

rity

Rel

ati

ve t

o o

ther

stu

den

ts i

n t

he

sam

e sc

ho

ol

Rel

ati

ve t

o n

ati

on

al o

r re

gio

nal

b

ench

mar

ks

As

a g

rou

p r

ela

tive

to

stu

den

ts i

n t

he

sam

e g

rad

e in

oth

er s

cho

ols

Rel

ati

ve t

o n

ati

on

al o

r re

gio

nal

ben

chm

arks

o

r as

a g

rou

p r

ela

tive

to

stu

den

ts i

n t

he

sam

e g

rad

e in

oth

er s

cho

ols

Test

s o

r as

sess

men

ts o

f st

ud

ent

ach

ieve

men

t

Teac

her

pe

er r

evie

w (

of

less

on

pla

ns,

as

sess

men

t in

stru

men

ts, l

ess

on

s)

Pri

nci

pal

or

sen

ior

staf

f o

bse

rva

tio

ns

of

less

on

s

Ob

serv

ati

on

of

clas

ses

by

insp

ect

ors

or

oth

er p

erso

ns

exte

rnal

to

th

e sc

ho

ol

Estonia 100% 83 100 100 73 100 98 82 26 67 85 72 78 61 74 32 43 74 16 87 27 52 25 58 27 52 25 58

Poland 100% 97 100 97 71 100 99 98 33 57 95 79 92 57 63 53 80 86 14 90 59 74 49 77 59 74 49 77

Slovenia 100% 24 99 100 54 99 98 96 24 53 92 40 75 43 62 36 73 38 15 69 16 41 4 43 16 41 4 43

Hungary 100% 76 98 99 62 99 97 83 52 69 87 60 65 61 73 33 56 82 13 50 62 40 21 45 62 40 21 45

Czech Republic 100% 89 99 94 56 99 97 89 40 65 89 60 84 62 72 31 54 79 4 56 62 61 32 63 62 61 32 63

Slovakia 100% 94 100 100 62 100 100 97 47 52 86 80 86 68 73 63 51 73 11 86 61 54 27 58 61 54 27 58

Latvia 100% 96 100 99 90 97 99 94 34 92 98 92 98 82 92 25 32 81 38 51 22 11 18 21 22 11 18 21

Lithuania 100% 85 100 99 61 99 99 94 55 55 95 71 81 48 62 25 38 73 15 66 41 a 19 19 41 a 19 19

Croatia 100% 68 100 97 44 100 99 89 59 72 94 55 82 66 81 22 19 45 13 84 63 a 29 29 63 a 29 29

Russian Federation

100% 84 100 90 88 99 100 96 52 83 99 98 100 95 98 76 82 97 65 99 69 77 51 84 69 77 51 84

Turkey 100% 72 99 96 85 99 94 71 73 73 84 71 55 72 83 50 46 72 31 77 91 72 70 81 91 72 70 81

Serbia 26% 46 100 99 37 99 95 84 61 35 96 63 79 57 63 56 50 78 19 84 93 27 36 47 93 27 36 47

Bulgaria 100% 92 100 97 48 99 100 79 42 84 90 92 70 78 86 33 58 87 30 78 88 65 56 69 88 65 56 69

Romania 78% 82 100 96 99 98 99 89 66 87 97 85 92 83 92 62 89 92 66 75 72 71 48 81 72 71 48 81

Montenegro 100% 59 100 100 48 100 40 12 21 38 55 56 57 45 47 76 82 96 24 98 87 35 27 39 87 35 27 39

Kazakhstan m 95 100 97 92 99 99 96 56 93 99 99 99 89 95 83 77 98 83 99 94 74 73 85 94 74 73 85

Azerbaijan 100% 98 99 91 71 96 97 96 81 82 89 94 89 85 93 86 91 96 68 81 99 92 83 95 99 92 83 95

Albania m 95 98 45 69 97 97 88 68 78 100 90 89 77 88 34 86 97 79 81 86 42 40 51 86 42 40 51

Kyrgyzstan 100% 98 99 92 92 98 94 78 63 84 98 99 90 85 93 66 86 96 85 98 91 84 63 90 91 84 63 90

OECD average 66% 76 98 94 77 99 98 78 50 54 77 47 77 46 59 37 36 45 33 66 47 47 23 52 47 47 23 52


94% 81 100 94 69 99 95 85 50 69 91 77 82 69 79 50 63 81 36 79 67 57 41 60 67 57 41 60

EU8 average 100% 80 99 98 66 99 99 92 39 64 91 69 82 60 71 37 53 73 16 69 44 48 24 48 44 48 24 48

CEE/CIS average 89% 81 100 91 70 98 92 80 58 73 91 82 82 76 84 58 70 87 51 87 85 64 52 68 85 64 52 68


-28% -5 -1 0 8 0 3 -7 0 -16 -14 -29 -5 -23 -20 -13 -27 -36 -4 -13 -21 -10 -17 -8 -21 -10 -17 -8

136



Table 8 resources and learning time.

Ind

ex o

f te

ach

er s

ho

rtag

e

Ind

ex o

f th

e q

ual

ity

of

the

sch

oo

ls’ e

du

cati

on

al r

eso

urc

es

Per

cen

tag

e o

f st

ud

ents

in

sch

oo

l wh

ere

ther

e is

a li

bra

ry

Per

cen

tag

e o

f st

ud

ents

in

sch

oo

ls w

her

e th

e p

rin

cip

al

rep

ort

ed l

ack

of

lib

rary

m

ater

ials

Per

cen

tag

e o

f st

ud

ents

usi

ng

a

lib

rary

to

bo

rro

w b

oo

ks f

or

ple

asu

re

Per

cen

tag

e o

f st

ud

ents

usi

ng

li

bra

ry t

o b

orr

ow

bo

oks

fo

r sc

ho

olw

ork

Students report on time spent for learning per week in regular school lessons in (hours:minutes):

Students report attending after-school lessons for at least one of the 3 subjects:

Ind

ex o

f sc

ho

ols

’ ex

tra

-cu

rric

ula

r ac

tivi

ties

bas

ed o

n

sch

oo

l pri

nci

pal

s re

po

rts

language of instruction mathematics science Total

Enrichment lessons

Remedial lessons

Estonia -0.11 0.04 97 26 66 85 3:21 3:45 3:13 10:21 24 35 0.44

Poland -0.78 0.29 97 19 65 82 3:49 3:24 3:08 10:21 57 35 0.46

Slovenia -0.72 0.48 97 7 67 87 2:55 2:43 3:20 8:59 25 27 0.91

Hungary -0.55 0.26 97 14 56 67 2:53 2:36 2:37 8:06 30 23 0.09

Czech Republic -0.02 -0.12 77 38 46 56 3:01 3:07 3:57 10:05 35 25 0.03

Slovakia -0.29 -0.46 87 49 46 59 2:56 2:49 3:35 9:21 31 30 0.71

Latvia -0.43 -0.11 98 15 63 80 2:43 3:41 4:25 10:50 54 14 0.77

Lithuania -0.37 -0.18 98 19 66 74 3:26 2:59 3:33 9:59 31 22 0.47

Croatia -0.19 -0.21 98 36 57 88 2:46 2:29 2:35 7:50 34 38 0.25

Russian Federation 0.13 -0.63 99 49 70 91 3:57 3:24 4:38 12:01 65 66 0.72

Turkey 2.05 -1.35 94 75 68 77 3:56 3:02 2:08 9:08 30 21 0.38

Serbia -0.64 -0.38 97 31 58 83 2:24 2:34 3:48 8:48 34 28 0.23

Bulgaria -0.64 -0.10 86 35 60 71 2:24 2:23 4:14 9:02 29 25 0.10

Romania -0.74 0.09 98 24 69 80 3:04 2:22 2:30 7:57 42 28 1.01

Montenegro -0.36 -0.79 98 39 62 73 2:38 2:35 2:14 7:28 40 20 0.62

Kazakhstan 0.47 -0.76 98 53 85 96 3:18 2:54 4:49 11:02 79 81 1.30

Azerbaijan -0.02 -0.58 98 36 81 87 2:49 3:39 5:10 11:39 68 61 0.90

Albania -0.06 -1.72 96 76 81 90 3:28 3:13 4:00 10:42 62 60 0.20

Kyrgyzstan 0.92 -0.74 86 70 84 79 3:36 3:51 3:54 11:22 62 40 0.72

OECD average -0.04 0.04 90 29 52 64 3:37 3:34 3:22 10:33 28 26 0.17

Study focus country average -0.12 -0.37 94 37 66 79 3:08 3:01 3:34 9:44 44 36 0.54

EU8 average -0.41 0.03 93 23 59 74 3:08 3:08 3:28 9:45 36 26 0.48

CEE/CIS average 0.08 -0.65 95 48 71 83 3:07 2:57 3:38 9:43 50 43 0.58

OECD-Study focus countries 0.08 0.41 -5 -8 -14 -16 0:29 0:32 -0:12 0:49 -15 -10 -0.37

Countries are ranked by country’s average performance on the three literacy scales. Source: OECD 2010 Tables IV.3.20, IV.3.23, IV.3.24, IV.3.16a, IV.3.17a, IV.3.19.

Index of teacher shortage based on school principals reports on whether they thought instruction in their school was hindered by a lack of qualified teachers and staff in key areas. Index of the quality of the schools’ educational resources based on school principals’ reports on the extent to which the school’s capacity to provide instruction was hindered by the shortage

ANNEX 2

137

Ind

ex o

f te

ach

er s

ho

rtag

e

Ind

ex o

f th

e q

ual

ity

of

the

sch

oo

ls’ e

du

cati

on

al r

eso

urc

es

Per

cen

tag

e o

f st

ud

ents

in

sch

oo

l wh

ere

ther

e is

a li

bra

ry

Per

cen

tag

e o

f st

ud

ents

in

sch

oo

ls w

her

e th

e p

rin

cip

al

rep

ort

ed l

ack

of

lib

rary

m

ater

ials

Per

cen

tag

e o

f st

ud

ents

usi

ng

a

lib

rary

to

bo

rro

w b

oo

ks f

or

ple

asu

re

Per

cen

tag

e o

f st

ud

ents

usi

ng

li

bra

ry t

o b

orr

ow

bo

oks

fo

r sc

ho

olw

ork

Students report on time spent for learning per week in regular school lessons in (hours:minutes):

Students report attending after-school lessons for at least one of the 3 subjects:

Ind

ex o

f sc

ho

ols

’ ex

tra

-cu

rric

ula

r ac

tivi

ties

bas

ed o

n

sch

oo

l pri

nci

pal

s re

po

rts

language of instruction mathematics science Total

Enrichment lessons

Remedial lessons

Estonia -0.11 0.04 97 26 66 85 3:21 3:45 3:13 10:21 24 35 0.44

Poland -0.78 0.29 97 19 65 82 3:49 3:24 3:08 10:21 57 35 0.46

Slovenia -0.72 0.48 97 7 67 87 2:55 2:43 3:20 8:59 25 27 0.91

Hungary -0.55 0.26 97 14 56 67 2:53 2:36 2:37 8:06 30 23 0.09

Czech Republic -0.02 -0.12 77 38 46 56 3:01 3:07 3:57 10:05 35 25 0.03

Slovakia -0.29 -0.46 87 49 46 59 2:56 2:49 3:35 9:21 31 30 0.71

Latvia -0.43 -0.11 98 15 63 80 2:43 3:41 4:25 10:50 54 14 0.77

Lithuania -0.37 -0.18 98 19 66 74 3:26 2:59 3:33 9:59 31 22 0.47

Croatia -0.19 -0.21 98 36 57 88 2:46 2:29 2:35 7:50 34 38 0.25

Russian Federation 0.13 -0.63 99 49 70 91 3:57 3:24 4:38 12:01 65 66 0.72

Turkey 2.05 -1.35 94 75 68 77 3:56 3:02 2:08 9:08 30 21 0.38

Serbia -0.64 -0.38 97 31 58 83 2:24 2:34 3:48 8:48 34 28 0.23

Bulgaria -0.64 -0.10 86 35 60 71 2:24 2:23 4:14 9:02 29 25 0.10

Romania -0.74 0.09 98 24 69 80 3:04 2:22 2:30 7:57 42 28 1.01

Montenegro -0.36 -0.79 98 39 62 73 2:38 2:35 2:14 7:28 40 20 0.62

Kazakhstan 0.47 -0.76 98 53 85 96 3:18 2:54 4:49 11:02 79 81 1.30

Azerbaijan -0.02 -0.58 98 36 81 87 2:49 3:39 5:10 11:39 68 61 0.90

Albania -0.06 -1.72 96 76 81 90 3:28 3:13 4:00 10:42 62 60 0.20

Kyrgyzstan 0.92 -0.74 86 70 84 79 3:36 3:51 3:54 11:22 62 40 0.72

OECD average -0.04 0.04 90 29 52 64 3:37 3:34 3:22 10:33 28 26 0.17

Study focus country average -0.12 -0.37 94 37 66 79 3:08 3:01 3:34 9:44 44 36 0.54

EU8 average -0.41 0.03 93 23 59 74 3:08 3:08 3:28 9:45 36 26 0.48

CEE/CIS average 0.08 -0.65 95 48 71 83 3:07 2:57 3:38 9:43 50 43 0.58

OECD-Study focus countries 0.08 0.41 -5 -8 -14 -16 0:29 0:32 -0:12 0:49 -15 -10 -0.37

or inadequacy of resources for instruction such as science laboratory equipment, textbooks, computers, internet connection, computer software, library and audio-visual materials. Index of schools’ extra-curricular activities based on school principals reports on whether the following extra-curricular activities are offered by the school: a band, orchestra or choir; school plays or musicals; a school yearbook, newspaper or magazine; volunteering or service activities; a book club; debating club/activities; a school club or competition for foreign language, mathematics or science; an academic club; arts club/activities; sports team/activities; lectures or seminars; collaboration with local libraries; collaboration with local newspapers.

138



Table 9 school climate.

Based on students’ reports: Based on principals’ reports:

Ind

ex o

f te

ach

er-s

tud

ent

rela

tio

ns

Ind

ex o

f d

isci

plin

ary

clim

ate

Ind

ex o

f te

ach

ers’

st

imu

lati

on

of

stu

den

ts’

read

ing

en

gag

emen

t

Ind

ex o

f st

ud

ent-

rela

ted

fa

cto

rs a

ffec

tin

g s

cho

ol

clim

ate

Ind

ex o

f te

ach

er-r

elat

ed

fact

ors

aff

ecti

ng

sch

oo

l cl

imat

e

Ind

ex o

f sc

ho

ol

pri

nci

pal

’s le

ader

ship

Par

ents

’ exp

ecta

tio

ns

for

hig

her

aca

dem

ic

stan

dar

ds

Estonia -0.04 0.05 0.06 -0.10 0.09 -0.15 14

Poland -0.35 0.07 0.29 0.05 0.47 0.58 11

Slovenia -0.42 -0.11 0.22 -0.39 0.02 0.31 27

Hungary -0.01 -0.02 0.23 0.18 0.51 -0.05 18

Czech Republic -0.24 -0.18 -0.12 -0.18 0.02 0.19 25

Slovakia -0.16 -0.02 -0.04 -0.25 -0.06 0.37 11

Latvia -0.03 0.25 0.24 0.03 0.21 0.33 10

Lithuania 0.14 0.30 0.31 0.20 0.60 -0.04 8

Croatia -0.17 -0.13 0.29 -0.51 -0.13 0.45 7

Russian Federation 0.07 0.44 1.14 -0.19 -0.31 0.50 22

Turkey 0.44 0.03 0.60 -1.66 -1.82 0.31 11

Serbia 0.16 -0.02 0.37 -0.32 -0.21 0.54 6

Bulgaria -0.01 0.02 0.32 -0.11 0.07 0.65 15

Romania 0.02 0.43 0.28 0.21 0.21 1.00 17

Montenegro 0.13 0.28 0.59 0.08 -0.05 0.98 3

Kazakhstan 0.41 0.78 1.22 -0.51 -0.54 0.41 13

Azerbaijan 0.53 0.57 0.72 0.83 0.14 0.87 23

Kyrgyzstan 0.27 0.35 0.89 -0.31 -0.53 0.33 24

Albania 0.67 0.53 0.59 0.84 0.51 0.71 26

OECD average 0.00 0.00 -0.00 -0.06 -0.09 -0.02 19


0.07 0.19 0.43 -0.11 -0.04 0.44 15

EU8 average -0.14 0.04 0.15 -0.06 0.23 0.19 16

CEE/CIS average 0.23 0.30 0.64 -0.15 -0.24 0.61 15


-0.07 -0.19 -0.43 0.05 -0.05 -0.45 3

Countries are ranked by country’s average performance on the three literacy scales. Source: OECD 2010 Tables IV.4.1, IV.4.2, IV.4.3, IV.4.4, IV.4.5, IV.4.8, IV.4.7.

The index of teacher-student relations is based on students’ agreement with the statements: ‘I get along well with most of my teachers’; ‘Most of my teachers are interested in my well-being’; ‘Most of my teachers really listen to what I have to say’; ‘If I need extra help, I will receive it from my teachers’; and ‘Most of my teachers treat me fairly’. The index of disciplinary climate is based on students reporting on the frequency of the following: ‘Students don’t listen to what the teacher says’; ‘There is noise and disorder’; ‘The teacher has to wait a long time for the students to quieten down’; ‘Students cannot work well’; and ‘Students don’t start working for a long time after the lesson begins’. The index of teachers’ stimulation of students’ reading engagement

ANNEX 2

139

is based on students reporting on the frequency of the following: ‘The teacher asks students to explain the meaning of a text’; ‘The teacher asks questions that challenge students to get a better understanding of a text’; ‘The teacher gives students enough time to think about their answers’; ‘The teacher recommends a book or author to read’; ‘The teacher encourages students to express their opinion about a text’; ‘The teacher helps students relate the stories they read to their lives’; and ‘The teacher shows students how the information in texts builds on what they already know’. The index of student-related factors affecting school climate is based on principals’ reports on whether student learning was hindered by ‘Student absenteeism’; ‘Disruption of classes by students’; ‘Students skipping classes’; ‘Students lacking respect for teachers’; ‘Student use of alcohol or illegal drugs’; and ‘Students intimidating or bullying other students’. The index of teacher-related factors affecting school climate is based on principals’ reports on whether student learning was hindered by ‘Teachers’ low expectations of students’; ‘Poor student-teacher relations’; ‘Teachers not meeting individual students’ needs’; ‘Teacher absenteeism’; ‘Staff resisting change’; ‘Teachers being too strict with students’; or ‘Students not being encouraged to achieve their full potential’. Last column refers to percentage of school principals’ reports that parental expectations are characterised by pressure on the school to achieve high academic standards among students from many parents.

figure 32 mean performance in pIsa 2009 vs. pIrls 2006 (4th grade).

PISA

PIR

LS

300 400 500 600300

400

500

600

The former Yugoslav Republic of Macedonia

GeorgiaRomania

Moldova

BulgariaLithuania

SlovakiaSlovenia

RussiaHungary

Latvia

Poland

Red lozenges are study focus countries participating in both PISA and PIRLS. Crosses on the vertical axis show the results in PIRLS of the CEE/CIS countries which did not participate in PISA. Blue squares are other countries which participated in both surveys. Source: OECD 2010 vol.1 Table I.A.; Mullis et al. 2007 Exhibit 1.2.

140



figure 33 mean performance in pIsa 2009 vs. TImss 2007 (8th grade).

300 400 500 600300

400

500

600

PISA

TIM

SS

8th

Armenia

Ukraine

Bosnia & Her.

Georgia

Turkey

RomaniaBulgaria

Serbia

LithuaniaRussia

CzechRHungary

Slovenia

Red lozenges are study focus countries participating in both PISA and PIRLS. Crosses on the vertical axis show the results in TIMSS of the CEE/CIS countries which did not participate in PISA. Blue squares are other countries which participated in both surveys. Source: OECD 2010 vol.1 Table I.A.; Martin et al. 2008, Mullis et al. 2008 Exhibit 1.2.

ANNEX 2

141

figure 34 mean performance in pIsa 2009 vs. TImss 2007 (4th grade).

200 300 400

PISA

TIM

SS

4th

500 600200

300

400

500Slovakia

CzechR

Hungary

Slovenia

Kazakhstan

Georgia

Armenia

Ukraine

Latvia

Lithuania

Russia

600

Red lozenges are study focus countries participating in both PISA and PIRLS. Crosses on the vertical axis show the results in TIMSS of the CEE/CIS countries which did not participate in PISA. Blue squares are other countries which participated in both surveys. Source: OECD 2010 vol.1 Table I.A.; Martin et al. 2008, Mullis et al. 2008 Exhibit 1.2.

Commissioned by the UNICEF Regional

Office for Central and Eastern Europe and

the Commonwealth of Independent States

(UNICEF RO CEE/CIS)

equity in learning?

Documents