how pisa came to rule the world.docx

8/10/2019 How Pisa came to rule the world.docx

1/5

How Pisa came to rule the world

feature | Published in TES magazine on 6 December, 2013| By: William Stewart

Last Updated:

6 December, 2013 Section:

feature

As the international studys latest results are released, its grip on

policymakers and school systems is tightening. But with great powercomes great responsibility, William Stewart reports

This week was a momentous one for governments across the globe as the latest, long-awaited results

from the worlds most influential international education study were published. The findings of the

Programme for International Student Assessment (Pisa) may come out only once every three years but

they are increasingly a very big deal indeed.

Even back at the beginning, however, Pisa had the power to make national leaders pay attention. The

first edition of the survey plunged a leading nation into a state of collective trauma over standards in its

schools (Germanys 2001 Pisa shock). And since then its influence has only grown. Pisa results have

been used to justify sweeping controversial reforms in England since 2010 and today are seen by a

growing number of countries as a guide to how to create the perfect school system.

More than half a million 15-year-olds in 65 different jurisdictions took the most recent Pisa tests last year.

But their performances will have an impact on many more students, some in countries that did not even

participate. Today, Pisa is at the centre of what has been dubbed by two US-based academics as a new

system of global education governance.

Pisa seems to be well on its way to being institutionalised as the main engine in the global accountabilityjuggernaut, which measures, classifies and ranks students, educators and school systems from diverse

cultures and countries using the same s tandardised benchmarks, Heinz-Dieter Meyer and Aaron

Benavot write in a new book, Pisa, Power, and Policy: the emergence of global educational governance.

The professors from the University at Albany, State University of New York argue that Pisa has allowed

the Organisation for Economic Cooperation and Development (OECD), which runs the programme, to

begin assuming a new institutional role as arbiter of global education governance, simultaneously acting

as diagnostician, judge and policy adviser to the worlds school systems.

The OECD readily acknowledges the reach of Pisa, noting that many countries now set goals and

benchmarks based on its results. But it emphasises the help the study can give to education ministers by

revealing what is possible rather than the power it has over their fates.

Few people would oppose the notion that collecting comparative data on the performance of national

school systems could prove useful. As Sir Michael Barber, who was education adviser to former British

prime minister Tony Blair when he was in office, told TES last year: It would be really surprising and

actually quite bizarre in an education field to argue that not researching something and not learning about

it is a better way to go than learning about it.


2/5

But once you go beyond that basic idea and look at what it might mean in detail, many potential problems

emerge. Chief among them must be the question of whether it is even possible to make useful

comparisons between such a variety of different countries. In the words of one critic, Professor Svend

Kreiner, a Danish academic who has challenged the statistical methods used for Pisa, it is meaningless

to try to compare reading in Chinese with reading in Danish.

Then there are the questions of exactly what should be measured, who decides and how they are heldaccountable. These will be asked more and more frequently as Pisa expands its activity. The latest round

of the assessment included a computer-based problem-solving test for the first time, alongside the usual

maths, reading and science tests. In 2015 that will be extended to include collaborative problem-solving.

Perhaps more significantly, a Pisa-based test for individual schools will be available from next year in the

UK, the US and Spain.

Cold War origins

It is not just the growing scope of Pisa that will lead to greater scrutiny. The programmes increasing

power and influence could present its own issues. There is already evidence that some countries - such

as Wales, which has gone through its own Pisa shock - are starting to tailor their education systemsspecifically to improve their Pisa rankings.

What began as an attempt at objective assessment is starting to become an active agent for change in

the same school systems it is trying to measure.

So how did things get to this point? Professor Daniel Trohler, from the University of Luxembourg, traces

Pisas origins all the way back to the Cold War and the launch of the Sputnik satellite by the Soviet Union

in 1957. The unprecedented feat of sending a 58cm-diameter metal sphere into space punctured the

Wests sense of superiority and triggered a crisis of confidence in school standards in the US. The federal

government began to pour money into education. But the control that local authorities were guaranteed

over state schools in the US presented the administration with a problem.

To invest billions of dollars and not be sure about the effects was understandably unsatisfying, Trohler

writes in his article The OECD and Cold War Culture: thinking historically about Pisa in Pisa, Power, and

Policy. If the federal government could not govern directly, it at least wanted to see what effects its

incentives had, and for this purpose a test instrument had to be developed.

The result, he says, was the 1964 proposal for a National Assessment of Educational Progress, which

developed tools of comparative testing that were used 35 years later at a global level in Pisa.

The first of these US national assessments took place in 1969, by which time the OECD was already

eight years old. The organisation, which defines itself as a forum of countries with a shared commitment

to market economies backed by democratic institutions, always saw education as a key part of its role.

But according to Trohler, it was only pressure from the US, and its threat to withdraw funding, that led tothe OECD pursuing the idea of an international comparative education study in the early 1980s. This was

despite opposition from most member states, who seemed to doubt the feasibility and usefulness of

international comparisons, he explains.

Stephen Heyneman, an education expert who worked for the World Bank for two decades, says that the

OECDs own directors of education research reacted with shock and deep suspicion to the plan to

collect education statistics. They believed it was unprofessional to try to quantify such indicators, and

that it would oversimplify and misrepresent OECD systems.


3/5

But it was the US view that prevailed, paving the way for the eventual publication of the first Pisa report in

2001.

This long gestation period meant that Pisa was not actually the first set of international tests to compare

school systems: the Trends in International Mathematics and Science Study (Timss) published its initial

results in 1995. But it is Pisa that has proved to be the most influential programme. One reason could be

that it comes out more often: every three years, compared with Timss four-year cycle.

The latter is run by the International Association for the Evaluation of Educational Achievement (IEA). In

2001, the IEA, an association of national research institutions and government research agencies, also

launched a reading test, the Progress in International Reading Literacy Study (Pirls), which comes out

every five years. But again this has a lower profile than Pisa.

Gabriel Sahlgren, research director at the Centre for Market Reform of Education in London, believes that

this is because Pisa has had better marketing and the OECD is better known than the IEA. But, he says,

both (sets of studies) are probably valuable in their own right.

Knowledge and real-life problems

Many agree with Sahlgren and view the assessments, which have important distinctions, as

complementary. Pisa is age-specific, testing 15-year- olds. For the IEA, it is the number of years that

students have attended school that is fixed. Timss tests fourth- and eighth-graders - roughly aged 9-10

and 13-14 - while Pirls is restricted to the fourth grade.

Even more significantly, whereas Timss and Pirls test topics that students are likely to have been taught

as part of their school curriculum, Pisa looks at how well students can apply knowledge to real-life

situations and problems.

This approach fits with the OECDs economic focus. The very meaning of public education is b eing

recast from a project aimed at forming national citizens and nurturing social solidarity to a project driven

by economic demands and labour-market orientation, Meyer and Benavot write.

More prosaically, this means that Pisa is not really a test of school effectiveness. Andreas Schleicher, the

senior OECD official who runs Pisa, acknowledged this point in a TES interview last year. There are

many different forms of students work - school is one, (but) it can be private tutoring, it can be learning

reading outside of school with parents - and we should look at this holistically, he said.

So, if Pisa is measuring what is going on in homes as well as schools, then could it be said that it is not a

fair judge of the school system?

I think that is true,Schleicher admitted. I agree with the criticism that you cant say that the school

system is entirely responsible for Pisa results.

That is a potentially awkward fact for a programme that is increasingly being seen as a measure of theeffectiveness of school systems, rather than the wider societal factors they contend with. And it could

become more awkward as the OECD begins to market Pisa - designed to assess the ability of students

across entire countries - as a way of rating schools.

But for Sahlgren the fact helps to explain some of the seeming inconsistencies with rival surveys. He uses

the example of Finland, a perennial Pisa high-flyer. The first time the country entered Timss in 1999, it

was placed at 14th for secondary students in maths. That was with seventh-grade students rather than

the eighth-graders entered by other countries. Nevertheless, Finlands next Timss entry in 2011 - it


4/5

decided not to participate in 2003 and 2007 - did not provide much encouragement. Again it entered

seventh-graders, but the average score they achieved in the test fell from 520 to 482.

Sahlgren argues that this is because the maths curriculum taught in Finlands schools has ignored

certain concepts that are not tested in Pisa. He adds: If you compare the Pisa mathematics with the

Timss mathematics, the Timss mathematics is much more relevant for higher studies in mathematically

intensive subjects like engineering, which are going to be very important for countries futures.

None of this has stopped educational tourists from around the world flocking to Finland to find out the

secret of its success. Because among todays ever more globally aware education policymakers, Pisa

rankings are increasingly being seen as the main measure of success.

Overrated rankings?

The way these rankings are calculated is coming in for increasing criticism, with academics including

Kreiner and Dr Hugh Morrison, from Queens University Belfast, arguing that flaws in the statistical

methods employed mean that the results are useless and meaningless.

Schleicher has mounted a vigorous defence. But it is not just the views of sceptical academics that shouldbe triggering concerns about the emphasis now placed on countries exact positions in the Pisa tables.

The OECD has said that large variation in single-ranking positions is likely because of the uncertainty

that results from sample data.

But it still publishes the league tables that become the focus of media and political discussion, giving Pisa

its high profile.

John Bangs, chair of the OECD Trade Union Advisory Committees working group on education, wants

the organisation to scrap the tables and start putting countries together in broad blocks or bands.

The OECD has always argued that the rankings are not the main point of Pisa, he says. But the reality

is that every country looks at the rankings even if the difference is a few points. It is turning into a football

league.

The OECD might say, It is not us, it is the way the countries use the results. But I would say that i t has

to do much, much more to stop Pisa being used for crude rankings.

Bangs also has major concerns about the Pisa-Based Test for Schools now being introduced by the

OECD. It should not fall into the trap of thinking it has got a marketable product that can be given to

schools, he says. The tests are supposed to be a system -wide evaluation.

He predicts the emergence of new elite league tables comprised of schools - most likely private ones -

that can afford to run the Pisa test. If the OECD wants toget into the test market for individual schools,

then they ought to do something entirely different and not use the tests developed for a system-level

approach, Bangs says.

The power of the Pisa brand will probably ensure that the school tests are a success despite those

concerns. But as Pisa grows, so will scrutiny of its findings and methods.

Only last month, TES revealed serious academic concerns about another crucial aspect of Pisa: the

quality of the contextual information submitted by schools participating in the programme (Fresh Pisa

claims shake trust in influential study, 15 November). There is also the risk that the higher the Pisa


5/5

stakes become, the more governments will try to do everything possible to improve their position in the

rankings.

Schleicher has admitted that in the past some countries, including Germany, Switzerland and Spain,

have tried to game the system. The OECD says that the way the study is set up means it is impossible

for them to succeed. But that is unlikely to stop further attempts in the future.

Today, Pisa has an unprecedented level of influence over global education. But it could be about to face

its biggest test yet.

how pisa came to rule the world.docx

Documents