Australian Government/ HEFCE/ European

Commission

16 October 2015, Singapore

Measuring Quality Outcomes in

Higher Education

Simon Marginson

Professor of International Higher Education

Director, ESRC/HEFCE Centre for Global Higher Education (Nov

2015)

UCL Institute of Education

University College London, UK

Coverage

• High Participation Systems (HPS) of higher

education: Participation and system stratification

• Research environment

• What are the systemic drivers of quality operating at

global level?

• Present global rankings

• The lacuna: Credible comparative metrics on learning

in higher education

• Problems of proxies and indirect measures

• Post-AHELO landscape

A time of movement and change in world higher

education

Participation is growing at 1% a year Gross Tertiary Enrolment Ratio 1970-2012

(UNESCO 2015)

1010101011111212121212131313131313

13131314141414151516

1717181920

2122232425

26272829

3132

0

10

20

30

40

50

60

70

80

90

1970

1971

1972

1973

1974

1975

1976

1977

1978

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

World NorthAmericaandWesternEurope Sub-SaharanAfrica

Same trend across the globe:

GTER

by world region, 1995/2012

15

4

6

23

14

10

17

33

60

32

8

23

25

26

31

43

71

79

0 10 20 30 40 50 60 70 80 90

WORLD

Sub-Saharan Africa

South & West Asia

Central Asia

Arab States

East Asia & Pacific

Latin America &

Carribean

Central & Eastern

Europe

North America &

Western Europe

2012 1995

Growth in participation to come World and Asian middle class 2009-2030 (billions)

Source: Brookings / OECD projection in 2010

Middle class persons are defined as persons living on USD $10-100 per

day, PPP

1.85

3.25

4.88

0.53

1.74

3.23

0

1

2

3

4

5

2009 2020 2030

World Asia

Tendency towards bifurcation and

stratification of HPS

Elite HEIs ‘student

selecting’

upward push (aspirations)

downward pull

(scarcity of resources and

status)

Middle Sector

Non-elite ‘demand

absorbing’

Stratifying effects in system design

• Competition

• World-Class University movement

• Under-regulation of quality in mass HE sectors

• More fragmented and diverse offerings, cross-border etc

• Indifference to equity issues (access to elite HEIs,

cognitive formation in mass sector, etc)

BUT POLICY COUNTER-WEIGHTS

• Horizontal system design features, system architectures

• Common mission designations (‘research university’)

• Sector-wide promotion nationally and offshore

• Funding parcels, nuanced specialisations sustain middle

sector

Students enrolled outside their country of

citizenship, millions, 1975-2012 OECD data, 2014

0.81.1 1.1

1.31.7

2.1

3.0

4.14.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1975 1980 1985 1990 1995 2000 2005 2010 2012

Students enrolled outside their country of

citizenship, millions, 1975-2012 OECD data, 2014

0.81.1 1.1

1.31.7

2.1

3.0

4.14.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1975 1980 1985 1990 1995 2000 2005 2010 2012

USA, 16%

UK, 13%

Australia, 6%

Germany, 6%

France, 6%Canada, 5%

Russia, 4%

Japan, 3%Spain, 2%

China, 2%

New Zealand, 2%

Italy, 2%

Austria, 2%

South Africa, 2%

Korea, 1%

Switzerland, 1%

Belgium, 1%

Netherlands, 1%

other OECD, 8%

other non-OECD,

17%

Leiden Ranking world top 15 Universities ranked 1-15 in world Papers

2010-2013

% papers in

top 10%

Number of papers in

top 1% top 10%

1 Harvard U USA 31,137 22.1 1026 6892

2 Stanford U USA 14,102 21.9 442 3083

3 U Toronto Canada 19,948 13.7 289 2738

4 U Michigan USA 17,283 15.1 264 2616

5 U California, Berkeley USA 11,804 21.8 360 2573

6 MIT USA 10,040 24.8 400 2486

7 U California, Los Angeles USA 14,002 17.4 301 2438

8 Johns Hopkins U USA 14,850 15.8 293 2348

9 U Oxford UK 12,935 17.8 293 2301

10 U Washington, Seattle USA 13,716 16.6 267 2276

11 U Pennsylvania USA 12,649 17.2 269 2178

12 U California San Diego USA 11,707 18.1 276 2124

13 U Cambridge UK 12,170 17.3 279 2100

14 Columbia U USA 11,807 17.5 261 2064

15 U California, S.

Francisco

USA 10,199 19.8 264 2017

Stronger research universities in Europe

European universities in ARWU top 80 in

2004 (14)

European universities in ARWU top 80 in

2015 (19)

27

39

41

45

46

48

51

57

59

63

68

72

74

79

Fed Instit Tech Zurich SWITZERLAND

U Utrecht NETHERLANDS

Paris 6 P&M Curie FRANCE

TU Munich GERMANY

Karolinska Instit SWEDEN

Paris 11 Sud FRANCE

U Munich GERMANY

U Zurich SWITZERLAND

U Copenhagen DENMARK

Leiden U NETHERLANDS

U Oslo NORWAY

U Helsinki FINLAND

Uppsala U SWEDEN

U Goettingen GERMANY

20

35

36

41

46

48

51

52

54

56

58 eq

58 eq

61 eq

67 eq

71

72

73

75

Fed Instit Tech Zurich SWITZERLAND

U Copenhagen DENMARK

Paris 6 P&M Curie FRANCE

Paris 11 Sud FRANCE

Heidelberg U GERMANY

Karolinska Instit SWEDEN

TU Munich GERMANY

U Munich GERMANY

U Zurich SWITZERLAND

U Utrecht NETHERLANDS

U Geneva SWITZERLAND

U Oslo NORWAY

Uppsala U SWEDEN

U Helsinki FINLAND

Ghent U BELGIUM

Ecole Normale Superieure FRANCE

Aarus U DENMARK

U Groningen NETHERLANDS

Stockholm U SWEDEN

Dynamism in China and Singapore: top 10% papers 2006-09 to 2010-13

(Leiden)

0

200

400

600

800

1000

1200

1400

2006-09 2007-10 2008-11 2009-12 2010-13

NU Singapore Nanyang UT Tsinhgua U Zhejiang U Peking U

Growth top 10% papers, 2006-09 to 2010-

13

university system 2006-09 2007-10 2008-11 2009-12 2010-13 growth

NU Singapore SINGAPORE 1042 1094 1173 1264 1374 31.9%

Nanyang TU SINGAPORE 568 640 776 910 1103 94.2%

Tsinghua U CHINA 819 875 953 1031 1217 48.6%

Zhejiang U CHINA 730 780 896 994 1182 61.9%

Peking U CHINA 622 705 773 867 1026 65,0%

Shanghai JT U CHINA 664 698 771 901 1020 53.6%

Fudan U CHINA 469 536 638 727 891 90.0%

U S&T China CHINA 503 509 536 576 675 34.2%

U Hong Kong HONG KONG 558 578 622 643 661 18.5%

Seoul National U KOREA 742 768 812 911 984 32.6%

National Taiwan

U

TAIWAN 604 613 647 660 691 14.4%

MIT USA 2091 2142 2260 2391 2486 18.9%

U Cambridge UK 1796 1867 1975 2080 2100 16.9%

What drives quality?

• Top end competition for research rankings (takes in only

top 500, i.e. T1 universities and aspirants)—generates

continuous increase in paper outputs and improved cite

rates

• Competition for foreign students, plus national QA and

management of student satisfaction—generates

continuous improvement in services for foreign students

• National competition, QA and management of student

satisfaction generates improvement in student servicing

in some HPS but absence of a global dynamic means

this is weaker than research effects

• Nothing drives continuous improvement in student

learning

Global rankings and quality

• ARWU big science. Works. Leiden and Scimago

allow fine-tuning of research management (REAL

METRICS MATTER)

• Times Higher more than two thirds research

driven

• QS and TH surveys no link to performance. No

inherent quality driver (there’s a marketing driver)

• Multi-indicator rankings of QS and TH incoherent,

no validity, no link to performance (separate indicators

would be OK)

• Teaching proxies are staffing ratios (???) and

surveys of ‘who is good at teaching’ (?????? Why do we

tolerate this?)

• U-Multi-rank provides best information,

performance drivers are less clear (dependence on

surveys)

Lacuna in learning measures in HE

• Impact of PISA as a driver of performance

• Not transparency in learning function

• No useful information to guide choice

• No performance drivers for T2 and T3 HEIs

• In the context of research-led competition and

absence of learning achievement measures talk

about parity (or primacy) of teaching is vapid

• Proxies create wrong incentives and we have no

way of telling they create right incentives

• Real, grounded metrics matter. Nothing else

works

Proxies don’t get us there

• Resource measures no necessary relation to quality

• Graduate employment data are important but do not give

us learning quality (human capital theory is a metaphor not

an individualisable technology and it cannot be reverse

engineered)

• Indirect outcomes assessment (student surveys of

learning behaviour, student satisfaction, student

engagement) etc a small part of what we need, BUT

• Management by satisfaction drives down cognitive

formation, standards (Arum & Roksa 2014, Armstrong & Hamilton

2014): ‘most students now believe it is not what you know it

is who you know’

• Cognitive formation does not sit well with consumer

sovereignty but is crucial to bright poor student

Measuring learning outcomes

• Three aspects

- general cognitive formation, CLA-type tests

- discipline-based learning

- work-related generic skills, nested in each occupation

• Robust faculty cultures to build discipline

measures and work skill measures on cross-

border basis

• Measures cannot provide holistic/total outcomes

• Use separate indicators and resist the pressure

for combined indicator-based tables

• Don’t combine with research measures in single

tables (unless through user customization)

Post-AHELO landscape

• Moving on

• Individual university route can’t work—like

governing on the basis of a UN Security Council

with 100 permanent members who all want to

say ‘no’

• Governments are crucial but only some can

deliver

• Move on CLA-type competency tests in advance

of the rest of the package

• Move in selected countries/regions (East

Asia/Singapore?) but crucial to include WCUs