opening the ‘black box’ was melbourne 2 oct 2015
TRANSCRIPT
OPENING THE ‘BLACK BOX’: ASSESSING TEACHING AND LEARNING
EXCELLENCE AS A GAME-CHANGE
Dirk Van DammeOECD/EDU/IMEP
2
Inputs Black box Outputs
3
Inputs Black box Outputs
4
Financial inputs in higher education increasing
• Total (public & private) financial investment grew– Between 2005 and 2011 on average across
OECD increase of 10% in per student expenditure and 27% in total expenditure
– With huge differences between countries, increases higher in countries with below-average expenditure, catching up
– Yearly per student expenditure is now 14 KUS$– Total expenditure increased from 1.3% GDP in
2000 to 1.6% GDP in 2011
Inputs
5
-30 -20 -10 0 10 20 30 40 50 60 706 000
10 000
14 000
18 000
22 000
26 000 United States
Switzerland
DenmarkSweden
Norway FinlandNetherlands Germany
JapanIreland
BelgiumFranceAustria
Spain
Israel
United Kingdom
BrazilItaly
KoreaPolandPortugalCzech Republic
HungaryChileSlovak RepublicMexico Estonia
Iceland
Russian Federation
New Zealand
SloveniaR² = 0.0789910852395619
Change in expenditure per student between 2005 and 2011 (%)
Annual expenditure per student (2011,
USD)
OECD aver-age
OECD aver-age
Financial inputs in higher education increasingInputs
6
• Private expenditure has increased a lot– 31% of total expenditure (0.5% GDP) comes from
private sources, mainly tuition fees– Increase from 25% in 2000– Total private expenditure increased with 32%
since 2005– >50% in Israel, US, Australia, Japan, UK, Korea
and Chile
Financial inputs in higher education increasingInputs
7
• Increase in total per student expenditure slows down since crisis– Negative growth in almost half of countries
between 2008 and 2011– Expenditure cannot catch up with increasing
student numbers• Increasing concerns about levels of private
expenditure, student debt– 5 million international students investing even
much higher amounts of money
But strong signs of stagnating fundingInputs
8
Estonia
Slovak Republic
Chile
Hungary
Korea
Czech Republic
Finland
Slovenia
Denmark
Russian Federation
Israel
Japan
United Kingdom
Italy
Poland
OECD averageEU21 average
NetherlandsSwitzerland
France
Sweden
GermanyAustralia
BrazilSpain
NorwayMexico
Belgium
PortugalAustria
United States
IrelandIceland
80
90
10 0
11 0
12 0
13 0
14 0
15 0
Chan ge i n ex pen di tu re Chan ge i n the nu mbe r o f stu den ts (i n ful l-t ime e qui va le nts) Cha nge i n ex pe ndi ture pe r s tu de nt
In dex of c han ge (2 00 8= 100 )
Chart B1.6. Changes in the num be r of students a nd c ha nge s in e x pe nditure per s tude nt by educ ationa l ins titutions , by le ve l of educa tion (2 00 5 , 2 01 0)In d ex o f c ha ng e b etwee n 2 0 05 a n d 20 10 (20 05 = 10 0 , 2 01 0 co ns ta n t p ric es )
But strong signs of stagnating fundingInputs
9
But strong signs of stagnating fundingInputs
• Efficiency and value-for-money become very important policy considerations– Both for governments and students/families– Cost of higher education becoming political issue
in many countries• What are students actually ‘buying’?
– Very weak relationship between cost and actual ‘product’, benefits and outcomes
– Value-for-money depends enormously on institution and field of study
10
Inputs Black box Outputs
11
Inputs Black box Outputs
12
• Share of tertiary educated in the population increased, but still huge disparities across countries– 34% of adult population in OECD now has a
tertiary qualification, up from 22% in 2000
Outputs Increases in graduate output
13
Outputs Increases in graduate outputCh
ina
Sout
h Af
rica
Indo
nesia
Braz
ilTu
rkey
Italy
Chile
Mex
ico
Port
ugal
Slov
ak R
epub
licCz
ech
Repu
blic
Colo
mbi
aAu
stria
Hung
ary
Pola
ndSl
oven
iaGr
eece
Germ
any
Latv
iaEU
21 a
vera
geFr
ance
Spai
nO
ECD
aver
age
Neth
erla
nds
Denm
ark
Icel
and
Belg
ium
Swed
enSw
itzer
land
Esto
nia
Norw
ayLu
xem
bour
gFi
nlan
dIre
land
New
Zea
land
Unite
d Ki
ngdo
mAu
stra
liaKo
rea
Unite
d St
ates
Israe
lJa
pan
Cana
daRu
ssia
n Fe
dera
tion
0
10
20
30
40
50
602000 2012%
14
• Employment benefits remain high• On average across OECD academic degree gives a
wage premium of 75% compared to upper secondary educated workers
• Total net present value over lifetime is around 160 000 US$ for man, 100 000 US$ for woman– But with huge differences across fields of study
• Or internal rate of return of 16%• High social benefits in health, interpersonal trust,
volunteering, political participation
Outputs Employment, earnings and social benefits remain high
15
Outputs Employment, earnings and social benefits remain high
Turk
eyD
enm
ark
Spa
inE
ston
iaS
wed
enN
ew Z
eala
ndG
reec
eK
orea
Japa
nC
anad
aS
lova
k R
epub
licP
olan
dN
orw
ayIs
rael
Cze
ch R
epub
licFr
ance
Aus
tralia
Finl
and
OE
CD
ave
rage
Por
tuga
lE
U21
ave
rage
Aus
tria
Uni
ted
Kin
gdom
Net
herla
nds
Italy
Bel
gium
Slo
veni
aG
erm
any
Uni
ted
Sta
tes
Hun
gary
Irela
nd
0
50 000
100 000
150 000
200 000
250 000
300 000
350 000
400 000
450 000
500 000
Private net returns Public net returns
Equ
ival
ent U
SD
16
• Is graduate output higher than the economy’s need for high-skilled labour?– Graduate unemployment– Filtering-down effect?– Over-qualification and over-skilling– Huge field-of-study mismatches
• Is polarization in labour markets, with high employment/high earnings because of skill-biased technological change going to last?
Outputs But increasing concerns about quality of output
17
Outputs But increasing concerns about quality of output
18
Outputs But increasing concerns about quality of output
• Enormous differences in the skills equivalent of tertiary qualifications– Foundation skills such as literacy and numeracy– Can societies accept that higher education
graduates have low literacy and numeracy skills?
19
Outputs But increasing concerns about quality of output
20
Japa
n
Finl
and
Net
herla
nds
Sw
eden
Aus
tralia
Nor
way
Flan
ders
(Bel
gium
)
Eng
land
(UK
)
Eng
land
/N. I
rela
nd (U
K)
Uni
ted
Sta
tes
Cze
ch R
epub
lic
OE
CD
ave
rage
Pol
and
Can
ada
Nor
ther
n Ire
land
(UK
)
Aus
tria
Ger
man
y
Irela
nd
Fran
ce
Den
mar
k
Est
onia
Slo
vak
Rep
ublic
Kor
ea
Rus
sian
Fed
erat
ion
Spa
in
Italy
0
5
10
15
20
25
30
35
40
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education
Proportion of 25-64 year-olds scoring at PIAAC numeracy level 4 and 5, by educational attainment of the population (2012)
Outputs But increasing concerns about quality of output
21
Outputs But increasing concerns about quality of output
• Concerns about the quality and added-value of a university experience– Academically Adrift: limited improvement in
academic skills– What is the relative contribution of selection
versus teaching and learning in the production of high-quality graduates; what is the actual ‘learning gain’
– Doubts on the quality of the teaching and learning experience at elite universities
22
Outputs But increasing concerns about quality of output
23
Inputs Black box Outputs
24
InputsIncreasing
investments by
taxpayers and families,
but concerns
about value-for-money
Black
box
Outputs
High numbers of graduates,
still high returns, but concerns
about quality
25
InputsIncreasing
investments by
taxpayers and families,
but concerns
about value-for-money
Black box
Outputs
High numbers of graduates,
still high returns, but concerns
about quality
?
26
• What do we know about how quality of teaching and learning results in high-quality output, and socially interesting outcomes?
• Information asymmetry: both public and private financers of higher education have very little understanding of what they actually are spending money for
• Increase of investment has not been accompanied by an empowerment of the input side to make smart choices through better information
Black box
Black box problem is problem of transparency
27
• How to improve transparency?– Performance management systems: essentially
bureaucratic control systems, very rarely comparative and tuned to better decision-making
– Quality assurance arrangements: increasingly relativistic (‘fitness-for-purpose’), bureaucratic and inward oriented (internal QA), unfit for external transparency, nor empowering students
– Student satisfaction surveys: empowering, but perception-based, largely unrelated to teaching and learning quality
Black box
Black box problem is problem of transparency
28
• How to improve transparency?– Input measures as proxies for teaching and
learning quality: above a threshold level input has a weak relationship to quality (cfr PISA)
– Research measurement and bibliometric indicators as proxies for teaching and learning quality: most inaccurate, unfair to wide range of institutions and institutional diversity
– Reputations: act as (too?) powerful tools, also in rankings, but inaccurate, often outdated, inimical to newcomers and innovators
Black box
Black box problem is problem of transparency
29
• Sound metrics of teaching and learning are very much needed– To reassure governments and families about the value-
for-money of investments– To reward institutions who invest in improving teaching
and learning and are not compensated through other measures
– To value institutional diversification– To reward and foster quality improvement through
mutual learning– To compensate for the over-reliance of rankings on
research and reputation metrics
Black box
Black box problem is problem of transparency
30
• What are the risks of not improving transparency?– Erosion of the symbolic power of degrees, the only
monopoly of the higher education sector• Employers turning to alternative modes of selection• Emergence of alternative modes of qualification (employer
credentials, badges, recognition of prior learning, etc.)
Black box
Black box problem is problem of transparency
31
Black box
Black box problem is problem of transparency
32
• What are the risks of not improving transparency?– Blaming the student becomes the main excuse– Decreasing trust of governments, employers, families
and wider society in the value of higher education• Degree inflation• Concerns about declining standards in some countries
– Gradual erosion of the financial health of higher education institutions if value-for-money concerns are left unanswered
• Financial bubbles of student debt
– Markets no longer accept non-transparency• Cfr Volkswagen
Black box
Black box problem is problem of transparency
33
InputsIncreasing
investments by
taxpayers and families,
but concerns
about value-for-money
Black box
Outputs
High numbers of graduates,
still high returns, but concerns
about quality
?
34
InputsIncreasing
investments by
taxpayers and families,
but concerns
about value-for-money
Black box
Outputs
High numbers of graduates,
still high returns, but concerns
about quality
Opening the black box
35
• We need to better understand what happens in the teaching and learning environments in universities– With the investment inputs provided– Explaining the benefits and returns and
understanding why they sometimes fail– In order to incentivize improvements in teaching
and learning– Empower students and families– Restore trust
Black box Opening the black box
36
• Comparative assessment of learning outcomes of graduates is the most promising approach to measure teaching and learning excellence– OECD’s AHELO project– National research projects in Germany, UK, Italy– CLA and various other initiatives in US– OECD-CEA partnership to implement CLA+ in
countries– European Commission supported CALOHEE
project in Tuning framework
Black box Opening the black box
37
• Strong resistance by parts of the academic community, but do they have a strong case?– No consensus on academic skills that matter– Risk of standardization– Institutional diversity too large to use limited number of
metrics– Methodological concerns– Cost and burden
• …exactly the same arguments used 20 years ago when the PISA programme was born
• …and very similar to arguments used 10 years ago against measuring research excellence
Black box Opening the black box
38
• Developing reliable metrics of teaching and learning excellence in universities is the next big systemic challenge in the development of higher education worldwide
• In the short term universities might think it’s not in their interest and that non-transparency is the better option
• But in the longer term that might be a very risky approach, in which the costs largely exceed the short-term profits
Black box Opening the black box
39
• The future progress of universities will not come from– Continued massification and ever higher numbers of
students– Continued increases in public and private funding
• But will be the outcome of better and more visible quality improvements in teaching and learning
• The impact of teaching and learning excellence will be similar to the way research excellence has changed the system
Finally
40
InputsExcellent teaching
and learning
Outputs
Opening the black box