Leaving Cert Maths and Performance inUniversity
Brendan HalpinDepartment of Sociology
December 1, 2010
What good is Maths?
I What do we want of it?I An important part of a rounded education?I A feed-stock for the knowledge economy
I Should we be worried about performance?
I Anxiety and resistance
Declining LC performance of UL entrants
35
40
45
50
55
60
1998 2000 2002 2004 2006 2008Year first observed
Maths Points Pct Higher
What do we want to know about it?
Key research question: Will encouraging LC HL maths improvethings in university?
I Do (where do) people who perform better at LC Mathsperform better in University?
I How much of this is due to prior ability, and how much is itamenable to policy intervention?
Maths bonus points as a policy lever
I Effective?I Will bonus points raise LC learning?I Will they raise student quality?
I Unfair?I Further penalise disadvantaged students?I Penalise female students?I Penalise the non-mathematical?
Nature of the bonus
0
20
40
60
80
100
120
140
D3 D2 D1 C3 C2 C1 B3 B2 B1 A2 A1
UL bonusOrdinary
HigherNew bonus
Nature of the bonus
I A flat 25 points for D3 plus, rather than 5-40 rising from C3to A1
I A good incentive for average students, but it doesn’t reallyreward excellence
I In so far as candidates respond to the incentive, the 25 pointswill be partly consumed by higher entry points
Structural disadvantage
I Different types of school have different provisionI Disadvantaged schools more likely not to offer higher level
mathsI Single-sex female schools more likely not to offer higher level
maths
I De-facto or innate gender differences in maths aptitude?
Gender and maths: How it Works . . .
Gender and maths: How it Works . . .
Credit: Randall Munroe; http://xkcd.com/385/
Leaving Cert Points and gender differences
40
45
50
55
60
65
70
75
80
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Female IrishMale Irish
Leaving Cert Points and gender differences
40
45
50
55
60
65
70
75
80
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Female EnglishMale English
Leaving Cert Points and gender differences
40
45
50
55
60
65
70
75
80
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Female MathsMale Maths
Leaving Cert Points and gender differences
40
45
50
55
60
65
70
75
80
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
MAS gender 1MAS gender 2
Gender and maths
I LC performance is poor relative to boysI University performance differs by department:
I In MAS, females do systematically better (somewhat maledept)
I In MAE, females do better than males recently (very maledept)
I In PHY, a tie (somewhat male dept)I In ECO, a tie (close to balanced)I In CES, females do better than males recently (somethat male
dept)
I In other words, no evidence of a debilitating mathematicaldisability
Maths and gender, LC grades
0.0
2.0
4.0
6.0
8
F E D3D2D1C3C2C1B3 B2 B1 A2 A1 F E D3D2D1C3C2C1B3 B2 B1 A2 A1
Ordinary Honours
F M
Density
Analysis: the research question
I The specific question I now address is, how much extra doesLC maths performance help us predict third level performance?
I This addresses a part of the larger question, but since LCperformance is due in part to innate ability and in part to the2nd level educational experience, it is not clear that changesin the educational process at 2nd level will cause changes at3rd level
Data and method
I The data:I From the Student Records SystemI Ten years of undergraduate grade data: 1999/2000 to 2008/9I All Autumn and Spring XY4000 grades that are normally
graded (no pass/fail, no Co-op)I Information about age, gender, LC performance, module size
etc.I Thanks to the VPAR, Student Services, QSU and Research
Office
I The method: Multi-level regression analysis with grade QCVas the dependent variable
Multi-level modelling of QCV
I Why analyse the determinants of grade?
I Retention and final award level are much more significantoutcomes
I However, the grade is the “atom” of performance – linksdirectly to the the module, the department (rather thanprogramme) as well as to the individual
I Repeated measurement at individual and module/departmentlevel is extremely useful – assess individual and structuralvariability
Nested structure of the data
Dataset
G1 G2 G3 . . . Gn
Nested structure of the data
Dataset
M1
G1 . . . Gk
M2
Gk+1 . . . Gl
M3
Gl+1 . . . Gn
Nested structure of the data
Dataset
D1
M
G . . . G
M
G . . . G
D2
M
G . . . G
M
G . . . Gn
Nested structure of the data
Dataset
F1
D1
M1
G1 G2
M2
G3 G4
D2
M3
G5 G6
M4
G7 G8
FN
D3
M5
G9 G10
M6
G11 G12
D4
M7
G13 G14
M8
G15 G16
Nesting as structure of teaching and learning
I Modules (within departments) are where the actual work ofteaching and learning and assessing goes on
I Module instances share much context: average level of ability,the specific curriculum, module size etc.
I Departments affect standards, resources, the identity of theteachers, etc.
I MLM that takes this nested structure into account dealscorrectly with higher level observed variables (e.g., modulesize)
I Also deals with higher-level unobserved heterogeneity – e.g.,the extent to modules differ from each other (and arerelatively homogeneous within) due to unobserved factors
Multiple observations per individual also
Dataset
F1
D1
M1
G1 G2 G3
M2
G4 G5
D2
M3
G6 G7 G8
M4
G9 G10 G11
F...
. . .
Multiple observations per individual also
Dataset
F1
D1
M1
G1 G2 G3
M2
G4 G5
D2
M3
G6 G7 G8
M4
G9 G10 G11
F...
. . .
MLM structure at the individual level too
I Repeated measurement at the individual level allows us totake account of individual variables correctly (e.g., same LCpoints for 40 different grades)
I It also accounts for individual-level unobserved heterogeneity,e.g., socio-economic status, motivation, aptitudes, bar bill
I This yields a “cross-classified” multi-level model
I A very important side effect: since individuals crossdepartments we get much more meaningful estimates ofdepartmental effects
Excursus: A web of departments
I Comparing disparate departments is hard: how to compareanalysing a marketing plan, and 18th century French text, orthe vibration pattern of a car exhaust?
I Experimentally we could assign matched samples of studentsto different programmes and compare results
I Observationally, we “control for” age, gender, LC points butdepartment differences could still be accounted for byunobserved factors
Grade pairs
I But since we observe the same students in differentdepartments we can control for this too
I But to what extent is the whole university linked by students?
I We can assess this by looking at each grade and linking it tothe same and other departments by pairing it with thatstudent’s other grades
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
Grade Pairs – an example
��������
����
��������
��������
~
LAW
PPALCS
LCS
LCS LAW PPA
LCS 2 2 2LAW 2 0 1PPA 2 1 0
Web of grades
LC
S
LA
W
SO
C
PP
A
HIS
AC
F
MM
A
PE
R
EC
O
PH
I
PS
Y
NM
I
PE
S
EP
S
LS
C
MS
T
AR
C
CS
I
EC
E
MO
E
PH
Y
MA
E
CE
S
MA
S
AR
T
PL
E
PLE
ART
MAS
CES
MAE
PHY
MOE
ECE
CSI
ARC
MST
LSC
EPS
PES
NMI
PSY
PHI
ECO
PER
MMA
ACF
HIS
PPA
SOC
LAW
LCS
0
2000
4000
6000
8000
10000
12000
14000
Web of grades (rate)
LC
S
LA
W
SO
C
PP
A
HIS
AC
F
MM
A
PE
R
EC
O
PH
I
PS
Y
NM
I
PE
S
EP
S
LS
C
MS
T
AR
C
CS
I
EC
E
MO
E
PH
Y
MA
E
CE
S
MA
S
AR
T
PL
E
PLE
ART
MAS
CES
MAE
PHY
MOE
ECE
CSI
ARC
MST
LSC
EPS
PES
NMI
PSY
PHI
ECO
PER
MMA
ACF
HIS
PPA
SOC
LAW
LCS
0
0.05
0.1
0.15
0.2
0.25
Tightly coupled
I AHSS and KBS are extremely tightly linked
I SEN and EHS are tightly linked but with some structure
I MIC’s two departments are, naturally, detached
I Within UL, only ARC stands out as being isolated
Numbers in the analysis
I 728,590 grades in the working data set, with LC points data
I Where Irish, English and Maths results are known:
I Total number of grades: 614,747I Number of individuals: 22,125I Number of module instances: 10,947I Number of departments: 26
Predicting QCVBase model Plus core LC
subjectsPlus
gender×maths
Intercept 1.270***LC points/100 0.595***
Calendar year -0.009***Male 0.055***Male by year interaction -0.022***Age at entry -0.004Years since entry
1 0.113***2 0.244***3 0.384***4 0.562***5 0.718***
Spring semester -0.014*Module size/100 -0.004Mean module points/100 -0.300***Dept proportion female 0.174
Note: ***: p < 0.005; **: p < 0.01; *: p < 0.05
Predicting QCVBase model Plus core LC
subjectsPlus
gender×maths
Intercept 1.270*** 1.348***LC points/100 0.595*** 0.536***Extra effect, Irish 0.003Extra effect, English -0.062*Extra effect, Maths 0.331***
Calendar year -0.009*** -0.006***Male 0.055*** 0.034***Male by year interaction -0.022*** -0.022***Age at entry -0.004 -0.002Years since entry
1 0.113*** 0.110***2 0.244*** 0.238***3 0.384*** 0.376***4 0.562*** 0.550***5 0.718*** 0.700***
Spring semester -0.014* -0.013*Module size/100 -0.004 -0.004Mean module points/100 -0.300*** -0.302***Dept proportion female 0.174 0.206
Note: ***: p < 0.005; **: p < 0.01; *: p < 0.05
Predicting QCVBase model Plus core LC
subjectsPlus
gender×maths
Intercept 1.270*** 1.348*** 1.362***LC points/100 0.595*** 0.536*** 0.535***Extra effect, Irish 0.003 0.004Extra effect, English -0.062* -0.060*Extra effect, Maths 0.331*** 0.311***Maths/gender interaction 0.049Calendar year -0.009*** -0.006*** -0.006***Male 0.055*** 0.034*** 0.006Male by year interaction -0.022*** -0.022*** -0.022***Age at entry -0.004 -0.002 -0.002Years since entry
1 0.113*** 0.110*** 0.110***2 0.244*** 0.238*** 0.238***3 0.384*** 0.376*** 0.376***4 0.562*** 0.550*** 0.550***5 0.718*** 0.700*** 0.700***
Spring semester -0.014* -0.013* -0.013*Module size/100 -0.004 -0.004 -0.003Mean module points/100 -0.300*** -0.302*** -0.302***Dept proportion female 0.174 0.206 0.206
Note: ***: p < 0.005; **: p < 0.01; *: p < 0.05
Modelling by departmentNo random slope Departmental random
slope
Intercept 1.3478***LC points / 100 0.5362***Extra effect, Irish 0.0028Extra effect, English -0.0618*Extra effect, Maths 0.3305***Calendar year -0.0063***Male 0.0339***Male by year interaction -0.0218***Age at entry -0.0024Years since entry
1 0.1103***2 0.2383***3 0.3759***4 0.5499***5 0.7000***
Spring semester -0.0132*Module size / 100 -0.0035Mean module points / 100 -0.3021***Dept proportion female 0.2064
Note: ***: p < 0.005; **: p < 0.01; *: p < 0.05
Modelling by departmentNo random slope Departmental random
slope
Intercept 1.3478*** 1.6829***LC points / 100 0.5362*** 0.4941***Extra effect, Irish 0.0028 0.0235Extra effect, English -0.0618* -0.0401Extra effect, Maths 0.3305*** 0.3792***Calendar year -0.0063*** -0.0049**Male 0.0339*** 0.1635***Male by year interaction -0.0218*** -0.0217***Age at entry -0.0024 -0.0040Years since entry
1 0.1103*** 0.1127***2 0.2383*** 0.2399***3 0.3759*** 0.3705***4 0.5499*** 0.5382***5 0.7000*** 0.6772***
Spring semester -0.0132* -0.0156***Module size / 100 -0.0035 0.0052Mean module points / 100 -0.3021*** -0.3209***Dept proportion female 0.2064 0.0338
Note: ***: p < 0.005; **: p < 0.01; *: p < 0.05
Departmental difference in maths effect
-1.5
-1
-0.5
0
0.5
1
1.5
2Dept random effect plus maths fixed effect
99% confidence interval
ARCEPS
PSYPHI
NMIHIS
LAWSOC
MMAPPA
PESMST
PLEPER
LSCMOE
ARTLCS
CSIECE
ACFCES
ECOPHY
MAEMAS
Departmental difference in maths effect
-1.5
-1
-0.5
0
0.5
1
1.5
2Dept random effect plus maths fixed effect
99% confidence interval
ARCEPS
PSYPHI
NMIHIS
LAWSOC
MMAPPA
PESMST
PLEPER
LSCMOE
ARTLCS
CSIECE
ACFCES
ECOPHY
MAEMAS
Protection against failure
Departmental difference in maths effect
-1.5
-1
-0.5
0
0.5
1
1.5
2Dept random effect plus maths fixed effect
99% confidence interval
ARCEPS
PSYPHI
NMIHIS
LAWSOC
MMAPPA
PESMST
PLEPER
LSCMOE
ARTLCS
CSIECE
ACFCES
ECOPHY
MAEMAS
Protection against failurePromoting excellence
Conclusion
I Maths performance at LC is a very important predictor ofperformance in UL
I This varies by department, but in half the effect is positiveand sometimes very large, and in nearly all the rest it isneutral in effect
I That is, maths ability makes an important contribution mostof the time
I Whether policies that improve performance at 2nd level willhave consequences for 3rd level is another question!