Learning Analytics and Accessibility – what can be done and pragmatic considerations

Martyn Cooper (IET)CALRG Conference 2015

Introduction

• Work is part of the eSTEeM project LA4DS-STEM

• Learning Analytics (LA) requires “Big Data”

• Particular interest in retention and pass rates

• Research Questions:• What can LA approaches tell us about the accessibility

(to disabled students) of modules?• If they tell us anything is the approach useful?

The Central Hypothesis

• In modules where the completion rate and the pass rate are significantly lower for disabled students than for nondisabled students then this is indicative of accessibility challenges in that module

Questions:• What is significance indicated by in this case?• How confident are we that we are not measuring other

factors that impact on performance such as: motivation; family circumstances; ability; educational background; etc.?

The Data Set

• The “big data”:–All Science and MCT Modules from presentation

2009B to and including 2013J (5 years)–1452 presentations in total–% Completion and % Pass Rates–Disabled students vs Non-disabled students–Data set needed some “cleaning-up” before analysis

      DisabledNo Yes

Module Presentation Total No. % complete % pass No. % complete % passA 2009B 1827 1605 63.4 61.6 222 56.3 52.7

B 2009J 2609 2282 67.4 66.3 327 59.0 56.9

C 2010B 1662 1492 61.5 60.1 170 60.6 58.2

… … … … … … … … …

Odds Ratios Explanation • Need a statistical useful comparison

–Using odds ratios [J.T.E. Richardson personal communication]

• If the probability of the members of Group 1 exhibiting a particular outcome is p then the odds of this are p/(1 − p)

• If the probability of the members of Group 2 exhibiting that outcome is q, then the odds of this are q/(1 − q)

• The odds ratio is the ratio between these odds (i.e. [p/(1 − p)]/[q/(1 − q)], which equals [p(1 − q)]/[q(1 − p)])

• Odds ratios vary from 0 (when p = 0 or q = 1) to infinity (when p = 1 or q = 0)

Odds Ratios Explanation cont.• An odds ratio of 1 means that there is no difference in

the odds of the two groups’ members exhibiting the outcome (when p = q)

• An odds ratio less than 1 means that the members of Group 1 are less likely to exhibit the outcome than are the members of Group 2; and an odds ratio greater than 1 means that the members of Group 1 are more likely to exhibit the outcome than are the members of Group 2

• N.B. - Whether an odds ratio is significantly different from 1 depends on the odds ratio itself and on the number of members in each group

Completion % All Modules

1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 10211081114112011261132113811441

-60.0

-40.0

-20.0

0.0

20.0

40.0

60.0

80.0

100.0

120.0

Disabled-Nondisabled % Complete

Disabled-Nondisabled % Complete

Cases % difference > 35% when low number disabled students registered

Completion % - Modules with > 25 disabled students registered

1 21 41 61 81 101121141161181201221241261281301321341361381401421441461481501521541561581601621641661

-30.0

-20.0

-10.0

0.0

10.0

20.0

30.0

40.0

Nondisabled-Disabled % Complete

Nondisabled-Disabled % Complete

Odds Ratios Completion All Modules in Data Set

1 64 127 190 253 316 379 442 505 568 631 694 757 820 883 946 1009107211351198126113240.000

2.000

4.000

6.000

8.000

10.000

12.000

14.000

16.000

18.000

20.000

Odds Ratios Complete (>1 if non-disabled outperform disabled)

Odds Ratios Complete (>1 if non-disabled outper-form disabled)

All cases Odds Ratio >10 occur when low number of disabled students registered

Thresholds for Decisions

• The learning analytics needs to lead to a decision about which modules include significant accessibility barriers for remedial action– The LA can tell you where a problem might be not what it is

• Thresholds for decision making are arbitrary but informed by the data– A reasonable threshold for identifying accessibility problems

seems to be an odds ratio of 4.0 or more

Future Work• Liaison with Science Accessibility Specialist particularly

with reference to S104 and planning for the new Level 1 gateway modules

• Liaison with the Science and MCT Data Wranglers to look for any correlation with the their data

• Focus group(s) with MCT and Science staff of mock-ups of Learning Analytics Dashboards

• Paper for LAK16 comparing with qualitative data from end of module surveys

Discussion Points• Learning Analytics approaches seem to be able to

identify major accessibility issues in modules– However this needs testing and only possible by a detailed

accessibility assessment of the module’s media and activities (this work not currently funded)

• LA approaches only valid on modules with a significant number of disabled students – suggest a minimum of 25

• Even with 25 disabled students per module really need to evaluate over multiple presentations to identify issues– Does this mean the approach is less useful than responding to

student complaints, or proper accessibility evaluation in production, etc.?

Questions

Institute of Educational TechnologyThe Open UniversityWalton HallMilton KeynesMK7 6AA

www.open.ac.uk