The Impact of Learning Analytics on eLearning Design

Patrick Devey, Ph.D.Chief Learning Officer

KnowledgeOne Inc.Montréal, Québec, Canada

THE MISSING STUDENTS CAPER

“I seem to be missing a significant number of students at the final exam…”

-FINA 200 Professor

0%

5%

10%

15%

20%

25%

30%

Voluntary Student Dropout

eConcordia FINA 200

“help…?”-FINA 200 Professor

WHERE DO WE BEGIN?

Student Logs (from LMS)

Over 50,000 entries!!!

Hazard Rate using Registration Data

1 2 3 4 5 6 7 8 9.000

.010

.020

.030

.040

.050

.060

FINA 200

Time (Week)

Haz

ard

Rate

2 3 4 5 6 7 8 9 10.000

.010

.020

.030

.040

.050

.060

Time (Week)

Haz

ard

Rate

Ass

ign

men

t 1

Ass

ign

men

t 2

Ass

ign

men

t 3

Hazard Rate using Registration Data

THE MISSING STUDENTS CAPER

…will return after these informative messages!

Learning Analytics

Data must be:• relevant to the learners and to the instructors,• focuses on individual learners or a course,• leads to informed decisions to improve learning

services or products at the learner/course level.

“The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.”

-The Society for Learning Analytics Research (SoLAR)

13

Pascarella & Chapman (1983)

Longitudinal Data: Path Analysis

R2 = .120

Data Sources

LMS

Google Analytics

Surveys

Industry Statistics

& Reports

EnrolmentHR System

xAPICustom

INDUSTRY STATISTICS & REPORTS

Easy to get, but do they apply to your situation?

http://en.wikipedia.org/wiki/Usage_share_of_operating_systems

Nov-07 Mar-09 Aug-10 Dec-11 May-130.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Browser History

Internet Explorer Firefox Chrome Safari Opera

Google Chrome Safari MS Internet Explorer Firefox Other0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Browser Comparison

W3 Concordia University

Windows Linux Mac Mobile0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

OS Comparison

w3 eConcordia

How could this data be used?

• What are the main browsers used by our learners?– Quality assurance testing– Compatibility (HTML5, plug-ins, etc…)

• What are the primary operating systems being used by your learners?

• Are your learners making use of mobile devices?• What are the trends?

– Should you be making preparations because of them?

LEARNING MANAGEMENT SYSTEM

It’s already being collected, might as well use it!

How could this data be used?

• When are learners most likely to be active?– Schedule live office hours– Schedule server maintenance

• Are learners achieving the learning objectives?– Adjustments to assessments– Adjustments to existing content– Addition of remedial content

• Is time-on-task correlated with performance?– Log-ins vs. performance– Identifying learners who have “stopped out”

GOOGLE ANALYTICS

Tuning in to the “Greatest Hits”

Demographic Data

Browser Statistics, OS, Mobile, etc…

Qui

z 1

Qui

z 2

Qui

z 3

Fina

l Exa

m

Qu

iz 1

Ass

ign

men

t 1O

rie

nta

tion

Se

ssio

n

Ass

ign

men

t 2

Qu

iz 2

Ass

ign

men

t 3

Qu

iz 3

Fin

al E

xamMidterm Break

031.aspx 042.aspx

How could this data be used?

• Is my visual design effective?– Do I have buttons that are not being used? Can the site be

optimized?• How are learners navigating through the course website?

– Are the navigation buttons being used as designed? Can they be improved or adapted?

• When are learners using particular course materials?– Is there an opportunity to send remedial content when it is

relevant?• How long are learners spending on individual sections?

– Is there an opportunity to improve the content?

DESIGN YOUR OWN DATA COLLECTION

If the data you want is not being collected, design a way to get it!

• Who are the students enrolling in online courses at Concordia University? How do they compare to the general student population at Concordia as well as to the distance education literature?

• Why are students enrolling in online courses? What expectations do they have about their online course?

• Why do students drop out of online courses? Are there demographic, attitudinal, integration, and/or behavioural variables related to persistence?

• When do students drop out of online courses?

• What interventions can the institution make to curtail attrition in their online courses?

What Patrick wanted to know

Timeframe of the Study

1 2 3 4 5 6 7 8 9 10 11 12 13

Weeks of the Semester

1. Web-Based Learning

Questionnaire

4. Registration Data3. Exit Survey

DNE DISCDropout Zone

2. Weekly Survey

44

THE MISSING STUDENTS CAPER

…and now, back to:

Finance: Course Satisfaction

2 3 4 5 6 7 8 9 1064

66

68

70

72

74

76

78

Week

Aver

age

Finance: Hazard Rate/Satisfaction

2 3 4 5 6 7 8 9 10.000

.010

.020

.030

.040

.050

.060

Time (Week)

Haza

rd R

ate

48

Ass

ign

men

t 1

Ass

ign

men

t 2

Ass

ign

men

t 3

2 3 4 5 6 7 8 9 1064

66

68

70

72

74

76

78

Week

Aver

age

The Facts

• Significant portion of the class were non-business students– Struggling with basic concepts such as time value

of money calculations• Older, worked full-time, studied part-time

– Did not have time to seek out instructor “in-between classes” or to go to campus

• Dropping out because of a fear of poor performance in the course

Interventions

• Live virtual office hours and tutorials scheduled when they were most needed– Dealt with the assignments– Sessions were recorded and posted on the course website

• Development of additional remedial content to address problem areas

• Provided links to the remedial content in the course content when they were relevant

• Altered assessments (rubric) for non-business students

CASE

CLOSED

(kind of)

WHAT DO YOU WANT TO IMPROVE?

What do you want to improve?

• Retention rate• Transfer of training• Achievement of learning objectives• Performance on assessments• Usability of the course website• Services to instructors and learners• ???

Take Action

1. Go beyond pass/fail to validate your learning solution.

2. Don’t let the system limit you. 3. Design the instruments you need to collect

the metrics you want.4. Don’t be intimidated by the numbers.5. Continuously seek ways to improve.

Thank you! Merci!

Questions?

patrick.devey@knowledgeone.ca @patrickdevey

http://survivingelearning.blogspot.ca

http://www.knowledgeone.ca