predicted project hatfield uk 2015 m glynn
TRANSCRIPT
![Page 1: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/1.jpg)
Student Data: Data is knowledge – putting the knowledge back in the students’ hands
Owen CorriganMark GlynnAisling McKennaAlan F. SmeatonSinéad Smyth
@glynnmark
![Page 2: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/2.jpg)
Outline
• Motivation and goals• Selecting the modules• Study by numbers• The interventions
- What the student sees• What the students said• The results
![Page 3: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/3.jpg)
Motivation
![Page 4: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/4.jpg)
Study by numbers
• 17 Modules across the University (first year, high failure rate, use Loop, periodicity, stability of content, Lecturer on-board)
• Offered to students who opt-in or opt-out, over 18s only
• 76% of students opted-in, 377 opted-out, no difference among cohorts
• 10,245 emails sent to 1,184 students who opted-in over 13 weekly email alerts
![Page 5: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/5.jpg)
Total Moodle Activity – notice the periodicity
![Page 6: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/6.jpg)
One example module – ideal !
![Page 7: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/7.jpg)
Modules which work well …
• Have periodicity (repeatability) in Moodle access• Confidence of predictor increases over time• Don't have high pass rates (< 0.95)• Have large number of students, early-stage
![Page 8: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/8.jpg)
No significant difference in the entry profiles of participants vs. non-participants overall
PredictEd Participant Profile
![Page 9: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/9.jpg)
LG116 – Predictor confidence (ROC AUC)
![Page 10: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/10.jpg)
SS103
![Page 11: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/11.jpg)
What did the students say?Students who took part were asked to complete a short
survey at the start of Semester 2 - N=133 (11% response rate)
Question Group 1 (more detailed email)
Group 2
% of respondents who opted out of PredictED during the course of the
semester4.5% 4.5%
% who changed their Loop usage as a result of the weekly emails
43.3% 28.9%
% who would take part again/are offered and are taking part again
72.2% (45.6%/ 26.6% )
76.6% (46% /30.6% )
![Page 12: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/12.jpg)
33% said they changed how they used Loop. We asked them how?
• Studied more– “More study”– “Read some other articles online”– “Wrote more notes”– “I tried to apply myself much more, however yielded no results”– “It proved useful for getting tutorial work done”
• Used Loop more– “I tried harder to engage with my modules on loop”– “I think as it is recorded I did not hesitate to go on loop. And loop as become
my first support of study.”– “I logged on more”– “I read most of the extra files under each topic, I usually would just look at
the lecture notes.”– “I looked at more of the links on the course nes pages, which helped me to
further my understanding of the topics”– “I learnt how often I need to log on to stay caught up.”
![Page 13: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/13.jpg)
Did you change Loop usage for other modules?
• Most who commented used Loop more often for other modules– “More often”– “More efficient”– “Used loop more for other modules when i was logging onto
loop for the module linked to PredictED”– “Felt more motivated to increase my Loop usage in general
for all subjects”
One realised that Lecturers could see their Loop activity“I realised that since teachers knew how much i was
using loop, i had to try to mantain pages long on so it looked as if i used it a lot”
![Page 14: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/14.jpg)
Subject Description Non-Participant ParticipantBE101 Introduction to Cell Biology and Biochemistry 58.89 62.05CA103 Computer Systems 70.28 71.34CA168 Digital World 63.81 65.26ES125 Social&Personal Dev with Communication Skills 67.00 66.46HR101 Psychology in Organisations 59.43 63.32LG101 Introduction to Law 53.33 54.85LG116 Introduction to Politics 45.68 44.85LG127 Business Law 60.57 61.82MS136 Mathematics for Economics and Business 60.78 69.35SS103 Physiology for Health Sciences 55.27 57.03Overall Dff in all modules 58.36 61.22
Average scores for participants are higher in 8 of the 10 modules analysed, significantly higher in BE101, and CA103. MS136
Module Average Performance Participants vs. Non-Participants
![Page 15: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/15.jpg)
Questions and discussion…
![Page 16: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/16.jpg)
Contact details
• [email protected]• @glynnmark
• http://enhancingteaching.com
• https://predictedanalytics.wordpress.com/
![Page 17: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/17.jpg)
Additional slides
![Page 18: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/18.jpg)
Student Interventions: Feedback
![Page 19: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/19.jpg)
The Interventions – Lecturers’ Experience
![Page 20: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/20.jpg)
LG116: Introduction to Politics
Students / year = ~110Pass rate = 0.78
![Page 21: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/21.jpg)
Importance of Ethics
• Ethics are important to ensure safety of participants and researchers
• Educational Data Analytics is a new area of research– Not much previous research to highlight possible ethical
issues– Requires extensive ethical consideration
• We have spent a lot of time this Summer preparing a DCU REC submission– We’ve submitted and had approval for a test case– We’ve met with REC chair to brief him
• We are following the 8 Principles set out by the Open University who are at EXACTLY the same stage as us
![Page 22: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/22.jpg)
So much student data we could useDemographics• Age, home/term address, commuting distance, socio-economic status, family
composition, school attended, census information, home property value, sibling activities, census information
Academic Performance• CAO and Leaving cert, University exams, course preferences, performance relative
to peers in school
Physical Behaviour• Library access, sports centre, clubs and societies, eduroam access yielding co-
location with others and peer groupings, lecture/lab attendance,
Online Behaviour• Mood and emotional analysis of Facebook, Twitter, Instagram activities, friends and
their actual social network, access to VLE (Moodle)
![Page 23: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/23.jpg)
Building classifiers for each week/each module
Training DataTesting
![Page 24: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/24.jpg)
Notes on model confidence• Y axis is confidence in AUC ROC (not probability)• X axis is time in weeks• 0.5 or below is a poor result• Most Modules start at 0.5 when we don't have much
information• 0.6 is acceptable, 0.7 is really good (for this task)• The model should increase in confidence over time• Even if confidence overall increases, due to randomness
the confidence may go up and down• It should trend upwards to be a valid model and viable
module choice
![Page 25: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/25.jpg)
LG116 MS136 LG101 HR101 LG127 ES125 BE101 SS103 CA103 CA1680%
20%
40%
60%
80%
100%WorkshopsWikisForumsAssignmentsQuizzesscormlessonchoicefeedbackdatabaseglossarywikiurlbookpagesfoldersfiles
Course content
![Page 26: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/26.jpg)
BE101: Intro to Cell Biology
Results / year = ~300Pass rate = 0.86
![Page 27: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/27.jpg)
BE101
![Page 28: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/28.jpg)
SS103: Physiology for Health Sciences
Results / Year = ~150Pass rate = 0.92
![Page 29: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/29.jpg)
MS136
![Page 30: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/30.jpg)
LG101
![Page 31: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/31.jpg)
HR101
![Page 32: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/32.jpg)
CA103
![Page 33: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/33.jpg)
Some unusable modulesModules where the ROC AUC increases slowly (e.g stays below 0.6) e.g. PS122
![Page 34: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/34.jpg)
Timescale for Rollout
• Still some issues on Moodle access log data transfer to be resolved
• Still have to resolve student name / email address / Moodle ID / student number
• Still to resolve timing of when we can get new registration data, updates to registrations (late registrations, change of module, change of course, etc.) …
• Should we get new, “clean” data each week ?
![Page 35: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/35.jpg)
Why did you take part?• The majority of students
wanted to learn/monitor their performance
• Many others were curious
• Some were interested in the Research aspect
• Some were just following advice
• Others were indifferent
![Page 36: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/36.jpg)
How easy was it to understand the information in the emails ?(1= not at all easy, 5 = extremely easy)
• Average 3.97 (SD= 1.07)
• Very few had comments to make (19/133)– Most who commented wanted more
detail.
![Page 37: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/37.jpg)
Week 3
Training DataTesting
![Page 38: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/38.jpg)
Week 4
Training DataTesting
![Page 39: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/39.jpg)
Week 5
Training DataTesting
![Page 40: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/40.jpg)
Week 6
Training DataTesting
![Page 41: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/41.jpg)
Week 7
Training DataTesting
![Page 42: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/42.jpg)
Week 8
Training DataTesting
![Page 43: Predicted project Hatfield UK 2015 M Glynn](https://reader036.vdocuments.net/reader036/viewer/2022070600/58d0b05b1a28ab1d3a8b4cf9/html5/thumbnails/43.jpg)
Week 9
Training DataTesting