deconstructing disengagement: analyzing learner subpopulations in moocs
TRANSCRIPT
![Page 1: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/1.jpg)
Deconstructing Disengagement:Analyzing Learner Subpopulations in
Massive Open Online Courses
René
Kizilcec
Chris
Piech
Emily
Schneider
![Page 2: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/2.jpg)
MOOCs (in this paper) are
instructionist + individualised
• 6-10 weeks long
• 2-3 hours of video lectures/week
• autograded assessments with regular
deadlines
• discussion forum
![Page 3: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/3.jpg)
Massive Open Online Courses
Heterogeneous population:
Learners join from anywhere in the
world, at any age, for any reason
![Page 4: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/4.jpg)
Defining Success for Open-Access Learners
Assessment scores are problematic:
• not comparable across courses
• not available for all learners because
test-taking is not aligned with learner
goals
![Page 5: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/5.jpg)
Defining Success for Open-Access Learners
Completion rates are highly problematic:
• numerator = certificate earners, i.e. learners who take assessments
• denominator = o total enrolled? overestimate; indicator of
interest and not participation
o total active? how defined?
• ignore plurality of learner intentions
• no nuance about subpopulations to help us design interventions or customized course features
![Page 6: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/6.jpg)
Process measures hold promise:
• conceptualize learning as an ongoing
set of interactions with learning objects
and other humans
• allow early detection and prediction
• indicate points for intervention
Defining Success for Open-Access Learners
![Page 7: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/7.jpg)
Defining Success for Open-Access Learners
Completion rates
Assessment scores
Process measures
![Page 8: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/8.jpg)
How to classify learners into
meaningful subpopulations?
![Page 9: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/9.jpg)
Classification Criteria
Classification methods for MOOC subpopulations:
Universal – valid across multiple courses
Theory-driven – reflect the processes of learning
Parsimonious – based on small, meaningful feature set
Predictive – suggest likely outcomes
Dynamic – account for new information over time
![Page 10: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/10.jpg)
Lens for Analysis
• Compare subpopulations
• Compare courses
![Page 11: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/11.jpg)
The Data
![Page 12: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/12.jpg)
Analyzed Three Courses
![Page 13: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/13.jpg)
Who took these MOOCs?
![Page 14: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/14.jpg)
A lot of data!
![Page 15: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/15.jpg)
Gender skew
![Page 16: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/16.jpg)
Interesting age group
![Page 17: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/17.jpg)
HDI skew
![Page 18: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/18.jpg)
Clustering
![Page 19: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/19.jpg)
Sub-populations basis?
Engaged
Not Engaged
![Page 20: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/20.jpg)
Engagement Ideal
Time
Enga
gem
ent
![Page 21: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/21.jpg)
Coarse Engagement Labels
(T) On Track: Did the weekly assignment on
time
(B) Behind: Did the weekly assignment, but
finished after the due date
(A) Auditing: Watched videos but did not do
the assignment
(O) Out: Did not interact with the course,
either through videos or assignments
We were able to predict who would take the final AUC = 0.96
![Page 22: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/22.jpg)
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
In this picture Out Is not to scale!
![Page 23: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/23.jpg)
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
5k
In this picture Out Is not to scale!
![Page 24: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/24.jpg)
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
In this picture Out Is not to scale!
7k
![Page 25: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/25.jpg)
Example Student 1
![Page 26: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/26.jpg)
Example Student 2
![Page 27: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/27.jpg)
Example Student 3
![Page 28: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/28.jpg)
Clustering Methodology
There were 21,108 paths in the
GS class
![Page 29: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/29.jpg)
Four Prototypical Trajectories
Cluster!
(k-means of L1 norm)
![Page 30: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/30.jpg)
Four Prototypical Trajectories
And?
![Page 31: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/31.jpg)
The Four Prototypical Trajectories
![Page 32: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/32.jpg)
Prototypical Trajectory 1: Completing
![Page 33: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/33.jpg)
Prototypical Trajectory 2: Auditing
![Page 34: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/34.jpg)
Prototypical Trajectory 3: Disengaging
![Page 35: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/35.jpg)
Prototypical Trajectory 4: Sampling
![Page 36: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/36.jpg)
Four Prototypical Trajectories
Consistent across three courses:
Auditing learners watch lectures throughout course, but
attempt very few assessments
Completing learners attempt majority of assessments offered
in course
Disengaging learners attempt assessments at beginning of the
course, but then sparsely watch lectures or disappear entirely
Sampling learners briefly explore course by watching a few
videos
![Page 37: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/37.jpg)
Four Prototypical Trajectories
The other courses?
![Page 38: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/38.jpg)
Four Prototypical Trajectories
![Page 39: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/39.jpg)
Four Prototypical Trajectories
<suspense>
![Page 40: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/40.jpg)
Four Prototypical Trajectories
![Page 41: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/41.jpg)
Four Prototypical Trajectories
Same pattern in all classes
![Page 42: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/42.jpg)
HS Composition [46k]C
om
ple
tin
g
Completing
Sampling
Disengaging
Auditing
![Page 43: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/43.jpg)
UG Composition [27k]C
om
ple
tin
g
Dis
en
ga
gin
g
Sampling
Auditing
![Page 44: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/44.jpg)
MS Composition [21k]C
om
ple
tin
g
Dis
en
ga
gin
g
Auditing
Sampling
![Page 45: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/45.jpg)
Validation
![Page 46: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/46.jpg)
Cluster Validation
• Different values of k (split by time)
• Including “assignment pass” (95%
overlap)
• Excluding “behind” (94% overlap)
• Silhouette of 0.8 (that’s pretty good)
• Pass the common sense test
![Page 47: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/47.jpg)
High Level
Clustering
Engagement in
MOOCs
Four
Prototypical
Patterns
![Page 48: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/48.jpg)
Results &
Recommendations
Comparing Trajectories
between Courses
![Page 49: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/49.jpg)
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
3.0 3.5 4.0 4.5 5.0
Overall Experience
Completing (and Auditing)
have best experience
Overall Experience
Identify subpopulations early
to customize course features
![Page 50: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/50.jpg)
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
0.1 0.51.0 2.0 4.0 7.0 10.0
Average Forum Activity
Completing learners are
most active on the forum
Discussion Forum
Reputation systems &
Social features
Causal relationship?
![Page 51: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/51.jpg)
Geographical Distribution
Trend confirmed by top four participating countries
United States, India, Russia, United Kingdom
![Page 52: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/52.jpg)
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
2 4 6 8 10 12 14 16
Odds Ratio (Male/Female)
Female Completing learners underrepresented in advanced courses
Gender
Frame assessments to
minimize stereotype threat
Stereotype threat? Spencer et al., 1999
![Page 53: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/53.jpg)
Future Directions
![Page 54: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/54.jpg)
Future Directions Experiments
Collaboration and Peer Effects
Interface Customization
Targeted Interventions
Nuanced Analytics
Auditing: MOOC-as-a-resource vs. MOOC-as-a-class
Disengaging: Early prediction for intervention
Reasons to enroll and trajectories
Engagement trajectories for real-time analytics in MOOCs
Dashboard visualizations
![Page 55: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/55.jpg)
Thank you!
Stanford Lytics Lab lytics.stanford.edu
Office of the Vice Provost for Online Learning
Roy Pea, Clifford Nass, Daphne Koller
Our LAK reviewers
Reference
S. Spencer, C. Steele, and D. Quinn. Stereotype threat and women’s math
performance. Journal of Experimental Social Psychology, 35(1):4–28, 1999.
![Page 56: Deconstructing Disengagement: Analyzing Learner Subpopulations in MOOCs](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a272a61a28abf46b8b45ca/html5/thumbnails/56.jpg)
More info?
René Kizilcec [email protected]
Chris Piech [email protected]
Emily Schneider [email protected]
Stanford’s Learning Analytics Group:
Lytics Lab lytics.stanford.edu
Paper: http://goo.gl/OSX72