boredom across activities, and across the year, within reasoning mind
DESCRIPTION
Boredom Across Activities, and Across the Year, within Reasoning Mind. William L. Miller, Ryan Baker, Mathew Labrum, Karen Petsche , Angela Z. Wagner. In recent years. Increasing interest in modeling more about students than just what they know. In recent years. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/1.jpg)
Boredom Across Activities, and Across the Year,
within Reasoning Mind
William L. Miller, Ryan Baker, Mathew Labrum, Karen Petsche,
Angela Z. Wagner
![Page 2: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/2.jpg)
In recent years
• Increasing interest in modeling more about students than just what they know
![Page 3: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/3.jpg)
In recent years
• Increasing interest in modeling more about students than just what they know
• Can we assess a broad range of constructs
![Page 4: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/4.jpg)
In recent years
• Increasing interest in modeling more about students than just what they know
• Can we assess a broad range of constructs• In a broad range of contexts
![Page 5: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/5.jpg)
Boredom
• A particularly important construct to measure
![Page 6: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/6.jpg)
Boredom is
• Common in real-world learning (D’Mello, 2013)
![Page 7: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/7.jpg)
Boredom is
• Common in real-world learning (D’Mello, 2013)
• Associated with worse learning outcomes in the short-term (Craig et al., 2004; Rodrigo et al., 2007)
![Page 8: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/8.jpg)
Boredom is
• Common in real-world learning (D’Mello, 2013)
• Associated with worse learning outcomes in the short-term (Craig et al., 2004; Rodrigo et al., 2007)
• Associated with worse course grades and standardized exam performance (Pekrun et al., 2010; Pardos et al., 2013)
![Page 9: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/9.jpg)
Boredom is
• Common in real-world learning (D’Mello, 2013)• Associated with worse learning outcomes in the
short-term (Craig et al., 2004; Rodrigo et al., 2007)• Associated with worse course grades and
standardized exam performance (Pekrun et al., 2010; Pardos et al., 2013)
• Associated with lower probability of going to college, years later (San Pedro et al., 2013)
![Page 10: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/10.jpg)
Online learning environments
• Offer great opportunities to study boredom in context– Very fine-grained interaction logs that indicate
everything the student did in the system
![Page 11: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/11.jpg)
Automated boredom detection
• Can we detect boredom in real time, while a student is learning?
• Can we detect boredom retrospectively, from log files?
![Page 12: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/12.jpg)
Automated boredom detection
• Can we detect boredom in real time, while a student is learning?
• Can we detect boredom retrospectively, from log files?
• Would allow us to study affect at a large scale– Figure out which content is most boring, in order
to improve it
![Page 13: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/13.jpg)
Affect Detection: Physical Sensors?
• Lots of work shows that affect can be detected using physical sensors– Tone of voice (Litman & Forbes-Riley, 2005)– EEG (Conati & McLaren, 2009)– Posture sensor and video (D’Mello et al., 2007)
• It’s hypothesized – but not yet conclusively demonstrated – that using physical sensors may lead to better performance than interaction logs alone
![Page 14: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/14.jpg)
Sensor-free affect detection
• Easier to scale to the millions of students who use online learning environments
• In settings that do not have cameras, microphones, and other physical sensors– Home settings• have parents bought equipment? • can they set it up and maintain it?
– Classroom settings• can school maintain equipment? • do students intentionally destroy equipment?• parent concerns and political climate
![Page 15: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/15.jpg)
Sensor-free boredom detection
• Has been developed for multiple learning environments– Problem solving tutors (Baker et al. 2012; Pardos et al. 2013)– Dialogue tutors (D’Mello et al. 2008)– Narrative virtual learning environments (Sabourin et al. 2011;
Baker et al. 2014)– Science simulations (Paquette et al., 2014)
• The principles of affect detection are largely the same across environments
• But the behaviors associated with boredom differ considerably between environments
![Page 16: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/16.jpg)
This talk
• We discuss our work to develop sensor-free boredom detection for Reasoning Mind Genie 2 (Khachatryan et al, 2014)
• Self-paced blended learning mathematics curriculum for elementary school students – Youngest population for sensor-free affect
detection so far• Used by approximately 100,000 students a
year
![Page 17: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/17.jpg)
Reasoning Mind Genie 2
• Combines – Guided Study with a pedagogical agent “Genie”– Speed Games that support development of
fluency
• Used in schools 3-5 days a week for 45-90 minutes per day
![Page 18: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/18.jpg)
Reasoning Mind Genie 2(a)
(b)
(c)
![Page 19: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/19.jpg)
Reasoning Mind Genie 2
• Better affect and more on-task behavior than most pedagogies, online or offline (Ocumpaugh et al., 2013)
• Still a substantial amount of boredom
• Reducing boredom is a key goal
![Page 20: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/20.jpg)
Role for affect detection
• If we can detect boredom in log files• We can determine which content is more
boring, and improve that content
![Page 21: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/21.jpg)
Related Work
• Evidence that specific design features associated with boredom in Cognitive Tutors for high school algebra (Doddannara et al., 2013)
![Page 22: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/22.jpg)
Related Work
• Evidence that specific design features associated with boredom in Cognitive Tutors for high school algebra (Doddannara et al., 2013)
• Evidence that some disengaged behaviors increase during the year (Beck, 2005)– Important to verify that differences in affect due
to actual content/design, not time of year
![Page 23: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/23.jpg)
Approach to Boredom Detection
• Collect “ground truth” data on student boredom, using field observations
• Synchronize log data to field observations • Distill meaningful data features of log data,
hypothesized to relate to boredom• Develop automated detector using classification
algorithm• Validate detector for new students/new
lessons/new populations
![Page 24: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/24.jpg)
BROMP 2.0 Field Observations(Ocumpaugh et al., 2012)
• Conducted through Android app HART (Baker et al., 2012)
• Protocol designed to reduce disruption to student– Some features of protocol: observe with peripheral
vision or side glances, hover over student not being observed, 20-second “round-robin” observations of several students, bored-looking people are boring
• Inter-rater reliability around 0.8 for behavior, 0.65 for affect
• 64 coders now certified in USA, Philippines, India
![Page 25: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/25.jpg)
Data collection
• 408 elementary school students
![Page 26: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/26.jpg)
Data collection
• Diverse sample important for model generalizability (Ocumpaugh et al., 2014)
• 11 different 8th grade classes• 6 schools– 2 urban in Texas, predominantly African-American– 1 urban in Texas, predominantly Latino– 1 suburban in Texas, predominantly White– 1 suburban in Texas, mixed ethnicity/race– 1 rural in West Virginia, predominantly White
![Page 27: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/27.jpg)
Affect coding
• 3 expert coders observed each student using BROMP
• Coded 5 categories of affect– Engaged Concentration– Boredom– Confusion– Frustration– ?
• 4891 observations collected in RM classrooms
![Page 28: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/28.jpg)
Building detectors
• Observations were synchronized with the logs of the students interactions with RM, using HART app and internet time server
• For each observation, a set of 93 meaningful features describing the student’s behavior was engineered
• Computed on actions occurring during or preceding an observation (up to 20 seconds before)
![Page 29: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/29.jpg)
Features: Examples
• Individual action features – Whether an action was correct or not– How long the action took
• Features across all past activity– Fraction of previous attempts on the current skill the
student has gotten correct• Other known models applied to logs– Probability student knows skill (Bayesian Knowledge
Tracing)– Carelessness– Moment-by-Moment Learning Graph
![Page 30: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/30.jpg)
Automated detector of boredom
• Detectors were built using RapidMiner 5.3• For each algorithm the best features were
selected using forward selection/backward elimination
• Data was re-sampled to have more equal class frequencies; models were evaluated on original class distribution
• Detectors were validated using 10-fold student-level cross-validation
![Page 31: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/31.jpg)
Automated detector of boredom
• Detectors were built using 4 machine learning algorithms that have been successful for building affect detectors in the past:– J48– JRip – Step Regression– Naïve Bayes
![Page 32: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/32.jpg)
Best One
• Detectors were built using 4 machine learning algorithms that have been successful for building affect detectors in the past:– J48– JRip – Step Regression– Naïve Bayes
![Page 33: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/33.jpg)
Machine learning
• Performance of the detectors was evaluated using
• A’– Given two observations, probability of correctly
identifying which one is an example of a specific affective state and which one is not
– A’ of 0.5 is chance level and 1 is perfect– Identical to Wilcoxon statistic– Very similar to AUC ROC (Area Under the Receiver-
Operating Characteristic Curve)
![Page 34: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/34.jpg)
Results
• A’ = 0.64
• Compared to similar detectors in other systems, validated in similar stringent fashion
System A'
Cognitive Tutor Algebra(Baker et al. 2012) 0.69
ASSISTments (Pardos et al. 2013) 0.63
EcoMUVE(Baker et al. 2014) 0.65
Inq-ITS(Paquette et al. 2014) 0.72
![Page 35: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/35.jpg)
Coefficient Feature
+0.212The standard deviation, across the clip, of student
correctness (1 or 0) on each action.
-0.013The number of actions in the clip that occurred on
Speed Game items.
-0.070The fraction of the total clip duration spent on Speed
Game items.
-0.073The number of actions in the clip on items where the
answer input was made by selecting an item from a drop-down list.
+0.290The minimum slip parameter (P(S) in Bayesian
Knowledge Tracing) on skills in the clip.
-0.260
The standard deviation, across the clip, of the action duration, normalized across all students, times the
presence (1) or absence (0) of a hint request on the previous action.
+0.123 Y-intercept.
![Page 36: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/36.jpg)
Using detectors
• Model applied to entire year of data from these classrooms
• 2,974,944 actions by 462 students• Includes 54 additional students not present
during observations
• Aggregation over pseudo-confidences rather than binary predictions– Retains more information
![Page 37: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/37.jpg)
1-Sep 8-Nov 15-Jan 24-Mar 31-May0.130
0.135
0.140
0.145
0.150
Boredom vs. Time of Year
Date
Ave
rage
Bor
edom
![Page 38: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/38.jpg)
Apparent downward trend
![Page 39: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/39.jpg)
Apparent downward trend
• Is it statistically significant?
![Page 40: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/40.jpg)
Apparent downward trend
• Is it statistically significant?
• Yes. • Students are less bored later in the year• F-test controlling for student• p<0.001
![Page 41: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/41.jpg)
Is it practically significant?
![Page 42: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/42.jpg)
Is it practically significant?
• No. • r = -0.06
![Page 43: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/43.jpg)
Is it practically significant?
• No. • r = -0.06
• With large enough samples, anything is statistically significant
![Page 44: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/44.jpg)
Kind of a positive thing
• At minimum, students aren’t getting more bored as the year goes on
• In other systems, students get more disengaged as the year goes on (Beck, 2005)
• And the overall level of boredom (~14%) is not very high
![Page 45: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/45.jpg)
Beyond this
• Curriculum is self-paced
![Page 46: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/46.jpg)
Beyond this
• Curriculum is self-paced
• Which means that predicting boredom by date may obscure real variation
![Page 47: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/47.jpg)
Beyond this
• Curriculum is self-paced
• Which means that predicting boredom by date may obscure real variation
• Instead, look at boredom by learning objective
![Page 48: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/48.jpg)
5.015.045.075.105.135.165.195.225.255.285.315.345.375.405.43
0.130
0.135
0.140
0.145
0.150
Boredom vs. Objective
Reasoning Mind Objective
Ave
rage
Bor
edom
(b)
![Page 49: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/49.jpg)
Predicting boredom by objective
• p<0.001• r=0.343
![Page 50: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/50.jpg)
If we cluster objectives into two groups
• “High boredom”• “Low boredom”
• Ignoring the one point in between the two groups
• Cohen’s D = 0.67
![Page 51: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/51.jpg)
Future work
• So… what is it that differentiates the higher boredom lessons from the lower boredom lessons?
![Page 52: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/52.jpg)
Future work
• So… what is it that differentiates the higher boredom lessons from the lower boredom lessons?– Nothing obvious, unfortunately…
![Page 53: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/53.jpg)
Future work
• So… what is it that differentiates the higher boredom lessons from the lower boredom lessons?– Nothing obvious, unfortunately…– May be necessary to develop a taxonomy of
potential differences, and see which are predictive– May be possible to build off prior work by
(Doddannara et al., 2013) that did exactly this for Cognitive Tutor
![Page 54: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/54.jpg)
Future Work
• Can we fix the more boring lessons?– Either by determining why they are boring– Or just by adding a little more “fun content”
![Page 55: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/55.jpg)
Eventual Goal
• Use precise assessments of boredom to help us enhance Reasoning Mind– Improving engagement– Improving learning outcomes
![Page 56: Boredom Across Activities, and Across the Year, within Reasoning Mind](https://reader035.vdocuments.net/reader035/viewer/2022070404/56813b86550346895da4b171/html5/thumbnails/56.jpg)
Thank you
twitter.com/BakerEDMLab
Baker EDM Lab
Baker EDM Lab
See our free online MOOT “Big Data and Education”All lab publications available online – Google “Ryan Baker”
“Data, Analytics, and Learning” – EdX, Fall 2014