Cristina ConatiDepartment of Computer ScienceUniversity of British Columbia

Beyond Problem-solving: Student-adaptive Interactive Simulations

for Math and Science

Overview

Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges

– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm

Conclusions and Future work

Intelligent Tutoring Systems (ITS)

Create computer-based tools that support individual learners By autonomously and intelligently adapting to their specific needs

StudentModel

Tutor

DomainModel

Adaptive Interventi

ons

ITS Achievements

In the last 20 years, there have been many successful initiatives in devising Intelligent Tutoring Systems

(Woolf 2009, Building Intelligent Interactive Tutors, Morgan Kaufman)

Mainly ITS that provide individualized support to problem solving through tutor-lead interaction (coached problem solving)– Well defined problem solutions => guidance on problem solving

steps– Clear definition of correctness => basis for feedback

Beyond Coached Problem Solving

Coached problem solving is a very important component of learning

Other forms of instruction, however, can help learners acquire the target skills and abilities– At different stages of the learning process– For learners with specific needs and preferences

Our Goal: Extend ITS to other learning activities that support student initiative and engagement: – Interactive Simulations– Educational Games

Overview

Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges

– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm

Conclusions and Future work

Challenges

Activities more open-ended and less well-defined than pure problem solving

– No clear definition of correct/successful behavior

Different user states to be captured (meta-cognitive, affective) in order to provide good tutorial interventions– difficult to assess unobtrusively from interaction events

How to model what the student is doing? How to provide feedback that fosters learning while

maintaining student initiative and engagement?

Our Approach

Student models based on formal methods for probabilistic reasoning and machine learning

Increase information available to student model through innovative input devices:– e.g. eye-tracking and physiological sensors

Iterative model design and evaluation

Overview

Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges

– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm

Conclusions and Future work

ACE: Adaptive Coach for Exploration

Activities organized into units to explore mathematical functions (e.g. input/ouput, equation/plot)

Probabilistic student model that captures student exploratory behavior and other relevant traits

Tutoring agent that generates tailored suggestions to improve student exploration/learning when necessary

(Bunt, Conati, Hugget, Muldner, AIED 2001)

Adaptive Coach for Exploration

EDM 2010

11

12

Adaptive Coach for Exploration

13

Adaptive Coach for Exploration

Before you leave this exercise, why don’t you try scaling the function by a large negative value?Think about how this will affect the plot

ACE Student Model(Bunt and Conati 2002)

Knowledge

Individual Exploration Cases

Exploration of Exercises

Exploration Categories

Exploration of Units

Iterative process of design and evaluation Probabilistic model of how individual exploration actions

influence exploration and understanding of exercises and concepts

e.g. (in Plot unit) • positive/negative slope• positive/negative intercept• large/small, positive/negative exponents…

Modeling Student Exploration

Our first attempt (Bunt and Conati, 2002)

Learning

Student Model

Number and Coverage of Exploratory Actions, e.g.• Positive/negative Y-Intercept• Odd/Even, Positive Negative Exponent....

Interface Actions

Preliminary Evaluation

Quasi-experimental design with 13 participants using ACE (Bunt and Conati 2002)

– The more exercises were effectively explored according to the student model, the more the students improved

– The more hints students followed, the more they learned

Because the model only considers coverage of student actions, it can overestimate student exploration

Need to consider whether the student is reasoning about the effects of his/her actions– Self-explanation meta-cognitive skill:

Revised User Model (Bunt, Muldner and Conati, ITS2004; Merten and Conati, Knowledge Based Systems 2007)

Learning

Student Model

Number and coverage of student actions

Self-explanation of action outcomes

Time between actions Gaze Shifts in Plot Unit

Interface Actions

Input from eye-tracker

Results on Accuracy

We evaluated the complete model against– The original model with no self-explanation– A model that uses only time in between actions as evidence of self-

explanation

Accuracy on SE Accuracy on Learning

50

60

70

80

No SE

SE (Time)

SE (Time + Gaze)

What’s Next (1)

Test adaptive interventions to trigger self-explanation (Conati 2011)

Discussion

ACE work provided evidence that• It is possible to track more “open ended” students’

behaviors than structured problem solving• eye-tracking can support the process

However, hand-coding the relevant behaviors, as we did

for ACE (knowledge-based approach)

• is time consuming

• likely to miss other, less intuitive patterns of interaction

related to learning (or lack thereof)

Alternative Approach (Amershi and Conati 2009, Kardan and Conati 2011)

Behavior Discovery Via Data Mining

Association Rules

MiningClustering

Actions LogsOther Data

Fe

atu

re

Ve

cto

rs

Vector of Interaction Features- Frequency Of

Actions- Latency Between Actions……………

Extract rules describing distinguishing patterns in each cluster

Groups together students that have similar interaction behaviors

Interpret in terms oflearning

• Experts• Performance

Measure(s)

Overview

Motivations Challenges of devising student-adaptive simulations Two examples of how we target these challenges

– ACE: interactive simulation for mathematical functions– CSP Applet: interactive simulation for AI algorithm

Conclusions and Future work

Tested with AI Space CSP applet

AISpace (Amershi et al., 2007)

– set of applets implementing interactive simulations of common Artificial Intelligence algorithms

– Used regularly in our AI courses– Google “AISpace” if you want to try it out

Applet for Constraint Satisfaction problems (CSP), visualizes the working of the AC3 algorithm

27

AISpace CSP Applet

Direct Arc Clicking

User Study (Kardan and Conati 2011)

65 subjects– Read intro material on the AC-3 algorithm– Pre test– Use CSP applet on two problems– Post test

13,078 actions More than 17 hours of interaction

Dataset

Features:– frequencies of use for each action– pause duration between actions (Mean and SD)– 7 actions 21 features

Performance measure for validation– Learning Gain from pretest to posttest

Feature vectors

Clustering

Behavior Discovery

Rule Mining

Found 2 clusters

Statistically significant difference in Learning Gains (LG)– High Learners (HL) and

Low Learners (LL) clusters

32

Feature vectors

Clustering

Behavior Discovery

Rule MiningClustering

Usefulness:Sample Rules

HL members: Use Direct Arc Click action very frequently (R1).

HL cluster:

R1: Direct Arc Click frequency = Highest (Conf =100%, Class Cov = 100%)

LL cluster:

R2: Direct Arc Click Pause Avg = Lowest (Conf =100%, Class Cov = 100%)

R3: Direct Arc Click frequency = Lowest (Conf = 93%, Class Cov=93.5%)

33

LL members: Use Direct Arc Click sparsely (R3) Leave little time between a Direct Arc Click and the next action

(R2)

Feature vectors

Clustering

Behavior Discovery

Rule Mining

Great, but what do we do with this?

We can use the learned clusters and rules to classify a new student based on her behaviors

Use detected behaviours for adaptive support– Promoting the behaviours conducive of learning– Discouraging/preventing detrimental behaviours

34

The User Modeling Framework

35

Association Rules

MiningClustering

Feature Vector

Calculation

OnlineClassifier

Adaptive Interventions

Behavior Discovery

User Classification

Actions LogsOther Data

F

e

at

u

re

New user’s

Actions

Vector of Interaction Features

If user is a LL and uses Direct Arc Click very infrequently (R3)

Then prompt this action

If user is a LL and pauses very briefly after a Direct Arc Click (R2)

Then take action to slow her down

Classifier Evaluation

Leave-one-out Cross Validation on dataset of 64 users For each user u in dataset

1. Remove user u

2. do Behaviour Discovery on the remaining 63

3. for each of u’s actions:» Calculate the feature vector uv

» Classify uv

» Compare with u’s original label

Accuracy as a function of observed actions

Discussion

User modeling framework for open-ended and unstructured interactions– Relevant behaviours are discovered via data mining

techniques instead being hand-crafted Very encouraging results with CSP applet

– Detected clusters represent groups with different learning gains

– Online classifier: good accuracy soon enough to generate adaptive interventions

– These interventions can be derived from the generated rules

Current Work

Applying the discovered rules to generate the adaptive version of the CSP applet

Adding eye-tracking input to the dataset

Conclusions

Research on devising student-adaptive didactic support for exploratory activities beyond problem solvingInteractive simulations

Challenges in modeling interactions with no clear structure or definition of correctness

Student modeling approaches based on probabilistic techniques and unsupervised machine learning very promising results

Shown how eye-tracking can help!We are also exploring it in relation to assessing engagement

and attention in educational games (Muir and Conati 2011)