An Evaluation of Learning Analytics To Identify Exploratory Dialogue in Online Discussions

Rebecca Ferguson, The Open University, UK

Zhongyu Wei, The Chinese University of Hong Kong

Yulan He, Aston University, UK

Simon Buckingham Shum, The Open University, UK

Discourse analytics

The ways in which learners engage in dialogue indicate how they engage with the ideas of others, how they relate those ideas to their understanding and how they explain their own point of view.

• Disputational dialogue• Cumulative dialogue• Exploratory dialogue

Exploratory dialogueCategory Indicator

Challenge But if, have to respond, my view

Critique However, I’m not sure, maybe

Discussion of resources Have you read, more links

Evaluation Good example, good point

Explanation Means that, our goals

Explicit reasoning Next step, relates to, that’s why

Justification I mean, we learned, we observed

Reflections of perspectives of others

Agree, here is another, makes the point, take your point, your view

Pilot study: LAK 2011Time Contribution

2:42 PM I hate talking. :-P My question was whether "gadgets" were just basically widgets and we could embed them in various web sites, like Netvibes, Google Desktop, etc.

2:42 PM Thanks, that's great! I am sure I understood everything, but looks inspiring!

2:43 PM Yes why OU tools not generic tools?

2:43 PM Issues of interoperability

2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages.

2:43 PM What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps!

2:43 PM My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model?

2:43 PM there are various different flavours of widget e.g. Google gadgets, W3C widgets etc. SocialLearn has gone for Google gadgets

Computational linguistics

Interdisciplinary field thatdeals with statistical and rule-based modelling of natural language from a computational perspective

Zhongyu Wei Yulan He

Three challenges

1. The annotated dataset is limited

2. Text classification problems are typically topic driven – this is not

3. Nevertheless, both dialogue features and topical features need to be taken into account

Self-training from labelled instances – a problem

Pseudo-label

Exploratory

Pseudo-label

Exploratory

Pseudo-label

Exploratory

Pseudo-label

Exploratory

✓

✓

✓

✗Including this

instance would degrade the

classifier

• For each turn in the dialogue, consider each unigram (word), bigram (2 words) and trigram (3 words)

• Exploratory or non-exploratory?• Take into account word-association probabilities

averaged over many pseudo-labelled examples

Self-training from labelled features

Pseudo-labelNon-

exploratory

✓Focusing on

features gives a more reliable classification

Bigrams

To improve labelling, take into account the classification of a number (k) of the nearest neighbours of that turn in the dialogue

Taking context into account

Pseudo-label

Non-

exploratory

Unlabelled turn in the dialogue, p1

Pseudo label for that turn, l1

Confidence value for that label, c1

0.272727271

Nearest neighbour pni1

Pseudo label lni1

Confidence level cni1

Nearest neighbour pni3

Pseudo label lni3

Confidence level cni3

Nearest neighbour pni2

Pseudo label lni2

Confidence level cni2

Let k = 3(look at 3 nearest

neighbours)

Pseudo-label

Non-

exploratory

Pseudo-label based on features is considered correct if support value (s)

based on context is high enough

Checking against context

Support value is calculated by taking into account the pseudo labels and confidence values of k nearest neigbours

?

Checking the pseudo-labels

Pseudo-label

Exploratory

Nearest neighbour 1Confidence level 0.333Pseudo-label

Non-

exploratory

Nearest neighbour 2Confidence level 0.999

Nearest neighbour 3Confidence level 0.666

s = 0.333 + 0 + 0.666

3

Because s < Rthis turn in the dialogueshould not be labelled ‘non-exploratory’

If the support valuefor this pseudo labelis greater than R then this turn in the dialogue can be labelled ‘non-exploratory’

Let R = 0.5

Pseudo-label

Non-

exploratory

Pseudo-label

Non-

exploratory

s = 0.333

?

Cue phrases from pilotAgreeAlsoAlthoughAlternativeAny researchAre weBecauseBut ifChallengeClaimDebateDefinitelyDependsDifficultDiscussionDo we haveDo youDoes that mean

Does this suggestDraftEvidenceExampleExceptMisunderstandingGood exampleGood pointGood thing aboutHave weHave you looked atHave you readHere is anotherHow areHow can[...]WhyYour view

94 cue phrases•Precise but•Low recall

Used to improve accuracy when classifying unannotated dataset

Dataset

Annotated•Elluminate text chat•Two-day conference•2,636 dialogue turns•Mean word tokens per turn: 10.14

Unannotated•Elluminate text chat•Three MOOCs•10,568 dialogue turns•Mean word tokensper turn: 9.24

Time Contribution

2:43 PM

Issues of interoperability

2:43 PM

The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages.

2:43 PM

What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps!

2:43 PM

My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model?

Manual coding of data subset

Category Description Examples include

Challenge A challenge identifies something that may be wrong and in need of correction

Calling into questionContradictingProposing revision

Evaluation An evaluation has a descriptive quality

AppraisingAssessingJudging

Extension An extension builds on, or provides resources that support, discussion

Applying idea to a new areaIncreasing range of an ideaProviding related resources

Reasoning Reasoning is the process of thinking an idea through

ExplainingJustifying your positionReaching a conclusion

Combining methods

• Train initial classifier on annotated dataset• Apply trained classifier to un-annotated data• Use self-learned features to find exploratory dialogue• Use cue-phrase matching to improve accuracy• Take context into account using k-nearest neighbours• Add selected instances to the training dataset

• Repeat for five iterations or until less than 0.5% of labels are changed

Evaluation criteriaOn a scale of 0 to 1…

Accuracy How many decisions were correct?Pilot 0.5389 SF+CP+KNN = 0.7924

PrecisionHow many ‘exploratory’ turns were actually exploratory?Pilot 0.9523 SF+CP+KNN = 0.8083

Recall How many exploratory turns were classified as exploratory?Pilot 0.4241 SF+CP+KNN = 0.8688

F1

Weighted average of precision and recallPilot 0.5865 SF+CP+KNN = 0.8331

Varying the value of k

k Accuracy Precision Recall F1

1 0.7868 0.8007 0.8666 0.8282

3 0.7924 0.8083 0.8688 0.8331

5 0.7881 0.8005 0.8685 0.8292

7 0.7586 0.7505 0.8640 0.8001

Looking at three nearest neighbours gives best results

Making use of the classifier

Each colour block represents 10 turns in the dialogueRed blocks are primarily exploratory, blue blocks primarily non-exploratory

Making use of the classifier

Total turns in the dialogue

Exp

lora

tory

tur

ns in

the

dia

logu

e

The line here is set to highlight anyone who had more than5/6 of their turns classified as exploratory

Analytics like these could be used to provide focused support to learners

Issues

Total turns in the dialogue

Exp

lora

tory

tur

ns in

the

dia

logu

e

Visual literacyHow can we share the maximum amount of information while making these analytics easy to use?

Assessment for learningHow can we use these analyticsto motivate and guide, rather than to discourage?

Participatory design How can we involve learners and teachers in learning discussions around these analytics?

Working in the middle space

Conclusion

• We proposed and tested a self-training framework

• Found it out-performs alternative methods of detecting exploratory dialogue

• Developed an annotated corpus for the development of automatic exploratory dialogue detection

• Identified areas for future research

• Identified ways of applying this work to support learners and educators

Yulan HeSenior Lecturer at the School of Engineering and Appied Science, Aston University, UK

SoLAR Storm webinarbit.ly/YSEVHG