An Initial Evaluation ofMetacognitive Scaffolding for

Experiential Training Simulators

Marcel Berthold, Adam Moore, Christina Steiner, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly,

Gary Donohoe, Gordon Power, Owen Conlan

• Who?• ImREAL Project• Evaluating Self-Regulated Learning and Metacognitive Scaffolding in Experiential

Simulator

• How?• Cross cohort comparison

• Baseline usage of sim vs scaffolded usage• Initial – no cross correlation / matching• Third year medical students

• What (did we find out)?• Performance• SRL behaviour• Scaffolding encounters & effects• Analysis of reflection text New! (Not in paper)

Overview

• create effective virtual reality training simulation tools that adapt to trainees’ past experiences or preconceptions

• closing the gap between the ‘real-world’ and the ‘virtual-world’

• Services should respond to users’ behaviour and adapt accordingly to the user model based on a pedagogical model

• Create services that can be connected to different simulators

Rationale of ImREAL

What do we want to know?

1. Is self-regulated learning supported?

2. Does the simulator augmentation through the service lead to better learning performance?

3. Does the simulator augmentation through the service increase motivation?

4. Is the service well integrated in the simulation and learning experience?

• SRL is composed of three cyclic learning phases: Forethought, Learning, Reflection (Zimmermann, 2002)

• Good SR learners use appropriate learning strategies and techniques

• Good SR learners achieve

better learning results and

are more motivated to learn

(Zimmerman, 2002; Veenmann, 2011)

SRL@ET workshop Kato Galatas, 15-06-2012

Self-Regulated Learning (SRL)

• Integrated SRL cycles + Peer experience

(Hetzner et al., 2011)

Pedagogical framework in ImREAL

Simulator

• Dialogic• Training for diagnostic interviews for psychiatric patients

Reflection Elicitation / Scaffolding

• Simulator has pre-existing reflection tool

• Can be triggered by learner• Prompts to reflect also added by

instructional designer

• In practice mode, cohort 2 students had ImREAL Metacognitive Scaffolding Service (MSS)

Phase 1 Trial

Scaffolding – Thinking Prompts

• Approximately 140 (cohort 1 = 131 / cohort 2 = 143) medical students, on average 21 years old (40% male vs. 60% female, 80% Irish). They performed the simulation as part of their third year medical training at Trinity College, Dublin (cohort 1 in 2011, cohort 2 in 2012).

• Cohort 2 additionally has scaffolding prompts in practice mode only

• Experience• No experience with ETU simulator• Experienced with interviews (97 %)• limited experience with interviewing psychiatric patients (15 %)

Cohorts

User

Reflection postings

Questionnaire Data

Log-Data

Assessment

Have you set a learning goal?

Methods

• Project evaluation question

o Can SRL be enhanced through

Metacognitive Scaffolding Services (MSS)?

• Formative evaluation approach

• Research foci of ImREALMSS

Integrated ImREAL services

• Investigate: Impact on SRL

reports, behaviour, qualitative

feedback

http://www.empowertheuser.ie

Methods: Evaluation approach in ImREAL (First User trail)

• No change in SRL compared to baseline• Highest usage of elaboration strategies

Memorizing strategies

Elaboration strategies

Organisation strategies

Planning strategies

Self-monitor-ing strategies

Time man-agement strategies

Achievement motivation strategies

Internal attri-bution

Effort strate-gies

First user trial

57.6455026455026

70.15873015873

62.6772486772486

59.1375661375661

60.005291005291

61.2169312169312

62.4047619047619

59.1111111111111

55.7777777777778

Baseline Evalua-tion

54.0687134502924

69.7953216374268

62.2368421052631

58.2353801169591

60.9078947368421

59.9269005847953

60.4385964912281

56.8728070175438

55.3004385964912

5

15

25

35

45

55

65

75

85

95

SRL strategies profil

SRL strategy scales

%

t20=3.34, p=.003

Results: SRL

• Scores are above centre line of the rating scale• Mentioned that prompts were not always presented at appropriate time

1

2

3

4

5

3.42 3.333.09

3.392.97 3.12 3.27 3.38 3.45 3.42 3.26

Learning Experience MSS

Items

Ra

tin

gs

Results: Learning Experience

• Students were motivated• To perform simulation• Learn about interviews• To apply skills learnt in simulation in real world

• State motivation after simulation higher in 1st user trial compared to baseline

Baseline Evaluation First User Trial0

1

2

3

4

2.48

3.35

Motivation

Trial

Ra

tin

gs

t118.47=-8.64, p<.001

Results: Motivation

• Comparison of expected and empirical MSS prompt distribution• More scaffolds in Information Management phase• Less scaffolds on reflection

Planning Informantion Management

Monitoring Debugging Reflection

Expected distri-bution

400.2 571.714285714286

400.2 301 343.028571428571

Empirical distri-bution

469 752 425 301 54

50

150

250

350

450

550

650

750

Expected vs. Empirical Distribution of MCSs

Phases

Fre

qu

enci

es

χ2(4,0.95)= 314.55, p<.001

Results: Scaffolding

• Positional – time, date, simulator, etc.• “Rapport and Initial inquiry have been completed. Must now concentrate on moving

the interview forward”

• Technical – material addressing the system, questions or underlying model • Great concept. Has a lot of potential, will be much better when more points to consider

and points of information are added to the overview that accurately reflect performance.

• Notes – observations of the patient • “Px. unable to quantify length of period she has been like this for.

• Reflections – reflective text • it is quite important to deal with the patient with empathy to make them comfortable

and also to try and illicit the cause of the bout of depression, in this case. I felt I did not do too bad as far as expressing empathy is concerned, allowing the patient to open up and try and formulate a management plan. More experience in history taking will be the key

Methods: Coding reflective text

• Corpus 1 (from cohort 1): had 358 unique text entries from 81 unique UserIDs giving an average of 4.41 entries per unique UserID• Engaged percentage = 57.9%

• Corpus 2 (from cohort 2): had 107 unique text entries into the note-taking tool from 50 unique UserIDs giving an average of 2.14 entries and 3418 views of the MST, resulting in 28 entries, 19 with text from 8 unique UserIDs• Engaged percentage = 35.7%

Results: Reflective Text

• percentage breakdown of each type of content within the note-taking tool across the two cohorts:

Position Technical Notes Reflection

Cohort 1 15 33 25 69

Cohort 2 17 57 16 66

Results: Reflective Text

• SRL• No correlation of log-data and SRL reports were observed – needs to be investigated

why? • No changes in SRL – long-term process

• MSS• Learners seem to need more assistance in effectively processing information by hints to

use more organizational, elaborative, summarizing or selective learning strategies• Are they more confident in the reflection phase and so pass the offer of scaffolds

• Time to engage• Less availability of simulators in cohort 2• Less lab time & shorter overall period

• Overall engagement and motivation increased• Higher percentage of time being scaffolded

Discussion

• Provide MS at appropriate times

• Keep learners longer in simulation

• Provide additional services to promote SRL

Lessons Learnt

What did we find out?

1. Is self-regulated learning supported?• Yes • Is it improved . . . . Maybe

2. Does the simulator augmentation through the service lead to better learning performance?

• No! But . . . . Yes!

3. Does the simulator augmentation through the service increase motivation?

• Yes

4. Is the service well integrated in the simulation and learning experience? • Yes

• Longitudinal study• SRL changes slow• Integration of changes into behaviour

• Fuller analysis of competency• Metacognitive / SRL as well as that trained by the simulator• Surfacing of reflection / competency – social?

• Affect support• Encouragement / engagement / motivation• Please take our survey!!!

Further Work

• The research leading to these results has received funding from the European Community's Seventh Framework Program (FP7/2007-2013) under grant agreement no 257831 (ImREAL project) and could not be realized without the close collaboration between all ImREAL partners.

Acknowledgements

http://bit.ly/smILEYCase sensitive!!!

Adam.moore@tcd.ie

@adam__moore

@ImREAL_project / www.imreal-project.eu

Thank-you!