Training complex judgmentThe effects of critical thinking instruction and contextual interference

Anne Helsdingen1, Tamara van Gog1, Jeroen van Merriёnboer1, 2, 1CELSTEC, 2Netherlands Laboratory for Lifelong Learning (NeLLL),

Open University of the Netherlands

An example

Your task: Judge the priority for the police

• Which variables to take into account?– E.g. weapon, condition of the victim, intention of perpetrator, any

camera pictures,…

• How do they relate to the priority?– Priority higher when victim is seriously wounded?

– Priority lower when there is no description or camara image of perpetrator?

• First priority estimate is a guess, but then you learn– On the basis of outcome feedback

– The relationship between relevant variables and the criterion

Complex judgment tasks

• Many judgment and decision problems are similar: – Military command and control

– Fire commander

– Medical diagnosis

• Studying experienced decision makers revealed: – Recognition based decisions

– Meta-cognition skills to monitor decision strategies

• Critical thinking instruction

Critical thinking instruction (Cohen, Freeman, & Thompson, 1998)

• 1) Story telling: try to incorporate all observations, all cues in one comprehensive explanation, one story. (templates)

• 2) Story testing: what is missing, what is conflicting with other information?

• 3) Story evaluation: the Devil’s Advocate tells you that your critical assumption is not true. Develop an alternative story that als explains all observations.

• 4) Develop plans and contingencies

Optimizing complex judgment training

• Does critical thinking instruction lead to better post training performance in complex judgment?

• How to prepare decision makers for the variety of problems?

• How to enhance transfer from learning task to other tasks?

• Manipulating sequence of learning tasks: Random sequence to enhance contextual interference

Contextual Interference

• Random sequence of learning tasks benefits learning (compared to blocked)

• In random: Learners compare between different tasks - elaborate

• In random: learners have to recollect different strategy vs having a strategy ready - reconstruct

• In blocked: an illusion of competence develops

Research questions

• Does critical thinking instruction enhance post training performance in complex judgment tasks?

• Does the contextual interference effect manifest itself in learning complex judgment tasks?

• What are the combined effects of critical thinking instruction and practice schedule?

Research questions

• Does critical thinking instruction enhance post training performance in complex judgment tasks?

Yes• Does the contextual interference effect manifest itself in learning

complex judgment tasks?

Sure

• What are the combined effects of and critical thinking instruction and practice schedule?

Critical thinking instruction in random and blocked practice

• Lab experiment (N=120), University students.• Task: estimate the priority (0-100) of a case for the police force

– Priority based on four dichotomous variables that were present in each case.

– Identify the variables and determine relationship with priority

– Outcome feedback only

• 2x3 between subjects design:– Blocked vs random sequence of learning tasks between subjects

– Critical thinking prompts: before task, after task and no prompts

• Performance measurement is difference between real and estimated priority

• During learning, on near transfer and far transfer

Critical thinking instruction

Critical thinking prompts - examples

• Are (Were) there any similarities between the following (the last) two cases? Which are (were) they? And what is (was) different between these cases?

• The following (last) two cases (will) differ (-ed) from the former two (preceding) cases on one specific aspect. Can you tell me what that is (was) ? And does (did) it have an effect on the priority of a case?

• Do you consider some aspects to be more important than others when estimating a case’s priority?

Hypotheses

• Transfer paradox:– Blocked practice yields better performance during learning

– Random practice leads to better performance on retention and transfer tests

• Critical thinking prompts enhance transfer test performance:– Blocked practice with proactive prompts yield higher transfer test scores than

without or with retrospective prompts

– Random practice with retrospecitive prompts yield higher transfer test scores than without or with proactive prompts

• During a blocked practice, learners develop an illusion of competence– Higher certainty during blocked practice learning than during random practice

– This is counteracted by critical thinking: blocked practice with proactive promts lower certainty than without prompts

Performance

• Performance during learning– Blocked practice showed better performance during learning phase

– In random practice, retrospective prompts yielded better performance

• Performance on near transfer– Random practice resulted in better performance than blocked

– After random practice with refrospective prompts better performance

• Performance on far transfer– After random practice better far transfer performance

– After random with retrospective prompts best scores

– After blocked practice with proactive prompts performance at similar level as after random practice

Certainty

• Certainty during learning

– No effect of practice schedule

– In blocked practice, proactive prompts resulted in higher certainty

– In random practice, no prompts condition yielded higher certainty

• Certainty on near transfer– After random practice higher certainty (+performance )

– After blocked practice with proactive prompts higher certainty (+performance)

• Certainty on far transfer– Higher certainty after random practice schedule (+performance)

– Higher certainty after blocked with pro-active prompts (+performance)

Discussion & conclusions

• √ Transfer paradox:

– Blocked practice yields better performance during learning √

– Random practice leads to better performance on transfer tests

• √ Critical thinking prompts enhance transfer test performance:– Blocked practice with proactive prompts yield higher transfer test scores than

blocked practice without or with retrospective prompts

– Random practice with retrospecitive prompts yield higher transfer test scores than random practice without or with proactive prompts

• ? During a blocked practice, learners develop an illusion of competence

– ᵡ Higher certainty during blocked practice learning than during random practice

– ᵡThis is counteracted by critical thinking: blocked practice with proactive promts lower certainty than without prompts

How about….• ….critical thinking?

• Better judgment strategy & deeper understanding of underlying mechanisms

• What aspects support learning: reflection, self-explanation, …..

• …contextual interference?• Benefits learning complex judgment: deeper understanding• Extra processing or illusion of competence ?

• ….combining the two?• Timing is everything

• Then even blocked practice may be adequate for learning

• It doesn’t hurt to try

• …future research• Are people capable to judge their own learning in these complex tasks?

• How can we measure judgments of learning continuously

and unobtrusively?