A Research Approach to Develop Measurable Competencies and Skills:

Videogames or Real-World Training

K. Anders Ericsson

Department of PsychologyFlorida State University

by

One of the crucial issues of traininginterventions is to be able to specify their goals so one can measure andevaluate objective outcomes.

I believe that videogames and simulatorscan provide us with the tools to specifythe goal and targets of educationfor everyday and professional life.

Getting away from paper-and-pencil or multiple-choice tests of the type of Scholastic Aptitude Tests (SAT)

Define outcomes with video games and simulations of everyday situations and tasks

TraditionalInstruction

On-the-jobexperience Target

Performance

“Retain skills and knowledge”“learning transfer”

“generalization”

TraditionalInstruction

On-the-jobexperience Target

Performance

“Retain skills and knowledge”“learning transfer”

“generalization”Alternatives,such as

SimulationsVideogames

Outline of My PresentationI. If the desired competency concerned a domain,

like Chess, Music, or Sports, how would expert performance be attained

II. Objective measurement of performance

III. Mechanisms, skills, and their acquisition through deliberate practice

IV. Acquisition of control and complex mental representations— Beyond automaticity V. Training and the fidelity assumption

I. If the desired competency concerned a domain, like Chess, How is Expert Performance Attained

How did chess players attain their performance?

Benjamin Franklinlearned to play chess from an acquaintancein 1733 He was one of the firstchess players in America. Although he was an avidplayer he reached a modestlevel of chess skill

Our Knowledge about Learning and Skilled Performance is Limited

How good is a chess player or an individual in a domain of expertise?

--Self ratings

Professors Drivers Doctors(94% above average) (80% above average) (Davies et al., 2006)

Self-Assessments on Relative PerformanceHow would you rate your own performance?bottom 10% below average average above average top 10%

Professors Drivers Doctors(94% above average) (80% above average) (Davies et al., 2006)

Self-Assessments on Relative PerformanceHow would you rate your own performance?bottom 10% below average average above average top 10%

How could a chess player in the chess club be able to keep improving?

Playing games

Watching better players play

Reading books

Improving One’s Chess Performance orPerformance in Tennis, Golf or SCRABBLE

Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)

Active participation Noin chess tournaments 6.1 correlation

Playing chess games Negativeoutside of chess 3.6 correlationtournaments

Serious analysis of Positivepositions alone 5.9 correlation

Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)

Active participation Noin chess tournaments 6.1 correlation

Playing chess games Negativeoutside of chess 3.6 correlationtournaments

Serious analysis of Positivepositions alone 5.9 correlation

Effects and Dose-Response Relations to Practice Activities

Professional teachers and coaches

* Monitor students’ development* design training tasks for individual students

Expert Performance

Rated Relevance for Improvement from Doing Xon a scale from 1 to 7 (Charness, Krampe, & Mayr, 1996)

Active participation Noin chess tournaments 6.1 correlation

Playing chess games Negativeoutside of chess 3.6 correlationtournaments

Serious analysis of Positivepositions alone 5.9 correlation

Simulated Play Against World Class PlayersStudy published games by chess masters

Make predictions for each next moveCheck if your prediction was correct, if not, study the chess

position until you understand why the correct move was played

Black on move

Position B

On-the-job supervised experience

Classroom Knowledge Acquisition Measurable

Outcomes

Part II. Objective Measurement of Performance

Independent ability to increase patients’ health

Kirkpatrick’sFour LevelModel ofTraining

Evaluation

The most respected experts (“expensive” stockbrokers) are not markedly better in picking stocks on New York Stock Exchangethan a random process, such as a monkey throwing darts (c.f. Wall Street Journal study)

Superior PerformanceChess

winning chess games

50 games (100-250 hours) for a rating

Psychotherapysuperior ability to increase

mental patients’ health (post-pre)50-100 patients (400-800 hours)

Teachingsuperior ability to increase

students’ performance (post - pre)

500-1000 contact hours

MinimalEffects of

Traditional Indicators

of Expertise

Long Education

Extended experienceas a professional

Teaching

Psychotherapy

Identify challenging and difficult situations, where experts are supposed to excel.

Adriaan de Groot (1914–2006)

Recreate the situation

and taskin laboratory

Actions and thoughts

of novicesand experts can

be directly compared

Select the best chessmove for this position

Type as much of thepresented text as possiblewithin one minute

Play the same piece ofmusic twice in samemanner

Chess

Typing

Music

Domain Presented Information Task

Selection

Speed

Control

Reproducible Soccer Performance

From cover story in Time Magazine (March 10, 2008) on The Science of Experience

Response to Critical Unexpected Problems

Salchow

Part III. Mechanisms, Skills, and their Acquisition through Deliberate Practice

The problem with learning during work and play

Andy Murray (Scotland) in 2010

The problem with learning during work and play

Immediacy of Informative Feedback

Sport training performance(1-2 years)

Investing value (10-20 years)

Medicine patient outcomes (12-36 months)

Increase in Complexity and Control as a Function of

Years of Piano Training

Yearsof

pianotraining

“individualized training activitiesespecially designed by a coach or teacher to improve specific aspects of an individual's performance through repetition and successive refinement.

To receive maximal benefit from feedback, individuals have to monitor their training with full concentration, which is effortful and limits the duration of daily training”.

(Ericson & Lehmann, 1996, pp. 278-279)

Deliberate Practice

Design and Sequencing of Training Activities

Professional teachers and coaches

* Monitor students’ development* design training tasks for individual students

Expert Performance

Accumulated Amount of Practice Alone During the Development

of Amateurs and Expert Musicians

The Body’s Resilience to Change The Protection of Equilibrium

Increase of Fitness

DeliberatePractice

Optimal

How many consecutive push-ups can someone perform without pause?

(H. H. Hart, 1974)

Range of modifiability with extendedDeliberate Practice

Charles Linster

Non-stop push-up records

http://www.recordholders.org/en/list/ulysses.html

6,006 Charles Linster (USA) 05-Oct-1965

…

10,507 Minoru Yoshida (JAP) Oct-1980

Fu Bingli, a kung fu master from China 12 press ups with right fingerhttp://www.telegraph.co.uk/news/newstopics/howaboutthat/6385592/Man-proves-he-has-worlds-strongest-fingers.html#Accessed on October 23rd, 2009

Part IV. Acquisition of Control and Complex Mental Representations—Beyond Automaticity

The Development of Increased ControlActively Avoiding Automating Control of Critical Aspects of Performance

Reproducibly Superior Performance is not automatic but is associated

with informative verbalizations of thoughts

Black on move

Position B* What aspect was overlooked* When could this aspect have been

discovered* How to avoid similar mistakes in the future* Develop new skills by deliberate

practice

Learning from Making an Incorrect Move

Alekhine beat most of the 30 skilled players while playing them simultaneously under blindfold conditions

Ref

P16A

http://images.google.com/imgres?imgurl=http://www.chessbase.com/news/2007/bilbao04.jpg&imgrefurl=http://www.chessbase.com/newsdetail.asp%3Fnewsid%3D4189&h=318&w=480&sz=36&hl=en&start=1&usg=__JwOiqXRmMEJcHCssYGF1N_5BqU4=&tbnid=DC9VzEbhAUofSM:&tbnh=85&tbnw=129&prev=/images%3Fq%3Dblindfold%2Bchess%26gbv%3D2%26hl%3Den%26sa%3DGAccessed on August 26, 2008

Blindfold Chess

Exceptional abilities

The Importance of Acquired Mental Representations

Expert Performers are Better Able to Represent and Analyze Situations

Slowing Down(Moultonet al.,2010)

(Ward &Williams, 2003)

(Tuffiash, Roring, & Ericsson, 2007)

(Kesting, et al., 2010)

V. Training and the Fidelity Assumption

It is frequently assumed that the best learning of skills occurs in the natural environment .

Hence, simulation should therefore reproduce the experience in the natural environment

What do we know about natural learning?How optimal is that type of natural learning?

Perceptual Performance on Cardiac Auscultation

Instruction and Experience

0-9 years 10-20 years

Over 20 years

General Practitioners

Student

Based on Butterworth & Reppert (1960)

Reviews (Choudhry, Fletcher, & Soumerai, 2005; Ericsson, 2004; Ericsson, Whyte, & Ward, 2007)

Perform

ance

No Correlations between Amount of Experience and Performance after the First Year

Benefits of Provided Experienceon Performance

Use of Cockpits with Visual and Kinesthetic Simulation

Modest transfer of performance—savings of training in real airplanes (except new procedures)

(Allerton, 2000; Roessingh, 2005; Rantanen & Talleur, 2005)

Focus on Fidelity and SafetyA airline captain needs 3,500 hours of flying plus 60 hours of simulation

(Parker, Johns, & Hellige, 2007)

Identifying the Crucial Elements of Effective Simulator Training - Deliberate Practice

A review of 109 studies of high-fidelity medical simulation showed thatthe characteristics of Deliberate Practice accounted for improvements

(Issenberg et al., 2005)

A review of 31 of those studies that examined repetitive practice, and “found a strong association (η2 = .46) between hours of practice on high-fidelity medical simulators and standardized learning outcomes.” (Issenberg et al., 2006, p. 792)

The Use of Simulators for Deliberate Practicewith Performers at Different Levels of Expertise

Allows exposure to challenging situations without danger

Allows exposure to difficult situations under conditionsoptimal to learning and performance - Individuals being focused and ready - Immediate feedback (and opportunity for repetition) - Presentation of related cases to facilitate discrimination

Allows the presentation of rare emergency situations

Deliberate Practicewith Simulators

Authentic test conditions with actors(Kneebone et al., 2005)

Reinstating actual scenariosin anesthesiology (Liu et al, 2009)

Measurement and TrainingCapturing the essence of expert performanceMeasuring it with representative tasks

Measuring Identification

current level of of optimal

performance training tasks

TraditionalInstruction

On-the-jobexperience Target

Performance

“Retain skills and knowledge”“learning transfer”

“generalization”Alternatives,such as

SimulationsVideogames

RecommendationsIdentify skills and analyze domains of activity

where performance can be measured with objective methods that capture on-the-job performance

with large individual differences in attained performance where increases in performance motivate major investment

Develop libraries of representative situations with appropriate feed-back about correct/appropriate actionswith scaled difficulty to maximize optimal training effects

Develop cognitive structures to support Deliberate Practiceassessment of representations for thinking, planning, and

evaluations in Long-term working-memory (LTWM)essential for “Retain[ing] skills and knowledge”

“learning transfer” “generalization”

The Complex Process of

Acquisition ofof Expert

Performance