Assessing the Teaching of Procedural Skills: Can Cognitive-

Task-Analysis Add to our Traditional Teaching Methods?

Maura Sullivan, PhD, Adrian Ortega, MD, Nir Wasserberg, MD, Howard Kaufman, MD,

Julie Nyquist, PhD, Richard Clark, EdD

University of Southern California

Cognitive-Task-Analysis (CTA)

Extends traditional task analysis to capture information about both the overt observable behavior and the covert cognitive functions behind it to form an integrated whole.

Schraagen, Chipman & Shalin, 2000

Acquiring ExpertiseDeclarative vs. Procedural Knowledge

Cognitive

Associative

Autonomous

Expertise Expertise is highly automated

Ericsson, et al. 2006; Gagne et al. 1993; Clark & Estes, 1996

Automaticity interferes with articulationClark et al., 2007, Feldon, 2004, Crandall & Getchell-Reitter,

1993, Chao & Salvendy, 1994

Learners need to be taught decision making strategies

Ericsson et al., 2006; Clark et al. 2007

Expertise cont.

Experts underestimate novice difficultyHinds, 1999

Experts not fully aware of 70% of own decisions Clark & Elen, 2006; Feldon & Clark, 2006

Experts’ decisions can be put into training materialsMerrill, 2002; Schaafstal et al. 2000

CTA Gains Overt observable behavior

Deconstruct automated knowledge into concrete steps

Covert cognitive functions Identification of decision points throughout

procedure Options related to each decision point

Ability to gain consensus amongst experts Provide residents with an advanced organizer

Purpose

To determine if surgeons omit relevant steps and decision points when teaching a colonoscopy

To determine if CTA can augment our traditional teaching methods

Methods Three experts videotaped teaching a

colonoscopy followed by “free recall”

Participated in CTA

Procedural Checklist and Cognitive Demands Checklist created

Videotape transcriptions transposed and omitted steps identified

Results

“What” to do

26-step Procedure

14 Decision Points

Surgeon A 61% 43%

Surgeon B 46% 25%

Surgeon C 50% 38%

Results cont.

26-step Procedure

14 Decision Points

Surgeon A 61% (50%) 43%

Surgeon B 46% (30%) 25%

Surgeon C 50% (26%) 38%

“What” to do“How to” do it

Conclusions

Experts did not articulate all relevant steps while teaching a colonoscopy or during free recall

CTA provides a means to capture omitted steps and decision points during traditional teaching

Limitations

Small N # of experts # of cases observed

No established “Gold Standard” No inter-coder reliability

Future Studies Quantify

educational effectiveness

Determine number of experts needed

Determine number of cases required

Establish reliability Accelerate the

learning curve

Implications

CTA can be used to develop more comprehensive curricula for teaching procedural skills

Assures exposure to all steps/decisions involved in a procedure

More complete training of residents

mesulliv@usc.edu(323) 442-2368