THE USE OF COGNITIVE TASK ANALYSIS TO INVESTIGATE HOW MANY

EXPERTS MUST BE INTERVIEWED TO ACQUIRE THE CRITICAL

INFORMATION NEEDED TO PERFORM A CENTRAL VENOUS CATHETHER

PLACEMENT

by

Craig W. Bartholio

A Dissertation Presented to the FACULTY OF THE USC ROSSIER SCHOOL OF EDUCATION

UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the

Requirements for the Degree DOCTOR OF EDUCATION

December 2010

Copyright 2010 Craig W. Bartholio

ii Dedication

There are many people that help support me during the entire three years of

completing this doctoral program and dissertation. I want to first thank my parents

for believing in me and supporting the whole process through multiple means. They

have always been a huge believer in my abilities even when I had my own doubts.

Secondly, I would like to thank my extended family for all their verbal

encouragement, baby sitting, inspiring phone conversations, editing, and all around

understanding of my multiple priorities and commitments.

To my three children, Chloe, Noah, and Hannah. Each of you has spent

several nights sleeping next to Daddy while he typed late into the night, and

sometimes into the next morning. I want to thank you for understanding all the times

I could not “play” with you because Daddy had to “study” or “write”. I have a lot of

playing activities to catch up on with each of you.

Lastly, I want to thank my wife for all her continued support. Without her

encouragement, frank realism, and editing abilities (far superior to mine), and the

ability to always understand the real perspective of importance in life, I share this

accomplishment with you.

iii ACKNOWLEDGEMENTS

I would like to take this time to express my deepest gratitude, appreciation,

admiration, and thankfulness to the faculty on my dissertation committee: Dr.

Richard Clark, Dr. Kenneth Yates, and Dr. Maura Sullivan without whom I would

not have been able to accomplish this huge undertaking.

To Dr. Richard Clark, I am in continual awe of your knowledge and ability to

express the complex in simple understandable terms. I want to offer sincere

gratitude for all of the time, patience, and sharing of his knowledge that allowed me

to accomplish this feat. I am thirsty for knowledge in the area of cognitive task

analysis and look forward to continuing to add to the current body of knowledge in

this field.

To Dr. Kenneth Yates, first of all thank you for inspiring me to switch from

the K-12 Leadership concentration to the Educational Psychology concentration. It

has been a wonderful and thrilling ride. I appreciate all the significant amount of

time effort you have provided me in achieving this accomplishment. You have

provided many wonderful insights into what it takes to write a dissertation. I am

deeply indebted to you.

To Dr. Maura Sullivan, without you initial support and guidance, I’m not sure

I would be writing this acknowledgement. Thank you for your patience and

understanding of this dissertation process. The insights on organizing my writing,

iv continual support and the sharing of your significant other to the good of the cause,

thank you.

Finally, to all my fellow cohort members Eko, Joon, Julia, Mary Ann, and

Lesile, and especially Patrick, thanks for your wonderful continual support.

v TABLE OF CONTENTS

Dedication ii

Acknowledgements iii

List of Tables vii

List of Figures viii

Abstract x

Chapter 1 1 Statement of the Problem 1 Review of the Literature 5

Current Trends in Surgical Training 5 Development of Expertise 7 Expertise in Medicine 8 Expertise in Surgery 9 Knowledge Types 10 Declarative Knowledge 11 Procedural Knowledge 13 Automaticity 14 Automaticity and Expert Recall 16 Automaticity and Expert Recall in Medicine and Surgery 18 Cognitive Task Analysis 20 Defining Cognitive Task Analysis 20 CTA Methodology 21 Effectiveness Studies using CTA 23 Effectiveness of CTA in Surgical Training 29 Limitations of CTA 35 Number of Experts Required for CTA 37 Summary 39 Purpose of the Study 40

vi Chapter 2: Method 42 Design 42 Subjects 43 Data Collection 44 Semi-structured CTA Interviews 44 CTA Coding Scheme and Procedure 45 CTA Protocol and creating a six-Subject Matter “Gold Standard” CVC Protocol 46 Data Analysis 47 Chapter 3: Results 52 Coding and Inter-rater reliability 52 Summary 74 Chapter 4: Conclusion s 76 Research Questions 76

Summary 84 Limitations and Implications 85 Conclusion 87 References 89 Appendices Appendix A: Non-Repeating Combinations of SME Protocols 98 Appendix B: CTA Coding Scheme 99 Appendix C: CTA Gold Standard for Central Venous Catheter 100

vii

LIST OF TABLES

Table 1: Excerpt from Central Venous Catheter Protocol Spreadsheet, Single Expert 48

Table 2: Excerpt from Central Venous Catheter Placement Protocol

Spreadsheet: Multiple Experts 50 Table 3: CVC Gold Standard Sections and Corresponding Number

Of Items per Section 53 Table 4: Percentage of Knowledge Acquired from an Expert When

Compared to the Six-Expert CVC Gold Standard Protocol (Experts A-F) 54

Table 5: The average percent of Total Knowledge, Action Steps,

Decision Steps per Number of Experts when compared to a Six-SME Gold Standard Protocol 56

Table 6: Average Percentage Increase of Total Knowledge, Action Steps,

And Decision Steps Acquired from Multiple Groups of Expert When compared to a Six-expert Gold Standard CVC Protocol 57

Table 7: Average Percentage of Objectives, Reasons, and Risks Acquired

From Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol 63

Table 8: Average Percentage of Indications, Contraindications, and

Standards Acquired from Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol 67

Table 9: Average Percentage of Equipment and Tasks acquired from

Multiple Groups of Experts when compared to a Six Subject Matter Expert Gold Standard Protocol 72

Table 10: Quantity of Experts Recommended if a 10% Marginal

Utility in Knowledge Acquisition is Expected 75

viii LIST OF FIGURES

Figure 1: Percentage of Acquired Knowledge Based on Six-Expert Gold Standard CTA CVC Procedure Protocol as a Function Of the Number of Experts 58

Figure 2: Average Percentage Increase of Knowledge as a Function

Of an Additional Expert for Total Possible Items in the Gold Standard CVC Procedure Protocol 59

Figure 3: Average Percentage Increase in Action Step Acquired

Knowledge as a Function of the Number of Experts 60 Figure 4: Average Percentage Increase of Action Steps as a

Function of Additional Experts 60 Figure 5: Average Percentage Increase of Gold Standard

Decision Steps as a Function of the Number of Experts 61 Figure 6: Average Percentage Increase of Decision Steps as a

Function of Additional Experts 62 Figure 7: Percentage of Knowledge Acquired of Risks as a

Function of the Number of Experts 63 Figure 8: Average Increase of Acquired Knowledge of Risks as a

Function of Added Experts 65 Figure 9: Percentage of Knowledge Acquired for Reasons to Perform

The CVC Procedure Correctly 65 Figure 10: Average Percentage Increase in Acquired Knowledge in the

Area of Reasons Based on a Six-Expert Gold Standard CVC protocol 66

Figure 11: Percentage of Indications to Perform the CVC Procedure

Based on a Six-Expert Gold Standard as a Function of the Number of Experts 67

ix Figure 12: The Average Percentage of Acquired Knowledge Gained as a

Function of Additional Experts in the Area of Indications to Perform a CVC Procedure 68

Figure 13: Percentage of Knowledge Acquired in the Area of

Contraindications as a Function of the Number of Experts 69 Figure 14: The Average Percentage Increase in the Amount of

Acquired Knowledge in the Area of Contraindications As a Function of the Number of Experts 70

Figure 15: Percentage of knowledge acquired for standards of CVC

Procedure as a function of the number of experts 71 Figure 16: Average Percentage Increase of Acquired Knowledge for

Standards Based on a Six-Expert Gold Standard CVC Protocol 71 Figure 17: The Average Percentage Gain of Acquired Knowledge of

Needed Equipment as a Function of Additional Experts 73 Figure 18: The Average Percentage Gain of Acquired Knowledge of

Needed Equipment as a Function of Additional Experts 74

x ABSTRACT

The purpose of this study was to examine the amount of relevant information

experts provide and fail to provide when asked to describe how to perform a

complex task in enough detail for students to perform the task. In this study,

medical experts where interviewed because their past successes and failures at

the task are known and so it could be determined that all experts had succeeded

consistently at the task being described. Past research has suggested that because

experts have both conscious and unconscious automated knowledge they may not

“know what they know” and so not be able to completely describe how to make

critical decisions during task performance. A version of Cognitive Task Analysis

designed to support training was used to interview medical school faculty and

analyze their description of a controversial trauma procedure in order to

determine the average percentage of knowledge that was acquired from a single

expert and how much additional knowledge is acquired from each succeeding

expert interviewed. After analysis, it was determined that the amount of

knowledge acquired from one expert was about 63 percent and the average

increase of acquired knowledge with the second expert was about 16% and the

third expert added another 8 percent. Past studies have reported considerably

lower percentages of relevant information captured with Cognitive Task Analysis

– about 30 percent of decisions are typically reported for example. Thus it was

xi hypothesized that information about the controversial procedure examined in

this study may have become more conscious to the experts interviewed because

they had been discussing it among themselves and reading accounts in journals.

1 CHAPTER 1

STATEMENT OF THE PROBLEM

Surgical skills training has evolved in recent years from the traditional “see one-

do one-teach one” model (Halsted, 1904) to methods that focus more on instruction

by expert surgeons and various forms of technology, such as computer based

learning; virtual reality (VR); and high-fidelity bench models (Vozenilek, Huff,

Reznek, & Gordon, 2004). These technologies increase surgical residents exposure to

a wide array of procedures while practicing their proficiency of technical skills prior

to real surgical exposure (Aggarwal, Grantcharov, & Darzi, 2007; Reznick &

MacRae, 2006;Vozenilek et al., 2004).

The common thread for both the traditional apprenticeship and current surgical

skills training programs is that expert surgeons are often relied upon as the “master

teacher” to inform and the curriculum content, training procedures, and simulation

software. However, research indicates that experts can omit up to 70% of

information when explaining to others how to perform a task (Clark, Pugh, Yates,

Early, & Sullivan, 2008). Additionally, the transfer of knowledge from experts to

novices is not easily accomplished (Hinds, Patterson, & Pfeffer, 2001). This may

have serious consequences for surgical training in which surgical experts may not

provide a full account of the knowledge and skills required to perform a procedure

(Abernethy, Poolton, Masters, and Patil, 2008). Thus, surgical residents may be

provided incomplete knowledge about the behavioral tasks of a procedure, but, more

2 important, the decision-making and cognitive processes that are inherently involved

with performing these procedures (Jacklin, Sevdalis, Darzi, & Vincent, 2008).

Cognitive task analysis is a generic reference to a variety of methods that have

been demonstrated to effectively capture the knowledge and skills experts use to

perform complex tasks (Crandall, Klein, & Hoffman, 2006; Hoffman & Militello,

2009; Clark, Feldon, van Merrienboer, Yates, & Early, 2008). There are many

definitions of cognitive task analysis (CTA). For example, Crandall, Klein, and

Hoffman (2006) define CTA as a “family of methods used for studying and

describing reasoning and knowledge [including] the activities of perceiving and

attending that underlie performance of tasks [and] the cognitive skills and strategies

needed to respond adeptly to complex situations” (p. 3). Whereas, Clark et al. (2008)

define CTA as the usage of “interviews and observation strategies to capture a

description of knowledge that experts use to perform complex tasks.” The common

thread among these definitions of CTA is that the goal of CTA is to capture the

underlying knowledge and skills experts use to solve complex tasks.

Information captured from experts through CTA has been shown to be effective

for training novice physicians (Velmahos, Toutouzas, Silin, Chan, Clark, Theodorou,

& Maupin, 2004; Sullivan, Brown, Peyre, Salim, Martin, Toowfugh, & Grunwald,

2007; and Luker, Sullivan, Peyre, Sherman, & Grunwald, 2008). For example,

Velmahos et al. (2004) found that surgical residents who received CTA based

instructions on how to perform a CVC procedure performed better on a 14 point

procedure checklist when compared to a control group taught through traditional

3 methods. Additionally, Sullivan et al. (2007) found that general surgery residents

performed significantly better on post -instruction technical assessments at both a

one-month and six-month benchmarks when instructed on how to perform a

percutaneous tracheostomy (PT) utilizing a CTA based curriculum. In another study,

Luker et al. (2008) found that although technical skills are developed through task

exposure, the insertion of a CTA based multimedia instructional program

significantly increased the knowledge and skills of the surgical residents completing

flexor tendon repair procedure. And, in a recent study, Tirapelle (2010) found that

randomly assigned surgical residents who were provided CTA-based instruction

improved in their knowledge and skills on how to conduct an Open Cricothyrotomy

when compared to a control group taught with traditional methods. These studies

emphasize the positive impact a CTA based curriculum has on surgical residents’

knowledge and skills in performing complex procedures.

It is often recommended in the CTA literature that multiple experts be consulted

to increase the validity and reliability of CTA results (Yates, 2007; Yates & Feldon,

in press). However, many researchers note that the recommendation to use multiple

experts creates a “bottleneck” due to the additional time and resources required

during the CTA process (Hoffman, Crandall, and Shadbolt, 1998; Hoffman, Shadbolt,

Burton, & Klein, 1995). As such, a question often asked is how many experts are

required to capture the optimal knowledge needed to perform a complex task? There

is a paucity of research that has studied this problem. An exception is Chao and

Salvendy (1994), who examined different techniques of acquiring knowledge to

4 diagnose computer software programs. While Chao and Salvendy (1994) found that

a single expert provided an average of 27% to 40%, depending on the software-

debugging task, the percentage of knowledge and skills increased as they interviewed

additional subject matter experts up to a total of six individuals. Chao & Salvendy

(1994) concluded three experts were needed to acquire the optimum critical

knowledge and skills needed to solve a complex task.

To date, there have been no published studies within the field of surgical training

that examine the number of surgical experts required to capture the optimum

knowledge and skills required to perform complex procedures. A study conducted

concurrently with this study sought to examine this question for the cricothyrotomy

procedure and found that 3 experts were required to capture the actions and decision

steps necessary to perform the procedure (Crispen, 2010). The current investigation

seeks to replicate the methodology used by Crispen (2010) to determine the optimal

number of experts required for the procedure to place a central venous catheter.

As such, this study poses two research questions: 1) How much information about

a central venous catheter placement procedure does a single expert provide when

compared to the combined contributions of a comprehensive “gold standard” protocol

based on interviews with six physician subject matter experts; 2) How much critical

information is gained from each additional interview about a central venous catheter

placement?

5 REVIEW OF THE LITERATURE

To understand the contributions of cognitive task analysis and its potential benefit

to medical education, we must first review the current trends in surgical training and

the inherent complexities of relying on expert physicians as instructors/mentors for

training surgical residents. A review of both declarative and procedural knowledge

types is critical to understanding how experts construct knowledge and sometimes

have difficulty communicating acquired knowledge. Often, subject matter experts

(SMEs) omit critical information when describing how to perform a task thus.

Research shows that surgeons can omit up to 70% of the critical information needed

to perform a procedure (Clark, Pugh, Yates, Sullivan, 2008). Thus, capturing this

information is a critical component to helping train surgical novices. Cognitive task

analysis provides a method to capture this omitted critical information. Therefore, a

review of the use of CTA and it’s reported effectiveness both within and outside the

medical field is presented. The review of the literature section will conclude with a

summary of recent studies that examined how many experts are required to capture

the critical knowledge needed to perform a complex task.

Current Trends in Surgical Training

Surgical training has experienced a paradigm shift over the last several decades

from the Halstedain-based apprenticeship model to the development of surgical skills

training facilities (Hamdorf and Hall, 2000). This shift is due to advancements in

technology and the need to expose surgical residents to multiple procedures to

increase the knowledge and skills of surgical residents outside the operating room

6 (Scott, Cendan, Pugh, Minter, Dunnington, & Kozar, 2008). As a result, university

medical education facilities are modifying their programs to utilize discrete,

procedure specific trainings, in an effort to provide the needed skills training for

proficient physicians (Sachdeva, Bell, Britt, Tarpley, Blair, & Tarpley, 2007). A

surgeon’s developing knowledge and skills in completing a variety of medical

procedures are determined by the frequency of practice. Surgical skills training

centers provide an opportunity for surgical residents to develop their skills to a level

of automaticity outside the operative room. The use of virtual reality and other human

performance simulators can help develop the knowledge and skills of surgical

residents (Reznick and MacRae, 2006).

Evidence of increased surgical knowledge and skills has been validated through

studies in short-term, specific procedure concentrated classes. Surgical interns who

were trained using a human patient stimulator (HPS) saw an increase the participant’s

trauma management skills and overall self-confidence in their abilities (Marshall,

Smith, Gorman, Krummel, Haluck, & Cooney, 2001). Additionally, third-year

residents indicated through self-reports that their knowledge and skills generally

increased as a result of participating in a small group training sessions covering four

surgical procedures [foley catheter placement; nasogastric tube insertion/removal;

I.V. placement; and arterial stick] (Meyers, Meyer, Stewart, Dreesen, Barrick, Lange,

& Farrell, In press). Grantcharov and Reznick (2008) indicate two common elements

within teaching procedural technical skills for surgery: pre-patient training and

training in a clinical situation. While the pre-patient focuses theory and simulation

7 curriculum, clinical situation training incorporates an expert instructor for

demonstration, instruction, and evaluation purposes.

The common element of all medical education is the reliance on surgical experts.

Experts are often characterized by the coordination of their highly organized

knowledge structure and their advanced technical skills within a specific domain

(Ericsson, Charness, Feltovich, & Hoffman, 2006). However, Sleeman and Brown

(1982) indicate that much of an expert’s knowledge is tacit and not readily available

to the expert during retrieval tasks. Therefore, relying on experts for educating

surgical residents is a major limitation to the current training methods. A brief review

of the characteristics of experts, experts in medicine, and the affect of experts on

medical training will be reviewed to establish an understanding of the complexity

involved with experts in the field of medicine and the underlying problems with the

expert’s ability to transfer their knowledge to novices.

Development of Expertise

Although expertise can be defined in many ways (Feldon, 2007), it is commonly

agreed upon that expertise is defined as the possession of a large body of knowledge

and procedural skills within a specific domain (Chi, Glaser, & Rees, 1982; Ericsson et

al., 2006; Feldon, 2007). Expertise includes the development of automaticity through

deliberate practice (Ericsson et al., 2006) that reduces the cognitive demand in

performing procedural tasks. Cognitive demand is lowered through automaticity by

reducing the number of decisions that require conscious thought. Therefore,

automaticity provides the benefit of processing speed in problem solving, but inhibits

8 conscious monitoring and modification of skills during task performance. The

inability to adapt routine behavior could lead to lower performance levels when faced

with atypical tasks. Feldon (2007) discussed the need for experts to have a developed

level of adaptivity when faced with atypical tasks (adaptive expertise). Adaptivity

provides the ability to adapt to novel task constraints while consistently demonstrate

optimal performance. In contrast, routine experts fail to maintain high levels of

performance when faced with changing task conditions. As a result of their high

performance level and cognitive efficiency in problem solving abilities, experts are

often called upon to provide information for training and expert systems, and are

revered socially as an authority figure within a specific field.

Expertise in Medicine

According to Ericsson et al. (2006) a significant portion of the research literature

on medical expertise focuses on diagnosis on medical problems. Boshuizen and

Schmidt (1992) investigated the different use of biomedical and clinical knowledge

by novice, intermediate, and expert-level participants in a study in clinical reasoning.

They defined biomedical knowledge as understanding the principle processes

underlying the manifestation of a disease (e,g, virus or bacteria, organ or organ

systems) and clinical knowledge as knowledge of the attributes in which diseases can

manifest themselves, the related overt and covert signs of the disease and overall case

management heuristics. Boshuizen and Schmidt (1992) found that with increased

levels of expertise, individuals utilized less biomedical knowledge and more clinical

knowledge while sequencing a clinical diagnosis. The increase in clinical knowledge

9 is due to an experts developed “illness scripts” which provide the ability of experts

to mediate the current presented task based on prior actual worked cases.

In a study to analyze how subject matter experts use personal causal models for

diagnostic purposes, Patel and Groen (1986) provided a written medical case to seven

cardiologist specialists. After a brief review of the case, the participants were asked

to provide a written account of the case and also provide an underlying

pathopysiology of the case including a diagnosis. Patel and Groen (1986) found that

the experts who provided an accurate diagnosis utilized forward reasoning in their

causal models, while participants who provided inaccurate diagnosis utilized

hypothetical testing, including backwards reasoning. Similarly, in their research on

expertise and problem solving, Chi et al. (1982) noted that one of the qualitative

differences between experts and novices is their approach to a presented task. While

novices solve problems while working backwards with hypothesis testing heuristics,

experts tend to perform an initial qualitative analysis of the problem and utilize

forward reasoning strategies mediating the current task with similarly stored

representations based on prior experience.

Expertise in Surgery

Acquiring surgical skills requires the development of cognitive and psychomotor

abilities (Hamdorf & Hall, 2000; Peters, Fired, Swanstrom, Soper, Sillin, Schirmer et

al., 2003). According to Hamdorf and Hall (2000), motor dexterity is developed

through three phases: cognition, integration, and automation. The cognition phase

consists of understanding the task. The integration stage incorporates the motor

10 movements specific to the procedure. Lastly, automation infers that the movements

needed for a procedure have become non-conscious and do not rely on continuous

external stimuli. As individuals develop surgical motor and cognitive skills it is

imperative to minimize error development (Porte, Xeroulis, Reznick, & Dubrowski,

2007; Abernethy et al., 2008).

In summary, the attainment of expertise requires the acquisition of extensive

conceptual knowledge and automated procedural skills that allows experts to solve

problems within minimal cognitive demand (Feldon, 2007). To obtain the level of

expertise, in any domain, takes about ten years of deliberate practice with continuous

feedback (Ericsson, 1996; Pavlik and Anderson, 2008). Therefore, to acquire the level

of expertise in medicine and surgery, in particular, requires extensive practice. Two

underlying themes that distinguish experts from non-experts are their knowledge base

and their rapid cognitive processing abilities (Chi et al., 1982).

Due to the automaticity of an expert’s knowledge and skills, conscious monitoring

of thought processes is difficult, therefore an expert’s description of problem solving

is typically limited to observable actions. An understanding of knowledge types

(declarative and procedural) and their function in developing expertise provides a

further understanding of the requirements for capturing needed information from

experts for training complex tasks, such as surgical procedures.

Knowledge Types

Knowledge is the representation of learned information that can be expressed in a

variety of settings for different purposes, actions, or means (Markus, 2001; Spender

11 1996; de Jong & Ferguson-Hessler, 1996; & Anderson, Krathwol, Airasian,

Cruikshank, Mayer, Pintirich, Raths, & Wittrock, 2001). Research indicates that

individuals have a limited capacity in the ability to consciously process information

and there is evidence that people have the ability to process four plus or minus one

bits of information within their working memory (Cowan, 2001). Working memory

is the theorized space that individuals “pay attention to” and encode new information

into long-term memory. Long-term memory is considered unlimited in its capacity

and is associated to encoded chunks of information and are accessible based on their

retrieval structures (Ericcson and Kintsch, 1995). In this section, two types of

knowledge – declarative and procedural – will be discussed and the interactions

between the two that comprise expertise

Declarative Knowledge

Cognitive psychologists have defined declarative knowledge in many ways.

Ormrod (2008) describes declarative knowledge as information retrievable from an

individual’s long-term memory into their short-term memory as concepts, principles,

general knowledge, and recollections of life events/ experiences. Schunk (2000)

suggests that declarative knowledge is comprised of facts, actions within a story, and

the organization of a passage, with reference to information obtained from a novel or

piece of literature. Another definition by Schraw (2006), postulates that declarative

knowledge encompasses facts, concepts, and the mediated relationships among

learned concepts that create an integrated holistic conceptualization within a specific

domain of knowledge. Declarative knowledge is often subdivided into two

12 categories, semantic knowledge comprised of organized schemata regarding

individual concepts (e.g., vascular system) and episodic knowledge (time specific

events or episodes in life, e.g. prom, wedding, & birth of a child) (Schraw, 2006).

Declarative knowledge is considered explicit knowledge (which is easily

retrievable) that is developed through encoding information from the environment.

Declarative knowledge is retrievable information that can articulate the answers to

question of what and why with regards to concepts and facts (Paris, Lipson &

Wixson, 1983; Anderson & Schunn, 2000; Hoffman & Militello, 2009). These

encoded chunks are retrievable from an individual’s long-term memory into their

short-term (i.e. working memory) for immediate needed purposes. Retrieved

declarative knowledge guides individuals in completing performance-based problem

solving tasks (Anderson, Bothell, Byrne, Douglas, Lebiere, & Qin, 2004). For

example, a surgeon knows what equipment to use for a procedure and why the

surgery is needed and has the ability to describe what and why

According to Anderson’s (1996) Adaptive Character of Thought (ACT-R) theory

declarative knowledge is developed as one encodes chunks of information from their

environment. Anderson (1996) describes the declarative chunks of knowledge as

“schema-like structures” that are categorically specific with tangible connections to

other related encoded content. Knowledge structures develop through the

accumulation and toning of individual units of information (chunks) that in whole

produce complex cognition. Although, these chunks of knowledge represent factual

information that an individual can express (Anderson & Schunn, 2000), the

13 acquisition of declarative knowledge does not occur in a vacuum, but rather in

conjunction with the acquisition of procedural knowledge.

Procedural Knowledge

Anderson (1996) contrasts declarative and procedural knowledge by describing

declarative knowledge as “chunks” of information, and procedural knowledge as

“production rules” that are based on the encoded chunks of information. Specifically,

Anderson (1996) suggests that production rules represent procedural knowledge of

the conditions (when) and actions (how) to perform a goal-oriented task that are

based on the individual’s declarative knowledge structures. Further, a production rule

can only be activated once a precise environmental state meets the set conditions for

the production rule to be carried out. For example, a surgical procedure consists of

multiple decisions that need to be made based on the presenting conditions within that

surgery. Thus, production rules symbolize the condition and action needed in

response to the attainment of specific goals.

Production rules consist of IF, THEN statements and are context embedded

(Anderson, 1996) In medical procedures, for example, IF there is bright red pulsating

blood in a syringe, THEN the needle must have punctured an artery. Thus, procedural

knowledge is “when and how to” perform an action or “make a decision” while

completing a task or solving a problem. These production rules (procedural

knowledge) are tacit and not easily retrievable to the individual. In surgery,

procedural knowledge consists of the decisions a physician makes on “when and

how” to perform certain actions during the surgery. In contrast, declarative

14 knowledge is the information the individual knows about the surgery and its

components (knowledge of the anatomy and the instruments used for a procedure.)

When they practice, individuals fine-tune their skills and declarative knowledge

gradually evolves into procedural knowledge (Pirolli & Anderson, 1985; Stadler,

1989; Anderson & Fincham, 1994; Anderson, Fincham, & Douglass, 1997;

Anderson, Fincham & Douglass, 1999; Pavlik & Anderson, 2008; and Knowlton &

Moody, 2008.). Ericsson et al. (2006) indicate that procedural knowledge is

considered to be the final state of the learning process: the combination of acquired

declarative knowledge used to solve problems that become automated through

deliberate practice.

In medicine, through deliberate practice, a highly experienced surgeon can

process the cues of a medical procedure and perform the tasks within the procedure to

a level of automaticity. This reduces the attention demands on the surgeon (What is

the next step in this procedure?) and frees up cognitive resources and enabling the

physician to shift attention to other cognitive actions such as directing the supporting

nurses, self-monitoring effectiveness of the actions taken, and the status of the patient

(changing conditions).

Automaticity

Automaticity is a means of restructuring same procedures so that working

memory is largely circumvented, freeing cognitive resources for other cognitive

demanding tasks (Ericsson, Charness, Feltovich, & Hoffman, 2006). Ericsson et al.,

(2006) discuss how automaticity is central to the development of expertise and

15 practice is the means to achieving automaticity. Expertise involves automated basic

strokes. Experts perform an extreme amount of practice. Such experience,

appropriately conducted, can yield effective, major behavior and brain changes.

Through the act of practice (with reflective feedback, self-regulation, & monitoring of

one’s progress) the character of cognitive operations changes in a manner that:

a) Improves the speed of the operations

b) Improves the smoothness of the operations

c) Reduces the cognitive demands of the operations.

Thus releasing cognitive resources (attention) for other functions such as

planning, self-monitoring, situational awareness. Appropriate planning includes the

use of a developed algorithm to problem solve. Self-monitoring focuses on the error

correction procedures conducted within a task. Situational awareness refers to being

observant of the current conditions during a task.

According to Ericsson et al. (2006), two major difficulties exist with the

development of automaticity. First, individual’s performing at this level of cognitive

automaticity are not as attentive to new cues that fall outside their learned procedure

and may overlook the otherwise salient cue, nor are individuals cognizant of all the

discrete steps involved within their actions. For example, while performing a medical

procedure, a surgeon may continue to follow previously learned behaviors when the

present conditions indicate alternative techniques may be more beneficial.

Additionally, when describing their actions multiple steps may be omitted due to their

automated knowledge. Second, once automaticity has been achieved, the ability for

16 individuals to increase their skills diminishes. Once individuals learn a task to the

level of automaticity, they lose the ability to augment their performance without

purposeful actions, both physically and mentally, which Ericsson calls deliberate

practice. Deliberate practice is 1) actively seeking out novel demanding tasks, and 2)

monitoring, through guided introspection, actions and decisions while performing a

task.

In summary, expertise is the interaction of declarative knowledge (the What, the

concepts, process, and principles involved in labeling items in the world) and

procedural knowledge (the how and when in terms of actions and decisions necessary

in completing a task). In terms of surgical residents, their declarative knowledge

consists of the conscious academic preparation and skills they have learned up to an

intermediate level of proficiency. Their procedural knowledge consists of utilizing

their declarative knowledge in completing medical procedures. As individuals

develop expertise, their knowledge becomes highly organized and integrated

structures within their domain. Their skills become highly developed through

deliberate practice, reflection, and corrective feedback.

Automaticity and Expert Recall

Cognitive psychology describes how individuals transform encoded information

through practice to develop heuristics in problem solving. These heuristics are often

automated procedures that are tacit and unconscious to the individual. Polanyi (1962)

initially referred to tacit knowledge as certain cognitive processes in context with

problem-solving behaviors that are inaccessible to the conscious mind. Tacit

17 knowledge refers to the individual’s inability to recall the explicit knowledge used

during a series of events while completing a task (Gourlay, 2004).

Expert instructors are often called up to provide needed information and resources

to novices through curriculum development and teaching. Hinds, Patterson, and

Pferrer (2001) found that due to the cognitive characteristics of experts, that is, a

highly organized, hierarchal system of interconnected abstract concepts, accompanied

by advanced procedural automated knowledge, the transfer of knowledge from

experts to novices has severe limitations. In their first study, Hinds et al. (2001)

investigated the different instructional styles used by beginner and expert instructors

when training novice individuals about how to understand simple electronic concepts

by completing an electronic circuit. Hinds et al. (2001) found that experts utilized

more abstract concepts and statements when compared to beginner instructors who

trained novices with more concrete statements. For instance, only 9% of experts

provided direct information about the requirement of properly connecting the wires to

the spring coil, which is essential for the electrical connection to be completed, while

90% of beginners provided this information. Hinds et al. suggest that experts

educating individuals in this process may omit critical information that novices need

to develop their conceptual knowledge within this domain. According to Hinds

(1999), as experts develop their automaticity, their simplification of facts and

concepts precludes them from recalling specific details and task imbedded

complexities that novices require for problem solving in this domain.

18 Automaticity and expert recall in medicine and surgery

An individual’s automated procedural (tacit) knowledge has systemic

consequences in the medical field. Surgical experts usually train novice physicians

(Vadcard & Luengo, 2004). Research indicates experts have greater perceptual,

cognitive, motor, attention and personal reflective feedback capabilities when

compared with non-experts (Abernethy, Poolton, Masters, & Patil, 2008). Physicians

develop their declarative and procedural knowledge through years of experience and

practice (Cauraugh, Martin, & Martin, 1999). Physicians develop and hone their skills

by conducting multiple procedures and develop ‘rules of thumb’ (Andre, Borgquist,

Foldevi, & Molstad, 2002) which are tacit and unconscious to the individual. As a

result of their continued practice, their skills becomes automated and transition from

being knowledge that is on a conscious level to knowledge that is tacit and illicit

without deliberate probing (Abernethy, Poolton, Masters, & Patil, 2008). Novice

medical students relying on the attending physician for critical information regarding

a procedure may not receive the knowledge required to perform adequately due to the

attending physician’s inability to recall the needed information (Hamdorf & Hall,

2000).

The consequences of expertise and automaticity on expert recall was clearly

establish through the research of Clark, Pugh, Yates, Early, & Sullivan (2008) in their

investigation of capturing the declarative and procedural knowledge from trauma

surgeons on how to perform an emergency femoral artery shunt procedure. Nine

trauma surgeons provided an unaided (no- CTA condition) account of the shunt

19 procedure. When their description of the surgery was compared to a gold standard

femoral shunt surgical protocol, it was determined that they omitted an average of

68.75% of the procedural steps performed in the surgery (Clark et al., 2008). As a

result of their automated knowledge and skills, the trauma surgeons within the study

were not able to recall the full accounts of the procedure. Theoretically, experts may

omit up to 70% of the procedural knowledge involved within a surgical procedure

during the instruction of novices. However, differing results were obtained when one

trauma surgeon was interviewed using CTA-based methodology. When the acquired

knowledge from the one trauma surgeon (CTA condition) was compared to the gold

standard protocol, Clark et al., (2008) calculated only a 30% omission of the

procedural steps. The significant variable in capturing the procedural steps from the

trauma surgeons was the use of an interview based on cognitive task analysis

methodologies, suggesting that cognitive task analysis shows promise as a

methodology to capture the automated, procedural (tacit) knowledge from experts.

The following sections includes a review of how cognitive task analysis is

defined; the appropriate CTA methodologies to capture the intended critical

knowledge and skills sought; the effectiveness of CTA outside medicine and within

surgical training; and the barriers and limitations to using CTA. Finally, the

importance in determining the number of experts and the methods utilized in

determining the optimal number of experts that are the most reliable and valid will be

reviewed.

20 Cognitive Task Analysis

Until the second half of the 20th century, behavioral task analysis was the primary

method to understand task performance by detailing the observable behaviors as an

individual performs a task. Although specific physical attributes of problem solving

skills could be quantified, there was little consideration for the unobservable thought

processes taking place inside the individual’s mind. Prior to the use of cognitive task

analysis, individuals developed and used a behavior task analysis to capture the

needed information from experts on how to complete tasks (Clark et al., 2008).

Although a behavior task analysis can describe the observable steps involved in

completing a task, it fails to capture the critical information about the cognitive

processes required for how to complete the task. Cognitive task analysis developed

out of the need to capture these previously unobservable decisions, analyses,

judgments, and other cognitive processes.

Defining cognitive task analysis

There are a considerable number of definitions for cognitive task analysis.

Cognitive task analysis (CTA) refers to a variety of tools and techniques for

describing the knowledge and skills required to perform a task that yields information

about the unconscious thought processes and goal structures that underlie the

observable task performance (Schraagen, Chipman, & Shalin, 2000). CTA

techniques capture a representation of the knowledge and strategies that have been

retained by the individual until some time after the event in question (Feldon, 2007).

Furthermore, cognitive task analysis (CTA) uses a variety of methods to capture

21 experts conscious and non-conscious automated knowledge needed to perform a

complex or critical task (Crandall, Klein, & Hoffman, 2006). The common theme

found in these definitions is the use of multiple techniques that elicit knowledge,

facilitate data analysis, and represent the content and structure of the participant’s

knowledge within a specific task domain.

CTA has been used to capture expertise in a variety of working environments.

Researchers have used a cognitive task analysis framework to examine expert

performance to capture the cognitive task demands on an Intensive Care Unit team

(Fackler, Watts, Grome, Miller, Crandall, & Pronovost, 2009); to capture the

cognitive demands and indicators of business pilot expertise when faced with various

weather conditions (Latorella, Pliske, Hutton, & Chrenka, 2001); to investigate the

characteristics and decision making of expert warning forecasters of severe weather

(Hahn, Rall, & Klinger, 2003); and to develop a preliminary visual design aide

(heuristic) and prescriptive evaluation criteria to assist intelligence analyst efficiently

sift through data overload and develop an appropriate response to situational

questions outside their area of expertise (Patterson, Woods, Tinapple, & Roth, 2001).

CTA methodology

Cooke (1994) identified over 100 different types of cognitive task analysis

methods. This enormous variety of CTA methodologies can make it difficult for

practitioners to choose the correct method for the end result purposes of the research.

Cooke (1994) grouped the CTA methods reviewed into three broad families of CTA

techniques: 1) Observations and interviews; 2) process tracing; and 3) conceptual

22 techniques. As noted in Clark et al. (2008), observations and interviews includes

talking with experts and watching them perform tasks. Process tracing involves the

use of think-aloud protocol or subsequent recall of an expert’s performance in a

specific task, while conceptual techniques are used to capture structured conceptual

knowledge within a domain (Clark et al., 2008).

Clark et al., (2008) reviewed the CTA literature and found that individual’s who

conduct CTA research typically follow a five-step procedure: 1) Collect preliminary

knowledge; 2) Identify knowledge representation; 3) Apply focused knowledge

elicitation methods; 4) Analyze and verify data acquired; and 5) Format results for the

intended application (Schraagen et al., 2000 and Clark et al., 2008,).

Collecting preliminary knowledge incorporates gaining initial knowledge of the

domain (bootstrapping) and developing a sequence of main steps that will guide the

CTA investigation. This initial step includes the analysis of documents and

identification of potential experts to be utilized within the study. The information

gained in the preliminary stage allows the analyst to examine the overall task in

question and identify possible sub-tasks and types of knowledge required for experts

to perform the complex task. Once the analyst has identified the knowledge

representations involved in the complex task, it is important to choose the appropriate

CTA method of knowledge elicitation (Clark et al., 2008; Yates, 2007) depending on

the knowledge type the analyst is trying to capture. Differing CTA methods elicit

either conceptual or procedural knowledge (or both) (Yates, 2007) with varying

efficiency (Hoffman, Crandall, and Shadbolt, 1998). One of the more effective ways

23 to conduct a CTA and capture both declarative and procedural knowledge is

through a semi-structured interview (Yates and Clark, in review).

After applying the appropriate focused knowledge elicitation methods, acquired

data analysis and verification involves multiple steps. This process includes a)

coding the transcribed interviews to identify the conceptual knowledge (concepts,

processes, and principles), procedural knowledge (action and decision steps), goals,

standards, equipment and materials (Clark, 2004); b) formatting the output for review

and verification by the interviewed subject matter experts (SME); and c) finally the

analyst aggregates all documents into one standard “gold standard” format (Clark et

al., 2008). The final CTA document details the declarative and procedural knowledge

required to complete the actions and decision steps involved with performing the task.

Often, these products are utilized in the development of instructional materials (see

Velmahos et al., 2004) and expert systems (Schraagen et al., 2000).

Research has demonstrated that CTA is effective to acquire the critical conceptual

knowledge and automated skills utilized by experts in task performance (Clark et al.,

2008; Yates, 2007). The following sections will discuss the effectiveness of using

cognitive task analysis methods within a variety of settings, concluding with a review

of CTA within surgical training.

Effectiveness Studies Using Cognitive Task Analysis

Webster’s dictionary (1999) defines effectiveness as “producing a definite or

desired result: efficient” (p.454). Research in the effectiveness of CTA has a variety

of components, such as the chosen knowledge elicitation technique (domain specific),

24 the type of knowledge captured or the usefulness of the knowledge and its

application capabilities (Hoffman, Crandall, & Shadbolt, 1998). In addition, the

effectiveness of a CTA study depends on the results meeting or surpassing some set

criteria in order to be considered effective. In experimental CTA research, the

effectiveness of a study is dependent on the acquired knowledge types utilizing a

specific methodology (Clark et al., 2008; Yates, 2007; Yates and Feldon, in press).

The use of cognitive task analysis has shown to be effective within a variety of areas,

including addressing data overload and human-computer interaction (Patterson,

Woods, Tinapple, & Roth, 2001), piloting under various weather conditions

(Latorella, Pliske, Hutton, & Chrenka, 2001); emergency response teams and white-

water rafting (O’Hare, Wiggins, Williams, & Wong, 1998); air traffic control

knowledge (Redding, Cannon, Lierman, Ryder, Purcell, & Seamster, 1991);

information retrieval through library resources (Pejtersen, 1989); and medical

education (Crandall & Getchell-Reiter, 1993; Johnson, Healey, Evans, Murphy,

Crawshaw, & Golud, 2005; Sullivan, Ortega, Wasserberg, Kaufman, Nyquist, &

Clark, 2008).

Redding et al. (1991) investigated the use of an integrated task analysis

methodology to extrapolate the expert knowledge structures of air traffic controllers

in an effort to produce instructional content for new curriculum. Five full

performance level air traffic controllers were videotaped solving four different

problem scenarios. Each participant provided a verbal protocol while reviewing their

performance giving a description of their immediate goals and decision-making

25 processes while solving each of the four problems. As a result of the collective

verbal protocols, mental models, task decompositions, and individual task models

were constructed. The results of the CTA indicated a significant difference between

expert and non-expert air traffic controllers. In comparison to novices, experts

utilized fewer management strategies at a greater effective rate than novices.

Although it was noted that experts had a vastly greater repertoire of strategies to draw

upon, their ability to synthesize the situation better than novices allowed them to

choose specific appropriate strategies to solve the problem at a faster effective rate,

unlike novices who sometimes exhausted their limited strategies to find a workable

solution. The results from the cognitive task analysis were integrated into a

standardized curriculum that provided novice air traffic controllers a consistent and

deeper understanding to solving problems within their field (Redding et al., 1991).

While the CTA conducted within the Redding et al., (1991) study resulted in a

standardized curriculum for future trainings, Johnson, Cumming, & Omodei (2008)

concluded that the knowledge and skills captured from bushfire fighters would be

inappropriately used in a universal training guide, but would be more appropriate for

training at different levels of leadership. Johnson et al., (2008) interviewed 90

experienced bushfire fighters from 2005-2006 in order to capture the conceptual

knowledge and the decision making process that occurred at various levels of

firefighting leadership positions with regards to a worst case scenario. Johnson et

al.’s, (2008) investigation was in reaction to the post hoc reality of poor decision

making in recent bushfires in the Australian countryside that lead to two fatalities.

26 Semi-structured interviews (adapted from the Critical Decision Method: CDM;

Klein, Calderwood, & MacGregor, 1989) were conducted in multiple probes. Initially

to capture the series of events with additional probes aimed at capturing the actions

and decisions occurring within specific incidents.

An analysis of the transcribed interviews indicated that the responses from the

bushfire fighters could be classified into three major themes: A fire focus; a fire

fighter focus; and a community focus. Additionally, Johnson et al. (2008) indicated

the results followed a trend that matched the level of leadership to the respondent’s

primary focus (i.e. higher levels of leadership displayed more of a community focus).

The CTA utilized captured the decision making process at all levels of leadership, but

due to the complexity at each level, a universal training protocol would not meet the

varied needs of the leadership positions. An interesting finding during the

investigation was the occurrence of experienced bushfire fighters omitting of any

references to a “worst case scenario” during their interviews which could lead to poor

decision making under certain contexts (Johnson et al., 2008). The omitted reference

to a ‘worst case scenario’ from experienced bushfire fighters is consistent with the

research literature on expertise and automated knowledge (Ericsson et al., 2006).

Cognitive task analysis provided a means for Johnson et al., (2008) to capture the

automated decision-making bushfire fighters would execute within a specific context

and for different leadership positions.

Cognitive task analysis has been shown to be effective in capturing decision-

making made by medical professionals (Crandall et al., 1993; & Jacklin, Sevdalis,

27 Darzi, & Vincent, 2008). For example, in an effort to elicit the critical cues

neonatal intensive care unit nurses used to evaluate the health and immediate future

needs of newborns within their care, Crandall and Getchell-Reiter (1993) conducted

semi-structured interviews with nineteen registered nurses. The semi-structured

interviews produced 33 incidents of immediate, life threatening situations, most often

cases of sepsis or systemic bacterial infection, requiring urgent care within minutes or

over the next few hours. The results indicated NICU nurses were alert to a number of

critical indicators of sepsis not found in the medical literature used to train neonatal

care nurses.

In a second study, Crandall & Getchell-Reiter (1993) conducted semi-structured

interviews with five experienced female NICU nurses utilizing a critical decision

method (CDM). Critical decision method is a retrospective knowledge elicitation

technique developed by Klein, Calderwood, & Macgregor (1989). Each nurse was

interviewed about three incidents that lead her to the correct diagnosis of infant

septsis, an incorrect diagnosis of the same nature, and a time when the baby

developed sepsis when not originally detected by the current methods. The

information obtained from the initial study along with the secondary study of sepsis-

related incidents was used to create a sepsis assessment guide that became a learning

aid within training NICU nurses.

As individuals gain expertise, their ability to communicate their knowledge to

novices diminishes (Hinds, Patterson, and Pferrer, 2001). Sullivan et al. (2008)

investigated whether expert surgeons omitted actions and decisions when teaching a

28 colonoscopy to second -year postgraduate residents. Three expert surgeons were

videotaped during an instructional session during which they provided a detailed

account of both the action and decision steps involved in a colonoscopy procedure.

Shortly after the expert’s instructional session, the experts participated in a free recall

of the colonoscopy procedure, focusing on the detail and completeness of the

information provided during the prior teaching session. Following their free recall, an

analyst conducted a CTA with each expert individually in a series of structured

interviews to capture the conceptual and procedural knowledge required to perform

the procedure. As a result of the knowledge captured through the CTA, a 26-step

procedural checklist and a 14-point ‘cognitive demands checklist’ (p.21) were

created. In comparing the experts free recall to the CTA checklist, Sullivan et al.,

(2008) found that the experts omitted 50% to 74% of the ‘essential how to steps’ and

57-75% of the critical decisions involved in the colonoscopy procedure. Although

the expert surgeons in the study were recognized as outstanding surgical educators,

they were not able to articulate all of the action and decision steps involved in the

task. This is consistent with the expertise literature that suggests the knowledge and

skills of experts are highly automated and not easily retrievable when prompted.

(Ericsson et al., 2006), and, moreover, because experts often make errors when

describing how to perform a task (Clark and Estes, 1996, Clark et al., 2008), they

often omit critical information they believe that they have communicated to their

students (Clark, 2006; Feldon, 2004).

29 In summary, the effectiveness of cognitive task analysis is often measured by

the type of knowledge captured and sufficiency for the intended application. The

studies reviewed provide insight about how the use of CTA successfully captures

both the conceptual knowledge and the procedural skills experts use to perform

complex tasks. While there are numerous studies that examine CTA for the purposes

of capturing expertise (See Cooke, 1994 and Yates, 2007) there is a paucity of studies

that investigate integrating the knowledge and skills of experts into training protocols

in the surgical domain (exceptions- see Luker et al., 2008; Sullivan et al., 2007;

Velmahos et al., 2004; and Tirapelle, 2010).

Effectiveness of CTA in Surgical Training

Cognitive task analysis (CTA) has been shown to make a significant difference in

increasing the knowledge and skills of surgical residents (Velmahos et al., 2004;

Bathalon, Martin, and Dorin, 2004; Johnson et al., 2005; Luker et al., 2008; Sullivan

et al., 2007; and Tirapelle, 2010). The following review of studies illustrates the

significant benefit CTA brings to surgical training. The proceeding sections will

discuss the benefits CTA provides to surgical training including capturing both action

and decision steps within a procedure; utilizing CTA to create training materials that

produce immediate acquisition of knowledge and skills when comparing CTA- based

curriculum versus traditional methods; and CTA and long term maintenance of

acquired knowledge and skills.

Cognitive task analysis is often utilized to capture the expertise in order to create

training protocols for novices. Johnson et al. (2005) followed CTA principles to

30 create training guides for five commonly performed interventional radiology

procedures. Each task analysis captured the action and decision steps involved in the

procedure. For example, the CTA conducted for an arterial needle puncture captured

101 action steps and 24 decision steps from the experts interviewed. Johnson et al.,

(2005) indicated the collective knowledge gained through conducting the five CTA’s

would provide a basis for establishing a standard of practice in conducting these

interventional radiology procedures. Johnson et al. (2005) proposed that the

knowledge and skills captured through CTA could be utilized to develop novice

training and assessment simulations focusing on both the action and decision steps

conducted during a procedure. As a result, CTA protocols would rapidly increase a

novice’s knowledge and skill development. Additionally, the CTA could be used as a

cue for experienced operators before they conduct a rarely performed procedure. An

expert’s automated knowledge and skills impacts their ability to modify ingrained

task performance (Ericsson et al., 2006). Therefore, reviewing the action and

decision steps within a procedure would help create a revised mental model to follow

during task performance. Overall, Johnson et al. (2005) hoped that their investigation

would lead to additional studies that focused on less common interventional radiology

procedures, thus highlighting the benefits CTA in its ability to capture automated

decision steps for training purposes.

In a study to examine the efficacy of CTA-based instructional protocols,

Velmahos et al. (2004) sought to determine if training new interns through instruction

developed with the results of cognitive task analysis made a difference in the

31 knowledge and technical skills of interns when compared to those trained with

traditional methods. Twenty-six new surgical interns were randomly assigned to

receive instruction on how to perform a central venous catheterization (CVC) through

traditional methods or through training based on a cognitive task analysis. Pre-test

scores indicated no differences between control (n=12) and experimental (n=14)

groups. Velmahos et al. (2004) found a significant difference between groups in the

posttest mean score when comparing the CTA group with the traditional group (11.0

+- 1.86 versus 8.64 +- 1.82, P=0.03). Velmahos et al. (2004) concluded that the

course taught with the results of cognitive task analysis was more effective in

increasing the cognitive knowledge and technical skills of interns on how to conduct

a CVC procedure.

In another study, Luker et al. (2008) investigated the use of a CTA based multi-

media instructional program and its effect on increasing the surgical skills knowledge

of residents in repairing a flexor tendon. Luker et al. (2008) utilized three flexor

tendon repair subject matter experts to construct a CTA detailing the knowledge and

skills needed to perform the surgery. The principal investigator designed a CTA-

based multimedia curriculum highlighting the critical decision points and skills

needed to complete a flexor tendon repair surgery. Ten surgical residents were asked

to perform flexor tendon repairs on three different occasions with a post-procedure

assessment to assess the residents understanding of the tasks and the critical decision

points within the procedure. Any increase in the participant’s knowledge and skills

from the first and second performance was deemed a result of practice and was

32 considered the “control group” within the study. A learning session using a CTA-

based multi-media instructional presentation was conducted in between the second

and third flexor tendon procedure. Luker et al., (2008) concluded that changes in the

post procedure scores resulted from the CTA-based presentation. The results

indicated that the mean difference between the first and second procedure (control

group) was 13.2 for conceptual knowledge and 8.5 for understanding the advantages

and disadvantages of each decision within the procedure. However, the mean

difference between the second and third procedure was 34.0 for conceptual

knowledge and 19.4 for decision points. Luker et al., (2008) discovered that although

the group improved due to shear practice, the use of CTA in surgical training

significantly increased the knowledge and skills needed to perform a flexor tendon

surgery.

In a recent study, Tirapelle (2010) investigated the effects of a CTA based

curriculum versus traditional surgical training methods on how to perform an Open

Cricothyrotomy. The composition of the twenty-six medical student participants

included 3rd year medical students, 2nd year post-graduate students, and 3rd year post-

graduate students. Pre-test measures showed no differences between the randomly

assigned experimental (n=12) and the control (n=14) groups in terms of experience

F(6,19) = .414; p = .860) or gender F(6,19) = .396; p = .873, or pre-test assessments

=5.4 (experimental) and =5.5 (control) out of 17 possible points. All assessment

and training materials for this investigation were developed from a gold standard

CTA report derived from interviewing six subject matter experts on how to perform

33 an Open Cricothyrotomy. Tirapelle (2010) utilized Clark’s (2004) Guided

Experiential Learning (GEL) protocol in the development of the training materials.

Both groups (experimental and control) received a 30-minute instructional session on

how to conduct the procedure. The control group received instructions through

existing course materials and course structure, while the experimental group was

provided a CTA-based curriculum. Both groups were provided opportunities to

practice an Open Cricothyrotomy on inanimate models following the instruction in a

guided practice format. Post-instruction analysis was conducted on the participant’s

acquired knowledge and skills through individual assessments and ratings from expert

instructors and surgeons participating in the presentation of the training materials.

While the post instruction assessment showed no significant differences in the

acquired conceptual knowledge between the groups ( = 14.5, experimental and 13.9,

control; (t(22) = 0.55, p = .59), a significant difference was found when Tirapelle

(2010) measured the acquired procedural knowledge ( =17.75, experimental and

=16, control; t(21) = 2.08, p = .050) between the groups. Tirapelle (2010)

concluded that CTA-based instruction had a significant positive effect on the

acquisition of procedural knowledge and performance when compared to traditional

expert-led surgical skills instruction.

While the research of Velmahos et al. (2004), Luker et al. (2008) and Tirapelle

(2010) all found CTA- based curriculum produced a greater immediate increase in the

knowledge and skills of surgical interns, Bathalon et al. (2004) provided evidence that

the use of CTA for surgical instruction has long-term benefits as well. Bathalon et al.

34 (2004) randomly assigned 44 first year medical students into three different

instructional groups on how to perform a cricthyroidotomy procedure. The first group

(n=16) received instruction from the traditional ATLS protocol. The second group

(n=13) learned the same procedure using both cognitive task analysis and kinesiology

principles. The third group (n=15) was instructed with a combination of CTA,

kinesiology principles, mental imagery practiced daily, and debriefing. The results

indicated that the groups taught with a CTA-based curriculum performed better

initially and also maintained their skill level over 12 months when compared to the

first group. The group who received training using a combination of CTA,

kinesiology principles, mental imagery practice, and debriefing showed the highest

skill acquisition and long-term maintenance.

In another study, Sullivan et al. (2007) investigated the effectiveness of a CTA-

based curriculum to instruct novices on the knowledge and skills required to

successfully complete a percutaneous tracheostomy (PT) placement as compared to

students trained by traditional methods. Sullivan et al. (2007) randomly assigned 20

postgraduate surgery residents into either the control group (N=11) and provided

traditional PT training, or the CTA group (N=9) and provided CTA-based training.

The results from Sullivan et al. (2007) study indicated that there were no significant

differences between the groups prior to instruction. However, post-instruction

assessment results indicated that a significant difference existed between the control

and experimental (CTA) group after instruction. The CTA group scored significantly

higher mean averages than the control group in the technical aptitude assessment at 1

35 month (CTA: 43.5+- 3.7, control 35.2 +- 3.9, P=0.001). Although an attrition of

skills was indicated at the 6-month reevaluation assessment for both groups (CTA: 39

+- 4.2, Control: 31.8 +- 5.8, P=0.004), the group that received a CTA-based

curriculum still performed significantly better than the group who received traditional

instructional methods. Therefore, the expertise CTA captures, when utilized in

instructional materials, plays a significant role in encoding knowledge and skill

development into long-term memory.

In summary, research has shown using CTA is effective for surgical training.

Cognitive task analysis is effective in capturing expertise to create training protocols

for standards of practice (Johnson et al., 2005). Additionally, others have developed

CTA-based instructional materials where the results showed significant gains in

knowledge and skills of surgical interns when compared to traditional methods

(Velmahos et al., 2004; Luker et al., 2008; and Tirapelle, 2010). While others

provided evidence CTA has beneficial acquisition of knowledge and skills with long-

term maintenance (Bathalon et al., 2004 and Sullivan et al., 2007). CTA can help

obtain insights into the automated expert conceptual and procedural knowledge of

domain specific tasks and separate the steps into individual discrete teachable units

for others to learn. While there are several positive outcomes when utilizing cognitive

task analysis methods, there are some limitations with CTA.

Limitations of Cognitive Task Analysis

Although the research on cognitive task analysis consistently shows that CTA

training is more effective than traditional training methods, there are several

36 limitations to using CTA techniques. An important limitation and the one most

commonly identified in the literature is the amount of time and effort involved in

conducting a CTA. Additional limitations include the automaticity of expert’s

knowledge and the ability to acquire their expertise. Next, these limitations are

discussed in detail.

The time necessary to conduct a cognitive task analysis can be time intensive

(Chao & Salvendy, 1994). According to Clark & Estes (2008), one hour of capturing

expertise requires approximately 30-35 hours of effort (Grunwald, Clark, Fisher,

McLaughlin, Piepol, 2004). Sullivan et al., (2008) estimated it took approximately 30

hours to complete the cognitive task analysis in their study. Additionally, Hoffman et

al., (2004) solidified the human cost of knowledge acquisition by documenting the

human hours required to complete components of their investigation utilizing CTA

methods. For instance, Hoffman et al., (2004) utilized CTA methods in transferring

the information in the Terrain Analysis Data Base (TADB) into 150 concept maps.

The total person hours to complete the 150 concept maps was calculated at 187.5-225

person hours. Each individual concept map took approximately 75-90 person hours

to complete.

One of the major limitations of cognitive task analysis focuses on the process of

knowledge acquisition (Hoffman et al., 1998). There are two main components to

knowledge acquisition including acquiring experts to interview and capturing their

expertise. An overall factor involved in these components is the time and effort

required in conducting a CTA. While there are enough experts to interview and

37 capture the desired procedural knowledge for training purposes, their time to

participate in a CTA may be limited. Patience, adaptivity to the expert’s schedule,

and the ability to mediate logistic situations are essential to obtaining an environment

to conduct the CTA. Once these issues have been resolved, a bottleneck still exists in

capturing the expertise due to their highly organized cognitive structures.

The automated knowledge and skills of experts is a natural barrier to acquiring

their expertise. Sleeman and Brown (1982) indicate that much of an expert’s

knowledge is tacit and not readily available to the expert during retrieval tasks. Glaser

(1985) derived that experts have the ability to discuss the “what” and “why” in

performing a task, but a conscious analysis of their verbal recall is required to capture

the “how and when”, that is the decisions and reasoning necessary to perform a task.

As a result experts automated knowledge may lead to retrieval inaccuracies during the

knowledge elicitation process (Feldon, 2007). As a consequence, dependence on a

single expert for a CTA investigation can provide incomplete results for intended

purposes. Therefore, the necessity of acquiring multiple experts may be an additional

limitation to capture the required knowledge.

Number of Experts Required for CTA

There is a scarcity of research indicating how many experts are required to

optimally capture the critical information necessary to problem-solve tasks. A notable

exception is Chao and Salvendy’s (1994) study recommending the use of three

experts. They randomly assigned twenty-four expert computer science students to

one of four knowledge elicitation techniques. The dependent variable was the

38 percentage of procedural knowledge captured through a knowledge elicitation

technique. The independent variables included the three computer programming

tasks (diagnosis, debugging, and interpretation) and the four knowledge elicitation

methods utilized (protocol, interview, induction, and repertory grid). The diagnosis

task combined with the repertory grid knowledge elicitation method indicated the

greatest gains in procedural knowledge from 40% from a single expert to 87% from a

total of six experts. For the other two tasks, the use of protocol knowledge elicitation

method showed the greatest gains in procedural knowledge obtained from one to six

subject matter experts: Debugging: 37% to 88%; Interpretation: 27% to 62% (Chao &

Salvendy, 1994). Chao and Salvendy (1994) found that the percentage of procedural

knowledge increased as they interviewed additional subject matter experts up to a

total of six individuals. As a result of their findings, Chao and Salvendy (1994)

recommended interviewing three subject matter experts based on a 10% marginal

utility cost-benefit analysis.

A review of the surgical training literature indicates a variability of the number of

experts utilized for CTA investigations. For example, Velmahos et al. (2004) made

use of two subject matter experts; Sullivan et al. (2007) gained knowledge from three

subject matter experts; Luker et al. (2008) utilized three subject matter experts in their

investigation; Johnson et al. (2006) employed two or three subject matter experts to

create their CTA protocols; and Sullivan et al. (2008) developed their CTA from three

subject matter experts. In an effect to create a decision map based on a diagnosis of

symptomatic gallstones, Jacklin, Sevdalis, Darzi, and Vincent (2008) utilized a

39 structured interview technique with 10 experienced physicians in the area of

gallstones. It appears common in the CTA research to utilize two to three experts for

the purposes of capturing expertise.

Currently, only two known studies exist recommending the number of subject

matter experts needed to conduct a cognitive task analysis: for solving computer

related problems (Chao and Salvendy, 1994): and for surgical procedures (Crispen,

2010). In a concurrent investigation, Crispen (2010) investigated the optimal number

of subject matter experts to be interviewed in acquiring the critical expertise required

to complete a surgical procedure. Crispen (2010) found that four subject matter

expert’s were optimal in capturing the knowledge and skills needed to perform an

Open Cricothyrotimy procedure. This was the first empirical study that has identified

a recommended number of subject matter experts to be interviewed to capture the

expertise needed to complete a surgical procedure.

Summary

Medical training centers are charged with producing competent physicians who

are technically proficient in a variety of procedures (Aggarwal, Grantcharo, & Darzi,

2007). Within every educational program, expert instructors are charged with

providing needed information and resources to help novices develop their expertise.

Due to an experts highly developed declarative and procedural knowledge, their

ability to share their expertise with novices is limited. Cognitive task analysis (CTA)

has been shown to be effective in capturing expert’s automated knowledge in a

variety of fields (O’Hare, Wiggins, Williams, & Wong, 1998; Redding, Cannon,

40 Lierman, Ryder, Purcell, & Seamster, 1991; Pejtersen, 1989; and Chao &

Salvendy, 1994). CTA investigations within the field of medicine are aimed at

capturing critical decisions an expert physician makes during a procedure. This

acquired knowledge has been proven beneficial in the development of expert-based

instructional protocols (Johnson et al., 2005). These CTA based training aids help

novice physicians develop declarative and procedural knowledge for a variety of

medical procedures.

Although there are significant benefits of CTA techniques, there are certain

limitations of this methodology. The two major limitations of cognitive task analysis

are 1) the process of knowledge acquisition and 2) the time required to conduct CTA

to provide a useable product for training purposes. The research literature has

multiple examples of studies indicating the benefits of using experts in capturing

expertise needed to perform a task/ procedure (Lyons, 2009). Currently, there is no

established recommended number of subject matter experts required when conducting

a CTA capturing the necessary conceptual and procedural knowledge for intended

purposes. Only two studies have provided recommended number of SMEs in CTA

research: a study conducted by Chao and Salvendy (1994) resulted in their

recommendation of three experts in solving computer based problems; and Crispen

(2010) recommended interviewing four experts for a surgical procedure.

Purpose of the study

Informed by the study conducted by Chao & Salvendy (1994), the purpose of this

investigation is to explore critical information that is gained from a single expert

41 position and the information gained from each additional expert CTA interview.

The information acquired from this investigation will allow us to propose an answer

to the following question: How many experts are recommended to interview in order

to collect the critical knowledge (procedural steps and decisions) needed to perform a

(surgical) procedure? The results of this investigation aims to establish a

recommended number of experts one would need to conduct CTA interviews with in

order to capture their expertise to develop a gold standard protocol to conduct a

medical procedure. Establishing a recommended number of experts to interview has

significant implications in long-term cost-benefit savings of time and effort inherent

in conducting CTA investigations.

To review, the research questions are:

1) How much information about a central venous catheter placement procedure

does a single expert provide when compared to a six-subject matter expert gold

standard protocol?

2) How much Critical Information is gained from each additional CTA interview

about the Central Venous Catheter Procedure?

42 CHAPTER 2: METHOD

The central venous catheter placement is a procedure that can be performed in

emergency and non-emergency situations. In medical procedures, a central venous

catheter ("central line", "CVC", "central venous line" or "central venous access

catheter") is a catheter placed into a large vein in the neck (internal jugular), chest

(sublcavian) or groin (femoral) . A CVC is used to administer medication or fluids,

and directly obtain cardiovascular measurements such as the central venous pressure.

As noted, the procedure is designed to gain access to a central vein for multiple

reasons, including rapid fluid infusion. The central venous catheter procedure was

used as the sample procedure to answer the proposed research questions. The current

study utilized a CTA based methodology. The proceeding sections review the overall

study design, how the subject matter experts were acquired, and how the data was

collected and analyzed.

Design

The current descriptive investigation, in general, replicates the research questions

proposed by Chao and Salvendy (1994), While Chao and Salvendy (1994) had 100%

knowledge of all possible errors within their study, the current investigation started

without a gold standard to measure expert’s elicited knowledge. Prior research

(Velmahos et al., 2004) on the central venous catheter placement produced a 14-point

checklist of tasks to complete during the procedure. This was created through the

collaborative work between two CTA experts and two experts on CVC. Although a

checklist has previously been created, it was never qualified as a “gold standard” for

43 how to perform the CVC procedure. The current investigation began by collecting

conceptual and procedural knowledge from experts discussing how to complete a

CVC procedure through semi-structured interviews. The combined experts’ elicited

knowledge was aggregated into a gold standard CVC protocol. The gold standard

protocol represented 100% of total knowledge about the CVC procedure. The gold

standard was utilized to quantify the total conceptual and procedural knowledge

acquired from each expert to answer the research questions. As indicated above this

data analysis was in part a replication of the work completed by Chao & Salvendy

(1994).

Subjects

A convenience sampling procedure was used to acquire the six expert physicians

who participated in this investigation. The participating physicians included four

trauma physicians; one critical care internist; and one anesthesiologist at whom work

in large medical centers in the Los Angeles area. The participants were considered

experts due to their extensive years of successfully performing the CVC procedure.

Prior CTA based investigations used a range between two to ten experts. Six experts

were utilized in the current investigation to determine the optimal number of experts

needed to capture the needed critical information and to have a large enough sample

to validate the average amount of information gained from additional experts without

expanding undue human time, effort, and cost. The research literature indicates

between two and five subject matter experts are recommended to complete a valid

CTA investigation.

44 Data Collection

The procedures on how to conduct the CTA for this investigation followed the

description provided by the work of Clark, Feldon, van Merrienboer, Yates, and Early

(2008) who provided details on the five most common elements of a CTA

investigation. The data collection for this investigation was conducted in four parts::

1) semi-structured CTA interviews of physicians with expertise on conducting CVC

procedure; 2) coding of interview manuscript; 3) creation of a six-expert based gold

standard CVC procedure protocol; and 4) analysis of the elicited knowledge from

both (a) single-expert and (b) non-repeated grouping of experts (i.e 2 SME’s, 3

SME’s up to 6 SME’s) against the gold standard.

Semi-structured CTA interviews

During the semi-structured cognitive task interview, the experts were asked a

series of questions that focused on the major tasks and potential problems a surgeon

could encounter when conducting a CVC procedure. Attention was focused on

obtaining the indications and contraindications on when to and when not to perform

the procedure. In addition to obtaining the overall procedural objective, the analyst

inquired about the expert’s knowledge of the benefits and any potential risks

performing the procedure. During their description of the CVC procedure, each

expert was asked to identify the equipment needed at the various stages within the

procedure. Upon confirmation of the major tasks, the experts were asked to describe

the specific actions they perform at each major task and sub task(s) within the CVC

procedure. During the interview process, the analyst asked probing questions about

45 the action and decision steps involved in the procedure to uncover any alternatives

to the decision steps being made and the criteria for choosing such alternative actions.

Lastly, the experts were asked to provide sensory information (touch, hear, or smell)

that a surgeon utilizes for an action or decision step. All interviews were recorded in

audio and transcribed into a manuscript.

CTA Coding Scheme and Procedure

Groups of two to three trained coders utilized a coding scheme developed by

Expert Knowledge Solutions (2009), to code the interview manuscripts. The analysts’

reviewed the transcriptions and coded items (words or phrases) into one of the

following categories: action or decision steps, equipment, indications to perform the

procedure, contra-indications on when not to perform the procedure, benefits, time

and accuracy details, along with any sensory information. Additional attention was

paid to uncover if any statements concealed covert action or decision steps or other

items that were considered critical information that required coding. Overall, The

coders compared their individual coding results and resolved coding disagreements

through discussion. Inter-rater coding reliability was calculated using Cohen’s Kappa

to assess the consistency of the coding process. Once the coding of the document was

completed and an inter-rater reliability calculation indicated a 99% inter-rater

reliability, the coded information was transferred into a CTA protocol.

46 CTA Protocol and creating a six-subject matter “gold

standard” CVC protocol

The coded items within each of the transcripts were formatted into individual

CTA protocols that listed all the relevant information involved in conducting the

central venous catheter placement (CVC) procedure from each expert interviewed.

Each expert reviewed their own CTA protocol for data verification and agreement.

Any necessary modification, deletions, or additions to the document were made at

that time. Upon inputting the changes provided by each expert, the experts reviewed

their own CTA protocols for verification of the edits and any other possible

corrections made by the expert. Upon final edits from the experts, the six individual

protocols were aggregated to create a gold standard CTA protocol.

The aggregation process started with combining similar statements that were

made by the experts regarding either action or decision steps. When different

statements were made regarding a similar task, they were molded into one action or

decision step (for example, “advance needle forward’ and ‘walk your needle’ are two

similar statements indicating the same action- pushing the needle forward into the

patient). Partial action steps were combined into larger steps to create complete and

efficient action steps. For example, the individual action steps of “gather materials’,

‘prepare equipment’, and ‘validate usability’ were combined to create one complete

action step. This aggregation process was completed for every section of the CTA

protocol.

47 After the six-subject matter expert CVC CTA protocol was created, it went

through two rounds of editing. In the first round, each expert reviewed the six-expert

CTA protocol independently for verification of data and editing purposes. All edits

were incorporated into the final CTA protocol by the researchers. A final editing

review was conducted by one of the original experts in the presence of an analyst for

immediate verification of data and sequencing of events within the CVC procedure.

These final edits were incorporated into the six-expert CTA job aid protocol. This

final version became the ‘gold standard’, which encompassed the complete details of

the central venous catheter placement procedure.

Data Analysis

The gold standard CTA protocol was used to calculate the percentage of

agreement from each of the individual subject matter expert CTA protocols. The

individual items within the CVC gold standard CTA protocol were individually

entered into a Microsoft word spreadsheet document. Each item was manually

graded for completeness in quantifying the information obtained from each expert.

Every discrete item that appears on the individual expert’s CTA protocol and matches

the ‘gold standard’ received one point. The discrete items included conditions,

equipment, action steps, and decision steps for example. Any item that was included

on the gold standard and not the subject matter expert’s CTA protocol, received zero

points. Zero points are awarded for any item that appears on the gold standard but not

on the subject matter expert’s initial CTA report (See table 1 for an example).

48 Table 1

Equipment CTA Gold Standard

Expert A Expert B Expert C Expert D Expert E Expert F

X-ray (A72; B1066; C48; F175; D381; E92)

1 1 1 1 1 1 1

Personal protective gear (A91; D60)

1 1 0 0 1 0 0

(Excerpt from CVC CTA protocol spreadsheet document.)

In an effort to provide evidence to answer the first research question, “How much

information does one expert provide when compared to a six subject matter gold

standard protocol? – an ‘acquired knowledge’ score for each subject matter expert’s

CTA protocol was calculated. An acquired knowledge score is the calculated

percentage of total knowledge obtained from each expert, per section of CTA

protocol, that was calculated based on the number of items captured from each expert

in comparison to the six subject matter expert gold standard protocol. For example,

per CTA interview, there are 30 pieces of equipment needed to complete the CVC

procedure, as identified by the six-SME gold standard CTA protocol. Each expert

provided varied amounts of equipment items in their initial interviews (i.e., Expert

“A”-23 equipment items; Expert “B”-7 equipment items; and Expert “C”-12

equipment items.) An acquired knowledge score for total equipment items was

calculated by dividing the number of items obtained by the total possible items, i.e. (#

of items per expert)/ 30). Therefore, the acquired knowledge from Expert A was

49 23/30 or 77%. In other words, Expert “A” provided 77% of the total number of

equipment items needed when compared to the CTA gold standard protocol.

Acquired knowledge scores were calculated for each of the sub sections within the

gold standard protocol, including action steps, decision steps, equipment, reasons for

performing the procedure, risks for not performing the procedure correctly,

indications to perform the procedure, and contraindications on when not to perform

the procedure or in determining site selection for the CVC procedure.

To answer the second research question- how much critical information is gained

from the addition of subsequent experts- an ‘acquired knowledge’ score was

calculated for each non-repeating groups of two, three, four, five, and six subject

matter expert groups for all sub-sections of the CTA CVC protocol. This calculation

will be represented by the cumulative total number of the procedure’s items captured

from each non-repeating group of subject matter experts divided by the total number

of items in the gold standard. The groupings were created by pairing the protocols

from the experts in non-repeating combinations, A SME combination refers to the

combined captured knowledge of the stated number of experts per grouping (i.e.,

Two-pair combinations: (Experts AB; AC; AD); Three SME combinations (Experts

ABC; ABD; ABE). There were 15 non-repeating two SME pair combinations, 20

non-repeating three SME combinations, 15 non-repeating four SME combinations, 6

non-repeating five SME combinations, and one six-SME combination (see appendix

‘A’ for complete list of two through six SME non-repeating group combinations).

50 Acquired knowledge scores were calculated for all subject matter combinations

by using the same procedure for the first research question: if a discrete item was

present in either of the experts CTA, that item was awarded one point and if not

present a zero will be inputted into the corresponding spreadsheet cell. Using an

excel spread sheet the total acquired knowledge for each paired SME combinations

were calculated for all 10 sections (i.e. objective, conditions, equipment, etc.) within

the CTA CVC protocol and the site location subsections (Internal jugular, subclavian,

and femoral sites). For each SME combination group an average was calculated from

adding up all the raw scores from the paired SMEs and dividing by the total number

possible for each sub-section (example- Standards). The same calculations were

performed for each subject matter combinations of three, four, five, and six expert

groups (see Table 2 for a two SME calculation for the sub-section: Standards). A sum

of the points from each of the subject matter expert pairings were divided by the total

number of gold standard items to provide the total percentage agreement with the

gold standard.

Table 2 Standards CTA Gold Standard SME A SME B Combined

The time frame for the CVC procedure ranges from 2 minutes (F50) -10 minutes (E86) with an average of 5 minutes (A445; B1101; E86)

1 1 1 1

Observable indications of success: Chest x-ray (E92) should show:

1 0 0 0

1. The catheter is in the superior vena cava. (B1081)

1 0 1 1

2. Clear lung fields (B1076) 1 0 1 1

51

3. Easily draw blood back from the catheter (E89)

1 0 0 0

4. Easily flush fluid into the catheter (E90)

1 0 0 0

Total # of Standards 5 1 3 3 Percent of Standard per SME 100 20 60 60

(Excerpt from CVC CTA protocol spreadsheet document.) (Table 2, Continued)

Since the ‘gold standard’ CTA protocol is a combination of the acquired

knowledge from all six physicians with expertise in conducting a CVC procedure, the

amount of total information acquired with each additional expert will increase up

until the sixth expert. In conducting cognitive task analysis interviews, there exists a

point of diminishing marginal return on the investment of time and human effort.

Chao & Salvendy (1994) utilized a marginal utility of 10% representing this point of

diminishing marginal return. The marginal utility of acquired knowledge was

distinguished as the change in increased acquired knowledge due to the additional

expert. In formula form, marginal utility = Change in total utility/ change in quantity.

In alignment with Chao & Salvendy (1994), the marginal utility for the current study

was set at the position where the additional acquired knowledge from an additional

expert was calculated to be less than 10%.

52 CHAPTER 3: RESULTS

Conducting CTA investigations constitutes significant investments in time and

effort to complete. An unanswered barrier to conducting a CTA research project is

deciding on the number of experts one would need to interview. Each expert

interviewed consists of a dedicated amount of time and effort to procure an expert,

capture their knowledge, analyze the findings, and represent their knowledge in a

working document. Establishing a standard number of experts to interview will

reduce the cost-benefit of conducting CTA investigations by providing future analyst

a set standard equated with reliable data. Knowing a set number of SMEs to interview

will minimize extraneous cost and effort from interviewing too many experts.

Coding and Inter-Rater Reliability

Each coder was instructed to review the transcript to capture the overall objective

of the procedure, any conditions for performing or contraindications, as well as noting

required action steps, decision steps, equipment, and the major tasks necessary to

perform the procedure. The analysts compared all documentation coding and the

inter-rater reliability was calculated at 99.46%. Each protocol document was

formatted into separate CTA protocols. Each CTA protocol was reviewed and edited

by the expert who was interviewed. The six individual CTA job aids were aggregated

to create a six-expert gold standard CTA CVC placement procedure protocol. Table 3

indicates the CVC gold standard CTA protocol subsections and the number of items

within each sub-section. The total items were calculated through adding the number

of individual concepts, actions, or decisions within each sub-section.

53 Table 3: CVC Gold Standard Sections and Corresponding Number of Items per Section.

Gold Standard Item Total Number Gold Standard Item

Total Number

Objective 1 Standards 5 Risks 17 Equipment 30 Reasons 3 Tasks 8 Indications 4 Action Steps 44 Contraindications 4 Decision Steps 14

Research question #1 : How much information about a central venous

catheter placement procedure does a single expert provide when compared to a six-

subject matter expert gold standard protocol?

The six-expert gold standard protocol was transferred to an Excel spreadsheet for

analysis. For each individual expert CTA protocol, a value of 1 was given to each

discrete item that represented a single action, decision step, condition, equipment, etc.

A column in the excel spreadsheet was established for each of the six expert CTA

protocols. The total score from each expert was calculated by adding all the values

together. The percentage of acquired knowledge was calculated by dividing the

obtained total for each of the six individual scores by the total possible number of

items from the six-expert CVC gold standard protocol. Sub-totals were also

calculated for the procedures action steps, decision steps, conditions, equipment,

reasons for performing the procedure, and risks for not performing the procedure

correctly.

To answer the first research question, the results of the study are displayed in

Table 4, which indicates the percentage of acquired knowledge obtained from

54 individual experts in comparison to the six-expert CTA CVC “gold standard”

protocol based on a single CTA interview. The range of total knowledge acquired

from a single expert when compared to a six-subject matter expert gold standard was

43% to 73% with an average of 57%. The range of acquired action steps from a

single expert when compared to a six-subject matter expert gold standard from a

single from 50% to 89% with an average of 70%. The average acquired knowledge

for decision steps from a single expert when compared to a six-subject matter expert

gold standard was 65% with a range of 57% to 71%.

Table 4 Percentage of Knowledge Acquired from One Expert When Compared to a Six Subject Matter Expert Gold Standard Protocol

SME 1 SME 2 SME 3 SME 4 SME 5 SME 6 Total Knowledge Acquired

65 48 43 73 61 54 57

Action Steps 75 66 50 89 70 70 70

Decision Steps 57 71 64 64 71 64 65

Objectives 100 100 100 100 100 100 100

Reasons 0 0 67 0 67 33 28

Risks 35 0 0 82 0 0 20

Indications 75 75 75 75 75 75 75

Contra-indications

50 50 25 25 50 75 46

Standards 20 60 0 0 60 0 23

Recommended Equipment

77 23 40 67 63 50 53

55 SME 1 SME 2 SME 3 SME 4 SME 5 SME 6

Tasks 100 100 75 100 100 88 94

(Table 4, Continued)

Research Question #2 : How much Critical Information is Gained from Each

Additional CTA Interview about the Central Venous Catheter Procedure?

In order to calculate the added percentage of acquired knowledge from each

additional subject matter expert (SME) a total composite score for each SME non-

repeating two, three, four, five, and six expert combination group was calculated. An

average percentage was determined by dividing the raw combination score obtained

from each paired SME combination by the total possible score based on the gold

standard. This calculation was completed for every section (Objective, Risks,

Reasons, Conditions, Equipment, Standards, Tasks, Action steps, and Decision steps)

including total number of items. With respect to Chao and Salvendy (1994), the

marginal utility was calculated as the change in an increased percentage of acquired

knowledge with the supplement of an additional expert. The point of diminishing

marginal return was calculated at the point where the marginal utility of the additional

expert was calculated at less than 10%.

Within this section, both tables and figures will be utilized to illustrate the

additional percentage of acquired knowledge for each of the sections within the CVC

gold standard. A table illustrating three categories will be followed by multiple

figures that help illustrate both the increase in acquired knowledge and the decrease

in marginal utility with each additional expert for individual categories. Table 5

56 indicates the total average percent of total acquired knowledge (all items on the

CTA protocol), action steps, and decision steps for each non- repeating group’s when

compared to the six-SME gold standard protocol. As anticipated, the average

percentage increases steadily with each additional expert up to six experts, whom

account for 100% of the items within the gold standard CTA protocol.

Table 5 The average percent of Total knowledge, Action steps, Decision steps per number of experts when compared to a six-SME gold standard protocol.

Table 6 illustrates the average percentage of acquired knowledge from a single

expert and the increased percentage of total knowledge, action steps, and decision-

steps as a result of an additional SME. Also noted within table 6 is the diminishing

marginal utility with each additional expert. The average amount of total knowledge

gained from zero experts to one expert was a 57.44% increase. With the addition of a

second expert, an average increase of 13.28% of total knowledge was noted. A third

expert provides and average increase of 8.59%, while a fourth expert provided an

increase of 4.59%. With the addition of the fifth and sixth experts, the average

increase of acquired knowledge was 6.62% and 9.49%, respectively. A single expert

provided an average increase of 70% for action steps and an average increase of 65%

Number of Experts

Total Knowledge

Action Steps Decision Steps

Per group Per group Per group 1 57% 70% 65% 2 71% 72% 88% 3 79% 78% 91% 4 84% 82% 92% 5 91% 84% 93% 6 100% 100% 100%

57 for decision steps. The addition of subsequent experts produces an increase in

acquired action steps and decision steps at varying increased rates. However, there is

clearly a diminishing utility with each additional expert.

Table 6 Average Increase in Percentage of Total Knowledge, Action Steps, and Decision Steps as a function of an additional Expert When Compared to a Six-Subject Matter Expert Gold Standard Protocol

Figure 1 shows the increase in percentage of total knowledge acquired as a function

of additional experts when compared to a six-expert gold standard CTA CVC procedure

protocol. Total knowledge consists of all the subsections measured, including action

steps, decision steps, conditions when to perform and contra-indications on when not to

perform the surgery. The average total knowledge increases from 57.44% with one

expert to 79.31% with three experts and 100% total knowledge gained from the six SMEs

interviewed.

Number of Experts

Total Knowledge Action Steps Decision Steps

increase increase increase 1 57.44 70.08 65.48 2 13.28 1.59 22.62 3 8.59 6.52 2.98 4 4.59 3.79 1.31 5 6.62 2.5 0.48 6 9.49 15.53 7.14

58

Figure 1. Percentage of acquired knowledge based on six-expert gold standard CTA CVC procedure protocol as a function of the number of experts.

Figure 2 shows the marginal utility or the percent of total knowledge acquired

with the addition of an additional expert, up to six total experts. One subject matter

expert provided an average increase of 57.44% of knowledge. When the data from

the second subject matter expert is combined with the first subject matter, there is an

average increase of 13.28% of acquired knowledge. When a third expert is added, an

average of 8.59% additional acquired knowledge is obtained. A fourth expert

provided an average increase of 4.59%. While a fifth expert added an additional

6.62%% increase in acquired knowledge. The sixth expert provided just an additional

9.49% of increased knowledge. Utilizing the established standard set by Chou and

Salvendy (1994) of 10% marginal utility to quantify how many experts are

recommended, figure 2 indicates that three experts are recommended to meet this

threshold level of acquired knowledge.

59

Figure 2. Average percentage increase of knowledge as a function of an additional expert for total possible items in the gold standard CVC procedure protocol.

Figure 3 shows the increase of percentage of action steps acquired [based

on a six-expert gold standard CVC procedure protocol] with each additional subject

matter expert. The accumulation of knowledge from additional experts indicates a

greater percentage of actions steps were acquired as a result of subsequent experts.

On average, the knowledge gained from additional experts exceeded the total

combined knowledge of the prior number of experts. For example, the amount of

acquired knowledge from two experts is greater than a single expert. This pattern of

increased acquired knowledge is repeated in future analysis of different subsections

within the gold standard protocol. A single expert provided an increase of 70.08% of

action steps while three experts provided an average of 78.18%, while six experts

provided 100% of action steps within the CVC gold standard procedure protocol.

60

Figure 3. Average percentage increase in action step acquired knowledge as a function of the number of experts. Figure 4 indicates the marginal utility for action steps in respect to

additional experts, based on a six-expert gold standard CVC procedure protocol. A

single expert provided an average increase of 70.08% of new knowledge. Figure 4

illustrates that marginal utility is achieved at two experts. The greatest percentage of

acquired knowledge is achieved with the initial expert. When the acquired

knowledge from additional experts is added, there is an increase in overall captured

knowledge. The variability of the average increase in acquired knowledge was a

product of the varying degrees of information acquired from the experts within the

investigation and their agreement with the gold standard protocol.

Figure 4. Average percentage increase of action steps as a function of additional experts.

61

Figure 5. Average Percentage increase of Gold Standard Decision Steps as a function of the number of experts.

Figure 5 shows the increase of acquired knowledge of gold standard

decision steps with each additional subject matter expert. One expert provided

65.48% of total gold standard decision steps, while three experts provide an average

of 91.07% and six experts provide 100% of gold standard decision steps. There were

a total of fourteen decision steps within the six-expert based gold standard protocol.

The majority of the decision steps were acquired with the information captured from

the first two experts, on average. The information captured from groups of three,

four, and five experts did not provide a significant increase of acquired decision steps

beyond the average of two experts.

62

Figure 6. Average percentage increase of decision steps as a function of additional experts.

Figure 6 shows the average percentage increase of decision steps as a function of

additional experts. A single expert provided an average increase of 65.48% of the

gold standard decision steps that are within the CVC placement procedure. With the

addition of a second expert, an average of 22.62% increase was noted. The acquired

knowledge from the third expert provided an additional 2.98% increase. Figure 6

illustrates that diminishing marginal utility of 10% was achieved with three experts.

As noted in figure six, the greatest percentage of acquired knowledge in decision

steps is achieved with two experts. Groups of three, four, and five experts did not

produce much of an increase in acquired knowledge. While six experts provided a

increase of acquired knowledge due to comprising of all captured knowledge.

63 Table 7 Average Percentage of Objectives, Reasons, and Risks Acquired from Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol

Table 7 indicates that the addition of subsequent experts did not increase the

percentage of information regarding the overall objective of performing the CVC

procedure; therefore a figure is not required. A total of one expert is recommended

based on a 10% marginal utility for the overall objective. The average percentage of

acquired knowledge in the areas of Risks and Reasons increases with each additional

expert up to six.

Figure 7. Percentage of knowledge acquired of risks as a function of the number of experts.

Number of Experts

Objective Reasons Risks

1 100 27.78 19.61 2 100 48.89 38.04 3 100 65 55.29 4 100 77.78 67.45 5 100 88.89 86.27 6 100 100 100

64 Figure 7 shows the percentage of acquired knowledge in the area of risks of not

performing a CVC placement procedure correctly. The percentage of acquired

knowledge of risks increases steadily with each additional expert. A single expert

provided an average of 19.61% of acquired knowledge. Two experts provide an

average of 38.04% while three experts provided an average of 55.29%. Four experts

provided 67.45% and five experts provided 86.27% of acquired knowledge. Six

experts provided 100% of total acquired knowledge in the area of risks of not

performing the CVC procedure correctly.

Figure 8 indicates the average percentage increase of acquired knowledge in the

area of risks as a function of added experts up to six experts. The percentage of

knowledge captured from experts, increases steadily as a result of additional experts.

The average acquired knowledge from a single expert was 19.61%. The addition of a

second and third expert provided an increase of 18.43% and 17.25% respectively.

The addition of a fourth expert indicated an average of 12.26%. The addition of a

fifth expert showed an elevated increase of 18.82% of acquired knowledge. This

increase was caused by the total inclusion of information from expert four, which had

the greatest agreement overall with the six-expert based gold standard. The sixth

expert provided an additional 13.73% increase. The marginal utility of 10% was

achieved at the sixth expert.

65

Figure 8. Average increase of acquired knowledge of risks as a function of added experts. Figure 9 shows the percentage of acquired knowledge in the area of reasons and

benefits of performing a CVC procedure correctly. On average, a single expert

provided 27.78% of acquired knowledge. The addition of a second and third expert

increased the acquired knowledge to 48.89% and 65% respectfully. Four experts

provided an average of 77.78% and five experts provided 88.89% of acquired

knowledge. Six experts provided 100% of total knowledge in the area of reasons to

perform the CVC procedure correctly.

Figure 9. Percentage of knowledge acquired for reasons to perform the CVC procedure correctly.

66

Figure 10. Average percentage increase in acquired knowledge in the area of reasons based on a six-expert gold standard CVC protocol.

Figure 10 displays the average percentage increase of reasons and benefits for

performing a CVC procedure correctly based on a six-expert CTA CVC procedure

gold standard protocol. On average, a single expert provided an increase of 27.78%

of new knowledge. Adding a second and third expert provided a 21.11% and 16.11%

percentage increase in acquired knowledge respectively. Four experts provided an

average of 12.78% increase, while both groups of five and six experts gave an

increase of 11.11% each. The marginal utility value of 10% was achieved at the sixth

expert.

67 Table 8 Average Percentage of Indications, Contraindications, and Standards Acquired from Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol

Number of Experts

Indications Contraindications Standards

1 75 45.83 23.33 2 92.5 65 42.67 3 92.5 71.25 59 4 95 78.33 73.33 5 100 91.67 86.67 6 100 100 100

Figure 11. Percentage of Indications to perform the CVC procedure based on a six-expert gold standard as a function of the number of experts

Figure 11 shows the average increase of acquired knowledge in the area of

indications to perform a CVC placement procedure as a function of the number of

experts up to six experts. Based on a six-expert gold standard, a single expert gave an

average of 79.17% of acquired knowledge. With the addition of a second and third

expert, the knowledge acquired increased to 96.67% and 98.75% respectfully. 100%

total acquired knowledge was achieved with the addition of expert four.

68

Figure 12. The average percentage of acquired knowledge gained as a function of additional experts in the area of indications to perform a CVC procedure.

Figure 12 shows the average increase of acquired knowledge as a function of an

additional expert. A single expert provided an average increase of 75%. The second

expert provided an increase of 17.5% of acquired knowledge. The third expert did

not provide any increase in acquired knowledge. The fourth expert provided only an

added 2.5% of acquired knowledge. Expert five provided 5% of additional acquired

knowledge while expert six did not provide any additional information. The marginal

utility was achieved with three experts.

69

Figure 13. Percentage of knowledge acquired in the area of contraindications as a function of the number of experts

Figure 13 displays the average acquired knowledge of contraindications as a

function of the number of experts based on a six-expert gold standard CTA CVC

placement protocol. A single expert provided an average of 45.83% of acquired

knowledge in contraindications on when not to perform the CVC placement

procedure or a particular side of patient to insert the CVC catheter. The third expert

provided an average of 71.25%. With five experts, an average of 91.67% of acquired

knowledge was obtained. Expert six provided 100% of total knowledge in

contraindications.

70

Figure 14. The average percentage increase in the amount of acquired knowledge in the area of contraindications as a function of the number of experts. Figure 14 shows the average increase in acquired knowledge as a function of the

number of experts, based on a six-expert gold standard CVC placement procedure in

the area of contraindications (when not to perform a procedure). The average

increase of acquired knowledge from the first expert was 45.83%. When the acquired

knowledge from a second expert was added, an additional 19.17% increase was

observed. The third and fourth expert provided an increase of 6.25% and 7.08%

respectfully. Interestingly enough, the addition of a fifth expert provided an average

increase of 13.33% of new acquired knowledge. The sixth expert provided an

increase of 8.33% of additional knowledge of contraindications. The marginal utility

of 10% threshold was achieved at the third expert.

71

Figure 15. Percentage of knowledge acquired for standards of CVC procedure as a function of the number of experts. Figure 15 displays the average increase of acquired knowledge in the area of

standards indicating that the CVC placement procedure was performed correctly. The

first expert provided an average of 23.33% of acquired knowledge. The addition of

the second expert provided 42.67% of acquired knowledge. The addition of expert

three and four provided 59% and 73.33% of acquired knowledge of the standards. A

total of 86.67% of knowledge was noted with the fifth expert and 100% of acquired

knowledge was achieved with the sixth expert.

Figure 16. Average percentage increase of acquired knowledge for standards based on a six-expert gold standard CVC protocol.

72 Figure 16 indicates the average percent increase of acquired knowledge in the

area of standards as a function of additional experts. A single expert provided

23.33% of new knowledge. The addition of the second expert provided an increase of

19.33% and expert three provided an average increase of 16.33%. The fourth and

fifth experts added 14.33% and 13.33% of acquired knowledge. The sixth expert

provided an increase of 13.33% of acquired knowledge in the area of standards

indicating the procedure was performed correctly. The 10% marginal utility was

achieved at the sixth expert.

Table 9 Average Percentage of Equipment and Tasks acquired from Multiple Groups of Experts When Compared to a Six Subject Matter Expert Gold Standard Protocol

Number of Experts Equipment Tasks 1 65.48 93.75 2 88.10 100 3 91.07 100 4 92.38 100 5 92.86 100 6 100 100

Table 9 indicates that the addition of the second expert increased the percentage

of information regarding the amount of tasks involved with performing the CVC

procedure to 100%; therefore a figure is not required. A total of two experts are

recommended based on a 10% marginal utility for the required tasks of the CVC

procedure.

73

Figure 17. Average Percentage increase of Gold Standard Equipment as a function of the number of experts. Figure 17 shows the average increase of acquired knowledge of necessary

equipment for the CVC procedure, based on the six-expert gold standard protocol. A

single expert gave an average of 65.48% of total equipment recommended to perform

the CVC placement procedure. Three experts gave an average of 91.07% while six

experts gave 100% of total acquired knowledge in the area of recommended

equipment for the CVC placement procedure. There was a significant increase

between a single expert and an additional expert, but there were only small changes

noticed from experts two through five and a slight jump of acquired knowledge of

equipment with the sixth expert.

74

Figure 18. The Average Percentage gain of acquired knowledge of needed equipment as a function of additional experts Figure 18 indicates the average percentage increase of acquired knowledge as a

result of adding the knowledge from an additional expert. With the assumption you

have no information with zero expert, a single expert will provide an increase of

53.33% of knowledge acquired in the area of recommended equipment for the CVC

placement procedure. When a second expert is added to the first expert, an average

of 22.22% of acquired knowledge was observed. A third expert added an average of

11.44% of additional knowledge, while the fourth and fifth experts added 2.11% and

8.11% respectfully. The sixth expert added an average of 2.78% of additional

acquired knowledge. The marginal utility of 10% was achieved at four experts.

Summary

On average, a single expert provided 57% of the total knowledge, 70% of the

action steps, and 65% of the decision steps in a central venous catheter six – SME

gold standard CTA protocol.. Table 10 indicates the point of diminishing marginal

75 utility for total knowledge acquired and all the other subsections of items acquired

within a six-expert CVC gold standard protocol.

Table 10: Quantity of Experts Recommended if a 10% Marginal Utility in Knowledge Acquisition is Expected

Item Measured Number of Experts for 10% Marginal Utility

Total Knowledge Acquired 3 Action Steps 2 Decision Steps 3 Objective 1 Reasons 6 Risks 6 Indications 3 Contraindications 3 Standards 6 Equipment 4 Tasks 2 Total Average 3.5

76 CHAPTER 4: CONCLUSIONS

The purpose of this study was two-fold: to investigate the average amount of

information obtained from interviewing an expert and the average amount of acquired

knowledge from each additional expert up to six experts. As a result of answering the

two research questions, a benchmark will be established for the optimal number of

experts one needs to interview to capture the knowledge and skills required to

complete a central venous catheter placement. The research literature that focuses on

the use of cognitive task analysis and experts has expressed the advantages and

benefits of utilizing experts and the knowledge captured through CTA knowledge

elicitation techniques. However, prior research has not focused on quantifying the

amount of knowledge CTA capture’s with respect to an overall task performance goal

and additionally, there is only one known study that specifically quantifies the

average amount of knowledge and skills captured from interviewing additional

experts (Crispen, 2010). As a result, there is limited research on the number of subject

matter experts needed to capture the acquired knowledge from experts (Chao and

Salvendy, 1994; Crispen, 2010).

Research Questions

Research question #1 : How much information about a central venous

catheter placement procedure does a single expert provide when compared to a six-

subject matter expert gold standard protocol?

The results of this study indicate that experts omitted critical information on how

to conduct a central venous catheter placement procedure. The range of acquired

77 knowledge obtained from the experts for total information in comparison to the six

–expert gold standard protocol was 43% to 73%, which means that the experts within

this study omitted a range of ~57%-27% of total information. Although this is not a

focus of this investigation, it is important to note that other research has reported

similar results. For example, Clark, Pugh, Yates, Early, and Sullivan (2008) found

that physicians omitted an average of 68.25% of procedural steps when the data from

their initial interviews was compared to a gold standard. Information regarding

subsets of knowledge was acquired for overall objective, risks of not performing the

procedure correctly, benefits and reasons to performing the procedure correctly,

conditions of when to perform the procedure (indications) and when not to perform

the procedure or a specific side/location (contraindications) and recommended

equipment. The results indicated that, on average, each of the experts were able to

provide the overall objective for the procedure (100%), an average of 70% of the

action steps, and 65% of decision steps. Additionally, each of the experts

interviewed provided an average of 28% of reasons and benefits, 20% of risks of not

performing the procedure correctly, 79% of indications of when to perform the

procedure, 46% of contraindications of when not to perform the procedure or choose

a side/location, and 56% of total equipment for the procedure, when compared to a

six-expert based gold standard protocol.

The findings of the current study are supported by previous research. Crispen

(2010) investigated the effects of the use of CTA in capturing the critical knowledge

from experts on how to conduct an open cricothyrotomy procedure. Crispen (2010)

78 found that a single expert provided only 56% of total knowledge, 66% of action

steps, and 28% of decision steps when compared to a six-expert gold standard on how

to perform an open cricothyrotomy procedure. The results of the current

investigation support previous research that utilized CTA knowledge elicitation

techniques in capturing critical knowledge from experts on specific tasks, in that

experts omit a significant amount of information when asked to describe how to

conduct a procedure (Clark et al., 2008). And yet the experts in this study reported

many more decision steps (65 percent) than in previous studies (28 percent in

Crispen, 2010 and Velmahos et al, 2004). It is possible that the difference is due to

the fact that the catheter procedure examined is highly controversial and undergoing

changes because of medical errors. This controversy may have led the experts

interviewed to become more conscious of the decisions necessary to perform.

Research question 2 : How much average ( ) critical information is gained

from each additional cognitive task analysis (CTA) interview in regards to a medical

procedure?

With each additional expert, the amount of acquired knowledge increased overall,

but at a diminishing rate of utility. When comparing the total knowledge acquired

from one to two subject matter experts, there was an average of 16 percent gain in

acquired knowledge. Three subject matter experts provided an average gain of 8% of

additional knowledge. The diminishing marginal utility effect of increased

percentage of acquired knowledge continued with the addition of the fourth expert

(4% increase), but jumped to a 5% increase when 5 subject matter experts were

79 utilized. This unexpected jump in percentage of additional acquired knowledge

from 5 experts could be explained by looking at the individual experts percentage of

acquired knowledge when compared to the gold standard. Expert 4 had the highest

agreement with the six-expert CTA CVC gold standard protocol at 73%. When the

groups of the five subject matter experts were created, the data from expert 4 was

included in five of the six of the averages created with this group. On the other hand,

when groups of 4 subject matter experts were created, expert 4 was include in only 10

of the 15 total groups of non-repeating experts possible. Therefore, the data expert 4

was present in only 75% of the total 4-expert possible groupings. As expected, when

the data from expert 4 was added to any grouping, the average of that group was

higher than groups that did not contain expert 4. The groups were created so no single

expert appeared in any combination more than once. No combinations appeared more

than once in any grouping level (2 SME, 3 SME, 4 SME, and 5 SME combinations).

One of the possible reasons for this unexpected elevated score from expert 4 could be

caused by a recency effect (Nairne, 1988) of performing the CVC procedure close in

approximation prior to their scheduled interview. Other factors include their

involvement with the overall training of resident physicians in how to perform the

CVC procedure. At the hospital where the investigation occurred, surgical residents

are not able to perform a CVC placement without completing a required surgical

skills course. In reviewing the interview transcripts, it was noted that one of the

experts indicated involvement as an instructor within a central venous catheter

placement surgical skills course. Due to the amount of information this expert

80 provided in agreement with the creation of the six-expert gold standard protocol

(73% agreement), this could lead to a possible criteria of who should be considered

an expert for the purposes of a cognitive task analysis interview. Future research

could investigate the relationship between teaching physicians versus non-teaching

physicians and the percentage of agreement with an expert based gold standard

protocol on how to conduct a medical procedure. Although research indicates

teachers often describe ‘what to do’ and not necessarily ‘how to do it’. Sullivan et al.,

(2008) found that while expert surgeons described between 46 to 61% on what to do

in a colonoscopy procedure: they reported only between 26-50% of how to perform

the same procedure. Additionally, Feldon (2004) found that experts have significant

omissions (48-89%) in their accuracy of self-reports. Therefore, honoring an

individual as an expert due to their position as a teacher needs to also include other

contributing factors as well. While the literature provides clear differences in both

the cognitive and task performance abilities between experts and novices (Ericsson et

al., 2006) future research could investigate other characteristics that qualify

individuals as experts.

In reviewing the data collected from each expert, it is noted that not every expert

interviewed provided information in all subset categories. This may be a human

variable associated with semi-structured interviews. While every semi-structured

interview used the same CTA interview protocol that provided the experts an

opportunity to share their knowledge and skills regarding the CVC procedure,

variance due to the individual characteristic differences between the multiple analyst

81 utilized in this investigation might have been a “contaminating variable” (Crandall

et al., 2006 p.261). An analyst could become a contaminating variable during the

interview process by not presenting themselves in a neutral presence and utilizing the

same tone and interpersonal communication actions equally to all individuals

involved in the research project. Since multiple analysts were utilized in this story

certain individual human differences and CTA interview skill level may have been a

contributing factor in the initial capturing of experts knowledge and skills. However,

each expert reviewed the information they provided, represented in a CTA protocol,

for verification of data and agreement purposes. As a result of reviewing the CTA

protocol, five out of six experts provided additional inserts of conceptual and

procedural knowledge and/or modifications to their initial interview data. Since the

experts were given an equal opportunity to provide such detail initially, their additive

and subtractive actions support previous research that experts tend to omit

information when explaining how to conduct a procedure (Clark & Estes, 1996).

Chao and Salvendy (1994) utilized a 10% marginal utility to determine when to

stop obtaining information from experts to solve computer application problems.

They indicated that although additional experts, beyond three, will provide an

increase in acquired knowledge, it is not worth the additional time and effort required

to gain the increase in knowledge from additional experts. The author utilized the

same 10% marginal utility threshold as the point where you would not need to

interview additional experts to acquire the needed knowledge on how to conduct a

central venous catheter placement. The results of the current investigation support

82 Chao and Salvendy (1994) given the significant costs human resources and time

involved with completing the current investigation and the average total acquired

knowledge from experts beyond three experts.

Table 7 indicates the recommended number of experts needed to interview to

capture the conceptual and/or procedural knowledge for each sub-section (i.e. risks,

standard, equipment, etc.) based on a 10% marginal utility. The data analysis

indicated three experts are recommended to capture all items (total acquired

knowledge in completing a CVC procedure. However, the data indicated different

amounts of experts are recommended to capture the knowledge and skills for each

subsection. As a result of averaging all the required experts for each subsection,

excluding total acquired knowledge, the average number of needed experts was 3.6.

For all realistic purposes, when reviewing the calculated average number of experts

needed for all subsections combined, four experts would be required. Similar results

were found by Crispen (2010). Crispen (2010) conducted a CTA investigation

focused on an open cricothyrotomy procedure in investigating how many experts are

needed to capture the critical information. Crispen (2010) found, on average, four

experts were recommended to interview in order to capture all critical knowledge and

skills needed to complete an Open Cricothyrotomy procedure.

Although there are several components to the central venous catheter placement

procedure, an argument can be made that the most critical components of the

procedure are the action and decision steps involved with the CVC protocol. When

reviewing the needed number of experts for both action and decision steps, based on a

83 10% marginal utility, the average number of experts needed is 2.5 experts.

Therefore, interviewing three experts is recommended to capture the desired

expertise.

The results of the current investigation support previous findings that Chao and

Salvendy (1994) obtained in their study. Chao and Salvendy (1994) focused on the

percentage of procedural knowledge (Action and Decision steps) required to

diagnose, debug, and or interpret a computer problem. When reviewing the required

number of experts needed for the action and decision steps for the CVC procedure it

is noted that three experts are required based on a 10% marginal utility. Additionally,

the current investigation provides support to the work previous conducted by Ashton

(1986). Ashton (1986) who built upon the work of R. M Hogarth (1978),

investigating how many experts and which ones are needed to be included in a static

group. Ashton (1986) concluded that group validity increases as a function of

increasing experts, however on average only three experts are needed to achieve the

most improvement of validity possible.

Performing a central venous catheter placement has greater human complications

than fixing a computer problem. Therefore, the analysis conducted within the current

investigation looked beyond just the procedural knowledge acquired from experts.

The current investigation looked critically at the contributions of each subsection

including: indications; contraindications; risks of not performing the surgery

correctly; benefit of performing the surgery correctly; and standards indicating a

correctly performed CVC placement procedure.

84 Summary

A cognitive task analysis interview was conducted with six expert physicians

regarding how to conduct a central venous catheter placement. The transcribed

interviews were coded and converted into individual CTA protocols listing the

knowledge and skills necessary to complete a central venous catheter placement. The

individual expert CTA protocols were combined into a six-expert gold standard

protocol through an iterative process that included analysis by both the experts and

analysts;, aggregation of individual reports into one, and a final validation of a

completely detailed procedure through a final expert review.

The results of the study indicated that on average experts provided approximately

68% of total acquired knowledge when compared to a six-person gold standard

protocol. When the acquired knowledge is added to the initial expert, it was found

that a second expert provided an average of additional 16% of acquired knowledge

and the third expert provided approximately 7% more information. The results of the

student indicate that four experts were needed to reach a 10% point of diminishing

marginal utility for the acquisition of additional knowledge based on combining all

sub-sections of a six-expert gold standard CTA CVC protocol, however only three

experts are recommended to acquire the most critical action and decision steps. This

study is one of the first investigations that have determined a required number of

experts needed to capture all the required knowledge to conduct a CVC procedure.

Future research will be needed to validate these findings.

85 Limitations and Implications for Future Research

The physicians who participated within the study were considered experts by their

longevity of being a physician, the number of times they have performed the central

venous catheter placement, and their activities in training new surgical residents.

However, no established criteria of how many times a physician needed to perform

this CVC procedure, in combination with years of experience, were set as parameters

in establishing quantitative data signifying who is an expert. Although each of the

physicians verbally expressed how many times they conducted the CVC procedure

and the approximate time they last performed the procedure, the exact quantity of

procedures performed and the exact date of their last performance of the procedure

was not obtained from all physicians. Therefore any correlation or regression analysis

that focused on the number of procedures conducted and future percentage of total

gold standard items or last date of performing the procedure and the percentage of

total gold standard items to measure the effects of time and recall was not available to

the current investigation. Future research may incorporate these demographic

characteristics to better quantify qualities of an expert physician in relation to other

measures of expertise for a more holistic view of what constitutes a medical expert.

Another limitation to this study was the expert’s ability to review documents with

a discerning eye checking for agreements and disagreements with their original and

gold standard protocols. If research indicates that experts have limited capacity to

recognize information they omit when providing instruction, there is reason to believe

that they would have similar difficulty recognizing omissions in a self-report based

86 protocol. Although every physician was notified that they would have the

opportunity to review the formatted protocols to verify accuracy and their ability to

make addition or deletions, only five out of six experts added or made any

modifications. One expert indicated the protocol looked fine the way it was. This

leads to a question; does the level of complexity cause the experts to have difficulty

in reviewing the document, or is time and effort on the part of the expert a

confounding variable? Johnson et al., (2006) acknowledged that in their iterative

verification and editing process of having the experts validate the acquired

knowledge; minimal to no response was provided by the experts in the second round

of editing. They concluded that this was primarily due to the busy schedules of the

expert radiologists and a limited understanding of the potential benefits CTA can

offer to training.

Finally, a significant limitation to this investigation could be tied to the context of

the procedure. While the current investigation indicates that three expert surgeons

need to be interviewed to capture the necessary knowledge and skills for a CVC

procedure, Crispen (2010) recommended that four expert surgeons are sufficient for

the Open Cricothyrotomy procedure. If the same CTA methodologies were utilized

for both the current study and Crispen (2010), then why were different numbers of

experts recommended? One of the contributing factors could be the frequency of the

procedures performed by the experts. While the Open Cricothyrotomy’s is

infrequently performed, the time delay in between procedures may affect the experts

ability to recall specific details conducted during the procedure.

87 The central venous catheter procedure is more frequently performed in

comparison to the Open Cricothyrotomy. Due to the frequency surgeons perform the

CVC procedure, it would be assumed that the experts would omit a greater percentage

of their knowledge and skills as a consequence of their automated knowledge.

However, the data indicated that the experts provided more critical information (70%

- action steps, and 65%- decision steps) than prior research has found (Clark et al.,

2008). Clark et al., (2008) found that expert trauma surgeons provided only 30% of

the critical information necessary to complete a medical procedure. A confounding

variable could be the increase of interest in the medical field related to the central

venous catheter procedure within the past year. The central venous catheter

procedure has recently become a procedure that requires physicians to pass a surgical

skills course in order to complete the procedure on a live patient and is continually a

topic of interest in the research literature.

Conclusion

This study aimed to contribute to the growing body of knowledge on the use of

cognitive task analysis and the benefits of working towards quantifying the optimal

number of experts needed to capture expert knowledge and the skills required to

perform complex medical procedures. Although this study indicates that three

experts are required to capture the critical information (action and decision steps)

needed to perform a central venous catheter placement procedure, future research will

be required to validate these findings. These findings could benefit future curriculum

development and additional research in the medical/ surgical education field by

88 limiting the costs of CTA by knowing how many experts are recommended to

capture critical information for a procedure. Additionally, Crispen (2010) found four

experts are recommended to capture the critical information related to the Open

Cricothyrotomy procedure, which differs from the results of the current investigation.

Since the primary factor involved with the current CTA research focused primarily on

human factors, future research needs to investigate and establish a standard for time

as a cost of conducting CTA investigations. While there are different CTA techniques

that capture the knowledge and skills from experts, reducing the time and effort

associated with the knowledge bottleneck inherent in the capturing of expert

knowledge to a usable form of knowledge representation- training protocols for

novice surgeons, is a compounding variable in CTA research that needs attention.

89 REFERENCES

Abernethy, B., Poolton, J. M., Masters, R. S. W., & Patil, N. G. (2008). Implications of an expertise model for surgical skills training. ANZ Journal of Surgery, 78(12) 1092-1095.

Aggarwal, R., Grantcharov, T. P., & Darzi, A. (2007). Framework for Systematic

Training and assessment of technical skills. Journal of American College of Surgeons, 697-705.

Anderson, J.R. (1996). ACT: A simple theory of complex cognition. American

Psychologist, 51(4) 355-365. Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., Qin, Y. (2004).

An integrated theory of mind. Psychological Review, 111(4) 1036-1060. Anderson, J. R. & Fincham, J. M. (1994). Acquisition of procedural skills from

examples. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20(6) 1322-1340.

Anderson, J. R., Finchman, J. M., Douglass, S. (1997). The role of examples and

rules in the acquisition of a cognitive skill. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(4), 932-945.

Anderson, L.W., Krathwohl, D. R., Airasian, P.W., Cruikshank, K. A., Mayer, R. E.,

Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom’s Taxonomy of Educational Objectives. New York: Longman.

Anderson, J.R., & Schunn, C.D. (2000). In R. Glaser (Ed.), Advances in instructional

psychology (5). Mahwah, NJ: Erlbaum. Andre, M., Borgquist, L., Foldevi, M., & Molstad, S. (2002). Asking for rules of

thumb: A way to discover tacit knowledge in general practice. Family Practice, 19(6), 617-622.

Ashton, R. H. (1986). Combining the judgments of experts: How many and which

ones? Organizational Behavior and Human Decision Processes, 38, 405-414.

Bathalon, S., Martin, M., & Dorion, D. (2004). Cognitive task analysis, kinesiology

and mental imagery: challenging surgical attrition. Journal of American College of Surgeons, S73.

90 Boshuizen, H.P.A., & Schmidt, H.G. (1992). On the role of biomedical knowledge in

clinical reasoning by experts, intermediates and novices. Cognitive Science, 16, 153-184.

Cauraugh, J. H., Martin, M., Martin, K. K. (1999). Modeling Surgical Expertise for

motor skill acquisition. The American Journal of Surgery, 177, 331-336. Chao, C-J. & Salvendy, G. (1994). Percentage of procedural knowledge acquired as a

function of the number of experts from whom knowledge is acquired for diagnosis, debugging, and interpretation tasks. International Journal of Human-Computer Interaction, 6(3), 221-233.

Chi, M., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. Sternberg

(Ed.), Advances in the psychology of human intelligence (Vol. 1). Hillsdale, NJ: Erlbaum.

Clark, R. E. (2004). Design Document For A Guided Experiential Learning Course.

Final report on contract DAAD 19-99-D-0046-0004 from TRADOC to the Institute for Creative Technology and the Rossier School of Education.

Clark, R. E. (2008, October 26). Knowledge and test types. Unpublished manuscript.

University of Southern California, Los Angeles, CA. Clark, R. E. & Estes, F. (1996). Cognitive task analysis. International Journal of

Educational Research, 25, 2, 403-417. Clark, R.E., Feldon, D., Van Merriënboer, J.J.G., Yates, K., & Early, S. (2008).

Cognitive task analysis. In J.M. Spector, M.D. Merrill, J.J.G. van Merriënboer, & M.P. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Clark, R. E., Pugh, C. M., Yates, K. A. and Sullivan, M. (2008). The use of cognitive

task analysis and simulators for after action review of medical events in Iraq. Technical Report 5-21-2008 developed for the Center for Cognitive Technology, Rossier School of Education, University of Southern California. Retrieved from http://www.cogtech.usc.edu

Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International

Journal of Human-Computer Studies, 41, 801-849.

91 Cowan, N. (2001). The magical number four in short-term memory: A

reconsideration of mental storage capacity. Behavioral & Brain Sciences, 24, 87-114

Crandall, B. & Getchell-Reiter. K. (1993). Critical decision method: A technique for

eliciting concrete assessment indicators from the intuition of NICU nurses. Advances in Nursing Science, 16(1), 42-51.

Crandall, B., Klein, G., & Hoffman, R.R. (2006). Working Minds: A practitioners

guide to cognitive task analysis. MIT Press: Cambridge, Massachusetts. Crispen, P. D. (2010). Identifying the Point of Diminishing Marginal Utility for

Cognitive Task Analysis Surgical Subject Matter Expert Interviews. Unpublished doctoral dissertation presented to the faculty of the USC Rossier School of Education, University of Southern California.

De Jong, T. & Ferguson-Hessler, M. G. M. (1996). Types and qualities of knowledge.

Educational Psychologist, 31, 2, 105-113. Ericsson, K. A. (2006). The influence of experience and deliberate practice on the

development of superior expert performance. In K. A. Ericsson, N. Charness, P. Feltovich, and R. R. Hoffman, R. R. (Eds.). The Cambridge Handbook of Expertise and Expert Performance (pp. 685-706). Cambridge, UK: Cambridge University Press.

Ericsson, K. A., Charness, N., Feltovich, P. J., Hoffman, R. R. (2006). The

Cambridge Handbook of Expertise and Performance. Cambridge University Press: New York, NY.

Ericsson, K. A. & Kintsch, W. (1995). Long-term working memory.

Psychology. Review. 102, 211–245. Expert Knowledge Solutions (2009). Cogntitive task analysis training aid. Fackler, J.C., Watts, C., Grome, A, Miller, T., Crandall, B., Pronovost, P. (2009).

Critical care physician cognitive task analysis: an exploratory study. Critical Care, 13(2), 1-8.

Feldon, D. F. (2004) Inaccuracies in expert self report: Errors in the description of

strategies for designing psychology experiments. Unpublished doctoral dissertation, Rossier School of Education, University of Southern California, USA.

92 Feldon, D. (2007). The implications of research on expertise for curriculum and

pedagogy. Educational Psychology Review, 19, 91-110. Gourlay, S. N. (2004). ‘Tacit knowledge’: The variety of meanings in empirical

research.. 5th European Organizational Knowledge, Learning and Capabilities conference; Innsbruck, Austria, 2-3 April.

Grantcharov, T.P. & Reznick, R.K. (2008). Teaching rounds: Teaching procedural

skills [Electronic Version]. BMJ: British Medical Journal 336, 1129-1131. Grunwald, T. Clark, D. Fisher S.S., McLaughlin, M. Narayanan, S., Piepol, D.

(2004). Using cognitive task analysis to facilitate collaboration in development of simulator to accelerate surgical training. Proceedingsofthe12thMedicineMeetsVirtualRealityConference,114‐120.

Hahn, B. B., E. Rall, and D. W. Klinger, (2003). Cognitive task analysis of the

warning forecaster task. Report RA1330-02-SE-0280. [Available from Klein Associates Inc., 1750 Commerce Center Blvd., North Fairborn, Ohio 45324-6362.]

Halsted, W.S. (1904). The training of surgeons. Bulletin of Johns Hopkins Hospital,

15, 267-275. Hamdorf, J. M. & Hall, J.C. (2000). Acquiring surgical skills. British Journal of

Surgery, 87, 28-37. Hinds, P. J. (1999). The curse of expertise: The effects of expertise and debiasing

methods on prediction of novice performance. Journal of Experimental Psychology: Applied, 5, 205-221.

Hinds, P. J., Patterson, M. & Pfeffer, J. (2001). Bothered by abstraction: The effect of

expertise on knowledge transfer and subsequent novice performance. Journal of Applied Psychology, 86(6), 1232-1243.

Hoffman, R. R. & Militello, L. G. (2009) Perspectives on Cognitive Task Analysis:

Historical origins and modern communities of practice. Psychology Press. New York, NY.

Hoffman, R.R., Crandall, B. W., & Shadbolt, N.R. (1998). Use of the critical decision

method to elicit expert knowledge: A case study in cognitive task analysis methodology. Human Factors, 40(2), 254-276.

93 Hoffman, R.R., Shadbolt, N. R., Burton, A.M., & Klein, G. (1995). Eliciting

knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62(2), 129-158.

Jacklin, R., Sevdalis, N., Darzi, A., & Vincent, C. (2008). Mapping surgical practice

decision making: an interview study to evaluate decision in surgical care. The American Journal of Surgery, 195, 689-696.

Johnson, C., Cumming, G., Omodei, M. (2008). How worst case scenarios are

considered by bushfire fighters: An interview study. Presented at the International Bushfire Research Conference, Adelaide, Australia.

Johnson, S., Healey, A., Evans, J., Murphy, M., Crawshaw, M., & Gould, D., (2006).

Physical and cognitive task analysis in interventional radiology. Clinical Radiology, 61, 97-103.

Klein G. A., Calderwood, R. & MacGregor, D. (1989). Critical decision method for

eliciting knowledge. IEEE Transactions on Systems, Man, Cybernetics, 19, 462-472.

Knowlton, B. J. & Moody, T. D. (2008). Procedural learning in humans. Learning

and Memory: A Comprehensive Reference, 3, 321-340. Latorella, K.A., Pliske, R., Hutton, R., & Chrenka, J. (2001). Cognitive task analysis

of business jet pilots' weather flying behaviors: Preliminary results. NASA-TM-2001211034. NASA Langley Research Center. Hampton, VA.

Luker, K. R., Sullivan, M. E., Peyre, S. E., Sherman, R., & Grunwald, T., (2008). The

use of cognitive task analysis-based multimedia program to teach surgical decision making in flexor tendon repair. The American Journal of Surgery, 195, 11-15.

Markus, M. L. (2001). Toward a theory of knowledge reuse: Types of knowledge

reuse situations and factors in reuse success. Journal of Management Information Systems, summer, 18(1), 57-93.

Marshall, R.L., Smith, J.S, Gorman, P.J,, Krummel, T. M., Haluck, R. S., & Cooney,

R. N. (2001) . Use of a human patient simulator in the development of resident trauma management skills. Journal of Trauma Injury, Infection and Critical Care, 5, 17–21.

94 Meyers M.O., Meyer A.A., Stewart, R.D., Dreesen, E.B., Barrick J., Lange, P.A.,

& Farrell, T.M, (In press). Teaching technical skills to medical students during a surgery clerkship: Results of a small group curriculum. Journal of Surgical Research.

Nairne, J. S. (1988). A framework for interpreting recency effects in immediate serial

recall. Memory & Cognition, 16, 4, 343-352. O’Hare, D., Wiggins, M., Williams, A., & Wong, W. (1998). Cognitive task analysis

for decision centered design and training. Ergonomics, 41(11), 1698-1718. Ormrod, J. E. (2008). Educational Psychology: Developing learners. New Jersey:

Pearson. Paris, S. G., Lipson, M. Y., & Wixson (1983). Becoming a strategic learner.

Contemporary Educational Psychology, 8, 293-316. Patel, V.L. & Groen, G.J. (1986). Knowledge based solution strategies in medical

reasoning. Cognitive Science, 10, 91-116. Patterson, E.S., Woods, D. D., Tinapple, D., Roth, E. M. (2001). Using cognitive task

analysis (CTA) to seed design concepts for intelligence analysts under data overload. In Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, Minneapolis, MN: October 8-12, 2001.

Pavlik, P.I., & Anderson, J.R. (2008). Using a model to compute the optimal schedule

of practice. Journal of Experimental Psychology: Applied, 14(2), 101-117. Peters, J. H., Fried, G. M., Swanstrom, L. L., Soper, N. J., Sillin, L. F., Schirmer, B.,

& Hoffman, K. (2003). Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery, 135(1), 21-27.

Pejtersen, A. M. (1989). A Library system for information retrieval on a cognitive

task analysis and supported by an icon-based interface. Association for Computing Machinery, 40-47.

Pirolli, P. L. & Anderson, J. R. (1985). The role of practice in fact retrieval. Journal

of Experimental Psychology: Learning, Memory, and Cognition, 11 (1), 136-153.

95 Porte, M. C., Xeroulis, G., Reznick, R. K., Dubrowski, A. (2007). Verbal feedback

from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. The American Journal of Surgery, 193, 105-110.

Polanyi M (1962) Personal Knowledge: Towards a Post-Critical Philosophy

University of Chicago Press. Redding, R. E., Cannon, J. R., Lierman, B. C., Ryder, J. M., Purcell, J. A. &

Seamster, T. L. (1991). The Analysis of Expert Performance in the Redesign of the en route Air Traffic Control Curriculum. In: Proceedings of the Human Factors Society, 35th Annual Meeting.

Reznick, R. K., & MacRae,H. (2006). Teaching surgical skills-Changesin the wind.

New England Journal of Medicine, 355, 2664-2669. Sachdeva, A. K.,Bell, R. H.,Britt, L.D., Tarpley, J. L., Blair, P.G., Tarpley, M. J.

(2007). National efforts to reform residency education in surgery. Academic Medicine, 82 (12), 1200-1210.

Schraagen, J. M., Chipman, S. F., and Shalin, V. (2000). Introduction to Cognitive

Task Analysis. In S. Chipman, V. Shalin (Eds.) Cognitive Task Analysis. Mahwai, NJ, Lawrence Erlbaum Associates.

Schraw, G. (2006). Knowledge: Structures and processes. In P. Alexander & P.

Winne, (Eds.) (2006). Handbook of educational psychology (2nd ed.). Mahwaj, NJ, Lawrence Erlbaum Associates.

Schunk, D. H. (2000). Learning theories: An educational perspective. Upper Saddle

River, NJ: Prentice-Hall. Scott, D. J., Cendan, J. C., Pugh, C. M., Minter, R. M., Dunnington, G. L., and Kozar,

R. A. (2008). The changing face of surgical education: Simulation as the new paradigm. Journal of Surgical Research, 147(2),189–193.

Sleeman, D.H. & Brown, J.S. (Eds.). (1982). Intelligent tutoring systems. New York:

Academic Press. Spender, J. C. (1996). Organizational knowledge, learning and memory: Three

concepts in search of a theory. Journal of Organizational Change Management, 9(1), 63-78.

96 Stadler, M. A. (1989). On learning complex procedural knowledge. Journal of

experimental psychology: Learning, Memory, and Cognition, 15(6) 1061-1069.

Sullivan, M. E., Brown, C.V. R., Peyre, S. E., Salim, A., Martin, M., Towfigh, S., &

Grunwald, T. (2007).The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement. The American Journal of Surgery, 193, 96-99.

Sullivan, M. E., Ortega, A., Wasserberg, N., Kaufman, H., Nyquist, J., & Clark, R.

(2008). Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional methods? The American Journal of Surgery, 195, 20-23.

Tirapelle, L.A. (2010). The effect of cognitive task analysis based instruction on

surgical skills expertise and performance. Unpublished doctoral dissertation presented to the faculty of the Rossier School of Education, University of Southern California.

Vadcard, L. & Luengo, V. (2004). Embedding knowledge in the design of an

orthopedic surgery learning environment. In: CALIE04, International Conference on Computer Aided Learning in Engineering education, Grenoble: 16-18 fevier, 2004, 1-6.

Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark, R. E., Theodorou,

D., & Maupin, F. (2004). Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. The American Journal of Surgery, 187, 114-119.

Vozenilek, J., Huff, S. Reznek, M., & Gordon, J. A. (2004) See one, do one, teach

one: advanced technology in medical education. Academic Emergency Medicine, 11, 1149-1154.

Webster’s New World College Dictionary (4th ed.). (1999). New York, NY:

Macmillan. Yates, K. (2007). Towards a taxonomy of task analysis methods: A search of

cognition and task analysis interactions. Unpublished doctoral dissertation presented to the faculty of the USC Rossier School of Education, University of Southern California.

Yates, K. A. and Clark, R. E. (In review). Cognitive Task Analysis. International

Handbook of Student Achievement.

97 Yates, K. A., & Feldon, D. F. (in press). Towards a taxonomy of cognitive task

analysis methods for instructional design: Interactions with cognition. Theoretical Issues in Ergonomics Science.

98 APPENDIX A

Non- Repeating combinations of SME Protocols

Two Subject Matter Expert Non- Repeating Pairs: AB, AC, AD, AE, AF, BC, BD, BE, BF, CD, CE, CF, DE, DF, EF Three Subject Matter Expert Non- Repeating Groups: ABC, ABD, ABE, ABF, BCD, BCE, BCF, CDE, CDF, ACD, ACE, ACF, ADE,

ADF, AEF, BDE, BDF, BEF, CEF, DEF Four Subject Matter Expert Non- Repeating Groups: ABCD, ABCE, ABCF, ACDE, ACDF, ADEF, BCDE, BCDF, BDEF, CDEF,

ACEF, ABDE, ABDF, ABEF, BCEF Five Subject Matter Expert Non-Repeating Groups: ABCDE, ABCDF, ABDEF, ACDEF, ABCEF, BCDEF Six Subject Matter Expert Group: ABCDEF

99 APPENDIX B

CTA Coding Scheme

Action = OA Conditions: Indications = OCI Contraindications = OCC Accuracy = OSA Time = OST Overall Reasons for Conducting Procedure = R Reason for Action or Decision step = r Equipment and Materials = EM Declarative Knowledge: Concepts = DC Processes = DPR Principles = DPN Procedural Knowledge: Classification = PDC Modification = PDM Sensory Cues: Hearing = SH Seeing = SS Touching = ST Steps: Action = A Decision step criteria for deciding = DSC (“IF” statements) Decision step alternatives = DSA (“Then” statements) Uncertain = Δ (Delta triangle)

100 APPENDIX C

Six-Expert Gold Standard Central Venous Catheter Placement Procedure Protocol

Cognitive Task Analysis:

Central Venous Catheter Placement

Gold Standard Protocol Procedure Description

Task Analysis

January 2010

Procedure Title: Central Venous Catheter Placement Experts: Surgeons A, B, C, D, E, F Task Analysts: Craig Bartholio, Joon Kim, Eko Canillas Interview Location: USC Keck Medical Center

1. Objective

(Write the overall terminal (end) performance objective of this procedure, using an “action” verb that describes what a resident does.)

• Perform a central venous catheter placement (A14; C8 F2; D10) to provide central venous access (A61; B42; E21) quickly administer drugs (D16), fluids, or bloods to a patient (B72) and to monitor the patient (D16).

2. Reasons

(What is the risk for not performing well? What are the benefits to the trainees if they do a good job?)

The risks of not inserting a central venous catheter properly could be: • Inadvertent carotid injury (IJ site) (A123) • Inadvertent pneumothorax (A132; D24) • Artery is inadvertently cannulated (subclavian site) (A308-309; D25) • Ectopic beats (A358) • Widening of the mediastinum (A420) • A hematoma (A420; D27) • Death (D33) • Damage to vein (D326) • Air embolus (D526) • Sepsis (D32) • Bleeding (D24)

101 • Physical damage to the area itself (D27) • Allergic reaction (D29) • Infection (D32) • Trauma to the vein (D326; D27) • Getting into the femoral artery by mistake (D391) • Guide wire becoming embolized (D545) The benefits of inserting a central venous catheter properly are: • Venous access (C8;F5) to provide fluids and drugs (C8;E24) • Access to central circulation to allow for hemodynamic monitoring (C8) • Deliver blood and nutrition (E24)

Indications: Perform the procedure if the patient exhibits one or more of these indications. .

• Patient requires long-term IV access (i.e. for administering medications and nutrition, etc.) (A45; B80; C25; D28; D42; E61; F6)

• Patient requires large volume resuscitation (A48; B75; E33; F8)

• Patient has inadequate peripheral IV access (A54; B83; C91; D16; F5)

• Patient requires direct access into central circulation system for Hemodynamic monitoring (C96; D16;D44; E53; F8)

Contraindications: Do not perform the procedure if the patient exhibits one or more of these contraindications. List the symptoms or conditions that make the particular treatment or procedure inadvisable.)

• Peripheral access is available for the patient (D49)

• Patient is coagulopathic (A56; B101; C126; E79; F15)

o Standard: First correct coagulopathy before performing CVC placement (E80)

4. Standards

(Are there any time limits, efficiency, or quality standards that must be met when performing this procedure? What are the observable indications of a successful procedure?)

• The time frame for the CVC procedure ranges from 2 minutes (F50) -10 minutes (E86) with an average of 5 minutes (A445; B1101; E86)

• Observable indications of success: Chest x-ray (E92) should show: 1. The catheter is in the superior vena cava. (B1081) 2. Clear lung fields (B1076) 3. Easily draw blood back from the catheter (E89) 4. Easily flush fluid into the catheter (E90)

5. Equipment

(What supplies and materials must be available to perform the task?)

102 • X-ray (A72; B1066; C48;F175; D381; E92)

• Personal protective gear (A91; D60)

o Cap (A91;C387; F309; D174; E97)

o Gown (A91;C387;F317; D174; E97)

o Sterile Gloves (A91;B171;C388;F310; D174; E97)

o Mask (A91;C387; D 174;F309; E97)

o Eye protection (C411; F310; D174; E99)

o Full sheet drape (C770; D182; F310; E100)

o Sterile Prep (includes chlorhexidine) (A92; B172; C383; D180; E101)

• Stethoscope (C719)

• Ultrasound machine (A123; B176; C232;; D112; E264; F408)

• Local anesthesia/one percent lidocaine (A188, A196;F527;D179; E298)

• Central Line Kit (A196;B118;C40;F309; D177; E97)

o Introducer needle (A69; A101; D305)

o Guide wire (A69; D255; E105)

o Central Venous Catheter (A70; D204; E105)

Single lumen central venous catheter-large lumen for large volume fluid or blood administration (A638, A644-A645; E44; D209)

Triple lumen- when large volume is no longer needed, when patient needs multiple medications (A647-A649;C577; D222; E369)

o Syringe (A101; D178; E104; F629)

o Tubing (B1054)

o Finder needle/20 gauge needle (A527)

o Dilator (A656-A657; D537; E105)

• Cardiac monitor (A355; B812)

• Scalpel (A380; F598; E105)

• Sterile saline (A409; F309; D192)

• Sutures (A411; D194; F631; E108)

• Sterile Dressing (A412; B1045; C773; D639; E107)

• Sterile towels, (F309), Extra towels (F309; D157), Large sheet (F310)

• Table to place equipment (F311)

• External lighting (D182)

• Non-scrubbed assistant (D176) (Optional)

103 • Extra kit (D183) (Optional)

6. Task List 1. Inform patient (B499; C38; D155; F120), Prepare Yourself (A90; B251; C39; D60;

E153; F133) & Equipment (B156; C39; D59; E153; F381) 2. Decide where to place the CVC from among three anatomic locations (Internal Jugular,

Subclavian, Femoral Vein) (A92; B254; C34; D49; E131; F82) 3. Prepare Patient (A92; B205;C40; D167; E275; F120) 4. Locate the vessel 5. Insert Introducer needle into the vein (A528; B257; D305; E104; F154) 6. Insert central venous catheter into the vessel (A70; B285; C44; D62; E154; F170) 7. Verify the placement of CVC (A71; B1066; C43; D64; E389; F175) 8. Secure catheter to the skin (A41; B296; C44; D63; E155; F179) 7. Procedure for Central Venous Catheter Placement Task 1: Inform patient (B499; C38; D155; F120), Prepare Yourself (A90; B251;

C39; D60; E153; F133) & Equipment (B156; C39; D59; E153; F381) For both single lumen and multi-lumen catheters Step 1.1: Determine the conscious state of the patient (A92; B205; C441; D286; E208,

E212; F286) IF the patient is awake, THEN inform the patient of the procedure (B499; C38; D155;

F120) AND obtain oral and/or written consent from the patient (D154; F118), AND go to step 1.2

IF the patient is unconscious OR non responsive, THEN go to Step 1.2 Step 1.2: Prepare yourself by donning protective gear (C39), i.e. cap, gown, (A91; D174;

E284; F133) gloves (A91; B171; F317), mask (A91; C39), and eye protection (D Edited Text) using universal precautions (A91; D174; E284)

Step 1.3: Determine the type of catheter to use (B156; C577; D493; E369; F109) IF it is an emergency situation, THEN use whatever kit is available (E167) AND go to

Step 1.4 Otherwise determine whether to use a single or multi-lumen kit (E168) based on the

following: IF: • The patient requires large volumes of blood or medication (F282), OR • The patient is unstable (F286), OR

104 • A Swan Ganz is required (F284) to monitor fluid and hemodynamic

parameters or temporary pacemaker, or for general resuscitation (F287), THEN use a single lumen (B157) AND go to Step 1.4 IF the patient requires: • Administering multiple medications over time (B158; F287), OR • Total Parental Nutrition (TPN) therapy (B160), OR • Antibiotic therapy (B160), OR • The patient is stable (B163) and requires medications that can only be given

through a large bore vein (F287), THEN use a multiple lumen catheter AND go to Step 1.4 IF you are unsure whether to use a multi-lumen catheter OR a single lumen catheter,

THEN choose a single lumen catheter (E185) Standard: A single lumen catheter can accommodate a multi-lumen catheter (D236;

E183) Step 1.4: Gather (D262; F308) and prepare all equipment needed for CVC procedure

(B478; C40; D175; E283; F381) by checking contents and usability (C40; B478; D175; E283; F308)

For single lumen catheter: Step1.4.1: Insert the dilator through the back of the catheter For Multi-lumen catheters: Step 1.4.2: Unlock the brown port For both single lumen and multi-lumen catheters: Step 1.5: Pull scalpel in and out of protective sheath (E293; F393) Step 1.6: Move wire back and forth from roll (E288; F389) Step 1.7: Determine if the Introducer needle will accommodate guide wire (C468), by

passing the wire through the needle. IF wire successfully passes through the Introducer needle, THEN assemble a 10 ml

syringe to the Introducer needle A549; F382), AND go to step 1.9. IF the wire does not pass through the Introducer needle, then start with a new kit (R). Step 1.8: Place a 5 ml syringe on your Finder needle (A528; C464; D288) Step 1.9: Prime catheter line(s) with saline (D526; F395)

105 Step 1.10: Place all equipment near patient or on a table (F383). Task 2: Decide where to place the CVC from among three anatomic locations (Internal

Jugular, Subclavian, Femoral Vein) (A92; B254; C34; D49; E131; F82) For both single lumen and multi-lumen catheters Step 2.1: Decide where to place the central venous catheter (A92, B334; C34; D149;

E131; F82) Select the subclavian location, unless any of the following occurs: IF • You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,

trauma, broken bone, infection, burns) (A57; B110; E76; F30) • You can’t find subclavian vein (A140), OR • It’s impossible to cannulate the subclavian vein (A140), OR • You cannot pass the guide wire during the subclavian procedure (A140), OR • You have a coagulopathic patient and are concerned with subclavian bleeding

(C279), OR • You are concerned about possible pneumothorax (C217), THEN choose internal jugular location unless any of the following occurs: IF • You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,

trauma, broken bone, infection, burns) (A57; B110; E76; F30) The patient’s internal jugular is unavailable (A140; B363), OR The patient has injury to internal jugular vein, OR The patient has thrombosis to the internal jugular vein, OR The patient has a C-Spine collar/injury (D150; F329), THEN, choose the femoral vein location, unless the following: IF • You are not familiar with the anatomy or anatomy is distorted (i.e. swelling,

trauma, broken bone, infection, burns) (A57; B110; E76; F30) • The femoral vein is not available, OR • There is pelvic trauma, THEN, you are unable to obtain central venous access.

106 Step 2.2: Decide between left OR right side of the site selected for patient to begin CVC

procedure (B208; D125; E249) For the subclavian location, the left side is preferred, unless (R): IF • The patient has a right-sided pneumothorax, OR • The patient has a venous thrombosis on the left side, OR • There is an arterial or venous injury to the left side, THEN use the right side. For the internal jugular location, the right side is preferred (R), unless: IF the patient has a left-sided pneumothorax, THEN use the left side. For the femoral vein location, there is no left or right preference, unless: If there is injury or thrombosis on one side, THEN use the other side. Task 3: Prepare Patient for CVC procedure (A92; B205; C40; D167; E275; F120) For both single lumen and multi-lumen catheters Step 3.1: Determine how to position the patient

IF the subclavian location is selected, THEN position patient in a Trendelenburg position (A169; F121), placing a roll of towels underneath the shoulder to open up the area (D157) and have assistant hold down patient arm at side (A169; B399; C394) AND go to Step 3.2

IF the internal jugular location is selected, THEN position patient in a slight Trendelenburg, head turned away from you (B398; D159; E433; F328) with their arms at their side (A169; B397; C394; E197) AND go to Step 3.2

IF the femoral location is selected, THEN position patient in a supine position (A169; B409; E228; F121) with legs rotated out to open up the area (A568; F441), AND go to Step 3.2

Step 3.2: Remove patient’s gown (D161; F315); drape the patient from head to toe (C389; D277; F123) lower guard rails (E270) and position bed at a height at which you are comfortable to perform the procedure (E271).

Step 3.3: Clean area with a sterile prep solution. (A92; B475; D167; E275; F316)

Step 3.4: Administer local anesthesia (A194; B503; C441; D284; F527) unless the patient is unconscious.

107 Procedure for Subclavian Vein Location (A) Task 4A: Locate the subclavian vein using physical landmarks For both single-lumen and multi-lumen catheters Step 4A.1: Palpate the clavicle and identify a point 2/3 distal from the sternal notch

(A245; C515; D281) on the clavicle and place thumb of non-dominate hand just inferior to the clavicle at that position (A245; B1128; C515; D281; F136) into the Delta-pectoral groove (E297), one to two finger-breadth(s) below the clavicle (D281; E297; F136)

Task 5A : Insert Introducer needle into the subclavian vein (A528; B257; D305;

E104; F154) For both single lumen and multi-lumen catheters Step 5A.1: Insert the Introducer needle at the location of your thumb. Standard: Subclavian vein sits deep (A283) underneath the clavicle (D372) therefore

keep the needle in a plane parallel to the floor to avoid inadvertent puncture of the apex of the lung (A248; C519; F476)

Step 5A.2: Advance the Introducer needle, under aspiration (A257; C522; D450; E321;

F160, F480), towards the sternal notch (A257-258; B1128; E316a) until it strikes the inferior surface of the clavicle (A253; C519; E316)

Step 5A.3: Pull back 1-2cm, angle the needle down, and guide the Introducer until you

can feel it pass underneath the clavicle (A254; B1134; C520; E317). Move forward to the subclavian vein (D373) until you see a nice drawback of blood into your syringe (D294) AND go to step 5B.6

IF you don’t get a flash of blood in your syringe, go back to Step 5A.1 (B1141) Step 5A.4: Confirm proper placement of Introducer needle in the subclavian vein by

removing the syringe from the needle (A347; B670; C563; D393; E336; F161) and visually assess presence of blood and blood flow

IF You have continuous and steady (not pulsating) bleeding (A348), OR You have dark slow dripping blood (D396; E337) THEN, assume that the needle is in the subclavian vein (A348; D396; E337)

AND go to step 5A.5 IF You see pulsatile, red blood (D398; E337)

108 THEN assume your needle is in the artery AND pull needle out (E340), apply

pressure (E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 5B AND try again on the same side without re-prepping (D399), OR repeat procedure using the other side (B670; C613)

Step 5A.5: Once the subclavian vein is cannulated, maintain a still (D461) position of

needle by firmly holding the needle at the hub with your non-dominate hand (D459; F536) to keep it in position (A345; B789; D459; F162)

Step 5A.6: Place thumb over top of needle to avoid air being introduced into the vein

(A350; C563; D459) Task 6A: Insert central venous catheter into the vessel. (A70; B285;

C44; D62; E154; F170) For both single lumen and multi-lumen catheters Step 6A.1: Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into

the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one hand on the guide wire (C566; E edited text)

Standard : Easily is a standard Leave a portion of the guide wire showing at the top of the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove

guide wire several centimeters until arrhythmias cease IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN

remove guide wire and Introducer needle and go back to Step 5A.1 Standard : Persisting with a stuck wire may shred the wire and embolize bits of the

wire. (D587) Step 6A.2: Remove the Introducer needle leaving only guide wire in place (A359; B233;

C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602) and sliding the Introducer needle off wire with your other hand (B883) so guide wire does not exit the vein (B885; C571)

Step 6A.3: Make a quarter inch incision with a scalpel where the wire enters the skin Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no

skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the dilator (A382; B899) for passage of the central venous catheter

Step 6A.4: Insert central venous catheter over guide wire into the vessel. (A70; B285;

C44; D62; E154; F170) For single lumen catheters:

109 Step 6A.4.1: Place the dilator through the single lumen catheter (R) Step 6A.4.2: Thread the single lumen catheter with dilator over the guide wire until the

hub contacts the skin (R) Step 6A.4.3: Remove the guide wire from the subclavian vein through the catheter (A70;

B277; D621; E382; F170) Step 6A.4.3.1: Use non-dominant hand to hold catheter and use dominant hand to remove

guide wire and dilator, together in one motion, from the catheter (A401; B979; D621; E382; F170). Go to Task 7A.

For multi-lumen catheters: Step 6A.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire

(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)

Standard: Do not let the wire get embolized (D545) Step 6A.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place

(D538; F603) Step 6A.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire

exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241)

Step 6A.4.7: Remove the guide wire from the subclavian vein through the catheter

(A70; B277; D621; E382; F170) Step 6A.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove

guide wire from the catheter (A401; B979; D621; E382; F170). Go to Task 7A. Task 7A: Verify the placement of CVC in the subclavian vein (A71; B1066; C43;

D64; E389; F175) For both single lumen and multi-lumen catheters Step 7A.1: Verify the correct placement of the central venous catheter by attaching a

syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in position in the vein (A401; B995; D622; E383; F179)

Step 7A.2: Attach a syringe to the port (single-lumen) OR ports (multi-lumen) and

aspirate. (R) IF you get blood flow, THEN proceed to Step 9A.2

110 IF you don’t get blood flow, THEN leave it and go back to Step 5A.1 Step 7A.2: Flush each port with a small amount of sterile saline, one to two cc’s per port

(A409; D622; E383; F178) Step 7A.3: Replace the caps on the catheter (A410; D626; F628) Step 7A.4: Verify the proper placement of the catheter inside the body through a chest x-

ray (A414; B1066; C753; D647; E389; F175). IF the chest x-ray indicates that the catheter is in the superior vena cava (A417; C753;

D647), THEN it has been placed properly AND go to Task 8A. IF the chest x-ray indicates a pneumothorax on the side of the central line (A418), OR a

widening of the mediastinum (A419), OR a hematoma indicating arterial injury (A420), OR indicates extravasations of blood into the right chest (A421), OR you see blood or air (B1086), OR you see the catheter in the wrong place, THEN notify the patient they have experienced a complication and will need further intervention, and notify attending physician.

Task 8A: Affix catheter to the skin (A41; B296; C44; D63; E155; F179) For both single lumen and multi-lumen catheters Step 8A.1: Affix the central venous catheter to the skin (C44) using two interrupted

sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387; F179)

Step 8A.2: Place an antibiotic-coated disk directly on the insertion site (D636) and some

sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound (B1044) Step 8A.3: Remove the drapes (E388) and place patient in a comfortable position (E388). End of Subclavian procedure.

Procedure for the Internal Jugular Location (B) Task 4B: Locate the internal jugular vein (A68; B220; C42; D291; F154) using

ultrasound (R) For both single lumen and multi-lumen catheters Step 4B.1: Place ultrasound probe near the head of the clavicle and move up the neck

(E439) to the apex of the sternocleidomastoid heads triangle (F411), positioning the probe so the internal jugular vein can be distinguished from the carotid artery by size and compressibility of the vein (A514; B1227; C416; D339; E450).

111 Step 4B.2: Once you have found the internal jugular through use of ultra-sound,

verify internal jugular vein by compressing the neck at the site you have located the vein (A517; C643)

IF the compressed vein is larger, oblong, changes with patient’s respiration (A519) AND

compresses (A520; C643) under pressure from the ultrasound probe, THEN it is the internal jugular (A517) AND proceed to Task 5B.

IF the vein appears completely round (A516) and remains patent under compression

(A517), THEN the vein is the carotid artery, Repeat Step 4A.2 until you have located the internal jugular, BUT, if you are unable to locate it after three attempts (B654), ask for assistance.

Task 5B: Insert Introducer needle into the internal jugular vein (A528; B257; D305;

E104; F154) For both single lumen and multi-lumen catheters Step 5B.1: Under ultrasound visualization, (R) advance the needle at an angle 30-60

degrees perpendicular to the skin into the internal jugular vein providing continuous aspiration on the syringe (A532; B561; C522; D293; E321; F161) until there is a flash or flow of blood (B561; D294; E325; F462)

IF you don’t get a flash of blood in your syringe, go back to Step 4B.1 (B1141) Step 5B.2 : Confirm proper placement of Introducer needle in the vein by removing the

syringe from the needle (A347; B670; C563; D393; E336; F161) and visually assess presence of blood and blood flow (R)

IF You have continuous and steady (not pulsating) bleeding (A348), OR You have dark slow dripping blood (D396; E337) THEN, assume that the needle is in the vein (A348; D396; E337) AND go to

step 5B.3 IF You see pulsatile, red blood (D398; E337) THEN assume your needle is in the artery and pull needle out (E340), apply pressure

(E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 4B and try again on the same side without re-prepping (D399), or repeat procedure using the other side (B670; C613)

Step 5B.3: Maintain the position of the needle by firmly holding the hub of the needle

next to the skin to keep it in position (A345; B789; D461; F162) with non-dominant hand (D459; F536)

112 Step 5B.4: Place thumb over end of needle to avoid any air embolus introduced into

the vein (A350; C563; D459) Task 6B: Insert central venous catheter into the internal jugular.

(A70; B285; C44; D62; E154; F170) For both single lumen and multi-lumen catheters Step 6B.1 : Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into

the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one hand on the guide wire (C566; E edited text)

Standard : Easily is a standard Leave a portion of the guide wire showing at the top of the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove

guide wire several centimeters until arrhythmias cease IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN

remove guide wire and Introducer needle and go back to Step 5A.1 Standard : Persisting with a stuck wire may shred the wire and embolize bits of the

wire. (D587) Step 6B.2 : Remove the Introducer needle leaving only guide wire in place (A359;

B233; C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602) and sliding the Introducer needle off wire with your other hand (B883) so guide wire does not exit the vein (B885; C571)

Step 6B.3 : Make a quarter inch incision with a scalpel where the wire enters the skin Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no

skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the dilator (A382; B899) for passage of the central venous catheter

Step 6B.4: Insert central venous catheter over guide wire into the internal jugular. (A70;

B285; C44; D62; E154; F170) For single lumen catheters: Step 6B.4.1: Place the dilator through the single lumen catheter (R) Step 6B.4.2: Thread the single lumen catheter with dilator over the guide wire until the

hub contacts the skin (R). Go to Step 6B.5 Step 6B.4.3: Remove the guide wire from the internal jugular vein through the catheter

(A70; B277; D621; E382; F170)

113 Step 6B4.3.1: Use non-dominant hand to hold catheter and use dominant hand to

remove guide wire and dilator, together in one motion, from the catheter (A401; B979; D621; E382; F170). Go to Task 7B.

For multi-lumen catheters: Step 6B.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire

(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)

Standard: Do not let the wire get embolized (D545) Step 6B.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place

(D538; F603) Step 6B.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire

exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241). Go to Step 6B.5

Step 6B.4.7: Remove the guide wire from the internal jugular vein through the catheter

(A70; B277; D621; E382; F170) Step 6B.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove

guide wire from the catheter (A401; B979; D621; E382; F170) Go to Task 7B. Task 7B: Verify the placement of CVC in the internal jugular vein (A71; B1066;

C43; D64; E389; F175) For both single lumen and multi-lumen catheters Step 7B.1: Verify the correct placement of the central venous catheter by attaching a

syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in position in the vein (A401; B995; D622; E383; F179)

IF you get blood flow, THEN proceed to Step 7B.2 IF you don’t get blood flow, THEN leave it and go back to Step 5B.1 Step 7B.2: Flush each port with a small amount of sterile saline, one to two cc’s per port

(A409; D622; E383; F178) Step 7B.3: Replace the caps on the catheter (A410; D626; F628) Step 7B.4: Verify the proper placement of the catheter inside the body through a chest x-

ray (A414; B1066; C753; D647; E389; F175).

114 IF the chest x-ray indicates that the catheter is in the superior vena cava (A417;

C753; D647), THEN it has been placed properly AND go to Task 8B. IF the chest x-ray indicates a pneumothorax on the side of the central line (A418), OR a

widening of the mediastinum (A419), OR a hematoma indicating arterial injury (A420), OR indicates extravasations of blood into the right chest (A421), OR you see blood or air (B1086), OR you see the catheter in the wrong place, THEN notify the patient they have experienced a complication and will need further intervention, and notify attending physician.

Task 8B: Affix catheter to the skin (A41; B296; C44; D63; E155; F179) For both single lumen and multi-lumen catheters Step 8B.1: Affix the central venous catheter to the skin (C44) using two interrupted

sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387; F179)

Step 8B.2: Place an antibiotic-coated disk directly on the insertion site (D636) and some

sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound (B1044) Step 8B.3: Remove the drapes (E388) and place patient in a comfortable position (E388). End of Internal Jugular procedure. Procedure for the Femoral Vein Location (C) Task 4C: Locate the femoral vein using physical landmarks and

ultrasound Step 4C.1 : Identify anatomic landmarks (D320) using Nerve-Artery-Vein-Empty

Space-Lymphatic’s from lateral to medial (A570) Step 4C.2 : Palpate the groin (E466) and femoral artery to locate femoral vein (A574;

B1157; C646; D331) Standard: Feel for arterial pulse and go medial (about 1 centimeter) (D331). The

femoral artery is just inferior to the inguinal ligament (A574; B1158; E467). Work your way away from the artery to find the insertion point (D331). The vein is one fingerbreadth (C425; E469) medial to this palpable pulsation (A574; B1158; E467). Stay below the inguinal ligament (D324)

Step 4C.3: Place the ultrasound at the location in step 4C.2 to confirm the location of the

femoral vein. Step 4C.4: Once you have found the femoral vein through use of ultra-sound, verify the

location of the vein by compressing the site where you have located the vein (A517; C643)

115 IF the compressed vein is larger, oblong, changes with patient’s respiration (A519)

AND compresses (A520; C643) under pressure from the ultrasound probe, THEN it is the femoral vein (A517) AND proceed to step 4A.5.

IF the vein appears completely round (A516) and remains patent under compression

(A517), THEN the vein is the femoral artery, Repeat Step 4A.2 until you have located the femoral vein, BUT if you are unable to locate it after three attempts (B654), ask for assistance.

Task 5C: Insert Introducer needle into the femoral vein (A528; B257;

D305; E104; F154) For both Single-lumen Introducer and Multi-lumen catheters Step 5C.1 : Using ultrasound visualization (R), insert Introducer needle into the skin at

the insertion point (D284) to cannulize the femoral vein (A581; B224; D291; E451) Step 5C.2 : Use continuous aspiration on the syringe until there is a return of venous

blood (A583; C1141; D294) IF you don’t get a flash of blood in your syringe, go back to Step 4C.4 (R) Step 5C.3 : Confirm proper placement of Introducer needle in the vein by removing the

syringe from the needle (A347; B670; C563; D393; E336; F161) and visually assess presence of blood and blood flow

IF You have continuous and steady (not pulsating) bleeding (A348), OR You have dark slow dripping blood (D396; E337) THEN, assume that the needle is in the vein (A348; D396; E337) AND go to

step 5C.4 IF You see pulsatile, red blood (D398; E337) THEN assume your needle is in the artery and pull needle out (E340), apply pressure

(E342) with a 4 by 4 dressing (C617) for 5 minutes (C618), go back to Task 4C and try again on the same side without re-prepping (D399), or repeat procedure using the other side (B670; C613)

Step 5C.4: Once the femoral vein is cannulated, maintain a still (D461) position of

needle by firmly holding the needle at the hub with your non-dominate hand (D459; F536) to keep it in position (A345; B789; D459; F162)

Step 5C.5 : Place thumb over top of needle to avoid air being introduced into the vein

(A350; C563; D459)

116 Task 6C: Insert central venous catheter into the vessel. (A70;

B285; C44; D62; E154; F170) For both single lumen and multi-lumen catheters Step 6C.1 : Insert the flexible side of guide wire (D558) easily (A354; B744; D672) into

the Introducer needle (A353; B675; C566; D462; E350; F154) while always maintaining one hand on the guide wire (C566; E edited text)

Standard : Easily is a standard Leave a portion of the guide wire showing at the top of the Introducer needle. Do not insert the guide wire completely (A361; B844; D565; E353) or it may cause heart arrhythmias (D558).

IF the guide wire induces arrhythmias in the patient (B816; C550; E361), THEN remove

guide wire several centimeters until arrhythmias cease IF the guide wire doesn’t go through the Introducer needle OR gets stuck (F456), THEN

remove guide wire and Introducer needle and go back to Step 5C.1 Standard : Persisting with a stuck wire may shred the wire and embolize bits of the

wire. (D587) Step 6C.2: Remove the Introducer needle leaving only guide wire in place (A359; B233;

C569; D492; E354; F162) by holding wire in one hand (B882; E Edited text; F602) and sliding the Introducer needle off wire with your other hand (B883) so guide wire does not exit the vein (B885; C571)

Step 6C.3: Make a quarter inch incision with a scalpel where the wire enters the skin Standard: Incision is wide enough (3 millimeters) (A391; D488) ensuring there is no

skin bridge (D596) to allow the connective tissue around the clavicle to be accessed by the dilator (A382; B899) for passage of the central venous catheter

Step 6C.4: Insert central venous catheter over guide wire into the vessel. (A70; B285;

C44; D62; E154; F170) For single lumen catheters: Step 6C.4.1: Place the dilator through the single lumen catheter (R) Step 6C.4.2: Thread the single lumen catheter with dilator over the guide wire until the

hub contacts the skin (R). Step 6C.4.3: Remove the guide wire from the femoral vein through the catheter (A70;

B277; D621; E382; F170) Step 6C.4.3.1: Use non-dominant hand to hold catheter and use dominant hand to remove

guide wire and dilator, together in one motion, from the catheter (A401; B979; D621; E382; F170). Go to Task 7C

117 For multi-lumen catheters: Step 6C.4.4: Hold the wire steady (C582; D535) and lead the dilator over the wire

(A381: B955; C583; D538; E365; F162) and run it back and forth to establish track to pass catheter (B915; C583; D497) and then remove the dilator (A386; E368; F163)

Standard: Do not let the wire get embolized (D545) Step 6C.4.5: Remove the dilator (B386; D539; E368) while holding the wire in place

(D538; F603) Step 6C.4.6: Guide the multi-lumen catheter (F604) (A396) over the wire until the wire

exits the brown port of the catheter (A399; C588; D523; E373). While maintaining control of the wire (C582; D535), advance catheter until the hub is at the skin (A398; B241). Go to Step 6C.5

Step 6C.4.7: Remove the guide wire from the femoral vein through the catheter (A70;

B277; D621; E382; F170) Step 6C.4.7.1: Use non-dominant hand to hold catheter and use dominant hand to remove

guide wire from the catheter (A401; B979; D621; E382; F170). Go to Task 7C Task 7C: Verify the placement of CVC in the femoral vein (A71; B1066; C43; D64;

E389; F175) For both single lumen and multi-lumen catheters Step 7C.1: Verify the correct placement of the central venous catheter by attaching a

syringe on all ports of the catheter and aspirating (A408) to verify that the catheter is in position in the vein (A401; B995; D622; E383; F179)

IF you get blood flow, THEN proceed to Step 7C.2 IF you don’t get blood flow, THEN leave it and go back to Step 5C.1 Step 7C.2: Flush each port with a small amount of sterile saline, one to two cc’s per port

(A409; D622; E383; F178) Step 7C.3: Replace the caps on the catheter (A410; D626; F628) Task 8C: Affix catheter to the skin (A41; B296; C44; D63; E155; F179) For both single lumen and multi-lumen catheters Step 8C.1: Affix the central venous catheter to the skin (C44) using two interrupted

sutures through the skin to the tabs on the hub of the catheter (A410; B1039; D360; E387; F179)

118 Step 8C.2: Place an antibiotic-coated disk directly on the insertion site (D636) and

some sterile ointment with a plastic occlusive dressing (C44; D639; E387) over the wound (B1044)

Step 8C.3: Remove the drapes (E388) and place patient in a comfortable position (E388). End of Femoral Vein procedure.