protagonists: henry ford, nikola tesla $...
TRANSCRIPT
11-‐04-‐18
1
Cognitive Task Analysis Carolina Wannheden, Doctoral student Health Informatics Center, LIME
”The expert’s bill”
Protagonists: Henry Ford, Nikola Tesla
$ 10,000
Marking wall: $ 1 Knowing where to mark: $ 9,999
2011-04-18 [email protected]
11-‐04-‐18
2
Discuss in pairs…
1. What is it that Tesla knows, and how does he know it?
2. How can we grasp this knowledge?
3. Why would we want to grasp this knowledge?
2011-04-18 [email protected]
Learning Outcomes
You should be able to
Explain when and why to perform a Cognitive Task Analysis Explain what distinguishes an expert and when knowledge-based
systems may be useful Explain different knowledge structures Apply the Critical Decision Method for the aquisition of expert
knowledge to inform the design of a CDSS
2011-04-18 [email protected]
11-‐04-‐18
3
The purpose of CTA…
…is to understand how people think and what they know (cognitive) to achieve some particular goal (task)…
…in order to Analyze incidents Develop training material Develop memory aids, decision aids, expert systems
2011-04-18 [email protected]
When to use knowledge-based systems
When real experts are
Scarce Expensive Inconsistent Unavailable on a routine basis
2011-04-18 [email protected]
11-‐04-‐18
4
Case from week 1
2011-04-18 [email protected]
Read the case and reflect on…
1. What makes the anesthesiologist’s decision challenging?
2. Why does the anesthesiologist choose the general anesthesia?
3. How would you describe the context? Which factors affect her decision-making?
4. What type of knowledge does the anesthesiologist need in order to make the right decision? Can you distinguish different types of knowledge?
2011-04-18 [email protected]
11-‐04-‐18
5
Experts know how, not just what!
Conceptual/factual knowledge (what) General anesthesia is for… Regional anesthesia is for…
Procedural knowledge (how) IF X, then provide the appropriate anesthesia and operate.
(Structural knowledge Guideline A is applicable for patients aged 15 or more)
2011-04-18 [email protected]
2011-04-18 [email protected]
11-‐04-‐18
6
Key aspects of CTA
1. Knowledge acquisition (KA) 2. Data analysis 3. Knowledge representation
2011-04-18 [email protected]
What CTA tries to capture
What people are thinking about What they are paying attention to The strategies they are using in making decisions What they are trying to accomplish What information they discard What they know about the way a process works
(Crandall, Klein, Hoffman, 2006)
2011-04-18 [email protected]
11-‐04-‐18
7
Techniques for knowledge acquisition
Interviews + Require a minimum level of resources + Can be performed in a relatively short time frame + Can yield a significant amount of qualitative knowledge - Lack of quantitative data - Bias due to selection of questions by researcher - Elicited knowledge may not correspond to what expert actually
does Think-aloud protocols Observations
Ethnographic evaluations to collect information in context Group techniques (e.g. brainstorming)
2011-04-18 [email protected]
Data analysis methods
Protocol and discourse analysis Elicit knowledge from individuals while they are engaged in
problem-solving or reasoning tasks Determine conceptual entities and relationships between them
Concept mapping Node-link structures of knowledge Concept maps can support the formation of consensus among
experts Verification and validation
Verification: fulfillment of perceived requirements (to define design) Validation: fulfillment of realized requirements (upon
implementation)
2011-04-18 [email protected]
11-‐04-‐18
8
Knowledge Representation
Narrative formats Chronologies Data organizers Process diagrams Concept maps
(Crandall, Klein, Hoffman, 2006)
2011-04-18 [email protected]
Decision Requirements Table
Treatment phase
Decision challenge
Cue/ Information
Strategy or practice
Novice Traps
Giving anesthesia before surgery
Choose adequate type of anesthesia
Age, medical history
2011-04-18 [email protected]
(Crandall, Klein, Hoffman, 2006)
11-‐04-‐18
9
Challenges in acquiring expert knowledge
Complex and resource-intensive Identification and access to domain expert with
Sufficient domain knowledge Interest in participating in knowledge acquisition process Minimal bias
(Greenes, 2006)
2011-04-18 [email protected]
Reasoning biases
Poor estimation of probabilities (Probability bias) Use terms like ”suggests”, ”supports”, ”goes against”, ”often”,
”evokes the possibility” to describes uncertainty
Estimation bias Recency bias (mistaking for frequeny) Anchor judgments on initial estimates Familiarity or stereotypic frequency over objective frequency Overestimate frequency of rare events
(Greenes, 2006)
2011-04-18 [email protected]
11-‐04-‐18
10
Critical Decision Method (CDM)
CDM was created to learn from specific incidents (Hoffman et al., 1998)
Described well in chapter 5, Working Minds: A Practitioner's Guide to Cognitive Task Analysis (Crandall, Klein, Hoffman, 2006)
2011-04-18 [email protected]
The CDM interview
Intensive in-depth interview (duration ~2 hrs) to elicit cognitive functions such as decision making, planning, sensemaking within a specific challenging incident
Conducted by 2 researchers 1. Primary facilitator (and note-taker) 2. Note-taker and time-keeper
Conducted in 4 sweeps (phases) 1. Incident identification 2. Constructing a timeline 3. Deepening 4. ”What if” queries
2011-04-18 [email protected]
11-‐04-‐18
11
Sweep 1: Incident identification
Goal: Try to identify an incident that will contain cognitive components beyond background and routine procedural knowledge Nonroutine, challenging events The participant has to have a role as a ”doer/decision maker” The participant’s decision making should have had a direct impact
on the outcome Critical event, time pressure
Ask the participant to provide a brief account of the story, from beginning to end
2011-04-18 [email protected]
Sweep 2: Constructing a timeline
Goal: Get a clear, refined, and verified overview of the incident structure, identifying key events and segments.
The interviewer diagrams the sequence of events on a timeline Identify critical points/”decision points” Try to note sequence and duration of events, actions, perceptions,
thoughts, decisions
2011-04-18 [email protected]
11-‐04-‐18
12
Sweep 3: Deepening
Goal: Get inside the expert’s head: ”[W]hat did they know, when did they know it, how did they know, and what did they do with what they knew?” (Crandall et al., 2006)
Based on the timeline probe critical points for the participant’s Perceptions Expectations Goals Judgments Confusions, uncertainties, concerns Options Information needed and used
2011-04-18 [email protected]
Sweep 4: ”What if” queries
Goal: Illuminate expert-novice differences and potential vulnerabilities for error in the domain
The interviewer poses hypotheticals about the event What if a novice had been at charge? What if [key feature] had been different? What training might have been an advantage? What knowledge, information, tools/technologies could have
helped?
2011-04-18 [email protected]
11-‐04-‐18
13
Discuss in your groups…
Would the Critical Decision Method be an appropriate method to elicit knowledge for the CDSS you intend to develop?
Would it be a feasible method?
2011-04-18 [email protected]
A final quote
”For now, […], it is the direct interaction among experts, and between experts and knowledge engineers, that will serve a crucial role in assuring the development of high quality and accepted knowledge bases that in turn enable the development and effective use of decision support systems.” (Greenes, 2006)
2011-04-18 [email protected]