Journal of Biomedical Informatics 44 (2011) 469–476

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier .com/locate /y jb in

Opportunistic decision making and complexity in emergency care

Amy Franklin a,⇑, Ying Liu a, Zhe Li a, Vickie Nguyen a, Todd R. Johnson a, David Robinson a,b,Nnaemeka Okafor a,b, Brent King a,b, Vimla L. Patel a, Jiajie Zhang a,⇑a Center for Cognitive Informatics and Decision Making, School of Health Information Sciences, The University of Texas Health Science Center at Houston, 7000 Fannin, Suite 600,Houston, TX 77030, USAb Department of Emergency Medicine, School of Medicine, The University of Texas Health Science Center at Houston, Houston, Texas, Houston, TX, USA

a r t i c l e i n f o

Article history:Received 31 March 2010Available online 15 April 2011

Keywords:Opportunistic decision makingOpportunistic planningComplexityEmergency departmentMethodologyTaxonomy of decision typesAcross-task decisions

1532-0464/$ - see front matter � 2011 Elsevier Inc. Adoi:10.1016/j.jbi.2011.04.001

⇑ Corresponding authors.E-mail addresses: Amy.Franklin@uth.tmc.edu (A. Fr

c.edu (J. Zhang).

a b s t r a c t

In critical care environments such as the emergency department (ED), many activities and decisions arenot planned. In this study, we developed a new methodology for systematically studying what are theseunplanned activities and decisions. This methodology expands the traditional naturalistic decision mak-ing (NDM) frameworks by explicitly identifying the role of environmental factors in decision making. Wefocused on decisions made by ED physicians as they transitioned between tasks. Through ethnographicdata collection, we developed a taxonomy of decision types. The empirical data provide important insightto the complexity of the ED environment by highlighting adaptive behavior in this intricate milieu. Ourresults show that half of decisions in the ED we studied are not planned, rather decisions are opportunis-tic decision (34%) or influenced by interruptions or distractions (21%). What impacts these unplanneddecisions have on the quality, safety, and efficiency in the ED environment are important research topicsfor future investigation.

� 2011 Elsevier Inc. All rights reserved.

1. Introduction

Emergency departments (ED) in hospitals are complex environ-ments that are high-workload, both information intensive andinformation poor, time sensitive, highly stressful, non-determinis-tic, interruption-laden, and life-critical. Optimal decision making,in these conditions, is difficult and sometimes impossible. As a re-sult, physicians frequently make their decisions based on heuris-tics they have developed, opportunities that arise, or otherexternal demands on their attention and activities. Effectivelyand efficiently managing cognitive, physical, spatial, and temporalresources in the ED environment is crucial for patient safety andquality of care.

In this paper, we begin with a description of our observationsconcerning the decisions made during task transitions in the ED.Next, we explain our taxonomy of decision types for task transi-tions based on the empirical study in the ED. Finally, we discussthe implications of task transition decision making by addressingthe complexity issues for critical care in the context of patientsafety, with a focus on opportunistic decision making.

The study of medical decision making is an increasingly influen-tial area of research in biomedical informatics. Current theories ofdecision making from classical models of risk and utility to contex-

ll rights reserved.

anklin), Jiajie.Zhang@uth.tm-

tual models of naturalistic decision making (NDM) all emphasizethe inherent factors of uncertainty and complexity in the medicaldecision process [1,2]. Task complexity, including that created byuncertainty and non-linearity, affects the efficiency of decision-making, as more complex tasks require more cognitive effort[3,4]. Decision support systems are one means of aiding and allevi-ating some of the difficulty [5]. However, much of the research ondecision making and decision support systems has focused on thechoices made during the care and treatment of a single patient.That is, these models revolve around a within-task choice in treat-ment or diagnostic reasoning. While this is a rich area for potentialsupport with technology and a point in care with significant riskfor error, physicians, particularly those in critical care environ-ments such as emergency departments (EDs), also make manydecisions across tasks (e.g. the selection of what to do next frommultiple choices after finishing, or being interrupted on one task).These dynamic decisions (e.g., a series of decisions in real time [6])during task transitions span a range of sub-goals, such as mainte-nance of situational awareness of the ED, concurrent care of multi-ple patients, and the training of medical students.

Our approach to decision making during task transitions isbased on the distributed cognition perspective, which considersthe ED as a distributed system composed of individuals and tech-nology situated on complex physical, social, organizational envi-ronment extended across space and time [7–9]. This first studyin our research program begins with the perspective of a singleindividual, the attending physician. We focus on how the environ-

470 A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476

mental factors including interactions within the network of careproviders and the artifacts in the ED affect what decisions aremade. Building off previous research exploring naturalistic deci-sion making from the impact of environmental factors on nursingtriage decisions [10], cognitive schema in evidence based practice[11] and the effect of multitasking and interruptions on workflow[12], we focus here on decision types rather than generalized deci-sion processes creating a broad perspective on critical care activi-ties as well as a unique view of the complexity of an emergencydepartment. Combining our novel method of categorizing physi-cians’ behaviors with a cognitive, ecologically based NDM [2] par-adigm, we aim to create a framework capable of capturing thedecisions made by the physicians as well as the environmental fac-tors that contribute to these choices. Our goal is to create a taxon-omy that elucidates the types of decisions made by physicians asthey balance situational factors, system-wide decisions, and thecare of individual patients. Such a classification system, highlight-ing the variability of the decisions made in complex medical envi-ronments, has important implications for critical care managementand cognitive support for across-task decisions that are not cov-ered by existing models of medical decision making.

2. Background and related studies

Traditional theories of decision making based on uncertainty,risk, and utility have been applied in medical domains [1,13,14].Everything from formal decision analysis using standard decisiontrees [15] to computational methods [16] have been employed torepresent problems and decisions in health care. While the goalof this paper is not to critique traditional versus contextual modelsof decision making (see Patel et al. [17] for such arguments), thedecisions covered by the above theories characteristically focuson the selection of an optimal or rational choice from an array ofpossible choices within a given task. For example, Karni’s utilitymodel [18] proposes that medical decision making ‘‘refers to thechoice of a course of action following a diagnosis of a patient’s con-dition. . . where the objective of both parties [doctor and patient] isto choose the treatment that is best from the viewpoint of the pa-tient’s welfare in all its relevant aspects.’’ Even Markov models de-signed to evaluate and simulate sequential decisions over timetypically focus on specific clinical treatment problems such asthe timing of a liver transplant [19], or the selection of a drug infu-sion plan [20].

Contextual or ecologically based paradigms of NDM are distinctfrom classical, normative models of decision theory in that NDMtheories focus on decision making in complex, natural environ-ments. NDM includes the influence of an individual’s knowledgeand experience on their ability to assess complex (and uncertain)situations and to act on their knowledge in light of environmentalfactors [21]. However, while NDM recognizes that individuals’decisions often deviate from the axioms of expected utility or prob-ability theory [22,23], NDM’s models of pattern recognition, analy-sis, simulation, and evaluation of scenarios and responses are alsooften studied within-task (e.g. anti-air warfare command and con-trol officers [24]; behaviors by fireground commanders [25] , pilots[26] and nurses [27]).

Although studies of NDM in medicine [17,28] have begun toshed light on the importance of viewing decision making as a com-plex, multi-layered situated process [11,17,28], there is a need tofurther develop methodologies that cover the full range of themedical decision making tasks including decisions during tasktransitions.

The awareness of contextual factors is often noted in the NDMliterature (e.g. external stressors), however, few studies considerthe influence of these variables on decision making strategies[29]. Bond and Cooper [30] note the importance of the contextual

factors on NDM in a generalized model, however, these contextualfactors are not integrated within their adaptation of their NDMmodel [31],

Hedberg and Larson [32] explored the environmental elements(that) have to be well considered before knowledge can be reachedabout decision-making in practice and found that interruptionsand work procedures influenced the decision making of expertnurses. This finding is similar to work by Hayes-Roth [33] concern-ing planning behavior. Hayes-Roths’ model predicts that differentsteps of the planning process are strictly ordered: their sequenceis determined partly by the structure of the problem and partlyby which information happens to be available, or which inferenceshappen to be activated, at which times. Although, not focused ondecision making but rather needed work-arounds, Wears and Perry[34] also note the impact of insufficient human-factors engineeringin emergency department workflow. Taken together these studiesprovide evidence that suggests the key to exploring the kinds ofdecisions made in complex medical environments lies in the cir-cumstances under which decisions are made. That is, doctors planand make decisions under the influence of environmental factors,which are often non-deterministic.

Therefore, to find the types of decisions made between tasks,we began by exploring the elements that influence behavior inthe critical care environment of the emergency department. Fromthere, we categorized the antecedents to events and their resultingdecisions. By creating a taxonomy of decision types, we can explorethe prevalence of these choices across physicians and environ-ments (e.g. events during any shift) to consider generalizable pat-terns. We will conclude with a discussion of the implications ofdecision making across the boundaries of individual patient care.

3. Methodology

3.1. Forming a framework to categorize decision types

Naturalistic decision making research involves observingbehaviors in the field and creating detailed descriptive accounts.Building on this methodology, we used a method combining an apriori classification framework with the provision of adding newcategories discovered inductively in the data using Grounded The-ory [35].

Decisions in the ED can be described at many levels of granular-ity. For example, X-raying a patient’s leg can be categorized at anabstract level as patient care, at a basic level of the decision as tothe next step in assessment/diagnosis, or at a fine grain level(sub-ordinate) of the selection of imaging techniques (see Rosch[36], and Smith and Medin [37] for discussions on categorization).Therefore, it is necessary to specify at what level of detail effortsshould be concentrated and analysis should occur. Using the mul-ti-stage iterative method described in the Hybrid Method to Clas-sify Interruptions and Activities (HyMCIA) developed by Brixeyand colleagues [38], we adopted a flexible framework that allowedfor categories to emerge both in data collection and analysis.

3.2. Data acquisition

Data was collected using shadowed observations. In shadowinga physician an observer records the details of the actions and inter-actions of the targeted individual. A convenience sample of fiveattending physicians were shadowed during their scheduled shifts.In the course of data collection, physicians were observed for fourto 8 hours of each observed shift. Seven sessions across the fivephysicians totaling over 40 h of observation provided rich datafor the analysis of workflow processes and decision making. Duringthe shadowing sessions, environmental elements in the ED were

Fig. 1. The stages of data acquisition and processing.

A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476 471

recorded, including the actions and the locations of the activitiesby physicians, the time, the participants involved in the task (e.g.the other parties the physician might be speaking with, caringfor, or interacting with), all observable antecedent events (e.g.being asked to attend to a patient, answer a call, responding toan alarm, etc.), and other ongoing events in the ED (e.g. arrival ofnew patients, consulting physicians from other departmentsappearing in the ED, number of beds filled, etc.). In addition toshadowed observation, our methods included a ‘think aloud’ narra-tion of the physician’s activities [39]. However, given the demandsof the ED, should a physician fail to provide this narrative no at-tempts were made to ask for clarification of the actions observed.To prevent any potential harm or alteration in the functioning ofthe ED, our observers did not interrupt or engage the physicians.Similarly, at a patient’s request, observers waited outside treat-ment rooms limiting data collection for infrequent spans of time.

All participants gave written informed consent prior to observa-tion. Activities were recorded as they occurred and often includedmultiple activities within a single minute. The observers recordedobservations in field note form. The level of granularity capturedby the recording system along with the features of the field noteform was developed over multiple iterations until a standard levelof recording was obtained. In addition to the handwritten notes, anelectronic form (UObserve) was developed on iPhone. This elec-tronic form included aspects of the semi-structure field note formalong with drop down menus capturing activities, participants,locations and free text comment space as well. This electronic fieldnote format was developed and tested in conjunction with thehandwritten notes. Upon further validation and reliability mea-sures, this method may be deployed exclusively.

The observers typically worked in teams of groups of at leasttwo. During the initial stages of project, all of the observers shad-owed a single target physician. In the later stages of the second halfof the study, each observer followed individual targets (i.e. attend-ing physician, resident physician, and trauma nurse.) The groupobservations were used in developing the study protocol and as ameans of training of new observers.

There were four observers for this study. Observer 1 is a mas-ter’s student in informatics as well as a trained MD. Observer 2 isa faculty member in the Center for Cognitive Informatics and Deci-sion Making. She has over 10 years experience in qualitative obser-vational methods but has no training as a health professional.Observer 3 is a graduate student in an informatics program withexperience in human-factors psychology. Observer 4 is a Ph.D. stu-dent in informatics with training in computer science. Observer 4created the iPhone application for the data collection. Each obser-ver received training prior to data collection in the current protocolof observation and recording. During the actual data collection, theobservers recorded their data independently and did not clarifyobservations (with each other or the physician) at that time. Thisindependent data collection that was used to calculate inter-raterreliability for observations completed at a later time.

3.3. Data analysis

Following Grounded Theory, analysis of the data occurred con-currently with data collection and refinement in data collectionprotocols. The stages of data collection and analysis follow.

Shadowing protocol

(1) At the start of each shift, the physicians to be shadowed (andother observed members of the ED personnel) gave informedconsent.

(2) Shadowed physicians were then followed, as permitted,throughout all activities in the trauma ED.

(3) While detailed information was collected on the major activ-ities of the shadowed physician, no recordings were madeconcerning the particular details of patient histories, patienttreatments, and personal conversations.

(4) Each observation session was recorded by hand and laterconverted in an excel file (see Fig. 1).

Initially, the data collected was read and reread to gather asense of the overall shift and gaps in the notes were noted[40,41]. Over multiple-observations, the protocol for data collec-tion became more refined as a standard for data collection wasachieved. Multiple observers worked together comparing initialobservations to determine the contextual factors needed to under-stand ED workflow (e.g. number of people in the waiting room, de-lays in imaging, etc.) post observation. From this comparison, alevel of specification in detail was determined as being optimalfor data collection (i.e. sparse enough level of detail to be capturedby hand recording) and of sufficient quality for post observation re-call (i.e. great enough detail for clarity). For example, a physicianordering a medication was an activity that was recorded, however,the drug, dosage, and directions for administration were not in-cluded in our field notes. Our data collection was centered at a ba-sic level of categorization [42] that subsumed the details of manytasks (e.g. entering passcodes, calling up patient record, enteringdata into fields are all part of charting). Such details includinginformation like the type of drug or its administration may be use-ful for a focused task analysis, however, concerning the decisionsmade across the care and management of an entire ED, a broaderview of decision making is needed to depict more generalizablepatterns in this initial study. In addition to selecting what aspectsof a task were or were not included in our notes, a number of datafields also emerged as central parts of our recording system. Foreach observation, our team members recorded the time, location,the roles’ of participants in activity (e.g. attending physician andresident), known situational factors (e.g. number of patients need-ing triage), activities and the perceived goal of each action.

3.4. Data cleaning

Following field recording, each file was cleaned for consistencyin the presentation of data elements and the data was transferredinto Excel files (with cells for time, location, participants, activities,and antecedent events.), which were compared across observerswith differences reconciled through discussion. For example, timestamps were used to determine the range of events. If multipleevents were coded as a single observation by one observer andas separate occurrences by a second observer, a discussion ensued.Typically, this resulted in the observations being corrected to fitthe multiple-observation format (see Fig. 2).

Fig. 2. Example of coding conflict concerning breaks between task transitions.

472 A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476

Clinical collaborators also provided insight as needed intounderstanding and describing the work processes observed. Asour observations involved task transitions (e.g. shifting betweenpatients, seeking information from others/computer records) lim-ited understanding of medical procedures is required. A smallnumber of data gaps were found (i.e. activities observed by onlyone coder (n = 13 of 285 observations used in final reliabilityassessment)). Once all transcribers agreed on the field notes, thesenotes were then analyzed for decisions types and environmentalfactors. Reliability was achieved, although disagreements in codingwere kept as points for discussion and consideration.

4. Results

4.1. Decision types during task transitions

According to our distributed cognition framework for decisionmaking, physician decisions in the ED are affected by a numberof factors including cognitive (mental processing, cognitive capa-bilities, etc.), clinical (e.g., severity levels, diagnosis, etc.), physical(e.g., physical proximity, ED layout, etc.), organizational (divisionof labor, reporting structure, etc.), and human and equipment (per-sonnel, patients, equipment, etc.) resources. Much of the priorwork on workflow and decision making in the ED has focused oncreating clinical pathways or elucidating naturalistic decisions inpatient treatment. Our observations suggest a need to exploredecision making at a different level. That is, what decisions do phy-sicians make when selecting between pending goals? We analyzedthe data to understand how these factors affect workflow throughdecisions during task transitions.

Based on our previous work [38,43], we began with a list ofcanonical activities physicians complete in the ED. This includescommon tasks such as patient assessment, observation, and com-munication. From our field notes, we analyzed these activities,the overarching goal for which each activity is conducted (e.g. careof patients, student teaching, etc.), the events surrounding eachactivity (e.g. patient arrival, X-rays complete), and the situationalfactors at that moment. Using these methods we determined thatthere are a number of task shifts in which a physician must selectwhat their next action should be. The most obvious of these be-

tween tasks choices is the decision of what to do following the com-pletion of a goal. As the chaos of the ED rarely allows a physician tosee a task (such as caring of a single patient) through from begin-ning to end without a break, the selection of between – task actionsmoves physicians from one goal to the next without a necessarycompletion of task. These shifts between goals are areas of poten-tial error.

In addition to decisions of what task to move onto following thecompletion of a goal, movement between the care of multiple pa-tients is a type of between task decisions. That is in the goal of car-ing for patient 1, the care of patient 2 is a switch in task.

A number of categories of behaviors emerged from our datasuch as a deciding on the next goal, moving between patients,switching between roles (physician as care giver versus physicianas teacher), and coping with environmentally forced breaks in task(interruptions, delays, and other disruptions). All of the aforemen-tioned decisions are considered to be concerned with between taskdecisions (or goal selection). Using these decision spaces, we thenconsider what types of decisions are made in these moments.

Three main types of decisions emerged from our analyses.Physicians made logical progression or planned decisions as they ac-tively selected their next action. Our iterative analysis quickly re-vealed that many activities follow a return to the logicalprogression of patient care after the completion of a different task.For example, although the attending physician may have seenother patients since the initial exam of the patient in Trauma Room1, when the attending physician views the patient’s X-rays and hethen decides to chart the findings, this is considered a plannedchoice (or logical progression) in treating this patient.

A second type of decision that emerged from the data is oppor-tunistic decisions. These decisions are a choice in action createdthrough unanticipated circumstances such as when moving be-tween two tasks. For example, if a doctor stops to check on a pa-tient sitting in the hall although he said in his think aloudprotocol that he was moving to the CT room, we consider this stopalong the way a decision of opportunity.

Similar unexpected choices or breaks in task, our third decisiontype, are sometimes forced upon a physician via an interruption ordisruption. These breaks can be momentary such as the disruptionof a pager going off (followed by a quick return to the previousactivity) or an interruption by a nurse whose need requires an

Fig. 3. Three types of observed decisions during task transitions.

A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476 473

immediate change in task. Brixey et al. [38] and Hedberg et al. [32]have both discussed the impact of interruptions on work flow anddecision making within critical care environments (see Fig. 3).

Beyond identifying decisions types based on the intent of thephysician (to select the next task = planned, respond to a breakin task = disruption or interruption, or take advantage of an unex-pected chance = opportunistic decision), we also must consider therole of the environmental elements or contextual factors that influ-ence decisions. That is, what led to these decisions types.

1 Opportunistic planning is limited to the reordering of priorities or strategies inachieving goals to take advantage of circumstance. Opportunistic decision making is abroader concept extending to can the creation and completion of new tasks outside ofany prior plan of action triggered by a congruence of unforeseen events.

4.2. Details of decision types

Further analysis of the details of our data led to the discoverythat within each broad decision type, additional contextual influ-ences could be identified. Planned decisions, which can followthe clinical pathway of treatment or the logical progression of care,can also be structured by the directions of a superior/colleague(e.g. an agreed upon course of action such as the order of complet-ing hand offs during a shift changed). We therefore broadly defineplanned decisions as those in which an individual selects the nextactivity regardless of the source of impetus. These sources of influ-ence can and do include:

� Planned decisions– Protocol/logical – next step in action series before the com-

pletion of a larger goal (e.g. following assessment there iscreation of a treatment plan).

– External forces – external forces that shape the selection ofactivity. This is typically patient needs/status/priority orsupervisor direction. This type of external force decision onlyoccurs when the next activity is to be selected – otherwisean urgent patient need during an ongoing activity is codedas an interruption.

– Preference – individual selection of next activity when noother outside forces influence the selection of the decision.This is a habitual choice rather than a single decision thatis likely to have some outside influence. (This could be achoice to physically walk around the department when loadis low rather than updating situational awareness usingMedhost as a proxy for walking the department (i.e. a ‘vir-tual’ walk around).

Similarly for breaks in task, the catalyst for the change in activ-ities can derive from a number of sources. Physicians are ofteninterrupted during a task by needs of others including nurses, stu-dents, and patients. On a number of occasions we have also ob-served physicians interrupting themselves. That is, the physician,either by action or through the think aloud protocol, projects oneintended goal, but prior to reaching the necessary location, stopsand redirects himself. Disruptions can also have a number of

sources (devices such as phones, pagers, or alarms) that causemomentary breaks in task. However, unlike an interruption, a dis-ruption does not involve a change in task. For example, in thecourse of taking a phone call a nurse may walk up to the physician.The doctor might hold up her index finger to indicate to the nurseto wait all without a break in speaking. We would consider thenurse’s approach as a disruption (although not an interruption asthe physician’s task is continued to completion). Breaks in taskare defined as seen below.

� Break in task– Interruption – occurs during ongoing activity and requires a

shift in task.

� By organizational design – the physical layout of the

workspace causes a break in work flow. For example, aphysician may intend to check on patient in Trauma 1and begin to walk to this treatment area, however, theirprogress is stymied because of chairs/beds/people block-ing the area causing a shift to a new task.

� By other/artifact – an outside entity causes ongoingactivity to be suspended and activity directed to anothertask (e.g. need to teach a student a procedure, need toprovide information for nurse, etc.)

� Self – an individual, independent of another person, sus-pends an activity to perform another activity (e.g. inter-rupting charting to locate additional information,stopping as you walk down the hall because you thinkof something).

– Disruption By Other/Artifact – outside entity disrupts/over-laps with ongoing activity but this ongoing activity is notsuspended (e.g. receiving a page but continuing with aprocedure).

Finally, opportunistic decisions arise from the conflation of sev-eral unforeseen events. This includes a doctor being in the rightplace to complete an unexpected task, someone having additionalresources available to them (such as unexpected personnel on theshift) or having a bit of free time when blocked from completing atask (e.g. waiting for an image to load.) Like opportunistic planning[33], in which individuals spontaneously take advantage of unfore-seen opportunities to achieve their goals, opportunistic decisionsare choices in action created through unanticipatedcircumstances.1

The three main sources of opportunity are proximity, time, andresources. While the above decision types are mutually exclusive,the antecedent or contextual factors need not be. It is possible tohave the right person, the right time, and the right resources simul-taneously to allow for a decision/activity that otherwise would nothave occurred.

In general, opportunities arise from:

� Proximity – use of physical location in decision-making. Near-ness makes desirable this course of action. Proximity is anopportunistic decision but not all opportunistic decisionsrequire proximity. For example, a doctor can select the nextpatient based on his or her proximity, but locate a piece ofneeded equipment when it is found unexpectedly on the doc-tor’s way to complete a different task.

00.10.20.30.40.50.60.70.80.91

Prop

ortio

n of

Dec

isio

ns b

y Ty

pe

Decisions by Session Observed

03|Break

02|opportunistic

01|Planned

Fig. 5. Proportion of observed decisions by type from each observed sessions acrossmultiple physicians.

474 A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476

� Time – often generated by artifact absence, lulls in work load orduring necessary delays (time during an X-ray). For instance, incaring for a patient, a physician must step away from the bed-side while X-rays are being taken. If the physician uses thesefew minutes to check on the patient in the adjoining bed, thisdecision is considered to be based on an opportunity of time.� Resources – staff, materials, and other resources influencing

decisions. For example, an additional attending physician dur-ing a shift may alter the distribution of demands (e.g. as youare available for that task, I can now do this other task.)

After determining the decision types (at both a general and con-textually specific level), we coded our transcribed field notes byincluding a decision type with each action. The figure below showsthe progression from the raw narrative form to the final data ele-ments including time, location, participants, activities, and decisiontype. A second decision is included to demonstrate the transition.

5. Reliability (see Fig. 4)

Following the creation and use of this coding method, inter-rater reliability was calculated to assess the robustness of our clas-sification system. All codings including disagreements were inde-pendently recorded and a group discussion assessed potentialsystem as well as individual biases. Just as our data collection pro-tocol was developed through iterative analysis, our decision classi-fication system also underwent revision. In its final form, threeindependent researchers reached ‘substantial’ reliability (Cohenkappa = .61) on coding behaviors with these decision types. Forexample, the observed behavior ‘Attending is at PACS (Nurse’s Sta-tion) reviewing CT image’ was coded by all three observers as aplanned decision/logical progression. For comparison all codersagreed that ‘Sees Resident 1 charting nearby, talks with Res1 aboutimage then goes onto discuss second patient’ is an opportunisticdecision of proximity.

After the completion of reliability measures, the remaining datawas coded for decision type and the data was compared acrosssessions.

6. Decision types

The decision types for the seven sessions are displayed below inFig. 5. Rather, that showing the variability within each of the broad

Fig. 4. The annotation and coding system for analyzing across-task decisions in ED.

decision categories, we focus on the higher-level decisions. Themore specific decisions including environmental factors are situa-tion-specific (e.g. the factors vary according to shift, physician, andpatients). (For more discussion on the distribution of these ele-ments across decisions see Liu [44].) Across seven attending physi-cian shifts, clear patterns emerged where on average 45% (sd .14)of the physicians’ decisions were planned, 34% (sd .15) were oppor-tunistic, and 21% (sd .6) were produced by a break in task.

It is only through the inclusion of decision types, rather thandescriptive decision processes that we can see that the choicesmade in the ER are most often (55% opportunistic + break) cre-ated by the environment, rather than by conscious selection ofthe physician. That is contrary to expectation, task transitiondecisions are not in most cases guided by protocol but are in-stead the result of situational factors. Further, as these decisionsare not based on choices in diagnostic reasoning, treatment op-tions or other well established guidelines, this research high-lights the need for new research on cognitive support at thislevel of decision making. For example, information display sys-tems showing the current status of all the patients in the emer-gency room may help physicians to better select the next patientto care for based on patient need (rather than the physiciansmemory of who needed assessment or the patient’s proximityto the physician). Similar clinical dashboards have been devel-oped for patient management in ICU care [45] and broader areasof resource allocation and project management. Our results sug-gest that work on the effect of opportunistic decision making onwork flow is also needed. As opportunistic decisions account forone-third of the activity in the ED, it is important to determine ifthese choices maximize the efficiency of care or increase thecomplexity of an already chaotic environment.

In addition to physician data, two sessions observing the deci-sions of nurses were also analyzed. The nurse data was collectedconcurrently with the above physician data using identical meth-ods. Like the physicians, the nurses had a large number of un-planned decisions, however, the proportion of planned decisionswas higher for the nurses (56% and 63% respectively). Given thissmall quantity of data available and the variability between nurses,it is difficult to compare nurses decisions between tasks to physi-cians. Considering the differences in tasks, responsibilities and de-gree of independence in their work, physicians may respond moreflexibly to their environment as reflected in non-deterministicdecisions. Nurses, on the other hand, may have more direct contactwith patients in procedural (i.e. planned) care. Such issues arebeing explored in current work.

Fig. 6. Adaption of the model of Recognition-Primed decision making.

A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476 475

7. Discussion

Our results indicate the need for refinements to NDM Models.While the generalized model of Bond and Cooper [30] and Klein[23] do not presuppose only within-task decisions, it might be use-ful to integrate environmental influences and across task consider-ations into the models. We propose a refinement of the NDMmodel below for our ED environment. At the highest level of themodel is the maintenance of situational awareness of the ED as awhole. This can include recognition of patterns concerning the careof individual patients or broad influences (e.g. knowing bars closed10 min ago, to the typical flow of ED traffic on rainy days). Our con-sulting physicians emphasized to our research team their desire toquickly create a treatment plan for each new patient in the ED aswell as the maintenance of a plan for the running of the entiredepartment. We would like to suggest that the consideration ofgoals includes a competition or at least awareness of multiple con-current aims (e.g. treatment of patients, supervision of personnel,training of students.) Beyond this step of recognizing the conflict,recursive loops are needed to allow for the simulation of one goal’sactions, modification, and potential implementation and then an-other goal’s simulation steps as part of goal resolution. To ascertainvariations in the mechanisms between models including depen-dence on heuristics or planning would require efforts beyond thescope of this paper. Similarly, future research is needed to explorethe demands of concurrent of conflicting needs on cognitive plan-ning. In the meantime, our current results suggest that cliniciansare guided by their situational awareness. These findings suggestthat the implementation of information displays (i.e. clinical dash-boards) may allow physicians to better manage both immediate/intermediate goals as well as longer term planning through greaterawareness of overall needs (see Fig. 6).

8. Conclusion

Our study has four implications for research on critical carecomplexity, medical decision making, and patient safety. First,our study focused on decision making during task transitions,which has important impact on workflow and ED complexity buthas not received much attention in the traditional research ondecision making. Second, we developed a new methodology for

the study of decision making based on the distributed cognitionframework that considers people and technology as an integratedsystem in complex physical, social, and organizational context.This methodology expanded the Recognition-Primed NDM frame-work by explicitly integrating a set of environment factors intodecision making process and identifying their roles in decisionmaking. Third, our study identified three major types of decisionsduring task transitions and this taxonomy is important in under-standing how physicians make decisions in the ED and for develop-ing decision support tools to improve decision making. Fourth, theempirical data provide important insight to the complexity or thechaotic nature of the ED environment. Our results show that halfof decisions in the ED we studied are not planned, including 34%opportunistic decision making and 21% forced decisions by inter-ruptions or distractions. How these unplanned decisions affectthe quality, safety, and efficiency of care in the ED environmentare important research topics that we are now investigating inour next phase of research.

References

[1] Beach LR, Lipshitz R. Why classical decision theory is an inappropriatestandard for evaluating and aiding most human decision making. In: KleinGA, Oransanu J, Calderwood R, Zsambok CE, editors. Decision making in action:models and methods. Norwood: Ablex Publishing; 1993. p. 21–35.

[2] Klein GA, Orasanu J, Calderwood R, Zsambok CE, Klein GA, Orasanu J, et al.Decision making in action: models and methods. Norwood: Ablex; 1993.

[3] Bystrom K, Jarvelin K. Task complexity affects information seeking and use.Inform Process Manage 1995;31:191–213.

[4] Chinburapa V, Larson LN, Brucks M, Draugalis JL, Bootman JL, Puto CP.Physician prescribing decisions: the effects of situational involvement and taskcomplexity on information acquisition and decision making. Soc Sci Med1993;36:1473–82.

[5] Wood RE. Task complexity: definition of the construct. Organ Behav Hum DecProcess 1986;37:60–82.

[6] Brehmer B. Dynamic decision making: human control of complex systems.Acta Psychol 1992;81:211–41.

[7] Rinkus SM, Walji M, Johnson KA, Malin J, Turley JP, Smith JW, et al. Human-centered design of a distributed knowledge management system. J BiomedInform 2005;38:4–17.

[8] Zhang J, Patel VL, Johnson KA, Malin J, Smith JW. Designing human-centereddistributed information systems. IEEE Intell Syst 2002;17:42–7.

[9] Patel VL, Zhang J, Yoskowitz NA, Green R, Sayan OR. Translational cognition fordecision support in critical care environments: a review. J Biomed Inform2008;41:413–31.

[10] Andersson A, Omberg M, Svelund M. Triage in the emergency department – aqualitative study of the factors which nurses consider when making decisions.Nurs Crit Care 2006;11:136–45.

[11] Falzer P. Cognitive schema and naturalistic decision making in evidence-basedpractices. J Biomed Inform 2004;37:86–98.

[12] Laxmisan A, Hakimzada F, Sayan O, Green R, Zhang J, Patel V. The multitaskingclinician: decision-making and cognitive demand during and after teamhandoffs in emergency care. Int J Med Inform 2007;76:801–11.

[13] Elstein AS, Rovner DR, Holzman GB, Ravitch MM, Rothert ML, Holmes MM.Psychological approaches to medical decision making. Am Behav Sci1982;25:557–84.

[14] Elstein AS. Naturalistic decision making and clinical judgement. J Behav DecisMaking 2001;14:353–84.

[15] Roberts MS. Markov process-based Monte Carlo simulation: a tool formodeling complex disease and its application to the timing of livertransplantation. In: Proceedings of the 24th conference on wintersimulation; 1992. p. 1034–40.

[16] Beck JR, Pauker SG. The Markov process in medical prognosis. Med DecisMaking 1983;3:419–58.

[17] Patel VL, Kaufman DR, Arocha JF. Emerging paradigms of cognition in medicaldecision-making. J Biomed Inform 2002;35:52–75.

[18] Karni E. A theory of medical decision making under uncertainty. J RiskUncertainty 2009;39:1–16.

[19] Alagoz O, Hsu H, Schaefer AJ, Roberts MS. Markov decision processes: a tool forsequential decision making under uncertainty. Med Decis Making2009;30:474–83.

[20] Hu C, Lovejoy WS, Shafer SL. Comparison of some suboptimal control policiesin medical drug therapy. Oper Res 1996;44:696–709.

[21] Klein G. An overview of naturalistic decision making applications. In: ZsambokC, editor. Naturalistic decision making. Mahwah: Lawrence ErlbaumAssociates Publishers; 1997. p. 49–59.

[22] Kahneman D, Slavic P, Tversky A. Judgment under uncertainty: heuristics andbiases. New York: Cambridge University Press; 1982.

[23] Klein GA. Naturalistic decision making. Hum Factors: J Hum Fact Ergon Soc2008;50:456–60.

476 A. Franklin et al. / Journal of Biomedical Informatics 44 (2011) 469–476

[24] Kaempf GL, Klein GA, Thordsen ML, Wolf S. Decision making in complexcommand-and-control environments. Hum Fact: J Hum Fact Ergon Soc1996;38:220–31.

[25] Klein GA, Calderwood R, MacGregor D. Critical decision method for elicitingknowledge. Syst, Man Cybern IEEE Trans 1989;19:462–72.

[26] Orasanu J, Fischer U. Finding decisions in natural environments: the view fromthe cockpit. In: Zsambok CE, Klein GA, editors. Naturalistic decisionmaking. Mahwah: Lawerence Erlbaum; 1997.

[27] Randel JM, Pugh HL, Reed SK. Methods for analyzing cognitive skills for atechnical task. Int J Hum Comput Stud 1996;45:579–97.

[28] Chisholm C, Collison E, Nelson D, et al. Emergency department workplaceinterruptions: are emergency physicians ‘‘interrupt-driven’’ and‘‘multitaking’’? Acad Emerg Med 2000;7:1239–43.

[29] Zsambok CE. Naturalistic decision making: where are we now? In: ZsambokCE, Klein GA, editors. Naturalistic decision making. Mahwah: LawerenceErlbaum; 1997.

[30] Bond S, Cooper S. Modelling emergency decisions: recognition-primeddecision making. The literature in relation to an ophthalmic critical incident.J Clin Nurs 2006;15:1023–32.

[31] Klein GA. A recognition-primed decision (RPD) model of rapid decisionmaking. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE, editors.Decision making in action: models and methods. Norwood: Ablex; 1993. p.138–47.

[32] Hedberg B, Larson US. Environmental elements affecting the decision makingprocess in nursing practice. J Clin Nurs 2004;13:316–24.

[33] Hayes-Roth B, Hayes-Roth F. A cognitive model of planning. Cognitive Sci1979;3:275–310.

[34] Wears R, Perry S. Human factors and ergonomics in the emergencydepartment. Ann Emerg Med 2002;40:206–12.

[35] Glaser B, Strauss A. The discovery of grounded theory. New York: AldinePublishing; 1967.

[36] Rosch E. Principles of categorization. In: Rosch E, Lloyd B, editors. Cognitionand categorization. Hillsdale: Erlbaum; 1978.

[37] Smith E, Medin D. Categories and concepts. Cambridge: Harvard UniversityPress; 1981.

[38] Brixey JJ, Robinson DJ, Johnson CW, Johnson TR, Turley JP, Patel VL, et al.Towards a hybrid method to categorize interruptions and activities inhealthcare. Int J Med Inform 2006;76:812–20.

[39] Newall A, Simon HA. Human problem solving. Englewood Cliffs: Prentice-Hall;1972.

[40] Silverman D. Qualitative research: theory, method and practice. London: SAGEPublications; 1997.

[41] Polit D, Hungler B. Nursing research principles and methods. Philadelphia: J.B.Lippincott Company; 1997.

[42] Rosch EH, Mervis CB, Johnson DM, Bayes Braem P. Basic objects in naturalcategories. Cognitive Psychol 1976;8:382–439.

[43] Zhang J, Patel VL, Johnson TR, Shortliffe EH. A cognitive taxonomy of medicalerrors. J Biomed Inform 2004;37:193–204.

[44] Liu Y. The impact of environmental factors on physician decision making inemergency department. Master’s Thesis, unpublished, University of TexasHSCH; 2010.

[45] Egan M. Clinical dashboards: impact on workflow, care quality, and patientsafety. Crit Care Nurs Quart 2006;29:354–61.