Contextual Inquiry in HCI: Lessons from Aeronautics

Sidney W. A. Dekker1 and James M. Nyce2

1 Department of Mechanical Engineering, IKP, Linköping Institute of TechnologySE - 581 83 Linköping, Sweden, sidde@ikp.liu.se

2 School of Library and Information Management, Emporia State UniversityEmporia, KS 66801, USA, jnyce@rocketmail.com

AbstractHuman-centered HCI systems can result if devel-opers pay heed to the orientations, expectations, andunderstanding of the (end-)users. Contextual inquiryhas emerged as one way since it can reveal what(computer ized) work means to the practitioners whodo it, but it needs to make the jump from the descrip-tion and analysis of current working practice to a de-sign language targeted at the future. In this paperwe use three examples from studies into the use offlight strips in air traffic control for their ability tomake this jump, extracting the lessons we still needto learn if we want to employ contextual inquiry as atool in creating HCI sys tems.

Contextual Inquiry in HCI

Human-centered HCI systems may result when at leastequal weighting is given to the orientations, expecta-tions, and understandings of the (end-)users as to thoseof the software engineers or developers. This is not un-problematic. Quite apart from the normal design pres-sures that seem to perpetually turn user-centered inten-tions into technology-centered systems (Anderson,1994; Forsythe, 1999), the question of what the orienta-tions, expectations and understandings of the (end-)user actually are is non-trivial. Involving (end-) usersin the design process is not the panacea (Giddens,1991; Singer & Dekker, 2000), nor is an overdose ofverification and validation (Woods & Dekker, 2001),nor is "getting human factors in early" in the designprocess. "Newell's catch" describes the basic paradoxof designing human-machine systems: The designermakes predictions about the impact of new technologyon an operating world. Testing this prediction ulti-mately requires fielding the system, but this means thatthe system must be realized at a variety of levels (spe-cific interfaces, software, training for users, etc.). Bythis time so much commitment and cost (psychologi-cal, organizational, political, financial) has becomeinvolved, that the potential for changing the designgiven the information and feedback is minimized. Dis-

Copyright © 2002, American Association for ArtificialIntelligence (www.aaai.org). All rights reserved.

satisfied with the inability to resolve Newell's catch,more HCI work has become "front-loaded". It focuseson understanding the nature of practice before begin-ning to develop, let alone fielding, new technology(Norros & Klemola, 1999; Hollan et al., 2000).

With the promise of privileged access to an under-standing of what activities mean to the people who dothem, designers have sought enlightenment in contex-tual enquiry (Beyer & Holtzblatt, 1998) and ethnogra-phy as a method to do it (Nyce & Lowgren, 1995;Harper, 2000). Contextual inquiry is finding out aboutpeople's work where they are doing that work, whilethey are doing it, and finding out what doing that workmeans to them. HCI design, by extension, is not somuch about the building of artefacts or systems, butabout designing new ways to work. The role of contextin shaping people's practices, and people's perceptionof their practices, is deemed crucial. This is in linewith the growing ecological commitment of many inhuman factors engineering: to them human perfor-mance problems are not about hypothesized internalmental structures that impose constraints on the proc-essing of information from the world, but rather aboutseeing how features of the engineered world imposeconstraints and affordances on goal-directed behavior(Vicente, 1999).

Ethnography Versus DesignMany, especially in CSCW (Computer Supported Co-operative Work), see contextual inquiry as the ultimateroad to human-centered HCI systems (Harper, 2000).After all, if you tailor the system to how users them-selves see their work and their world, you cannot get itwrong. It has to be the summum of human-centeredness. But designers are not ethnographers—thekind of professionals equipped par excellence to con-duct contextual inquiry. In fact, most designers don'teven want to be ethnographers and cannot be botheredto learn all that it takes, especially the analytic partthat makes for 'strong' ethnography (Anderson, 1994;Forsythe, 1999). At the same time ethnographers havetried, but almost equally often failed, to provide mean-ingful input into designers' choices. This is mostly be-cause ethnographers cannot make the jump from thedescription and analysis of current working practice toa design language targeted at the future (Woods et al.,

HCI-Aero 2002 161

From: HCI-02 Proceedings. Copyright © 2002, AAAI (www.aaai.org). All rights reserved.

1996). We could call these "designable fu-tures"—descriptions of (future) cognitive work that al-low designers to tailor for that kind of work proactively.Or, for their part, ethnographers cannot be bothered tomake the jump to designable futures because it is notin their professional doctrine (Plowman et al., 1995).Harper (2000) in fact denies any role for ethnographyin requirements capture.

The deeper issue, however, is this: those involvedin design and development (including ethnographers)are frequently tempted to equate that which informantsdo or tell them with what ethnography is and what itcan tell them. Confounding informant understandingwith ethnographic analysis has implications for thecredibility and contribution of ethnography in creatingHCI systems. Of course, informants can claim privi-leged access to their operating world: what they knowand can tell us always has to be right in a sense, oth-erwise they could not carry out their work. Yet this isnot the same as ethnography, nor is it the same asguiding HCI (re-) design. Tying ethnography to futuredesign hinges on understanding the features and objectsof work—not cast in the unpacked language of an in-formant, but by extracting, revising and verifying thecategories into which informants parse their world. Italso hinges on finding ways to "build out" this revised,nuanced understanding into a designable future. Strongethnography is more than record and replay. Analysisrequires the ethnographer to move back and forth be-tween informant understanding (native categories) andanalytic senses of the work that designers can begin toread as human-centered requirements for the (future)system.

Three Contrasting Studies

Below we contrast three different studies aimed at cre-ating HCI systems in Air Traffic Control (ATC). Thestudies were all concerned with the so-called "flightprogress strip", its role in controlling practice, andwhether new ATC systems could do without it (or,rather, could handle some computerized version). Allwere contextual inquiries to a greater or lesser extentin the sense that they sought to document the role ofthe flight progress strip where and while controllerswere working with them (or without them) and whatrole the strips had for the controllers using the artefactsin actual practice. Examining this tiny paper artefact,the three studies form a small part of a body of workthat is positively huge (see Mackay, 2000) and spec-tacularly inconclusive. There is no consensus onwhether controllers can safely or effectively controltraffic without strips (an acute question as various airtraffic control systems are due to be replaced or up-dated with stripless versions). The three studies are rep-resentative of this body of work, and instructive withrespect to HCI systems not because they converge on averdict about the fate of the paper strip as opposed to

computerized variants (for they do quite the contrary).They present lessons on how to unconfound informantunderstanding from ethnographic analysis; on how touse contextual inquiry in the present for the design ofthe future.

Study ISo what are strips for, really? There is no one answer,and any answer would be so locked into a descriptionof how the controllers themselves see their strips, givethem meaning and employ them across operationalsituations, that it would take an ethnography by itselfto get there. This did not deter (Albright et al., 1996).Pulling out an impressive arsenal of human factorsmethods including direct observation, communicationmonitoring, biographical questionnaires, time and mo-tion measurements, system performance evaluation andsubjective workload assessments, the researchers setout to study how controllers managed traffic withoutflight strips. The researchers' entry assumption was thatflight strips support "information retrieval", and thatcontrollers will have to "compensate" for their removalby retrieving information about flights from othersources such as the radar screen (p. 1). The research-ers, however, never left an analytic trace for others tocritique of follow. The research does not reveal how itcame to "information retrieval" as central category inthe first place. Throughout the study, however, thestarting category of "information retrieval" was neverchallenged, or revised, and only superficially un-packed. In fact, results derived from stripless controlstudies were cast in terms of controller information re-trieval too, for example the number of times a control-ler now "requested information from the pilot" (Albrightet al., 1996, p. 5). Results, cast in such terms, showedthat controllers could retrieve information elsewherewithout negative consequences for their perceivedworkload or ability to manage the traffic. Replacingflight strips with more automated systems, in conclu-sion, would be feasible. Although a contextual inquiryin the narrow methodological sense, the entry categoryimposed by the researchers throughout, pre-ordainedthe nature of the things they saw, the data they gath-ered and analyzed. The whole endeavor was profoundlyetic (looking in from the outside) in perspective. Bynever challenging or revising the "information re-trieval" category, and by mistaking their own reality forthe controllers', the research produced foregone conclu-sions and interpretations. The flight strip is merely oneartifact that supports information retrieval, and an inef-ficient one at that. It can be substituted by other arti-facts, such as the radar screen, and its lack can becompensated for by engaging in other activities, suchas asking other participants in the system.

The message from such work is that designers cansubstitute computer-based artifacts for paperbasedones, they can replace human work with machine workwithout any further consequences to the larger human-

162 HCI-Aero 2002

machine ensemble or human expertise required tomake it all work. This is the substitution myth (Holl-nagel, 1999). The idea is that computerization trans-forms the tools people work with and forces them toadapt their practice ("compensate"). In this sense, de-signers only need to cater for the controller activity "in-formation retrieval" in new ways. Reality, however, hasmore often been inverted: computerization fundamen-tally transforms human work, and people in turn adaptthe new artifacts so that they fit the demands of actualpractice. Indeed, others who studied flight strips (e.g.Mackay, 2000) have seen controllers not only read(and retrieve) what is on flight strips. They saw control-lers write on the strips, mark them up with all kinds ofsymbols in different colors, pass them along to eachother, tap them or point at them, cross data out onthem, talk about them while holding them, fondlethem, stack them, sort them, separate them, displacethem, categorize them, shove them back and forth,look at them on the neighbor's console, and more. Re-moving the opportunity to do all this could disrupt tra-ditional controlling strategies in ways not anticipatedand challenge practitioners to develop profoundly newways of working the traffic. On the other hand, ofcourse, all could be activities associated with retriev-ing (in some way) information (of some kind). Butfrom study I, we would never know that. Information re-trieval was taken as inherently meaningful category,never unpacked or more closely specified, aroundwhich they entire study and future system would bebuilt.

Study IIHailed as the mother of contextual flight progress stripstudies (Harper, 2000), the Lancaster University pro-ject spent many manmonths on the ethnographic obser-vation and analysis of air traffic controller practice,with a particular focus on the use of the flight strip(Hughes et al., 1993). The grand conclusion of this"celebrated" project (see Harper, 2000) was that strips" are the means by which controllers see and note whatis happening, what they have already done, what needsto be done. They are an essential feature of 'getting thepicture', 'organising the traffic', which is the means ofachieving the orderliness of traffic. The strips and theirorganisation are a proxy orderliness of the configurationof the traffic flow" (Hughes et al., 1993, p. 133).

Strips help controllers 'get the picture'. Such moth-erhood insight should not have taken longer to arrive atthan half an afternoon spent observing air traffic con-trollers (see indeed Harper, 2000). But it took months.Further, if strips are the means for a controller to knowwhat is going on, then there is no point in automatingor developing anything new. Such ethnography is Lud-dism, however unintended, in a new cloak—and notunique among ethnographers, (see Mackay, 2000). Itshould be no wonder that designers often think they cando just as well, or better, themselves (Forsythe, 1999).

As if to confirm the point, the developers, having lostpatience with the ethnographers, presented a set ofguiding questions for the ethnographers to answer, sothey (the developers) could finally get on with buildingtheir thing. Here they are (Hughes et al., 1993, p. 135):

• "what characteristics of the existing manual sys-tem are unimportant and need not be supported ina computerised system?

• what are important manual activities which neednot be supported in a computerised system becausethe activities are a consequence of the fact that nocomputer support is available?

• what characteristics of the manual system must bereplicated without change in a computerised sys-tem?

• what activities from the manual system may besupported in a way which is different from thatused in the manual system?"

The developers not only completely missed thepoint. They did so in a way that was impeccable andeloquent to a fault; so obvious in its simplicity as to beirrevocable. All the old biases of optimistic engineeredpositivism were there for the ethnographers to see. Thedevelopers can simply substitute computers for pa-per—just tell them which parts you would likeswapped. Conceptually, the questions were no depar-ture from the misguided substitution myth that has gov-erned function allocation for decades. Confronted witha set of questions of such mechanistic developer way-wardness, the ethnographers never quite recovered.They were unable to formulate a meaningful reply inthe remaining pages of their paper and remainingmonths of their project, and the entire ethnographic-cum-design effort fizzled out on the backburner of mu-tual misunderstanding. It migrated outside the fringes oftechno-push development, resulting only this year(2002) in an ATC system that was hugely over budget,immensely over time, and entirely without flight pro-gress strips.

The failure of ethnographers and developers to getalong or even understand each other is not at all uniqueto this study. It did not have its source in the develop-ers' imperative desire to, well, develop the system.Though emic in perspective (looking out from insidethe native), the study presents a particularly naive formof ethnography. Such ethnography does not interrogatethat which is common sense to the native. It takespractitioner categories as canonical and inherentlymeaningful. Of course, informant competence, as ex-pressed in their own words, is strong and valid. Butconfusing it with contextual inquiry or analysis doesnot lead to strong ethnography. It also does not lead todesign guidance.

Indeed, Beyer & Holtzblatt (1998) explicitly pro-hibit the use of native categories in design discussions,lest these characterizations trap developers into theirentry assumptions or lure them into believing that na-tive motherhoods (such as "flight strips help me get the

HCI-Aero 2002 163

mental picture") can actually double as informativeanalytics. Tellingly, the title of the paper on study IIwas "from ethnographic record to system design", notfrom "ethnographic analysis...". Clearly, a mere ethno-graphic recording does not cut the contextual mustard.The jump from ethnographic record to system design istoo large, conceptually and analytically, to be left forthe designers to deal with. As the study shows, design-ers cannot and do not want to deal with it—indeed itshould not be their job. Talking to designers meaning-fully requires the one who does the contextual inquiryto engage strong ethnography, strong particularly withits analysis. Only such higher-order analytical work canlead to designable futures. The question is: how?

Study IIIInformant remarks such as "flight strips help me get themental picture" should serve as the starting point of acontextual inquiry, not as its conclusion. But how canwe move from native category to analytic sense? Therevision of categories is a hallmark of strong ethnogra-phy, and Ross (1995) (Study III here) has the seedlingsof a good example. Surprisingly, it did not do the kindof all-encompassing data gathering of the previous twostudies (a massive array of human factors studies ormany manmonths of ethnographic observation). It wasa simple survey, derided by many ethnographers as aninstrument that imposes the researcher's meaning ondata rather than bringing out the native's (e.g. Hugheset al., 1993). But Ross (1995) is characterized by thestrongest analysis and synthesis of all three. He slowlytreaded through the lowly masses of context-dependentsurvey data, abstracting and categorizing as he wentalong , relying on previous categorizations for help (seeDella Rocco et al., 1990), in fact comprising research-ers from (Albright et al., 1996). Using these, Ross al-lows us to see how we can avoid jumping to conclu-sions in a big unverifiable leap. Instead, we must as-cend from the masses of data, and away from them, insmall, traceable steps, through multiple intermediatelevels of analysis and make it all very explicit andopen. Such analysis and synthesis leaves others a traceto follow and critique. It is a consistent reminder to thehuman factors community (Woods, 1993; Xiao & Vin-cente, 2000) that if we want to responsibly draw con-clusions from our context-specific data (extracted ei-ther from the lab or the field), we have to conductsome sort of epistemological analysis. In going fromlocal, particular behaviors to generalizable universalpatterns of cognitive strategy, we have to be deliber-ate, slow and cautious, and above all, explicit. Thetypical way proposed is to move up from the context-specific details to the concept-dependent generals in aseries of successive steps, each more abstract than theprevious one; each less in domain terms and more incognitive terms than the previous one (Xiao &Vicente, 2000). Many refer to this as the process of in-duction—reasoning from the particular to the general

(e.g. Beyer & Holtzblatt, 1998).Indeed, epistemological analysis, the careful

movement from context to concept, is crucial for mi-grating to designable futures; for making a link be-tween ethnographic analysis and design guidance. Thisis exactly what study III encourages us to do. For ex-ample, context-specific controller activities such as"entering a pilot report; composing a flight planamendment" reveal a cognitive strategy (Della Roccoet al., 1990) at a slightly higher level of analysis: thatof the "transformation or translation of information forentry into the system" which, at an even higher level ofanalysis, could be called "coding", together with othercognitive strategies (Ross, 1995). Part of this coding issymbolic, in that it uses highly condensed code lan-guage (red underlinings, black circles) to mark up theflight strip in ways that are meaningful only to the con-troller him/herself and a few initiated others. Only fromthere can we make the (then no longer so large) jumpto the highest level of abstraction—helping us identifya major cognitive role for the flight strip: the compres-sion of complexity. Unable to keep all the details ofwhat a flight would do stable in the head, the controllercompresses this complexity, or amortizes it, as DavidKirsh would say, by letting one symbol stand for com-plex concepts and interrelationships, some even tempo-ral.

Other high level, concept-dependent roles of theflight strip would be the anticipation of dynamics (whatcomes next) and the support of coordination (e.g. in so-called handovers of flights to other controllers). Notehow the language is no longer cast in that of the con-text-specific details. It is no longer locked into that of apractice intimately connected to the artefact. It is alanguage that developers can begin to look at as a des-ignable future. Complexity and dynamics, as well ascoordination, are critical to what makes air traffic con-trol what it is, including difficult. Whatever developerswant to develop, they will have to take into accountthat controllers use their artefact(s) to help them dealwith complexity, to help them anticipate dynamic fu-tures, and to support their coordination with other con-trollers. This, backed up by its detailed analysis andsynthesis, is a useful contour of how to create a desig-nable future. It is all in sharp contrast to Study I whichmade its jump from context-specifics to one central(and misguided) concept in one large leap, leaving notrace for others to follow or critique, and in contrast toStudy II which got stuck in the context-specifictrenches, parroting the language of the native. Study IIIshows us the paradox in contextual inquiry for HCI:Creating designable futures requires extreme sensitivityto context. Yet it asks us to extract the description ofpeople's work away from the current context that helpsshape it. Otherwise designers will not understand whatwe are trying to say, and will not know what to do next.

164 HCI-Aero 2002

Main Points• Contextual inquiry is seen by many, especially in

CSCW, as one route to human-centered HCI sys-tems. It can reveal what cognitive work means tothe people who are doing it, where they are doingit and while they are doing it. If designers tailor thesystem to how users themselves see their work andtheir world, then this has to bring designers closerto creating human-centered systems.

• Contextual inquiry is difficult. In order to use con-textual inquiry for purposes of requirements cap-ture, we have to make the jump from our descrip-tion of people's current practice to a design lan-guage targeted at the future. We have to untanglewhat informants do and tell us from what ethnogra-phy is and what it can tell us.

• Strong contextual inquiry can make this jump. Todo so, it has to contain an explicit epistemologicalanalysis. Typically this is a process of induction,moving from context-specific particulars to con-cept-dependent cognitive descriptions of perform-ance, through multiple cautious steps. The re-useof native categories must be resisted, as must bethe making of single leaps from context-specificsto cognitive constructs.

• Methodological imperialism is counterproductive.The contrast between the three studies in this pa-per shows that if human performance data gather-ing—however long or however wide—is notbacked up by strong analysis, then much goes towaste and designers become misguided as a result.Contextual inquiry is methodologically non-dogmatic. What matters is not so much how youget the data, but what you do with them.

• "Designable futures", and by extension human-centered HCI systems, can result if we succeed indescribing people's work in universal cognitiveterms that allow designers to start tailoring for thechallenges of that work in a proactive way. De-signers don't build artefacts or systems so much asthey create new ways in which practitioners musthandle the challenges associated with their work.

References

Albright, C. A., Truitt, T. R., Barile, A. B., Vortac,O. U., & Manning, C. A. (1996). How controllerscompensate for the lack of flight progress strips (Fi-nal report DOT/FAA/AM-96/5). National TechnicalInformation Service, Virginia.

Anderson, R. J. (1994). Representations and re-quirements: The value of ethnography in systemdesign. Human-computer interaction, 9, 151-182.

Beyer, H., & Holtzblatt, K. (1998). Contextual de-sign: Defining customer-centered systems. San Di-

ego, CA: Academic Press.

Della Rocco, P. S., Manning, C. A., & Wing, H.(1990). Selection of air traffic controllers for auto-mated systems: Applications from current research(DOT/FAA/AM-90/13). National Technical Infor-mation Service, Virginia.

Forsythe, D. E. (1999). 'It's just a matter of com-mon sense': Ethnography as invisible work. CSCW,8, 127-145.

Harper, R. H. R. (2000). The organisation in eth-nography: A discussion of ethnographic fieldworkprograms in CSCW. CSCW, 9, 239-264.

Hollan, J. Hutchins, E., & Kirsh, D. (2000). Dis-tributed cognition: Toward a new foundation forhuman-computer interaction research. ACM Trans-actions on Computer-Human Interaction, 7(2), 174-196.

Hollnagel, E. (1999). From function allocation tofunction congruence. In S. W. A. Dekker, & E.Hollnagel (Eds.), Coping with computers in thecockpit, pp. 29-54. Aldershot, UK: Ashgate Pub-lishing Co.Hughes, J. A., Randall, D., & Shapiro, D. (1993).From ethnographic record to system design: Someexperiences from the field. CSCW, 1, 123-141.

Giddens, A. (1991). Modernity and self-identity.San Francisco, CA: Stanford University Press.

Mackay, W. E. (2000). Is paper safer? The role ofpaper flight strips in air traffic control. ACM/Transactions on computer-human interactions, 6(4),311-340.

Norros, L., & Klemola, U. M. (1999). Methodo-logical considerations in analysing anaesthetists'habits of action in clinical situations. Ergonomics,24(11), 1521-1530.

Nyce, J. M., and Löwgren, J. (1995). Toward foun-dational analysis in human-computer interaction. InP. J. Thomas (ed.), The social and interactional di-mensions of human-computer interfaces. Cam-bridge, UK: Cambridge University Press.

Plowman, L., Rogers, Y, & Ramage, M. (1995).What are workplace studies for? ECSCW '95. Pro-ceedings of the Fourth European Conference onComputer Supported Cooperative Work, Stock-holm, Sweden, 10-14 September, 1995. Dordrecht,NL: Kluwer, pp. 309-324.

Ross, G. (1995). Flight strip survey report. Can-berra, Australia: TAAATS TOI.

Singer, G., & Dekker, S. W. A. (2000). Pilot per-formance during multiple failures: An empiricalstudy of different warning systems. Journal ofTransportation Human Factors, 2(1), 63-76.

Vicente, K. J. (1999). Cognitive work analysis: To-ward safe, productive and healthy computer-basedwork. Mahwah, NJ: Lawrence Erlbaum Associates.

HCI-Aero 2002 165

Woods, D. D. (1993). Process tracing methods forthe study of cognition outside the experimentalpsychology laboratory. In G. A. Klein, J. Orasanu,R. Calderwood, & C. E. Zsambok (Eds.), Decisionmaking in action: Models and methods, pp. 228-251.Norwood, NJ: Ablex.

Woods, D. D., Patterson, E. S., Corban, J. M.,Watts, J. C. (1996). Bridging the gap between user-centered intentions and actual design practice.Proceedings of the Human Factors and ErgonomicsSociety 40th Annual Meeting, pp. 967-971.

Woods, D. D., & Dekker, S. W. A. (2001). Antici-pating the effects of technology change: A new eraof dynamics for Human Factors. Theoretical Issuesin Ergonomics Science, 1(3), 272-282.

Xiao, Y., & Vicente, K. J. (2000). A framework forepistemological analysis in empirical (laboratoryand field) studies. Human Factors, 42(1), 87-101.

166 HCI-Aero 2002