international diary studies amber light

8/6/2019 International Diary Studies Amber Light

1/9

Diary Studies as a Tool for Efficient Cross-Cultural Design


Robert Gillham

Amberlight Partners

58 Bloomsbury Street

London

WC1B 3QT

UK

[email protected]

Abstract

Lab-based usability testing and location-based ethnography are time consuming

and costly. We introduce diary studies as a useful design alternative for cross-

cultural work. The relative strengths of the methods are assessed and some

lessons from our own work with diary studies are shared.

1 Introduction

There is considerable debate within the field of Human Computer Interaction

(HCI) regarding the relative merits of classic laboratory-style usability studies

versus the use of ethnographic methods for investigating user requirements

within different cultures.

Neither approach is without its drawbacks. Both approaches suffer from theneed to conduct studies in different countries in order to reach members of the

target culture. This normally incurs considerable expense as fieldwork outside

ones own country usually requires hiring local expertise for the purposes of

recruitment, facility hire and translation of materials (Dray & Siegel 2005).

In this paper we look at the advantages and disadvantages of both approaches in

cross-cultural design and go on to discuss an alternative, the diary study, which

combines elements of both.


2/9

R.Gillham

1.1 Usability Studies

Lab-based usability studies have long been the staple methods of investigating

interface issues between cultures. Some of the earliest works which specifically

address interface design in cross-cultural settings are based on the assumption

that usability testing will be employed (e.g. Nielsen 1990, Nielsen 1996).

Usability testing is a lab-based activity with its roots in cognitive sciences. As

such it has certain advantages. It is a highly focussed technique which allows

the facilitator dedicated time with the user interacting with the product in

question. This represents highly efficient use of time to investigate specific

issues with the product interface.

However, usability studies are not without their shortcomings. Most pertinently

for cross-cultural studies, the main arguments against their use is that:

Usability studies seek to benchmark performance against knownheuristics. When the issues are unknown, as they often are when testing

product use in a foreign culture, the heuristics' usefulness is reduced

(Dicks 1994). There must be something to test, whether this be a non-localised version

of the product, or a prototype. User requirements capture should take

place before concrete design begins in order to arrive at a culturally

appropriate interface (Ghosh & Chavan 2004). Usability testing may

demand that certain assumptions about the target culture are built into

the interface before user requirements are, in fact, fully understood.

Usability testing data concerns subjects in a contrived situation (Rieman1993). Nielsen (1994) cautions that the environment created in testing

may be biased towards western modes of behaviour and may distort the

picture of participants from other cultures.

1.2 Ethnographic methods

The major drawbacks identified with usability testing in an international setting

are the assumptions that the heuristics are well-known and that the original

product will require nothing beyond translation and a few tweaks in order to

meet requirements. In fact this is often not the case. Culture affects interactionwith technology at levels significantly deeper than the superficial (Hoft 1996,

Ito & Nakakoji 1996, Mahemoff & Johnston 1998, Yeo 2001). Preliminary

investigation into requirements, to explore unknown cultural issues has been

identified as a necessary precursor to more formal cultural analysis (Gunter &


3/9


Randall 2003). Rse (2005) identifies the need to establish the applicability and

impact of even well-known cultural factors within the context of use of a

specific product.

Ethnography, and methods borrowed from ethnography have become popular

as an alternative means of investigating user requirements within different

cultures. Ethnomethodology allied to more broadly observational methods have

become widespread within the design community as they offer rich, detailed

insights into the context in which people interact with technology (Gunter &

Randall 2003).

There has been significant take-up of ethnography within business since the lateNineties (Masten & Plowman 2003). However, it is questionable whether these

companies are employing ethnographic methods in their strictest sense, or

simply applying those ethnographic techniques that can be cost effectively

applied in the time allowed (Plowman 2003). Ethnography is notoriously labour

intensive, and in its strictest application takes months (or even years) to

complete, creating vast reams of data for analysis. The time and resources to

complete such a study is not a luxury that is often afforded to practitioners

within industry.

Gillham (2004) identifies a need for methods for localisation which meet

certain requirements for localisation. Principally, that the methods need to:

1. be understood and applied by designers who may not havebackgrounds in localisation or cultural research

2. help designers uncover factors which lie beneath the level of merelinguistic translation

Whilst ethnographic and observational methods clearly meet the second

requirement by uncovering the covert cultural factors which usability testing

does not, their application also demands skills and resources which many

designers and their organisations lack, and thus fails the first criteria for

localisation methods.

In the following sections we explore diary studies, an alternative method which

has growing currency in the design world and has been successfully employedin cross-cultural HCI studies.


4/9

R.Gillham

2 Diary Studies

2.1 Diary studies in HCI

The diary study is a method that has its roots in both psychological and

anthropological research. In its simplest form, it consists of a representative

sample of subjects recording information about their daily lives for a given

period. The data captured can then be analysed in a variety of ways depending

upon the nature of the data, although most often this will take some form of

taxonomy of user behaviour (Brown et al, 2000). Diary studies can take many

different forms and have been widely used in such varied domains of

application as medicine, education and architecture, as well as technology use.

Broadly, there are two approaches to diary studies, each seeking to uncover

different types of information.

The psychological diary studies tend to require that participants record the

frequency of events in their daily lives which are of interest to the researcher.

This is highly effective in allowing researchers to understand not merely theevents that one can observe in a given time with the participant, but the

importance and impact of events to the participant due to the frequency with

which they occur in their everyday life. HCI frequency studies have been

conducted on such diverse topics as mobile device use (Palen & Salzman

2002), task switching and interruptions (Czerwinski et al 2004) and even the

impact of different technology media on the frequency of lying (Hancock et al

2004).

The anthropological approach gives subjects a wide remit to record any

information about their day-to-day activities or environment which they feel is

important to them. This has also been called a 'cultural probe', and the subjects

media for recording information may not be constrained to paper, but may

include cameras or other devices in order to record anything which they may

encounter which has salience. Gaver et al (1999) claim this approach can be

used to "undermine the expectations of researcher and participants." Palen &

Salzman (2002) advocate this approach because it 'moves beyond objectives of

counting events to focus on descriptive accounts of activity.' This type of studyhas been used effectively in inclusive design projects where the requirements of

the target user group may be unfamiliar to the designer (Blythe et al 2003).


5/9


2.2 Diary studies in cross-cultural work

Little if nothing has been written about the possible uses of diary studies in a

specifically cross-cultural setting. This is surprising, given the insights that

diary studies can bring to the process of designing for users of another culture.

Diary studies offer cross-cultural researchers several advantages:

Diary studies are ecologically valid, as they can be used to recordtechnology use within context over time (Palen & Salzman 2002).

Context of use is an important and often hidden cultural factor which

must be understood by designers (Mahemoff & Johnston 1998).

The unexpected information which comes out of diary studies can helpchallenge designers assumptions (Gaver et al 1999). This is useful in

localisation where many beliefs about certain cultures can be based on

unhelpful stereotypes or urban myths.

Diary studies are cheap to carry out in comparison to lab or ethnographicstudies. The period the diary covers is constrained. Setup costs consist of

the recording materials, recruitment of participants, incentives and

translation of instructions to participants. Local suppliers can usually be

safely charged with directing subjects, as this is a less demanding taskthan, for example, conducting an ethnographic interview, and can be

easily managed from abroad. Depending on the nature of the study and

the questions the research seeks to answer, the need to visit the target

country may be removed altogether.

It has been suggested that international studies should be carried out inparallel and opposed to serial studies, as this allows for side-by-side

comparison and analysis (Dray & Siegel 2005). Diary studies are highly

replicable and multiple studies can be carried out in parallel with no

appreciable increase in costs. This is not the case with other forms of

parallel study, which require multiple teams of researchers and careful

monitoring for standardisation.

Diary studies can be employed to record any sort of behaviour, whetherusing technology, or simply everyday events which suggest the

requirement for technology. Thus diary studies circumvent the need for

an extant product to test.


6/9

R.Gillham

3 Recommendations for Cross-Cultural Diary Studies

At the time of writing, Amberlight Partners are conducting diary studies invarious locations, the results of which will help assess further the usefulness of

diary studies in an international context. In the meantime, we offer the

following lessons from our own experiences:

Provide participants with digital voice recorders to record their entries.Digital recorders provide the best format for access, sharing and reuse of

the data. There are many affordable models available in most developed

countries. If the study is to be carried out in locations where such

equipment is expensive, consider reusing the units from a different leg of

the research.

Make full receipt of the participation fee contingent on the participantcompleting the diary period and returning the device!

Provide a semi-structured format for recording entries and be prepared toalter the format across locations. Highly structured formats can prevent

users from recording information or events that are unanticipated but still

of interest to the researcher. Structured formats may also beinappropriate in South American countries, where participants often

ignore the correct formats. Equally, unstructured formats can result in

participants failing to record important information. They can also be

inappropriate in countries such as Japan, where users tend to want

guidance on the correct way to record data.

Provide participants with alternative means to record data, such as anemail address. Often participants may not be able to record data verbally

(in the workplace, for example). Users in some countries may be morereticent to be seen speaking to the recorder in public and will wait till

the end of the day to record a summary of the days events.

It is recommended that the diary study is followed up by in-depthinterviews with participants. As mentioned, diary studies are useful in

throwing up unexpected information that is often culturally-specific, and

vital to understanding the context of use. However, this information is

often of a mundane nature to the participant, and they may not elaborate

upon it sufficiently in their diary entry.

Interviews can be carried out via phone, but visits to users in situ arerecommended. These add considerably to cost, but are still important as

a form of discount ethnography in establishing some basic background

about the participants home and work life which informs understanding


7/9


of the context of use. The cost of interviewing is still considerably lower

than the cost of facility-based testing, or ethnographic observation, of a

similar sample.

The correct choice of local vendor will be crucial to the success of thestudy as they will be required to handle recruitment, instructing

participants, and possibly translation and transcription of diary entries.

This is often beyond the usual scope of a local market research-style

agency and care should be taken to ensure that the vendor is willing and

able to provide these services. See Dray & Siegel (2005) for more advice

on selecting vendors for international studies.

4 Conclusion

There is a clear and identified need to establish requirements for different target

cultures prior to beginning localisation work. Lab-based testing of extant

products are not always suitable for this purpose. Ethnographically-informed

observation techniques have been shown to be useful in understanding

requirements of users in different cultures, but are also time-consuming and

expensive, and the skills are hard to learn.

Diary studies show promise as a means of extracting some of the rich

information provided by ethnographic methods, but demanding less costs, time

and skilled resources. This is an important consideration in an industry where

designers acknowledge the need for localisation, but complain of a lack of

knowledge or design support to inform them (Rse 2001, Gillham 2004).

There appears to be little in the way of literature on the subject of diary studies

in cross-cultural interface research, but many of the claims for diary studies inother areas of HCI and design suggest that they meet many of the needs of the

localisation community.

Diary studies are not without their limitations, and these should be explored

further alongside the claims of ethnography and other methods. There is also a

need for case studies and validation of the arguments made for diary studies,

and the authors are conducting ongoing studies which will form the basis of

future work for publication.


8/9

R.Gillham

5 References

Blythe, M., Baxter, G., Wright, P., Cheverst, K., Clarke, K., Dewsbury, G.,Rouncefield, M. & Sommerville, I. (2003) Gathering Requirements for

Inclusive Design, Second BSC HCI Group Workshop on Culture and HCI

Bridging Cultural and Digital Divides, p.65-71.

Brown, B., Sellen A. & O'Hara, K. (2000) A Diary Study of Information

Capture in Working Life, Proceedings of ACM CHI 2000 Conference on

Human Factors in Computing Systems, p.438-445.

Czerwinski, M., Horvitz, E. & Wilhite, S. (2004) A Diary Study of Task

Switching and Interruptions, Proceedings of the ACM HCI 2004

Conference on Human Factors in Computing Systems, p.175-182.Dicks, R.S. (2002) Mis-usability: on the uses and misuses of usability testing.

Proceedings of the 20th annual international conference on Computer

documentation, Toronto, Canada.

Dray, S. & Siegel, D. (2005) "Sunday in Shanghai, Monday in Madrid?!" Key

Issues and Decisions in Planning International User Studies, in Aykin, N.

(Ed) Usability and Internationalization of Information Technology,

Lawrence Erlbaum, New Jersey.

Gaver W., Dunne, A. & Pacenti, E. (1999) Cultural Probes, Interactions, v.6(1), p.21-29.

Ghosh, K. & Chavan, A.L. (2004) Collaborating on ethnography & design

research: center for ethnography & contextual innovation at HFI,

Extended abstracts of the ACM CHI 2004 conference on Human Factors

in Computing, p.1069-1070.

Gillham, R. (2004) Towards a Model of Effective Design Support for

Localisation, Proceedings of the Fifth International Workshop on the

Internationalisation of Products and Systems, p. 103-114.

Gunter, K. & Randall D. (2003) "Is it Cheaper Online?" Cultural Motivations

for Internet Shopping, Second BSC HCI Group Workshop on Culture and

HCI Bridging Cultural and Digital Divides, p.57-64.

Hancock, J., Thom-Santelli, J. & Ritchie, T. (2004) Deception and Design: The

Impact of Communication Technology on Lying Behaviour, Proceedings

of the ACM HCI 2004 Conference on Human Factors in Computing

Systems

Hoft, N. (1996) Developing a Cultural Model. In Nielsen, J. & Del Galdo, E.

(Eds) International User Interfaces. New York, US: Wiley.Ito, M. and Nakakoji, K. (1996) Impact of Culture in User Interface Design. In

Del Galdo, E. and Nielsen, J. (Eds) International User Interfaces. New

York, US: Wiley.


9/9


Mahemoff, M.J. & Johnston, L.J. (1998). Software Internationalisation:

Implications for Requirements Engineering. Fowler, D. and Dawson, L.

(Eds), Proceedings of the Third Australian Workshop on Requirements

Engineering, p.83-90. Deakin University: Geelong.

Masten, D. & Plowman, T. (2003) Digital Ethnography: The next wave in

understanding the next consumer experience, Design Management

Journal, v.14, n.2, p.75-81.

Nielsen, J. (1990) Designing user interfaces for international use, Advances in

Human Factors/Ergonomics Volume 13, Elsevier, Netherlands.

Nielsen (1996) International usability testing. In Del Galdo, E. and Nielsen, J.

(eds) International User Interfaces. New York, US: Wiley.

Palen, L. & Salzman, M. (2002) Voice-Mail Diary Studies for Naturalistic DataCapture under Mobile Conditions, Proceedings of CSCW 2002, p.87-95.

Plowman, T. (2003) Ethnography and critical design practice. In Laurel, B.

(Ed) Design Research, Methods and Perspectives, MIT Press, MA.

Rieman, J. (1993) The Diary Study: A Workplace oriented research tool to

guide laboratory efforts. Proceedings of the ACM INTERCHI 1993

Conference on Human Factors in Computing Systems, p.321-326.

Rse, K. (2005) The Development of Culture-Oriented Human Machine

Systems: Specification, Analysis and Integration of Relevant InterculturalVariables, in Kaplan, M. (Ed) Cultural Ergonomics, Advances in Human

Performance and Cognitive Engineering Research, Volume 4. Elsevier,

Netherlands.

Rse, K. & Zhlke, D. (2001) Culture-oriented design: Developers knowledge

gaps in this area. In Johannsen, G. (Ed), 8th IFAC/IFIP/IFORS/IEA

Symposium on Analysis, Design and Evaluation of Human-Machine

Systems, p.11-16.

Yeo, A.W. (2001) Global-software development lifecycle: An exploratory

study. Proceedings of the ACM CHI 2001 Conference on Human Factorsin Computing, p.104-111.

international diary studies amber light

Documents