Diaries

KSE966/986 Seminar - Fall 2012/Spring 2013 March 29, 2013Uichin Lee

ContentsWhy use diaries in HCI?Challenges with diariesParticipants for a diary studyTypes of Diaries (feedback,

elicitation)Data collection When to record an entry?Analysis of diaries

What is a diary?A diary is a document created by an

individual who maintains regular recordings about events in their life, at the time that those events occur

Diaries are useful for recording information that is fluid and changes over time◦User mood, feelings, such as frustration, anger,

and pain, (mobile) information needs (any types of multiple events occurred within a day)

If recall after-the-fact will be hard, inaccurate, or subject to biases, diaries can be a useful data collection method

What is a diary?Informal diaries:

◦Blogs, status updates using Facebook, or “tweets” on Twitter

◦Entries are made as they occur, in real-time (although there are no stated research purposes)

Diaries have been adopted, from sociology and history, for use in human-computer interaction research

What is a diary?Example: time diary

◦A time diary is when there is a focus on how time is used, and entries need to be made on a regular basis (with a specified dimension of time)

◦Time diaries are useful, because people often remember time inaccurately

Diary is a useful approach as much of research in HCI focuses on understanding how we interact software and hardware artifacts in our daily lives

Why use diaries in HCI?Diaries fill the gap between observation in

naturalistic settings and fixed laboratory settings, and surveys

Users may have different reactions when being observed, and observers may not always understand what is going on

If interested in collecting data that is fluid and changes over time (rather than factual data), surveys can lead to biased data due to biases in recall

Multi-method research is often the best approach◦ as all methods have strengths and weaknesses

Why use diaries in HCI? Diaries allow for more collecting more detailed research

data than surveys Survey has predefined questions w/ little flexibility given

to the respondents ◦ Mostly appropriate if we ask users to recall info that does not

change over time Diaries are good at understanding not only what users

are doing, but why they are doing it Dairies are useful for collecting:

◦ Fluid data: occurring only at a specific time, and changing, e.g., mood, feelings, perception, time, or response

◦ Responses: needing a very short time period between the occurrence of the event and the recording of the event

Time is an important dimension, because asking users to recall after-the-fact how much time they spent or wasted will lead to inaccurate answers

Why use diaries in HCI? Useful if data cannot be accurately collected

as we do in experimental research (measuring human performance), or observational means (e.g., automatic data collection)

Survey/diary: ask users about themselves◦ How did they perceive a certain experience with

the computer or device?◦ How did they feel?◦ How did they respond?◦ How much time did it take them?◦ How did it impact on their mood?◦ When did they use it?

Why use diaries in HCI? Diaries can investigate the use of

technology that exists in multiple stages◦Tech that does not exist but could

(where researchers investigate communication or info usage patterns, separate from the tech)

◦Tech that exists but needs to be improved (how people use existing technology)

◦Prototypes of new tech that need to be evaluated

Why use diaries in HCI?Diaries are good for recording user-

defined incidents◦When users intended to perform a task,

but decided not to do so◦When users feel that they have learned

something newDiaries are also good at researching

situations where users move around and don’t stay in one place◦Mobile phones, GPS devices, hand-held

tech

Challenges with diariesUsers sometimes are not introspective

and are not even aware of the specifics of what they are doing and therefore may have trouble recording it in a diary entry

Users may not follow through and record (via paper or electronic) a sufficient number of entries

Time recording may still be less accurate for time diaries than for controlled laboratory setting or automated data collection

Challenges with diariesGenerally harder to recruit users

for a diary study than for something less intrusive, like a survey

Since data are both qualitative and quantitative, data analysis may take a long time

Hard to strike a balance between a frequent-enough series of diary entries, and infringement on daily activities (user participation may then trail off)

Participants for a diary studyLike any other type of research, prior

planning and testing are a must to ensure a valid outcome

First step is to determine who will take part in the study (e.g., demographic, education, computer experience)

Try to get a representative group of participants, but it’s more important to have users who can provide useful insight

Potential diarists must understand the purpose, be motivated and have competence in using any required technology for diary entries

Participants for a diary studyDiary study:

◦Must be structured so that it yields useful data, without imposing an unreasonable burden on users e.g., employment, health, relationships with

others◦Should not negatively impact on

employment, health, or relationshipsParticipants:

◦Should be paid for taking part in the diary study

◦Need to be informed of their rights, including to remain anonymous

Types of DiariesFeedback diary: the data recorded in the

diary is itself the purpose of the research (i.e., meaningful to the researcher)◦Users make entries when a certain event or

threshold occurs, or on a stated time basis

Elicitation diary: the users record only basic information about important events occurring in their day (i.e., meaningful to the user; also the researcher)◦These data points are used as prompts for

expansion at a later time

Types of Diaries: Feedback diaryHow often a diary entry is made?

◦ What event, time, or threshold triggers?Examples:

When you feel frustrated with an interface When you completed a certain task

Users can also make a recording at a set time every day (say 9PM), or during a specific time period (say 12-6PM)

Feedback diaries can be structured or unstructured◦ Likert scales, checkboxes, time recording (e.g., every 15

minutes)◦ General reflection (“How are you feeling right now about

your computer?; How did you respond? How did you think the tech could be improved?”)

Types of Diaries: Elicitation diaryData points recorded in an

elicitation diary are quick and simple◦Short snippets of text◦Short audio records◦Short video clips or pictures

Should not interrupt what users were doing

Then, later (in an interview, on a web site, or other format), users expand on their entries, describing in more detail

Elicitation diary (examples)Development of a new handheld document

scanner (Brown et al. 2000)◦ 22 users were asked to take a photo any time they

felt that there was a document they would like to capture

◦ Over 7 days, they made 381 entries◦ Pictures were then used during a series of semi-

structured interviews to prompt users Mobile information seeking (Sohn et al., 2008)

◦ Users were asked to send in a short text message, identifying when they had an info need

◦ This short text entry is then used to remind the users of answering a series of questions on a web site (e.g., Where were you? What were you doing?)

Types of Diaries: Hybrid approachExample: student’s use of

transportation (Carter and Mankoff, 2005)◦For two weeks, users were asked to

call a specific number every time they made a transit decision (location is automatically captured using GPS)

◦During a later interview, the users were presented with the recordings and were prompted to expand their thoughts on specific decisions

Cf) Experience Sampling Capture participants’ in-situ responses to hypothetic

requests◦ Precisely controlled timing◦ Tracking compliance◦ Characteristics of rating behavior (w/ additional data collection)◦ Reduction of human error (in data management)

Example: understanding location disclosure (Consolvo et al., CHI 2005)◦ Phase1: identify a buddy list ◦ Phase 2: participants receive 10 randomly timed

questionnaires everyday for two weeks from 9AM-9PM on weekdays; 10AM-10PM on weekends Questionnaire: Context (Where are you? What are you doing? With

whom are you?) + What the user wants to disclose about location (based on the hypothetical request from a social relation on the buddy list)

◦ Phase 3: participants reflected their experiences in Phase 1&2

Data collectionHow will the diaries be recorded?

◦Paper?◦Electronic?◦Voice?◦Smart phones?

Often now, technology is being used to record diary entries

Crucial to choose the media that will be most natural for the diarists in their everyday life

Two weeks is often an appropriate length of time for a diary study

When to record an entry?Participants should be given clear

guidance on when to perform an entry in the diary◦What activities are of interest?◦What events, incidents, or feelings should

result in a diary entry?◦How often should diary entries be made?◦How detailed should the entries be?

Make sure NOT to pay participants based on the number of diary entries◦Any payment should be for regular

participation but should not be linked directly to # entries

When to record an entry?If diary reports are turned in during

the study period, monitor the incoming reports (e.g., who’s not reporting or not providing useful data)

Reminders and feedback can be sent during the period of the diary study◦e.g., 5 text messages were sent a day to

remind users to send in diary entries (Sohn et al. 2005)

Analysis of diariesTransfer all records to an easy-to-

analyze electronic formatDo statistical analysis on

quantitative dataPrepare and examine qualitative

data, potentially do a content analysis

With qualitative data, you can contact the participants after the fact to see if your interpretations are correct

SummaryWhy use diaries in HCI?Challenges with diariesParticipants for a diary studyTypes of Diaries (feedback,

elicitation)Data collection When to record an entry?Analysis of diaries