Augmenting the Remote Control: Studies in Complex Information

Navigation for Digital TV

Aseel Berglund

ISBN 91-7373-940-5 ISSN 0345-7524 Department of Computer and Information Science Linköpings Universitet SE-581 83 Linköping, Sweden Printed in Sweden by UniTryck Linköping, 2003.

To Shawshan, Akram, and Erik

�

�

�

�

�

�

�

�

�

����������������������������� ������������

Abstract

The transition to digital TV is changing the television set into an entertainmentas well as information supplier device that provides two-way communicationwith the viewer. However, the present remote control device is not appropriatefor navigation through the huge amount of services and information provided bythe future digital TV, presumably also a device for accessing the Internet. Onepossibility for coping with the complex information navigation required by TVviewers is an augmentation of the interaction tools currently available for TV.Two approaches to such an augmentation are investigated in this thesis: linkingpaper-based TV guides to the digital TV and enhancing the remote control unitwith speech interaction.

Augmentation of paper-based TV guides is a futuristic research approachbased on the integration of paper-based TV guides into computation technol-ogy. This solution provides interactive paper-based TV guides that also functionas a remote control for the TV. A prototype system is developed and explorativestudies are conducted to investigate this approach. These studies indicate thebenefits of integrating paper-based TV guides into the TV set. They also il-luminate the potential to provide innovative solutions for home informationsystems. Integrating familiar physical artefacts, such as paper and pen into TVtechnology may provide easy access to information services usually providedby PCs and the Internet. Thus, the same augmentation needed for TV as anentertainment device also opens up new communication channels for providingsociety information to citizens who do not feel comfortable with conventionalcomputers.

The thesis also reports on studies of speech interfaces for TV informationnavigation. Traditional speech interfaces have several common problems, suchas user acceptance and misinterpretation of user input. These problems areinvestigated in empirical and explorative studies with implementation of mock-ups and running research systems. We have found that the pragmatic solutionof augmenting remote control devices by speech is a suitable solution that easesinformation navigation and search.

i

ii

Acknowledgements

I am deeply grateful to many people for their support during this long journey.It is not possible to mention all of them, but I want to take the opportunity toexpress my sincerest gratitude to those who ought to receive special attention.

I would like to express my deepest gratitude to my supervisor Sture Hagglundfor his support, criticism and encouragement. I would also like to thank ArneJonsson and Kevin McGee for their generous contribution of time and expertise.Many thanks to Hans Marmolin for starting up this thesis project.

This research has been supported by the Foundation for Advancement ofKnowledge and Competence (KK-stiftelsen) through the Industry ResearchSchool in Applied IT and Software Engineering at Linkping University withNokia and Santa Anna IT Research Institute. Support was also received fromthe HMI-graduate school and IDA.

Special thanks to Lillemor Wallgren, Birgitta Franzen, Helene Wigert andBritt-Inger Karlsson for their administrative support. I also wish to extend mygratitude to the TUS group for their computer support.

I would like to thank my colleagues for the joint effort on some of the studies;Erik Berglund, Magnus Bang, Pontus Johansson, Anders Larsson, Jonas Lund-berg and Pernilla Qvardfordt. I also wish to thank Lars Ohlsson and SophiaHager for the joint effort on Paper V.

Thanks to my fellow PhD students at the Division of Human-Centered Sys-tems. In particular, I would like to thank David Dinka for always being there,especially when things were down. I also wish to thank Fredrik Arvidsson,Christina Olvingson, and Sonia Sangari, it is always a pleasure talking to you!Thank you Ola Leifler for your Latex support!

My gratitude goes also to my wonderful friends. Special thanks to MariaMinassian for listening and encouraging me. You are always close despite thelong distance!

Then of course, thanks to the LUPA board, the fantastic adept group, andmy supporting as well as encouraging mentor Ewa Ljunge.

Finally and most of all, I would like to thank my wonderful family. My twofantastic sisters Susan and Sally – who are my friends and supporters. Youare the best sisters anyone can wish for. Special thanks to my parents in lawMargareta and Bage. I always look forward to the Sunday dinners. My sweetHoffe – you are the best dog! My adoring parents Shawshan and Akram – thankyou for always being there especially when I needed you, for your enormous

iii

support, and for believing in me! I have been very fortune to have you. Myloving husband Erik – thank you for supporting me when I was in doubt andwhen things were really difficult! Thank you for all your help and scientificsupport! You have generously contributed with your time even in situationswhen you did not have any time. Thank you for sharing my life! You are myeverything.

This thesis is dedicated to Shawshan, Akram, and Erik. I love you.

Aseel BerglundLinkoping in April 2004

iv

Contents

Abstract i

Acknowledgements iii

List of Papers vii

Introduction 11 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . 23 Research Question . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4.1 Brief history of TV interfaces . . . . . . . . . . . . . . . . 44.2 Paper interfaces . . . . . . . . . . . . . . . . . . . . . . . 64.3 Speech interfaces . . . . . . . . . . . . . . . . . . . . . . . 7

5 Research Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 85.1 Data Collection Methods and Techniques . . . . . . . . . 85.2 Methods Applied in the Different Papers . . . . . . . . . . 11

6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Summaries of the Papers . . . . . . . . . . . . . . . . . . . . . . . 14

7.1 Paper I: Speech Enhanced Remote Control for Media Ter-minal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7.2 Paper II: Multimodal Dialogue Systems for InteractiveTV Applications. . . . . . . . . . . . . . . . . . . . . . . . 16

7.3 Paper III: Using Speech and Dialogue Systems for Inter-active TV Navigation . . . . . . . . . . . . . . . . . . . . 16

7.4 Paper IV: Error Resolution Strategies for Interactive Tele-vision Speech Interfaces. . . . . . . . . . . . . . . . . . . . 17

7.5 Paper V: The Paper Remote: An Augmented TV Guideand Remote Control. . . . . . . . . . . . . . . . . . . . . . 17

8 Summary and Concluding Discussion . . . . . . . . . . . . . . . . 18References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Paper I:Speech Enhanced Remote Control for Media Terminal 29

v

Paper IIMultimodal Dialogue Systems for Interactive TV Applications 41

Paper IIIUsing Speech and Dialogue for Interactive TV Navigation 53

Paper IV:Error Resolution Strategies for Interactive Television SpeechInterfaces 83

Paper VThe Paper Remote: An Augmented TV Guide and RemoteControl 97

vi

List of Papers

Papers Included in this Thesis

The following five papers are included in the thesis and will be referred to bytheir Roman numerals. The papers are printed in their original state exceptfor format changes and some minor errata corrections. Papers I and II werepublished before I got married when my sir-name was Ibrahim:

I. Ibrahim A., Lundberg J. and Johansson J (2001) Speech Enhanced RemoteControl for Media Terminal. In Proceedings of Eurospeech’01, Aalborg,Denmark, Volume 4, pp. 2685 – 2688.

II. Ibrahim A. and Johansson P. (2002) Multimodal Dialogue Systems forInteractive TV Applications. In Proceedings of the 4th IEEE InternationalConference on Multimodal Interfaces 2002 (ICMI’02), Pittsburgh, USA,pp. 117 - 122.

III. Berglund A. and Johansson P. (2004) Using Speech and Dialogue for Inter-active TV Navigation. Submitted to the international journal: UniversalAccess in the Information Society (UAIS), Springer-Verlag Heidelberg.

IV. Berglund A. and Qvardfordt P. (2003) Error Resolution Strategies for In-teractive Television Speech Interfaces. In Proceedings of the Ninth IFIPTC13 International Conference on Human-Computer Interaction, INTER-ACT 2003, Zurich, Switzerland, pp. 105 – 112.

V. Berglund A., Berglund E., Larsson A., and Bang M. (2004) The PaperRemote: An Augmented TV Guide and Remote Control. Submitted tothe international journal: Universal Access in the Information Society(UAIS), Springer-Verlag Heidelberg.

Other Publications by the Author

The following published papers are not included in the thesis. However, PaperIII is an extension of Paper VI. Papers VI, VII, and VIII were published beforeI got married when my sir-name was Ibrahim:

vii

VI. Ibrahim A. and Johansson P. (2003) Multimodal Dialogue Systems: A casestudy for Interactive TV In Proceedings of the 7th ERCIM workshop (UserInterfaces for All), 2002 in Paris, France, LNCS 2615, pp. 209 – 218.

VII. Hultcrantz J. and Ibrahim A. (2002) Contextual Workshops: User Partic-ipation in the Evaluation of Future Concepts. In Proceedings of Partic-ipatory Design Conference 2002 (PDC2002), Malmo, Sweden, pp. 344 –348.

VIII. Lundberg J., Ibrahim A., Jonsson D., Lindquist S., and Qvarfordt P.(2002) ”The snatcher Catcher” -an interactive refrigerator. In Proceed-ings of the second Nordic conference on Human-computer interaction(NordiCHI 2002), Aarhus, Denmark, pp. 209 – 212.

viii

Introduction

1 Motivation

The primary multimedia electronic device in the home is the television (TV) setwith its input device, the infrared remote control. During the last two decades,the transition to digital TV has changed the television concept; new services andapplications are provided. There are attempts to integrate the computer andthe TV to provide a home terminal that functions as both an entertainment aswell as an information supplier device (Voutsas and Halverson, 2000). Theseattempts have faced various problems, e.g. economic, technical and usabilitydifficulties.

One of the main problems that faces the home terminal, i.e. the future TV,is the complex information navigation through the remote control device. Thisproblem is caused by the attempts to design the future TV as a computer butwith the same conceptual model of the so-called ’traditional TV’ which com-plicates the interaction. For example, Internet pages are navigated by movinga marker controlled by means of four arrow keys on the remote control. Thus,compared to the ’traditional TV’ which consists mainly of TV channels, futureTV devices will contain very large amounts of information and provide complexfunctions through which viewers can make use of the information and services.As an example, let us consider the user of a common digital TV application;the Electronic Program Guide (EPG), that provides information about TV pro-grams and channels. The user’s goal is to find an interesting movie and watchit. The user has to first search for movies, either by browsing or through somekey-based search functions. Having found a set of satisfying alternatives, theuser must analyze the description of each movie and decide which one is mostsuitable. Throughout these steps, many things can go wrong. For example, theuser may not find what s/he is looking for, and may not succeed in enteringsearch text. Moreover, the steps can take a long time to accomplish and theuser may feel that it is a waste of time. The complexity of these steps is causedby the complexity of the interaction due to the huge amount of informationcontrolled by the traditional interaction device and model; the remote control,which is better suited for the ’traditional TV.’

Most TV development has been technology driven (Eronen, 2003). In gen-eral, home technology providers have sound knowledge of the technology they

1

Figure 1: Digital and interactive TV systems usually consist of a set-top box (STB)connected to the TV set. Generally, two remote control devices are included: one forcontrolling the TV set itself and one for the set-top box.

produce but not on the social context in the use of technology (Venkatesh, 1996).There are some studies focused on understanding household and domestic envi-ronments (Harper, 2003, Hektor, 2001, Lee and Lee, 1995, O’Brien et al., 1999,Morrison and Krugman, 2001, Venkatesh, 1996). These studies provide impli-cations for the future of domestic appliances.

As pointed by many researchers, augmentation of existing artifacts is a strat-egy that can be applied to help users (Mackay, 1998). Help is important foruniversal usability, which is about making information services usable for ev-eryone (Stephanidis and Savidis, 2001). Help for future TV usage is needed bymore and more people because TV is one of the most important devices for thehome and is used by a wide range of people.

In this thesis we argue that augmenting existing tools instead of adding newones is a suitable approach to helping users interact with the future TV. Theexisting tools we focus on in this thesis are paper-based TV guides and theremote control device.

2 Problem Description

This research started at Nokia Home Communications in Sweden which de-veloped set-top boxes (digital box receivers) and media terminals (digital boxreceivers and interactive TV devices) for the home. These devices are elementsin digital and interactive TV systems that provide, among other things, digi-tal TV channels, Internet services, games and communication applications, seeFigure 1. In other words, these devices are the home terminal of the future,they are entertainment and information suppliers for the homes. This researchstarted with the problem that Nokia Home Communications was facing: The fu-

2

ture TV is designed as a computer with the ’traditional TV’ interaction model.This approach causes complex information navigation by TV viewers throughremote control devices that tend to be big, consisting of many buttons, and notso easy to use (Nelson, 1999). Figure 2 shows a few examples of existing remotecontrol devices to illustrate how they can differ in design, shape, functionality,size and ease of use.

One possibility to cope with complex information navigation is to investigateaugmentation of interaction tools available for TV today. In this thesis, twoapproaches to such an augmentation have been investigated: linking paper-based TV guides and the digital TV as well as enhancing the remote controldevice with speech interaction.

TV guides printed on paper are common and familiar to TV viewers, how-ever, they are currently isolated from TV sets. Paper Remote is a concept forlinking such guides with the digital TV. This is a futuristic research approachfocused on adding computation augmentation to paper-based TV guides. ThePaper Remote concept connects paper-based TV guides with the TV set andallows people to access TV content with a digital pen. Thus, this future TVguide functions both as an interactive paper-based guide and as an interactioninterface for TV sets. In this thesis, we have studied the current usage of TVguides (Paper V). We have, then, developed a prototype system and conductedexplorative studies to investigate the benefits of the concept (Paper V). Usingpaper as an interface to technology has already been studied, however, mainlyin work contexts (see Section 4.2).

This thesis has also explored a realistic commercial solution to the com-plex information navigation. The solution is focused on enhancing the exist-ing interaction through speech interaction. The research is focused on designissues from the user’s point of view and not from the technological point ofview. The research is based on user studies of both working research systemsand mock-ups. Previous studies confirm the benefits of speech interfaces ingeneral (Marshall, 1992, Martin, 1989, Visick et al., 1984). However, to ourknowledge little is known about speech interfaces for TV sets. This thesis re-ports on the experiences and lessons learned from designing speech interfacesfor the TV environment that is social and noisy. Unlike the computer environ-ment, the TV is characterized by a big screen with not very high resolution,and placed at a distance in front of the viewer. We have studied the potentialof speech command input (Paper I), the benefits of spoken natural languageand dialogue for TV navigation (Paper II and III), and strategies that can beapplied when the viewer input is misrecognized by the system (Paper IV).

3 Research Question

The simple days of 1-3 channels of TV are gone. Researchers are now facing thedaunting task of developing an easy-to-use and multifaceted interaction systemfor the future TV providing natural interaction for a highly heterogeneous view-ing audience. In addition, the development also includes examining the design

3

Figure 2: Different types of remote controls.

of such a system using the tools available in the TV context today. The overallresearch question addressed by this thesis is:

How can commonly available TV artifacts or elements of the TV view-ing context be developed to provide a desirable interaction design for thecomplex and multifaceted future TV?

The elements of the TV viewing context include: the TV room, the remotecontrol device, the paper-based TV guide, gestures, eye tracking and the VCR.This thesis is limited to two of these areas, namely:

− What are the prospects of integrating familiar physical artifacts, such aspaper and pen, with TV technology?

− How suitable are speech interfaces in general and natural language in par-ticular for TV information navigation?

We have studied how users of future TV may react to the interaction avail-ability through interactive paper-based TV guides and speech. We have con-ducted experimental and explorative studies with implementations of mock-upsand running research systems.

4 Related Work

In this section we will provide a brief review of TV, paper and speech interfaces.

4.1 Brief history of TV interfaces

Television is one of the central entertainment devices in homes(Herigstad and Wichansky, 1998). For example, in the United Kingdom 99% ofhouseholds had TVs in 2000 – 01 (ONS, 2002). During the last two decades,most of the development of TV has been directed towards digitalization aswell as integration with computers (Frenkel, 1989, Zong and Bourbakis, 2001,Seel, 2003). Thus, new concepts have been exploited, such as interactive digital

4

TV (Jacobs and Dransfield, 1998, Frenkel, 1989, Kim, 2001, Lougos et al., 2002,Nelson et al., 1995, Press, 1993). Despite the fact that television is one ofthe central devices in households, relatively few publications address inter-active digital TV in system design literature. The development of domesticdevices in general and TV devices in particular have been driven by techno-logical advances rather than by understanding the interaction between peopleand technology (Taylor and Harper, 2003). A small number of studies havebeen conducted to explore television viewing and the relationships betweentelevision and people’s daily lives (Adams, 2000, Clancey, 1994, Cohen, 2002,Conway and Rubin, 1991, Gauntlett and Hill, 1999, Heeter and Greenberg, 1988,Taylor and Harper, 2003, Lee and Lee, 1995, Logan et al., 1995, Silverstone, 1994,O’Brien et al., 1999, Jensen and Toscan, 1999).

Interactive digital TV is an entertainment device that offers TV channels,Internet services, such as banking, e-mail, and e-shopping and other interactiveapplications, such as games, electronic program guides and pools (Furht, 1996,Dransfeld et al., 2000, Kerschbaumer, 2000). There are attempts to evolve thefuture TV into an enhanced entertainment as well as information supplier de-vice. However, the interaction tools with this device will still be the traditionalremote control device. The degree of interactivity varies from a simple remotecontrol button press to sending information back and forth to service providersby means of a return path (Perera, 2002, Gill and Perera, 2003a). Accordingto Toscan and Jensen, interactive TV is a combination of technologies thatallows viewers greater control over what they watch and when they watch it(Toscan and Jensen, 1999).

Nevertheless, there are three main problems that face interactive digital TV.First, the paradigm for user interaction is drawn from the menu-driven world ofpersonal computers (Ali and Lamont, 2000, Klein et al., 2003, Koller et al., 1997).Therefore, people with limited computer experience, the elderly in particular,are excluded. Second, the interaction design is poor (Freeman and Lessiter, 2003).Therefore even people familiar with new technologies find interactive TV non-intuitive and exhibiting inconsistencies (Gill and Perera, 2003b, Klein et al., 2003).Last, the increased number of TV channels make the navigation and selec-tion difficult, thus channel surfing and information gazing become impractical(Black et al., 1994, Chorianopoulos, 2003, Ehrmantraut et al., 1996) Currently,the main ways of changing channels on interactive digital TVs are:

− Scrolling by using the remote control, which can be time consuming(Ehrmantraut et al., 1996).

− Entering channel numbers from the remote control, which results in mem-ory overload for more than 50+ channels.

− Using an on-screen TV guide application called Electronic Program Guide(EPG) which is usually complex [ (Daly-Jones and Carey, 2000),(Ehrmantraut et al., 1996, Taylor and Harper, 2003)]. EPGs are oper-ated by point-and-click maneuvers from the remote control or wirelesskeyboards.

5

These problems result in complex viewer interaction that can cause frustra-tion and irritation. Therefore, it is essential to deal with these problems. Easeof use is one of the most important factors for interactive TV (Black et al., 1994,Choi et al., 2003, Freeman and Lessiter, 2003, Kang, 2002, Jan and Loes, 2001).Another important factor for adopting interactive TV in the home is enjoyment(Choi et al., 2003). In general, previous research has shown that entertainmentis relatively important as regards technology adoption in the home environment(Venkatesh and Brown, 2001) and TV is one of the most important entertain-ment devices in the home(Herigstad and Wichansky, 1998).

There are many approaches to solving these problems. One approach fo-cuses on the remote control units that control the TV interfaces. Ease of useof TV interfaces is affected by the design of these units. Some attempts havebeen made to develop usable remote controls (Logan, 1994, Logan et al., 1994,Logan and Lenzi, 1995, O’Modhrain and Oakley, 2003). However, the start-ing point for these attempts has been based on the so-called ’traditional TV’where the number of channels and services is not as great as it is for thefuture TV. There have also been attempts to build altered remote controls(Bretan and Kroon, 1996, Enns and MacKenzie, 1998, Robertson et al., 1996),(Myers et al., 2002, Thoma and Nissler, 1999). Another approach that has beenapplied to solve the problem applies personalization technology to ensure thatthe right people are provided with the right information (Smyth et al., 2002,Goren-Bar and Glinansky, 2004, Westerink et al., 1998). Applying speech is anadditional approach (Portolan et al., 1999, Goto et al., 2003). The latter isstudied in this thesis. A more innovative approach that has not been previouslystudied is merging TV and digital paper to provide interactive paper-based TVguides that also function as a remote control for the TV set. This approach isa second factor studied in this thesis.

4.2 Paper interfaces

In general, many researchers have noticed the significant merits of paper. It isubiquitous, highly portable, easy to use in a wide range of environments, inex-pensive, can be annotated easily, and provides excellent readability properties(Hansen and Haas, 1988). So far, paper as an interface has mainly been stud-ied in the work environment (Johnson et al., 1993, Sellen and Harper, 1997,Sellen and Harper, 2001) and compared with computer interfaces(Hornbek and Frokjer, 2003, Harper et al., 1991). However, to our knowledgelittle is known about the importance of paper in the home environment and theTV area.

Systems and prototypes have been developed to demonstrate the potentialof augmented paper, such as the PaperLink system which is an electronic fillingsystem for paper documents (Arai et al., 1997), and the Protofoil system thatis a document management system that can be accessed by both paper andcomputer workstations (Rao et al., 1994). There are variations of augmenteddesk systems that make a physical desk more like a computer workstation and

6

that support computer-based interaction with paper documents (Wellner, 1993,Koike et al., 2001, Mackay et al., 1993, Ishii and Ullmer, 1997). A typical ex-ample is the DigitalDesk (Wellner, 1993).

Previous research has shown that the use of paper in the modern world ispersistent because of paper’s physical properties, not just despite these proper-ties (Johnson et al., 1993). Johnson et al. (1993) claim that paper has utilitythat will not disappear with the increase in the electronic world, therefore themain goal should be integrating, not removing paper from the electronic world.Furthermore, Koike et al. show that the integration of paper and digital infor-mation is effective (Koike et al., 2001). Sellen and Harper predict that paperin support of reading tasks will be one of the hardest paper-based tasks to shiftto the digital world (Sellen and Harper, 1997). They also identified four affor-dances of paper supporting the flexible interweaving of reading and writing; thepaper’s tangibility, its spatial flexibility, its tailorability, and the manipulabilityof paper (Sellen and Harper, 2001).

4.3 Speech interfaces

In general, previous research has shown that there are many benefits of speechinterfaces in various areas, such as computer, telephone, and web-based inter-faces (Damper and Wood, 1995, Newell et al., 1990, Stifelman et al., 1993),(Kurematsu, 1992). However, little is known about whether speech interfacesare suitable for TV interfaces and about how to design a system appropriateprimarily for entertainment purposes and secondarily for information searchpurposes.

Previous research has reported a 20-40% efficiency increase using speechsystems compared with other interface technology, such as key board input(Martin, 1989, Visick et al., 1984). Furthermore, research has also shown thatspeech increases productivity and improves accuracy (Marshall, 1992). Apply-ing speech-only interfaces may not be beneficial for the TV set that includesa large screen and remote control units. There are various ways of combiningmodalities when building multimodal interfaces. Combining speech with graph-ical interfaces is a common strategy for multimodal interfaces involving speech(Mane et al., 1996). Multimodal interfaces can ease the access to a system sincethey permit a broader range of users to use the system than the traditional in-terface (Oviatt, 2002). Efficiency is another important advantage of multimodalinterfaces (Cohen et al., 2000).

To our knowledge, applying speech interfaces in general and multimodalinterfaces in particular to the TV environment is not common. Little researchhas been done on the TV context (Goto et al., 2003, Hauptmann et al., 1995,Portolan et al., 1999). More study is necessary to investigate how best to designsuch interfaces for the TV set. This thesis focuses on speech design issues fordigital TV.

7

5 Research Method

This section will deal with the methods used in this research, the choices thathave been made and the consequences of these choices. The research is dividedin two types of studies: explorative and empirical. Section 5.1 describes themethods applied for data collection in general, discussing their strengths andweaknesses. Section 5.2 presents which of these methods were used in the 5different papers included in this thesis.

5.1 Data Collection Methods and Techniques

The methods and techniques used in this thesis are interviews, observations,workshops, Wizard-of-Oz techniques, and prototypes as well as research sys-tems.

Interviews

The interview is a central technique in qualitative research that provides richdata (Bryman, 2001, Kvale, 1996, MacNealy, 1997). Facts and opinions of po-tential users are discovered in interviews. Interviews can be unstructured orsemi-structured (Kvale, 1996). The objective of the interviews is to understandthe participants’ point of view. In unstructured interviews, the interviewer hasa list of topics to cover, while in semi-structured interviews the interviewer hasan interview guide. It is important that questions in the interview guide aresufficiently specific but not leading (Kvale, 1996).

Observations

Observations can often provide useful additional information about the studiedtopic. Participants can be observed while they interact with a system to collectdata about reactions (Preece et al., 1994) and to capture descriptions of the wayactivities are performed (Newman and Lamming, 1995). The method allowsthe observer to view what users actually do and to focus on areas of interest. Adisadvantage of observations is that the participants’ behavior may be alteredbecause of the presence of the observer. Observation may be direct or indirect.With direct observation, the investigator is actually present during the taskwhile with indirect observation the task is viewed by some other means such asa video recording.

Workshops

Workshops are useful for generating group creativity by gathering a group ofpeople and focusing on a problem or an issue. Workshops can be consideredas structured brainstorming. Workshops can be conducted in different phases(Arvidsson et al., 2002, Jungk and Mullert, 1996). One of the benefits of work-shops conducted in groups is that more data can be collected in the same amountof time compared to methods based on individual participation. This advantage

8

is characteristic for methods in which data is collected using qualitative meth-ods conducted in groups such as focus groups. Moreover, a group discussioncan generate new ideas because when one person starts talking about an ideait can trigger associations, thoughts, and ideas in other people. However, thedynamic of the group can affect the participants negatively. In the presence ofdominant persons there is a risk that their opinion will affect the rest of thegroup and consequently the discussion will take another direction than it wouldhave in another group constellation. In such cases, the workshop leader has tobe more active.

Focus groups

The focus group method is a comprehensive research method that is used tocollect data through structured and controlled group interviews (Morgan, 1993,Morgan, 1997). Information and insights are collected through interaction be-tween the moderator and the group, as well as between group members. Onecriticism of focus group methods is that they often do not include real users butgatekeepers, such as supervisors, managers and others who made decisions aboutthe products (Hackos and Redish, 1998). In a focus group session, groups tendto compromise a lot, in a way that makes their decisions always end up in somekind of no-man’s-land, conformity (Beyer and Holtzblatt, 1998, Brown, 1988).Thus, the decision the group takes is actually something no single member ofthem will stand behind. This is common in groups of people that do not knoweach other well, everybody wants to be polite, especially when the experienceof the discussed issue differs a lot between the group members. It is also pos-sible that the group, if they do not have that much experience of the topic forthe discussion, start to make decisions that are much more extreme than thegroup members would express if they had been interviewed alone, polarization(Beyer and Holtzblatt, 1998, Brown, 1988). This is simple to avoid if the focusgroup is used only to obtain the participants’ opinions and not force them tomake decisions that will be representative for the group.

Wizard-of-Oz techniques

In research, there are several situations in which implementing a complex systemis not possible. Simulating a system is common in such situations. Accordingto Ericsson, two of the most common arguments for building a simulated sys-tem (Ericsson, 1999) are: First, the system is too costly to build. Therefore,research is needed to investigate the usefulness and need for the system beforedesigning it. Second, the technology required to build the system is not avail-able, but within reach. The purpose for the simulated study is to evaluate theneed for future research.

Thus, the main benefit of this approach is that ideas and concepts canbe evaluated before they are implemented. In the Human Computer Inter-action domain, the Wizard-of-Oz approach has been described as an exper-imental evaluation technique (Coutaz, 1994), or as a rapid prototyping for

9

Figure 3: The human operator (to the right) simulates the system when the partici-pant (to the left) interacts with the system.

systems that are costly to build or require new technology (Landauer, 1987,Maulsby et al., 1993)

In this approach, a human operator (a wizard) simulates the system and actsthe way the system would act (Dahlback et al., 1993, Fraser and Gilbert, 1991).The wizard mediates the interaction while participants are led to believe thatthey are using a real system. However, in fact the system is simulated by thewizard. A typical Wizard-of-Oz experiment setup is depicted in Figure 3. Thewizard is located in a room adjacent to the room where the study is conducted.

This approach has, of course, disadvantages and potential problems. How-ever, Research in the domain of natural language has shown that people whoknow about the human operator behave differently and use different dialoguestrategies than those who believe they are using a real system (Dahlback et al., 1993,Fraser and Gilbert, 1991). The main potential source of validity and/or relia-bility problems for this approach is the human operator – the wizard. Theoperator’s expectations cause a bias. The human operator can become biaseddue to the hypotheses used in the study. This can be avoided by providing theoperator with a formal set of behavior guidelines and not with a set of hypothe-ses. These guidelines help to reduce the bias. Another solution is to use anoperator who is not involved in the research, an operator who is not aware ofthe goals of the experiment.

An important issue regarding the Wizard-of-Oz technique in general is ethics.

10

Participants in the experiment think they are interacting with a real system andit might be argued that this is unethical. However, the obvious risk is that bytelling the participants about the experiment setup their behavior would beaffected. However, Fraser and Gilbert (Fraser and Gilbert, 1991) suggest thatWizard-of-Oz experiments should tell half the truth instead of an outright lieand leave the participants draw their own conclusions.

Prototypes and research systems

It is important to involve the users in the early stages of the design process inorder to get feedback and to understand their needs as well as expectations(Jacobson et al., 1992, Schuler et al., 1993). Prototyping is an iterative ap-proach used throughout the entire development process (Avison and Fitzgerald, 1995)and is necessary to make sure that the developed system will meet users’ needs(Nielsen, 1993). A prototype serves as a communication medium between thedevelopers and the users (Jacobson et al., 1992). Further, prototypes make itpossible to test a work practice that does not yet exist and reveal future possibili-ties resulting from the new system (Newman and Lamming, 1995). With proto-types, different design alternatives can be tested (Jacobson et al., 1992). Mock-ups are one kind of prototype and are often used in the early stage of the devel-opment process (Muller et al., 1997). Advanced mock-ups can vary from eithersimple paper sketches to advanced working prototypes (Muller et al., 1997).

We have built working prototypes that have been used in controlled ex-periments with Wizard-of-Oz techniques to gather information for quantitativeanalysis (see Paper I and IV). We have also built two working research sys-tems that have been used to gather information for qualitative analysis. Thefirst system is an advanced EPG dialogue research system (see Paper II andIII). The second research system is called Paper Remote which is an interactivepaper-based TV guide (see Paper V).

5.2 Methods Applied in the Different Papers

For some issues in this thesis, a quantitative approach has been natural, whilea qualitative approach has been natural for others. Two types of studies havebeen conducted: explorative user studies and experimental studies. The explo-rative study approach is used to explore design implications and prospects ofconcept design that are new and require more knowledge. The data collectedusing this approach is based on participants’ needs, experiences, attitudes andexpectations. In contrast, the experimental study approach is applied to eval-uate issues that are difficult to attain knowledge about using solely theoreticalanalysis. Therefore, these issues are evaluated empirically.

Explorative user studies

Paper II, III, and V presents results from two explorative user studies to provideinitial design insights. Figure 4 illustrates the work process. These studies began

11

Figure 4: The work process of the explorative user studies reported in Paper II, III,and V.

by surveying the literature that raised research issues. To our knowledge, littleresearch has been performed in the area. Therefore, we decided to conductexplorative studies to obtain knowledge that would form the foundation forupcoming work. According to Basili (Basili, 1996) the software-engineeringdiscipline requires a cycle of model building, experimentation and learning touncover or develop knowledge. For this reason, research systems were builtand used in the studies to demonstrate the concepts for the participants. Thesystems were tested by the participants and qualitative data were collected withinterviews, observations, workshops and focus group meetings.

Results presented in Paper II and III are based on observations and inter-views while results in Paper V are based on interviews, workshops and focusgroup meetings. In all these papers, running research systems were developedand used in the studies.

Empirical studies

Paper I and IV reports on two empirical studies. The process of the researchis described in Figure 5. The work began by surveying the research literaturethat raised certain questions. Answers to these questions could not be basedon theoretical analysis. Therefore, we decided to evaluate these questions em-pirically. Prototypes were implemented and evaluated from the Wizard-of-Ozapproach. There are two major reasons for choosing this approach. First, thenature of the problems in Paper I and IV are not yet suitable for full-scaleimplementation and evaluation since we still need to collect more informationabout the situation. This approach helps us to observe different users, needsand behavior as much as it supports the collection of empirical data that per-tains to our research questions. Second, it has been shown that this method isviable for similar research, especially for natural language communication with

12

Figure 5: The work process of the experimental studies reported in Paper I and IV.

computers (Jonsson and Dahlback, 1988, Carroll and Aaronson, 1988).The problem of validity and/or reliability caused by the human operator was

managed differently in these studies. In Paper I, clear guidelines were providedwhile in Paper IV the operator was not aware of the goals of the experiment.

Ethics is an important issue for Wizard-of-Oz techniques in general. Webelieve that our experiments do not deceive the participants since:

− After the experiment, the aim and actual setup of the experiment wereexplained to the participants as well as the reasons for using the Wizard-of-Oz technique.

− Participants’ permission to use the collected data was requested and re-quired. If participants did not give permission, the collected data was tobe destroyed and not used. This did not happen in our experiments.

The empirical studies were supplemented with individual post-test interviewsto collect the participants’ assessments of the concept and experiences frominteracting with the interface. It is important to mention that the purpose ofthe interviews was not to evaluate the prototype . What was explored was theparticipants’ subjective response to such types of interfaces.

6 Contributions

The research reported here has contributed to understanding:

− Exploration of merging TV technology and digital paper and the prospectsof integrating paper interfaces into home technology (explained in PaperV).

− TV habits of the Swedish audience and their usage of TV guides. Fourtypes of TV watching were identified: TV as a background, TV as a mainactivity, TV as a partial activity, and TV as an accompaniment activity.The levels of TV guides usage were also identified as well as importantqualities for current TV guides. These are presented in Paper V.

− The appropriateness of speech for viewer interaction with future TV in-terfaces. We have investigated spoken commands, natural language and

13

Figure 6: Current interaction model is based on direct manipulation by means ofremote control (left). Future interaction is based on the Paper Remote concept as wellas on speech input in addition to direct manipulation by means of remote control

dialogue interaction. Results of Paper I indicate the benefit of providingspoken commands for information navigation. Further, Paper II showshow dialogue and natural language input is suitable for information navi-gation.

− Description of situations when speech is not appropriate and when theremote control is more suitable (explained in Paper III).

− Potential error resolution strategies that can be applied when speech rec-ognizers misinterpret the users’ input. These strategies can also be appliedto other home technology systems with a large screen and a distance be-tween it and the user, see Paper IV.

We have also identified two interaction models for TV based on researchresults. The first model is presented in Paper V and indicates the potential ofinteractive paper-based guides that are also integrated into the TV system, seeFigure 6. The second model is presented in Paper II and concerns a systemwith spoken natural language input in combination with visual output. Themodel is based on a dialogue approach in which two partners communicate witheach other about a subject. The model suggests a clear distinction between theinvolved partners and the discussed subject, see Figure 4.

7 Summaries of the Papers

Five papers that address various interaction design aspects of future TVs areincluded in this thesis. On a more detailed level, the papers can be divided intotwo groups: one that addresses speech interaction (Paper I, II, III, and IV) andone that addresses the Paper Remote concept (Paper V). The contribution ofeach paper is described in this section.

14

Figure 7: The three-entity interaction model for multimodal TV dialogue sys-tem.

7.1 Paper I: Speech Enhanced Remote Control for MediaTerminal.

In Proceedings of Eurospeech’01, Aalborg, Denmark, Volume 4, pp. 2685 – 2688.Co-authored by:– Jonas Lundberg, Department of Computer and Information Science, LinkopingUniversity, S-581 83 Linkoping, Sweden, email: jonlu@ida.liu.se– Jenny Johansson, Nokia, Diskettgatan 11, SE-583 35 Linkoping, Sweden,email: jenny.johansson@nokia.com

This paper focuses on the navigation difficulties inherent in the huge numberof television channels. The aim of this study is to investigate whether or notspoken commands could solve the navigation problem. In this study, two inputtechniques are tested empirically: remote control and speech input.

The study showed that subjects preferred to use shortcuts rather than nav-igate, and that the speech command group could use shortcuts in situationswhere the remote control group had to resort to navigation. However, subjec-tive data showed that people preferred to have a remote control unit.

In summary, this study indicates that speech command input is more effec-tive as the steps required to complete tasks are fewer and shortcuts can be usedmore often in the speech condition. However, subjective data indicate that peo-ple are more satisfied with the remote control input. In conclusion, multimodalinteraction, in which speech input complements the remote control unit is rec-ommended. Speech control should not be a substitute for the remote controldevice.

15

7.2 Paper II: Multimodal Dialogue Systems for Interac-tive TV Applications.

In Proceedings of the 4th IEEE International Conference on Multimodal Inter-faces 2002 (ICMI’02), Pittsburgh, USA, pp. 117 – 122.Co-authored by Pontus Johansson, Department of Computer and InformationScience, Linkoping University, S-581 83 Linkoping, Sweden,email: ponjo@ida.liu.se

This paper focuses on revealing the design knowledge of a multimodal elec-tronic program guide for interactive TV. Multimodal interface is defined as acombination of spoken natural language and visual representation. In this study,a running research system is developed and tested. A qualitative study based onobservations and interviews was conducted to examine the users’ experiences ofthe system interaction. The system was designed by adding speech interactionto an already existing commercial electronic program guide controlled by meansof a remote control.

Study results indicate that spoken natural language input combined withvisual output is preferable for TV applications. Furthermore, user feedbackrequires a clear distinction between the dialogue system’s domain result andsystem status in the visual output. Consequently, an interaction model is pro-posed and consists of three entities: user, domain results and system feedback,see Figure 4. This model is created from a dialogue approach in which twopartners (here the user and the system) communicate with each other aboutsomething (here TV channels, programs, descriptions, etc.)

7.3 Paper III: Using Speech and Dialogue Systems for In-teractive TV Navigation

Submitted 2004 to the international journal: Universal Access in the Informa-tion Society (UAIS), Springer-Verlag Heidelberg.Co-authored by Pontus Johansson, Department of Computer and InformationScience, Linkoping University, S-581 83 Linkoping, Sweden,email: ponjo@ida.liu.se

This paper focuses on the complex information navigation required by TVviewers. The paper focuses on providing spoken natural language and dia-logue in combination with visual representation for information navigation inthe interactive TV context. A running system is built and users’ reactions areevaluated. Data is collected by testing the prototype, observing subjects andinterviewing them afterward. User experiences are analyzed and structured intofive categories that capturing essential aspects of TV interaction. These cate-gories are; interaction style, information load, data access, effectiveness and in-teraction initiative. From the analysis, design considerations relevant for speechand dialogue information systems for television interfaces emerge.

The results of this study indicate that dialogue interfaces have a significant

16

potential for TV interaction. However, results also indicate that the remotecontrol device may still play an important role. A highly heterogeneous viewingaudience can access the TV set by providing both speech and remote controlinteraction.

7.4 Paper IV: Error Resolution Strategies for InteractiveTelevision Speech Interfaces.

In Proceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction, INTERACT 2003, Zurich, Switzerland, pp. 105 – 112.Co-authored by Pernilla Qvardfordt, Department of Computer and InformationScience, Linkoping University, S-581 83 Linkoping, Sweden,email: perqv@ida.liu.se

This paper explores possible error resolution strategies in the case of mis-recognizing user input to a TV set. From previous research two techniqueshave been identified: repetition of input by the user, called here correction, andchoice of a suitable interpretation from several alternatives, called here clari-fication. From these two techniques, four alternative strategies suitable for aTV environment are designed. These four strategies are tested experimentallywith a Wizard-of-Oz method: one with repetition (correction) and three basedon choosing the best alternative from an n-best list given in audio and visualmode, alone or combined (clarification audio, clarification visual and clarifica-tion audio & visual).

An evaluation of the four error recovery strategies shows that people preferclarification to correction for error correction. Between visual and audio clar-ification strategies, the visual n-best list provided better performance becauseit offered the flexibility to say the number of the item or to repeat the inputwhile audio clarification was found tedious by the participants. When audio andvisual clarification were combined, no further performance advantage was foundand participants felt that the additional audio information was distracting.

In summary, results show that displaying an n-best list gives the most effi-cient interaction in a TV environment. Furthermore, redundant audio feedbackdoes not influence the performance.

7.5 Paper V: The Paper Remote: An Augmented TVGuide and Remote Control.

Submitted 2004 to the international journal: Universal Access in the Informa-tion Society (UAIS), Springer-Verlag Heidelberg.Co-authored by Erik Berglund, Anders Larsson, and Magnus Bang, Depart-ment of Computer and Information Science, Linkoping University, S-581 83Linkoping, Sweden, email: eribe, andla, magba@ida.liu.se

TV guides printed on paper are commonly used to search and find infor-mation about TV channels and programs. This paper focuses on studying the

17

potential of augmenting such guides by linking them with TV technology. Threestudies are reported in this paper.

The first study focuses on studying the actual usage of paper-based TVguides and how this usage is related to actual TV watching. People who usuallyuse paper-based TV guides were interviewed about their usage habits, needs,and expectations. Results show that paper-based guides are easy to access,read and find. However, they are passive, isolated from the TV environment,and do not provide cognitive help. Thus, disadvantages and advantages of suchguides are related to the physical properties of paper. The disadvantages can beeliminated by adding digital technology to provide interactive and TV-connectedpaper-based guides. This study provides design implications for a new conceptcalled Paper Remote which is an interactive paper-based TV guide and a remotecontrol for the TV. Consequently, a working research system was developedaccording to these implications and used in the two subsequent studies.

The second study focuses on evaluating the Paper Remote concept. Work-shops are conducted in which the prototype is tested and the concept is evalu-ated both individually and in groups.

In the third study, existing guides are criticized and users’ interests of futureTV guides are examined. This study is explorative where adapted cooperativescenario building future workshops are applied.

Results of this paper indicate that pen and paper can, to some extent, besuitable for TV interaction. The properties of paper-based TV guides, basedessentially on properties of paper in general, are desired. The Paper Remoteconcept preserves the properties of paper and at the same time adds interac-tion and computation technology which augment viewers’ interaction. Anotherinteresting result is the conceptual model of the Paper Remote, see Figure 6.Further, integrating paper and pen with TV technology may provide easy ac-cess for information services usually provided by personal computers and theInternet.

8 Summary and Concluding Discussion

Viewers and users of digital TV face the problem that information navigation bymeans of remote control is complex. The transition to digital TV has changedthe television set into an entertainment as well as information device that pro-vides two-way communication with the user. However, the present remote con-trol device is not appropriate for navigation through the huge amount of servicesand information provided by the future digital TV, presumably also a devicefor accessing the Internet. One possibility for coping with the complex informa-tion navigation required by TV viewers is an augmentation of interaction toolsavailable today, based on already existing TV elements and artifacts.

The research reported in this thesis took its departure in an interest forenhancing the remote control with speech interaction for TV information nav-igation. Traditionally, speech interfaces have exhibited several common prob-lems, such as user acceptance and misinterpretation of user input. We thus set

18

out to study prospects for improving navigation in the information space of-fered by digital TV by using speech interaction and dialogue. Possible solutionswere investigated in empirical and explorative studies with implementations ofmock-ups for Wizard-of-Oz studies as well as usability studies of running re-search systems.

Our studies confirm the conjecture that the pragmatic solution of augment-ing remote control devices with speech may provide a suitable means that easesinformation navigation and search. In particular, a result of our studies is docu-mentation of how the remote control device can be augmented by adding speechand connected dialogue technology. The thesis also illuminates the importanceof understanding user attitudes towards speech interaction in order to designacceptable dialogue interfaces. An interesting result of the explorative studies isthe indication that the structure of the underlying interaction model may be im-portant for the perceived ease of understanding. Thus, sometimes a three-entitymodel for interaction, with a clear distinction between the dialogue partners onthe one hand and the object domain of the dialogue on the other is preferred.More research is, however, needed in order to explore the importance of thisconcept.

Nevertheless, the main focus of this thesis project was gradually transferredinto a study of more futuristic approaches to substitute the traditional remotecontrol device, thus allowing augmentation and integration of other commonartifacts in the home with digital media. TV guides printed on paper mayprovide such a solution with the potential to also function as a remote controlfor the TV. In order to study this concept, we developed a prototype system, thePaper Remote, and conducted explorative studies to investigate the potentialof this approach. The idea of the Paper Remote is to prepare for a future with’digital paper’ realized either by an electronic reading device, such as the Anotopen or by paper with printed electronic circuits. The studies in this thesis usedAnoto technology and specially prepared TV guides printed on paper.

User studies that linked digital paper to the TV for everyday informationnavigation illuminated the potential to provide innovative solutions also forhome information systems. Integrating familiar physical artifacts, such as pa-per and pen, into TV technology may provide easy access to information servicesusually provided by PCs and the Internet. Thus, the same augmentation nec-essary for TV as an entertainment device also opens up new communicationchannels for providing community information to citizens who do not feel com-fortable with conventional computers. The solutions and results of the studiesin this thesis provide a first step in this interesting direction.

In summary, the research presented in this thesis contributes to the task ofproviding interaction solutions to the complex information navigation requiredby future TV viewers. These solutions are based on the augmentation of alreadyexisting TV elements and artifacts.

19

Refrences

Adams, W. J. (2000). How people watch television as investigated using focusgroup techniques. Journal of Broadcasting & Electronic Media, 44(1):78 –93.

Ali, A. and Lamont, S. (2000). Interactive television programs: Current chal-lenges and solutions. In Proceedings of the 8th Annual Usability Profession-als’ Association Conference, UPA’00, pages 14 – 18, North Carolina.

Arai, T., Aust, D., and Hudson, S. E. (1997). Paperlink: a technique for hyper-linking from real paper to electronic content. In Proceedings of the SIGCHIconference on Human factors in computing systems, pages 327–334, At-lanta, Georgia, United States. ACM Press.

Arvidsson, F., Ihlstrom, C., and Lundberg, J. (2002). Visions of future news- consensus or conflict? In Proceedings of the 25th Information SystemsResearch Seminar in Scandinavia, Bautahoj, Denmark.

Avison, D. and Fitzgerald, G. (1995). Information Systems Development:Methodologies, Techniques and Tools, 2nd ed. London: McGraw-Hill Com-panies.

Basili, V. R. (1996). The role of experimentation in software engineering: past,current, and future. In Proceedings of the 18th international conference onSoftware engineering, pages 442 – 449, Berlin, Germany. IEEE ComputerSociety.

Beyer, H. and Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centered Systems. San Francisco: Morgan Kaufmann.

Black, A., Bayley, O., Burns, C., Kuuluvainen, I., and Stoddard, J. (1994).Keeping viewers in the picture: real-world usability procedures in the de-velopment of a television control interface. In Conference companion on Hu-man factors in computing systems, pages 243 – 244, Boston, Massachusetts,United States. ACM Press.

Bretan, I. and Kroon, P. (1996). Concurrent engineering for an interactiveTV interface. In Conference companion on Human factors in computingsystems, pages 117–118, Vancouver, British Columbia, Canada. ACM Press.

Brown, R. (1988). Group processes: Dynamics within and between groups. Ox-ford: Blackwell Publishers Ltd.

Bryman, A. (2001). Social Research Methods. Oxford, UK: Oxford UniversityPress.

Carroll, J. and Aaronson, A. (1988). Learning by doing with simulated intelli-gent help. Communication of the ACM, 31(9):1064–1079.

20

Choi, H., Choi, M., Kim, J., and Yu, H. (2003). An empirical study on the adop-tion of information appliances with a focus on interactive tv. Telematicsand Infomatics, 20(2):161–183.

Chorianopoulos, K. (2003). The digital set-top box as a virtual channel provider.In CHI ’03 extended abstracts on Human factors in computing systems,pages 666–667. ACM Press.

Clancey, M. (1994). The television audience examined. In Journal of advertisingresearch, volume 34, pages 78–87.

Cohen, J. (2002). Television viewing preferences: programs, schedules, and thestructure of viewing choices made by Israeli adults. Journal of Broadcasting& Electronic Media, 46(2):204–221.

Cohen, P., McGee, D., and Clow, J. (2000). The efficiency of multimodal in-teraction for a map-based task. In the Proceedings of the Applied Natu-ral Language Processing Conference (ANLP’00), pages 331–338, Seattle,Washington.

Conway, J. C. and Rubin, A. M. (1991). Psychological predictors of televisionviewing motivation. Communication Research, 18(4):443 – 463.

Coutaz, J. (1994). Evaluation techniques: Exploring the intersection betweenHCI and Software Engineering. In Proceedings of the Workshop on SoftwareEngineering and Human Computer Interaction (ICSE’94 Workshop on SE-HCI. Tylor, R. N. and Coutaz, J., editors, Lecture notes in ComputerSience, vol 896,.

Dahlback, N., Jonsson, A., and Ahrenberg, L. (1993). Wizard of oz studies –why and how. Knowledge-Based Systems, 6(4):258–266. Also in: Readingsin Intelligent User Interfaces, Mark Maybury & Wolfgang Wahlster (eds),Morgan Kaufmann, 1998.

Daly-Jones, O. and Carey, R. (2000). Interactive tv: a new interactionparadigm? In CHI ’00 extended abstracts on Human factors in computersystems, pages 306 – 306, The Hague, The Netherlands. ACM Press.

Damper, R. I. and Wood, S. D. (1995). Speech versus keying in command andcontrol applications. International Journal of Human-Computer Studies,42:289–305.

Dransfeld, H., Pemberton, J., and Jacobs, G. (2000). Quantifying weightedexpert opinion: The future of interactive television and retailing. Techno-logical Forecasting and Social Change, 63(1):81–90.

Ehrmantraut, M., Harder, T., Wittig, H., and Steinmetz, R. (1996). The per-sonal electronic program guide – towards the pre-selection of individualtv programs. In Proceedings of the fifth international conference on Infor-mation and knowledge management, pages 243–250, Rockville, Maryland,United States. ACM Press.

21

Enns, N. R. N. and MacKenzie, I. S. (1998). Touchpad-based remote controldevices. In CHI ’98 conference summary on Human factors in computingsystems, pages 229–230, Los Angeles, California, United States. ACM Press.

Ericsson, M. (1999). Supporting the Use of Design Knowledge - An Assess-ment of Commenting Agents. PhD thesis, Dissertation No. 592, Linkoping,Linkoping studies in science and technology.

Eronen, L. (2003). Digital television for all: User preferences and designers’?views on what would suit the user. In N. Carbonell and C. Stephanidis(Eds.): User Interfaces for All, LNCS 2615, pages 179–186. Springer-VerlagBerlin Heidelberg.

Fraser, N. and Gilbert, N. S. (1991). Simulating speech systems. ComputerSpeech and Language, 5:81 – 99.

Freeman, J. and Lessiter, J. (2003). Using attitude based segmentation tobetter understand viewer’s usability issues with digital and interactive tv.In Proceedings of the 1st European Conference on Interactive Television:from Viewers to Actors? (EuroITV), pages 19 – 27.

Frenkel, K. A. (1989). The next generation of interactive technologies. Com-munication of the ACM, 32(7):872–881.

Furht, B. (1996). Interactive television systems. In Proceedings of the 1996ACM symposium on Applied Computing, pages 7–11. ACM Press.

Gauntlett, D. and Hill, A. (1999). TV Living: Television, culture and everydaylife. Routledge.

Gill, J. M. and Perera, S. A. (2003a). The accessibility of interactive digitaltelevision. In Craddock G M, McCormack L P, Reilly R B & Knops H TP Assistive Technology - Shaping the Future. IOS Press, ISBN 1383 813X,pages 500 – 504.

Gill, J. M. and Perera, S. A. (2003b). Accessibility of universal design of inter-active digital television. In In Proceedings of the 1st European Conferenceon Interactive Television: from Viewers to Actors? (EuroITV), pages 83 –89.

Goren-Bar, D. and Glinansky, O. (2004). Fit-recommend ing tv programs tofamily members. Computers & Graphics, 28(2):149–156.

Goto, J., Komine, K., Kim, Y.-B., and Uratani, N. (2003). A television con-trol system based on spoken natural language dialogue. In Proceedings ofProceedings of the Ninth IFIP TC13 International Conference on Human-Computer Interaction, INTERACT 2003, pages 765 – 768, Zurich, Switzer-land.

22

Hackos, J. T. and Redish, J. C. (1998). User and Task Analysis for InterfaceDesign. New York: John Wiley & Sons.

Hansen, W. J. and Haas, C. (1988). Reading and writing with computers: aframework for explaining differences in performance. Communication ACM,31(9):1080–1089.

Harper, R. (2003). Inside the smart home. Springer-Verlag London Limited.

Harper, R. H. R., Hughes, J. A., and Shapiro, D. Z. (1991). Working in harmony:An examination of computer technology in air traffic control. In Bowers,J. M. and S.D.Benford, e., editors, Studies in Computer Supported Coop-erative Work: Theory, Practice and Design, pages 225 – 234. Amsterdam:Elsevier.

Hauptmann, A. G., Witbrock, M. J., Rudnickyand, A. I., and Reed, S. (1995).Speech for multimedia information retrieval. In UIST-95 Proceedings ofUser Interface Software and Technology, pages 79–80.

Heeter, C. and Greenberg, B. (1988). Cableviweing. Norwood, NJ: Ablex Pub-lishing Company.

Hektor, A. (2001). What’s the Use? Internet and Information Behaviour inEveryday Life. Dissertation, Linkoping. Linkoping University, (Departmentof Technology and Social Change).

Herigstad, D. and Wichansky, A. (1998). Designing user interfaces for television.In CHI 98 conference summary on Human factors in computing systems,pages 165–166. ACM Press.

Hornbek, K. and Frokjer, E. (2003). Reading patterns and usability in visual-izations of electronic documents. ACM Transactions on Computer-HumanInteraction, 10(2):119–149.

Ishii, H. and Ullmer, B. (1997). Tangible bits: towards seamless interfacesbetween people, bits and atoms. In Proceedings of the SIGCHI conferenceon Human factors in computing systems, pages 234–241, Atlanta, Georgia,United States. ACM Press.

Jacobs, G. and Dransfield, H. (1998). Scenarios for interactive tv – europe’suncertain future. Long Range Planning, 31(3):396–405.

Jacobson, I., Christerson, M., Jonsson, P., and Overgaard, G. (1992). Object-Oriented Software Engineering: A Use Case Driven Approach. ACM Press:Addison-Wesley Publishing Co.

Jan, V. D. and Loes, D. V. (2001). Searching for the holy grail: Images ofinteractive television. new media & society journal, 3(4):443 – 465.

Jensen, J. F. and Toscan, C. (1999). Interactive Television: TV of the Future orthe Future of TV? (Media and Cultural Studies). Aalborg University Press.

23

Johnson, W., Jellinek, H., Leigh Klotz, J., Rao, R., and Card, S. K. (1993).Bridging the paper and electronic worlds: the paper user interface. In Pro-ceedings of the SIGCHI conference on Human factors in computing systems,pages 507–512, Amsterdam, The Netherlands. ACM Press.

Jonsson, A. and Dahlback, N. (1988). Talking to a computer is not like talkingto your best friend. In Proceedings of The first Scandinivian Conference onArtificial Intelligence,, Tromso, Norway.

Jungk, R. and Mullert, N. (1996). Future Workshops - How to Create DesirableFutures. Institute for Social Inventions, London.

Kang, M.-H. (2002). Interactivity in Television: Use and Impact of an In-teractive Program Guide. Journal of Broadcasting and Electronic Media,46(3):330–345.

Kerschbaumer, K. (2000). Fulfilling the promise: The cable industry is poised tocapitalize on the concept’s long-awaited potential. Broadcasting & Cable,7(10):22 – 34.

Kim, P. (2001). New media, old ideas: The organizing ideology of interactivetv. Journal Of Communication Inquiry, 25(1):72–88.

Klein, J. A., Karger, S. A., and Sinclair, K. A. (2003). Digital televi-sion for all - a report on usability and accessible design. Available at:http://www.digitaltelevision.gov.uk/dtv for all.html.

Koike, H., Sato, Y., and Kobayashi, Y. (2001). Integrating paper and digitalinformation on enhanceddesk: a method for realtime finger tracking on anaugmented desk system. ACM Transactions on Computer-Human Interac-tion, 8(4):307–322.

Koller, F., Burmester, M., and Wohr, A. (1997). User interfaces for interac-tive tv – a case study with end users. In Proceedings of the ECMAST 97conference, pages 327–341.

Kurematsu, A. (1992). Future perspective of automatic telephone interpreta-tion. IEICE Transaction Communication, 1:14 – 19.

Kvale, S. (1996). Interviews: An introduction to qualitative research interview-ing. Thousand Oaks, CA: Sage Publications.

Landauer, T. K. (1987). Psychology as a mother of invention. In Proceedingsof the SIGCHI/GI conference on Human factors in computing systems andgraphics interface, pages 333 – 335, Toronto, Ontario, Canada. ACM Press.

Lee, B. and Lee, R. S. (1995). How and why people watch tv: Implicationsfor the future of interactive television. Journal of Advertising Research,35(6):9–18.

24

Logan, R. J. (1994). Behavioral and emotional usability: Thomson consumerelectronics. In Wiklund, M. E., editor, Usability in Practice: How com-panies develop user friendly products., pages 59–82. Academic Press, Inc.,Boston.

Logan, R. J., Augaitis, S., Miller, R. H., and Wehmeyer, K. (1995). Livingroom culture - an anthropological study of television usage behaviors. InProceedings of the Human Factors and Ergonomics Society.

Logan, R. J., Augaitis, S., and Renk, T. (1994). Design of simplified televisionremote controls: a case for behavioral and emotional usability. In Proceed-ings of the Human Factors and Ergonomics Society 38th Annual Meeting,pages 365 – 369.

Logan, R. J. and Lenzi, L. E. (1995). Innovations in rca user interface design.Design management journal, 6(4):16 – 20.

Lougos, C., Vassilopoulou, K., and Vrechopoulos, A. (2002). Interactive digitaltv services - viewers’ perceptions. In Proceedings of the eBusiness andeWork 2002 Conference: Challenges and Achievements in E-Business andE-work.

Mackay, W., Velay, G., Carter, K., Ma, C., and Pagani, D. (1993). Augmentingreality: adding computational dimensions to paper. Communications of theACM, 36(7):96–97.

Mackay, W. E. (1998). Augmented reality: linking real and virtual worlds: anew paradigm for interacting with computers. In Proceedings of the work-ing conference on Advanced visual interfaces, pages 13–21, L’Aquila, Italy.ACM Press.

MacNealy, M. S. (1997). Toward better case study research. IEEE Transactionon Professional Communication, 40(3):182 – 196.

Mane, A., Boyce, S., Karis, D., and Yankelovich, N. (1996). Designing theuser interface for speech recognition applications. ACM SIGCHI Bulletin,28(4):29–34.

Marshall, J. P. (1992). A manufacturing application of voice recognition forassembly of aircraft wire harnesses. In Proceedings of Speech Tech/VoiceSystems Worldwide, New York. Media Dimensions, Inc.

Martin, G. L. (1989). The utility of speech input in user-computer interfaces.International Journal of Man-Machine Studies, 30(4):355–375.

Maulsby, D., Greenberg, S., and Mander, R. (1993). Prototyping an intelligentagent through wizard of oz. In ACM Conference on Human Factors inComputing Systems, INTERCHI’93. ACM Press.

25

Morgan, D. L. (1993). Successful focus groups: Advancing the state of the art.Newbury Park, California: Sage Publications, Inc.

Morgan, D. L. (1997). Focus Groups as Qualitative Research. Focus Groups asQualitative. Thousands Oaks, California: Sage Publications, Inc.

Morrison, M. and Krugman, D. M. (Winter 2001). A look at mass and computermediated technologies: Understanding the roles of television and computersin the home. Journal of Broadcasting & Electronic Media, 45(1):135–161.

Muller, M., Haslwanter, J. H., and Dayton, T. (1997). Participatory practicesin the software lifecycle. In M.G. Helander, T.K. Landauer, P. P. E., editor,Handbook of Computer-Human Interaction, 2nd ed., pages 255–297. ElsevierScience B.V., Amsterdam.

Myers, B. A., Bhatnagar, R., Nichols, J., Peck, C. H., Kong, D., Miller, R., andLong, A. C. (2002). Interacting at a distance: measuring the performance oflaser pointers and other devices. In Proceedings of the SIGCHI conferenceon Human factors in computing systems, pages 33–40. ACM Press.

Nelson, M. (1999). Remote controls. DigiPoints: The Digital Knowledge Hand-book, 3(4).

Nelson, M. N., Linton, M., and Owicki, S. (1995). A highly available scalableitv system. In Proceedings of the fifteenth ACM symposium on Operatingsystems principles, pages 54–67. ACM Press.

Newell, A. F., Arnott, J. L., Carter, K., and Cruickshank, G. (1990). Listen-ing typewriter simulation studies. International journal of man-machinestudies, 33(1):1–19.

Newman, W. M. and Lamming, M. G. (1995). Interactive System Design.Addison-Wesley Longman Publishing Company Inc.

Nielsen, J. (1993). Usability Engineering. Academic Press, Inc.

O’Brien, J., Rodden, T., Rouncefield, M., and Hughes, J. (1999). At homewith the technology: an ethnographic study of a set-top-box trial. ACMTransactions on Computer-Human Interaction (TOCHI), 6(3):282 – 308.

O’Modhrain, S. and Oakley, I. (2003). Touch tv: Adding feeling to broad-cast media. In Proceedings of the 1st European Conference on InteractiveTelevision: from Viewers to Actors? (EuroITV).

ONS (2002). Family spending: a report on the 2000-01 family expendituresurvey, london: the stationery office.

Oviatt, S. L. (2002). Multimodal interfaces. In The Human-Computer In-teraction Handbook: Fundamentals, Evolving Technologies and EmergingApplications, J. JACKO AND A. SEARS, Eds. Lawrence Erlbaum Assoc.,Mahwah, NJ. Chap. 14, pages 286 – 304.

26

Perera, S. (2002). Interactive digital television services for people with lowvision. Available at: http://www.tiresias.org/itv/itv1.htm.

Portolan, N., Nael, M., Renoullin, J.-L., and Naudin, S. (1999). Will we speakto our tv remote control in the future? In Proccedings of the 17th Inter-national Symposium on Human Factors in Telecommunication, HFT’99,Copenhagen, Denmark.

Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., and Carey, T. (1994).Human-Computer Interaction. Addison-Wesley.

Press, L. (1993). The internet and interactive television. Communication ofACM, 36(12):19–23.

Rao, R., Card, S. K., Johnson, W., Klotz, L., and Trigg, R. H. (1994). Proto-foil: storing and finding the information worker’s paper documents in anelectronic file cabinet. In Proceedings of the SIGCHI conference on Hu-man factors in computing systems, pages 180–185, Boston, Massachusetts,United States. ACM Press.

Robertson, S., Wharton, C., Ashworth, C., and Franzke, M. (1996). Dual deviceuser interface design: Pdas and interactive television. In Proceedings of theSIGCHI conference on Human factors in computing systems, pages 79–86,Vancouver, British Columbia, Canada. ACM Press.

Schuler, D., Namioka, A., and (Eds.) (1993). Participatory Design: Principlesand Practices. Hillsdale, NJ: Lawrence Erlbaum Assoc.

Seel, P. B. (2003). Who wants dtv? Journal of Broadcasting & ElectronicMedia, 47(1):149–152.

Sellen, A. and Harper, R. (1997). Paper as an analytic resource for the design ofnew technologies. In Proceedings of the SIGCHI conference on Human fac-tors in computing systems, pages 319–326, Atlanta, Georgia, United States.ACM Press.

Sellen, A. J. and Harper, R. H. R. (2001). The Myth of the Paperless Office.Cambridge (MA): MIT Press.

Silverstone, R. (1994). Television and everyday life. Routledge.

Smyth, B., Cotter, P., and Ryan, J. (2002). Evolving the Personalized EPG - AnAlternative Architecture for the Delivery of DTV Services. In Proceedingsof the 2nd Workshop on Personalization in Future TV, 2nd InternationalConference on Adaptive Hypermedia and Adaptive Web Systems, Malaga,Spain.

Stephanidis, C. and Savidis, A. (2001). Universal access in the informationsociety: Methods, tools, and interaction technologies. Universal Access inthe Information Society, 1(1):40–55.

27

Stifelman, L. J., Arons, B., Schmandt, C., and Hulteen, E. A. (1993).Voicenotes: a speech interface for a hand-held voice notetaker. In Pro-ceedings of the conference on Human factors in computing systems, pages179–186. Addison-Wesley Longman Publishing Co., Inc.

Taylor, A. and Harper, R. (2003). Switching on to switch off. In (ed.), H. R.,editor, Inside the smart home, pages 115 – 126. Springer-Verlag LondonLimited.

Thoma, V. and Nissler, J. (1999). Evaluation of a pc/itv interface for onlineservices. In Proceedings of the Active Web conference.

Toscan, C. and Jensen, J. F. (1999). Interactive Television: TV of the Futureor the Future of TV? (Media and Cultural Studies), chapter 1, pages 11 –23. Aalborg University Press.

Venkatesh, A. (1996). Computers and other interactive technologies for thehome. Communications of the ACM, 39(12):47–54.

Venkatesh, V. and Brown, S. A. (2001). Longitudinal investigation of personalcomputers in homes: Adoption determinants and emerging challenges. MISQuartely, 25(1):71 – 102.

Visick, D., Johnson, P., and Long, J. (1984). The use of simple speech recog-nizers in industrial applications. In Proceedings of INTERACT’84.

Voutsas, D. and Halverson, C. (2000). Surfing the home with your tv. In Pro-ceedings of the eighth ACM international conference on Multimedia, pages452–455. ACM Press.

Wellner, P. (1993). Interacting with paper on the digitaldesk. Communicationsof the ACM, 36(7):87–96.

Westerink, J. H. D. M., van der Korst, M., and Roberts, G. (1998). Evaluatingthe use of pictographical representations for tv menus. In CHI 98 confer-ence summary on Human factors in computing systems, pages 217–218, LosAngeles, California, United States. ACM Press.

Zong, L. and Bourbakis, N. G. (2001). Digital video and digital tv: A compar-ison and the future directions. Real-Time Imaging, 7:545–556.

28