d.7.1 user evaluation planfp7-hermes.eu/uploads/media/deliverables/hermes_d... · criteria for...

HERMES – Cognitive Care and Guidance for Active Aging FP7-ICT 216709 Specific Targeted Research or Innovation Project Start date of project: January 1, 2008 Duration: 36 months

D.7.1 User Evaluation Plan

Norman Höller, Arjan Geven, Manfred Tscheligi (CURE)

Mari Feli Gonzalez (INGEMA)

Version: 1.0 Date: 30/03/2009 Dissemination level: (PU, PP, RE, CO): PU Project Co-Funded by the European Commission within the 7th Framework Programme


/ 28

Abstract This deliverable gives an overview of the evaluation procedure of the prototypes developed in the HERMES project. It describes the main research questions with regards to the HERMES system, i.e. what are the goals of the evaluation, followed by a procedural description of the evaluation. Within the evaluation, the following components receive attention: 1. Simplicity / Complexity Evaluation 2. Information Visualization Evaluation 3. Game Experience Evaluation 4. Multimodal Reminder Evaluation 5. Technology Acceptance Evaluation The procedural evaluation contains the methods for evaluation, the inclusion and exclusion criteria for participant selection, as well as a time schedule for the multinational evaluation in Austria and Spain.


/ 28

Table of Contents

D.7.1 USER EVALUATION PLAN ......................................................................................................................... 1

1. INTRODUCTION ............................................................................................................................................ 4

1.1 BACKGROUND ........................................................................................................................................... 4 1.2 SCOPE OF THIS DELIVERABLE .................................................................................................................... 5

2. RESEARCH QUESTIONS TO BE ANSWERED IN THE USER EVALUATIONS ................................ 6

2.1 TECHNOLOGY ACCEPTANCE....................................................................................................................... 6 2.1.1 Technology acceptance of the HERMES system .................................................................................. 7

2.2 INTERFACE COMPLEXITY........................................................................................................................... 8 2.2.1 Interface complexity in the MyPast application ................................................................................... 8

2.3 INFORMATION VISUALIZATION ................................................................................................................... 9 2.3.1 Information Visualization in the MyPast application........................................................................... 9

2.4 GAME EXPERIENCE.................................................................................................................................... 9 2.4.1 Game experience in the cognitive games ........................................................................................... 10

3. HERMES PROTOTYPE............................................................................................................................... 11

3.1 OVERVIEW OF THE FIRST PROTOTYPE......................................................................................................11 3.1.1 HERMES MyPast Application............................................................................................................ 14 3.1.2 HERMES MyFuture ........................................................................................................................... 14 3.1.3 HERMES Cognitive Games................................................................................................................ 14 3.1.4 HERMES PDA ................................................................................................................................... 14

3.2 OVERVIEW OF THE SECOND HERMES PROTOTYPE.................................................................................. 15

4. USER RECRUITMENT AND TIMING ...................................................................................................... 18

4.1 RECRUITMENT ......................................................................................................................................... 18 4.2 TIMING .................................................................................................................................................... 19

5. EVALUATION PROCEDURE..................................................................................................................... 20

5.1 EVALUATION PROCEDURE....................................................................................................................... 20 5.2 EVALUATION OF MY PAST ........................................................................................................................ 21 5.3 EVALUATION OF THE MYFUTURE FRONT-END ........................................................................................ 22 5.4 EVALUATION OF COGNITIVE GAMES ....................................................................................................... 23 5.5 EVALUATION OF THE HERMES PDA...................................................................................................... 24 5.6 INFORMATION VISUALIZATION ................................................................................................................. 24 5.7 USER ACCEPTANCE QUESTIONNAIRE....................................................................................................... 24

6. EVALUATION OF THE SECOND PROTOTYPE.................................................................................... 25

7. DATA USED IN THE EVALUATION ........................................................................................................ 26

REFERENCES......................................................................................................................................................... 27


/ 28

1. Introduction

1.1 Background Usability and user acceptance of technology play a critical role for the success of the overall HERMES system. Accordingly it is essential to understand how end-users experience technologies with its social and cultural meaning which implicitly trace the value of technological products in elderly people’s lives, as well as to observe their context of use [1]. In order to verify the end-users’ acceptance of the HERMES system the evaluation process will happen in different countries and cultures. The field of HCI highlights the importance of the social and cultural impact, as well as the effects of pleasure, fun, emotional effects and aesthetics have on the experience of use tantamount to the overall usability criteria as metrics for a technology evaluation process. To ensure a most usable outcome that is driven by real user needs, several means are implemented within the HERMES project to take care of these aspects. In this section the approach for the evaluation methodology of the complete HERMES system will be described. In the 1st prototype trials the HERMES system will be evaluated and tested, in lab environment, to study its level of compliance with the users’ requirements hierarchy i.e. end-users’ needs priorities and requirements engineering refinements. The 1st prototype trials consist of tests with different user interfaces, different ways of visualizing audio and video data and usability evaluations. The relevance and applicability of the system for the interaction with the elderly people will be evaluated. The focus of the evaluation will be put on user-system interaction evaluation, accessibility standards and the way the system is perceived by the users as meeting their needs. Furthermore it will reveal information about the perceived user comfort and quality-of-experience, as well as the level of conformance of the system with all relevant guidelines. Regarding evaluation procedures with users involved, which will be the usability evaluations of usage contexts, this document will be enhanced with detailed descriptions of the evaluation procedures as the required technical details are available to provide an operational manual for trials conduction. The 1st prototype trials will be conducted in Austria with an aim to involve 16 people in the facets of the evaluation. Due to the possibility of people dropping out of the evaluation we will set a minimum of 12 participants in Austria. In Spain, due to the more intensive nature of the study there (including the speech processing components which are available only in Spanish), we evaluate the system with 8-10 persons. After the reengineering of the first prototype based on the results of the 1st user trials, the 2nd prototype trial is due to test the effectiveness of the system within several scenarios. These trials will happen again with real users but this time in real environments. The 2nd prototype trials will be conducted in Austria with the same 12-16 persons in a mainly lab setting and in Spain with the same eight users where possible directly in their homes, including their family and surroundings in the evaluation of the HERMES system.


/ 28

1.2 Scope of this Deliverable Inputs for this deliverable are mainly those deliverables that are part of the work package 2, particularly D.2.1 summarizing user requirements and D.2.3 which lists the scenarios that guide the technology developments. Of course, the evaluation also heavily leans on the technology and interfaces developed in the work packages 3, 4 and 5. This deliverable is concerned with the evaluation of the HERMES prototypes in the two scheduled lab- and field trials. The document describes the research questions, the methodology, the first prototype, the evaluation procedure, the user recruitment and the timeframe of the evaluation.


/ 28

2. Research Questions to be answered in the User Evaluations This section provides an overview of the research questions that we hope to progress on by conducting the user evaluations, focusing on the following elements: • Technology Acceptance of the HERMES system in general • Interface Complexity of the Past application • Information Visualization of audiovisual contents • Game Experience of the cognitive games

2.1 Technology acceptance User acceptance can be defined as the willingness of a certain user group to employ a certain system for the task it was designed to support [3]. One of the earliest attempts and a key model for the existing technology acceptance models was “Theory of Reasoned Action”(1975), which takes its roots from social psychology and explains persons actual behaviour through her/his behavioural intention; attitudes and subjective norms are the effecting factors for the intention [4]. The innovation diffusion theory (IDT) or Diffusion of Innovations (DOI), coined by Rogers [5] concentrates on the diffusion of ideas and possible social factors that might affect the diffusion’s level. According to Rogers [5] the following five characteristics of a technology determine its acceptance:

• relative advantage (the extent to which it offers improvements over available tools), • compatibility (its consistency with social practices and norms among its users), • complexity (its ease of use or learning), • trialability (the opportunity to try an innovation before committing to use it), • observability (the extent to which the technology's gains are clear to see).

Diffusion studies have demonstrated that innovations affording relative advantages, compatibility with existing practices and beliefs, low complexity will be more extensively and rapidly accepted than an innovation with the opposite characteristics. Particularly, three of these characteristics seem to have the greatest influence: relative advantage, compatibility and lack of complexity. While the diffusion model has broad appeal, there are concerns that the characteristics Rogers lists are too loosely defined to provide a sound basis for a complete theory [3].

Characteristics of users accepting a certain technology vs. users rejecting the same technology were researched by Alavi and Joachimsthaler [6]. Their findings suggest that the most important factors with regard to user acceptance are cognitive style, personality, demographics, and user-situational variables. To summarize, there is no single model or theory explaining or even predicting acceptance of a system. But key variables in the technology and the user are being understood better: “technology must satisfy basic usability requirements and be perceived as useful by its intended user community. User experience and training will impact acceptance levels [3]”.


/ 28

2.1.1 Technology acceptance of the HERMES system The usability and senseful-ness of HERMES system will be evaluated on the one hand through the perceived usefulness and on the other hand through the perceived ease-of-use combined with the real ease-of-use. The evaluation of the end-user acceptance of the HERMES system will be carried out by using the Unified Theory of Acceptance and Use of Technology by Vankatesh, Morris, Davis, and Davis (2003) [1], which has been built upon the well established Technology Acceptance Model (TAM) and has been extended through unifying various models of IT acceptance by combining and integrating the elements of eight prominent models: 1. Theory of Reasoned Action, 2. TAM – Technology Acceptance Model, 3. Motivational Model, 4. TPB – Theory of Planned Behaviour, 5. Combined TAM-TPB, 6. Model of PC Utilization, 7. Innovation Diffusion Theory, and 8. Social Cognitive Theory.

Figure1: Unified Theory of Acceptance and Use of Technology, Vankatesh (2003)

According to UTAUT, there are four main constructs namely: • performance expectancy • effort expectancy • social influence, and • facilitating conditions. Gender, age, experience, and voluntariness of use are affecting directly the four key constructs’ on usage intention and behaviour.

In addition, the reminders generated by the system are to be evaluated with respect to their acceptance. A well-placed reminder provides the user with the support in the right context, whereas an ill-placed reminder might annoy, embarrass or distract the user from his or her task and in turn lead to mistakes.

The reminder system of HERMES is set to be available in the final prototype and will be available on the home system and the HERMES PDA. The reminders that will be generated by the system will be context dependent – they can be person based, location or time based. On the home system reminders might be triggered by time or upon input from the cameras and/or the microphones. Time triggered reminders are not new, but reminders triggered by


/ 28

sensory information like cameras and microphones are not usual as of today. In the evaluation acceptance of such reminders will be assessed.

2.2 Interface complexity In the world of component-based software engineering (CBSE) efforts exist to come up with indicators for the complexity of interfaces of software components [7]. These efforts are made to have a way for managing complexity levels of software components to ultimately enhance quality and productivity of these components. The same applies in the HCI domain where – originating from safety-critical or mission-critical systems – interface complexity plays a large role in the real and perceived effectiveness of user interaction. Users often have to take correct decisions under time limitations. High complexity of user interfaces may result in errors and negative consequences in the worst case or “only” frustrations in the best case, depending on the domain of the application. To minimize both negative consequences and error-associated negative affect, it is an essential area of study within HCI to identify the factors that create user-perceived or real interface complexity. Ways to assess complexity of user interfaces include a) the evaluation of user satisfaction using HCI, by questionnaire survey [8]; and b) the evaluation of a system’s complexity through objective observation of the user’s ability and difficulties using HCI, by an independent scientist following a specific questionnaire. Coskun and Grabowski [9] measured user interface-related issues using metrics such as • ease of understanding displays, • consistency of display terminology, • sequence of information presentation in displays • consistency between displays and required tasks Dix, Finley, Abowd, and Beale [10] also suggest three points for user interface evaluations: • To assess the extent of the system’s functionality • To assess the effect of the interface on the user, and • To identify any specific problems with the interface.

2.2.1 Interface complexity in the MyPast application HERMES is designed to support older adults to lead an active and independent life-style for as long as possible. However, the complexity of the interface used for HERMES is also critical for the success of the system. It is our intention to identify user interface complexity indicators’ effect on older adults. Three different interface realizations of HERMES MyPast will be evaluated with interface complexity questions in mind. In particular, a combination of theoretical and user-based evaluation with respect to interface complexity will be conducted with the MyPast application to evaluate the following aspects: • number of options presented on an interface, • number of hierarchical levels used on the interface and • amount of information being displayed and presented to user


/ 28

in order to evaluate the aspects of organization of information, easy access to information, and adequacy of information levels.

2.3 Information visualization The visual representation of large amounts of non-numerical, abstract information is the topic in information visualization. The field itself derives from different communities (e.g. information graphics, statistics, user interface community). When designing interfaces for users the problem of representing vast amounts of data is a prominent issue. Especially data describing audio is hard to visualize. One approach is to quantitatively render time and/or frequency content of an audio signal, using methods such as the oscillograph and sound spectrograph [16, 17]. Donath et alter [18] describe ways to visualize social structures of a conversation by providing patterns about bursts of activity, arrival of new members or the evolution of conversational topics. Which methods are suited best for the users of the HERMES application to tell apart speakers and to quickly retrieve information from the presented data is up to the field trial to find out. Information visualization is a central aspect of the interface. Because of the life logging aspects of the system over time huge amounts of audio and video data will populate the system. For end users it can become very tedious to search for e.g. a conversation when the search returns many similar audio objects. A clever way to represent a huge amount of data would help with this issue. Video data is relatively easy to visualize, one can use selected frames to summarize video data. The visualization of the content of audio data is trickier.

2.3.1 Information Visualization in the MyPast application The MyPast application provides a way to visualize speech information as highlighted in the deliverable D.5.1, which aims to visualize more information than simply the audio waveform which does not provide the user with the information specifically requested. Metadata that associates the waveform is therefore used to try to improve the way users can see the contents of the audio without actually having to listen to the whole file, which is a very inefficient way to browse through the audio data. Fast-forward functions only to a very limited amount as speech is quickly transformed into inaudible content where only the time-display (current vs. remaining) provides useful information about the position in the audio file. Visualizations try to overcome this, however, the question remains whether the visualization is understood by users or whether users prefer alternative visualizations of audio, or even prefer the slower but possibly more accurate playback of the audio file in order not to miss important aspects.

2.4 Game experience In recent years the computer gaming world has become increasingly large [11]. Computer gaming has become a widely used and accepted medium. In synch with this development also methods to evaluate computer games of all kinds with regard to user experience have been developed. Methods for game evaluation have been designed by adopting techniques from the usability field such as usability tests and heuristic evaluations. According to literature, user experience in games can be measured using the following qualitative and quantitative methods [13, 14, 15]:


/ 28

• physiological measurements, • expert evaluation (heuristics etc.), • subjective, self-reported measures and • usability tests. Psycho-physiological measurements that have been used in the field of assessing game experience typically use EMG and/or GSR to measure arousal changes and with that, emotional reactions to game interaction. Electromyogram (EMG) measures muscle activity by detecting surface voltages that occur when a muscle is contracted, used mainly to measure facial expressions [19]. EMG has an advantage over video analysis because low evocative emotions are difficult to recognise visually. The facial action coding system (FACS) is essential to measure muscle activity in the face. Partala [20] registered EMG data of two facial muscles for ten subjects watching series of emotionally arousing pictures and videos. Based on the changes in the activity of the two facial muscles, it was possible to distinguish between ratings of positive and negative emotional experiences at a rate of almost 70% for pictures and over 80% for videos in real time. Galvanic Skin Response (GSR) is said to be a linear correlate to arousal. Even though moderating effects occur through room temperature, humidity and participants activities, GSR as an arousal indicator can also serve as indicator for design issues [22]. Modulating GSR with Heart Rate gives additional information that provides the possibility to evaluate quality of game interaction [19].

2.4.1 Game experience in the cognitive games Cognitive training is a key functionality of HERMES. HERMES Cognitive Games will provide end users with the possibility to train their cognitive abilities. The system serves as a platform allowing for the development of different kinds of cognitive games. The first HERMES prototype will feature at least one cognitive game. To asses the user experience of HERMES cognitive games we will use the following methods:

1. game heuristics as developed by Köffel et al [12] 2. Psycho-physiological methods, mainly Electromyogram (EMG) and Galvanic Skin

Response (GSR) are used to evaluate the games with users 3. Game experience questionnaires in order to assess the results in the other two elements 4. Questionnaires to assess game experience (e.g. the Game Experience Questionnaire,

GEQ [23]) These measures provide us with indicators for game performance with respect to user experience of the games, which in turn provide us assistance in determining how to improve the games in order to better address the requirements of the users.


/ 28

3. HERMES Prototype The first HERMES prototype is being built on the input of the requirements analyses and the results of the evaluation of the scenarios. This section describes functionalities of the prototype and the hardware interface concept of the system; following sections will describe the evaluation procedure, the recruitment of test-subjects, the timing of the user trials and data used in the user trials.

3.1 Overview of the first prototype The first prototype is expected to support many of the functionalities described in the scenarios, however some of the elements will only be available for the second prototype. The home-based component prototype will consist of three GUI front-ends to which the seven HERMES scenarios, described in D.2.3, are mapped:

• HERMES MyPast • HERMES MyFuture • HERMES Cognitive Games

The HERMES MyPast, MyFuture and Cognitive Game applications will be accessed through a touch table interface. In addition, the MyFuture application has a mobile extension which is used for interaction with the user when the user is not in the home (i.e. focused on the context-based reminders as well reminder setting) Table 1 summarizes the mapping between scenarios and HERMES GUI front-ends.

Scenarios

Applications

1 Facilitation of episodic memory

• MyPast

2 Cognitive Training

• CognitiveGames

3 Advanced Activity Reminding • Setting a reminder at home • Setting a reminder on-location • Setting a shopping reminder

• MyFuture • PDA

4 Conversation support

• MyPast

5 Mobility support

• MyFuture • PDA

6 Facilitation of lexical access to people´s names

• PDA

7 Searching for common use objects

• MyPast

Table 1: Mapping between scenarios and HERMES GUI-front-ends


/ 28

As part of the first prototype, the following technologies will be available: Technology Name

Description Main Partner(s) involved

Person / Face Identification Identifies Actors entering HERMES space AIT People Counting (Visual) Counts People within the room AIT Speech Info Indexing & Retrieval

Search in spoken data IBM

Voice Activity Detection Detects Human Speech AIT Visual Person Tracking Tracks people movement within the

HERMES space (in-door) AIT

Speech-to-text Transcription of spoken conversations (Spanish only)

IBM

MemoryAid Functionality for searching and filtering database items based on various criteria

UniBrad

MyPast Application Presents the user interface functionalities in a GUI to the user and allows the user to retrieve and edit information that has been captured by the input processing components (as described in the deliverable D.5.1)

CURE/Unibrad

MyFuture Application Presents the user interface functionalities in a GUI to the user and allows the user to • set appointments • store notes as addition to appointments • edit appointments • synchronization with HERMES PDA

imports appointments from the PDA and exports appointments to PDA (within a defined time interval)

CURE/TXT

Fingertips tracking for the Multi-Touch Surface

Track users’ fingertips to enable multi-touch surface interface

AIT

Cognitive Games Application

Cognitive Training Games Logic (as Defined in D6.1)

AIT

Mobile POI Manager • GPS integration • Possibility to add POIs basing on the

current location

TXT


/ 28

Mobile device synchronisation

Synchronisation with the home-based workstation with focus on:

• Available audio transcriptions related to audio notes attached to appointments

• Audio recordinds for Conversation Support.

• New/Updated stored POIs (using the “locate mode”)

• Appointments (to and from the PDA and the home-based workstation)

• Audio notes attached to the appointments

TXT

Mobile Shopping List Manager

Allow for the user to manage shopping lists includes GPS integration (the user is alerted when close to the food store)

TXT

Speech-to-text Transcription of spoken appointments to make them searchable and to make to display them in textual form

IBM

Speech info indexing The speech transcripts are stored in indexed form for fast search

IBM

Image/Video Tagging and Retrieval

Stores images/video with appropriate tags / Retrieves video segments based on queries

AIT


UniBrad

Audio recorder • The mobile application allows the user to record the audio note and attach it to a chosen event (Appointment Dictation)

• The mobile application allows also to record a discussion (Conversation Support)

TXT

Hermes PDA Application

The mobile application allows the user to • insert/delete/edit/browse/notify

appointments (including information like datetime, notes, POIs, etc.)

• record discussions • attach audio notes to appointments • display post processed speech

transcriptions • take pictures • browse HERMES-registered people

profiles • manage shopping lists

TXT

Video summarisation Summarisation of videos through frame analysis

UniBrad

Table 2: Overview and description of technologies available in the first prototype


/ 28

3.1.1 HERMES MyPast Application HERMES MyPast supports end users in capturing those pieces of their lives that are important to them. The application relies on sensory input coming from microphones and video cameras. The start of the recording can happen automatically (relying on sensory information) or being triggered manually, depending on the preferences of the user. The same is true for the ending of the recording. Once the recording is finished HERMES automatically starts with the post processing of the newly acquired data. To retrieve data from HERMES the user is provided with different search possibilities: names, time and date intervals, emotions, pictures and keywords. The user has the possibility to retrieve content in different formats: audio only, video only, audio and video and a transcript of the audio. Regarding the first prototype, HERMES MyPast will deliver the functionalities described with the constraint that online speech processing will not be available yet – which is taken into account in the set up of the evaluation of the prototype.

3.1.2 HERMES MyFuture HERMES MyFuture in its final version will support end users in planning their lives. MyFuture enables users to set appointments of different kinds (time based, location based), create to do’s and store notes for future use both at home and while on the go. Notes might constitute of recorded audio or typed words. The HERMES PDA will support users on this subject while on the go. To assure consistency with the home based part of the system, HERMES will automatically synchronize the PDA with the home-system. All data concerning future appointments and to do’s will be available on both the home based system and the HERMES PDA. This provides the user with access to future appointments wherever s/he might be. Regarding the first prototype, the MyFuture application provides the functionalities of setting appointments and reminders as described – the creation and presentation of multimodal reminders to the users is scheduled for the final prototype.

3.1.3 HERMES Cognitive Games HERMES Cognitive Games will provide end users with the possibility to train their cognitive abilities using HERMES. The system serves as a platform allowing the development of different kinds of cognitive games. For the first HERMES prototype one cognitive game will be implemented on the HERMES system.

3.1.4 HERMES PDA The HERMES PDA will support the following functionalities:

• Possibility to record audio notes • Possibility to set a GPS based reminder • Possibility to set an appointment • Synchronization with the HERMES system


/ 28

All the functionalities will be implemented in the HERMES PDA at the time of the first user trial.

3.2 Overview of the second HERMES prototype The second HERMES prototype will feature technologies developed for the first prototype as well as technologies that were not ready then. Technologies from the first prototype will be refined for the second prototype. Main differences are to be expected in the reminding system of HERMES, since reminders in the first prototype are not expected. Additionally, voice emotion detection and the use of far field microphone array processing will differentiate the first from the second prototype. At this stage in the project the following technologies are planned to be integrated in the second HERMES prototype. Technology Name

Description – episodic memory Main Partner(s) involved

Person / Face Identification Identifies Actors entering HERMES space AIT People Counting (Visual) Counts People within the room AIT Voice emotion recognition For emotion based search IBM Speech Info Indexing & Retrieval

Search in spoken data IBM

SpeakerID based on far-field array

Identified Speaker as online component

AIT

Voice Activity Detection Detects Human Speech AIT Far field microphone array processing

Cleans Recordings from Noise Locates Speaker (i.e. to boost ASR)

AIT/IBM

Visual Person Tracking Tracks people movement within the HERMES space (in-door)

AIT

Offline speaker segmentation & identification

• The segmentation is needed to enhance the ASR.

• Identification is needed for speaker id based search (offline)

IBM

Speech-to-text Transcription of spoken conversations (Spanish only)

IBM


UniBrad

Rule-based semantic analysis A rule based method for text searches in the database using semantically related words, related to a given search keyword

UniBrad

Lexical semantic analysis Utilises a lexical resource to find semantically related terms for searches

UniBrad

MyPast Application • Presents the user interface functionalities in a GUI to the user and

• allows the user to retrieve and edit information that has been captured by the input processing components (as described in the deliverable D.5.1)

CURE/Unibrad


/ 28

Fingertips tracking for the Multi-Touch Surface

Track users’ fingertips to enable multi-touch surface interface

AIT

MyFuture Application Presents the user interface functionalities in a GUI to the user and allows the user to • set appointments • store notes as addition to appointments • edit appointments • create reminders manually and

automatically • synchronization with HERMES PDA

imports appointments from the PDA and exports appointments to PDA (within a defined time interval)

CURE/TXT

Cognitive Games Application

Cognitive Training Games Logic (as Defined in D6.1)

AIT

TTS HERMES speaks to Maria using synthesized voice

IBM

Mobile POI Manager • GPS integration • Possibility to add POIs basing on the

current location

TXT

Mobile device synchronisation

Synchronisation with the home-based workstation with focus on:

• Available audio transcriptions related to audio notes attached to appointments

• Audio recordinds for Conversation Support.

• New/Updated stored POIs (using the “locate mode”)

• Appointments (to and from the PDA and the home-based workstation)

• Audio notes attached to the appointments

• Pictures • People profiles

TXT

Mobile Shopping List Manager

Allow for the user to manage shopping lists (i.e. add, edit, delete), includes GPS integration (the user is alerted when close to the food store)

TXT

Data logging and mining for reminder generation

Analyses the data collected and analyses which elements provide useful knowledge for reminder generation.

AIT

Rule Engine and semantic query

Automatic reminders based on rules AIT

Speaker identification Speaker id tags are a part of the speech transcripts

IBM

Image/Video Tagging and Retrieval

Stores images/video with appropriate tags / Retrieves video segments based on queries

AIT


/ 28

Audio recorder • The mobile application allows the user to record the audio note and attach it to a chosen event (Appointment Dictation)

• The mobile application allows also to record a discussion (Conversation Support)

TXT

Hermes PDA Application

The mobile application allows the user to • insert/delete/edit/browse/notify

appointments (including information like datetime, notes, POIs, etc.)

• record discussions • attach audio notes to appointments • display post processed speech

transcriptions • take pictures • browse HERMES-registered people

profiles • manage shopping lists

TXT

Picture browser and searcher The mobile application enables to user to browse through all the faces by: • Picture • Category • Keyword (name)

TXT

Profile detail retrieval The mobile application presents the user with a form containing the person details.

TXT

Video summarisation Summarisation of videos through frame analysis

UniBrad

Table 3: Overview and description of technologies available in the second prototype At this stage in the project it is not possible to specify the exact evaluation plan for the second prototype. Reasons for this are that results of the evaluation of the first prototype will have an impact on the outcome of the second prototype. Therefore, a second iteration of the test-plan for the second prototype will be performed, presumably after the analysis of the results of the first user trial.


/ 28

4. User Recruitment and Timing The project will use two evaluation phases. This chapter describes the recruitment procedure and estimated planning of the two user evaluation field trials.

4.1 Recruitment In Austria the tests are planned to be carried out with 12 to 16 potential HERMES users and in Spain 8-10 potential HERMES users from the target group are expected to be involved. To be included in the user trials the following inclusion criteria have to be met by potential HERMES end users: • users are over 60 years of age, • are diagnosed with AAMI or MCI, • do not suffer from any severe sensorial and/or motor problems and • are living independently in their own homes. Additionally, the main target group of HERMES is people who have retired, however it is not a specific inclusion or exclusion criterion. Users will as far as possible be recruited from the pool of users that was generated during the requirement analysis phase of the project. We expect a 90% to 95% overlap with the persons selected during requirement analysis phase, in order to build upon the personalized system that is created and to allow personalized data to be revealed to the users as well as to allow adaptation to the speaker to be performed.


/ 28

4.2 Timing The HERMES user evaluation trials will take place in Vienna and in San Sebastian in accordance with the overall projects planning. The following table summarizes the actions planned for HERMES and their timings.

VIENNA Scheduled First Evaluation Q3 2009 Evaluation in lab (with 16 selected members of HERMES user group)

Q3 2009

Field test with HERMES PDA (4 members of HERMES user group)

Q3 2009

Interviews (with users, care personal and memory experts)

Q4 2009

Second Evaluation Q4 2010 Evaluation in lab (with 12 to 16 selected members of HERMES user group)

Q4 2010


Q4 2010

SAN SEBASTIAN Scheduled First Evaluation Q3 2009 Evaluation in lab (with 12 selected members of HERMES user group)

Q3 2009


Q3 2009

Interviews (with users, care personal and memory experts)

Q4 2009

Second Evaluation Q4 2010 Evaluation in the field (up to 8 extended homes; at this point we can not say in how many homes exactly we will evaluate HERMES prototype 2 )

Q4 2010

Field test with HERMES PDA (ideally with the users of the homes having HERMES installed)

Q4 2010

Table 4: Timing/Schedule of the Evaluation Lab and Field Trials


/ 28

5. Evaluation Procedure The definition of the evaluation procedure is essential for verifying how well the HERMES system delivers the expected functionalities according to its design specifications, matching to user requirements. Before testing the prototype with potential end users, a range of technical tests at component, sub-system and system level will be carried out to establish the degree to which the system is (mal)functioning and to ensure that the system delivers its functional requirements as described in deliverable D.3.3 (System verification plan). The evaluation procedure of HERMES with participants is described in this chapter.

5.1 Evaluation Procedure The user trials will be carried out in Austria and in Spain. All tests regarding the first prototype of HERMES main system will be carried out in the lab in Austria and in Spain. The HERMES PDA will be tested in the field and in the lab. While the focus in Austria lies on the general interaction, the evaluation of HERMES with Spanish users also includes the speech components. The system will be personalized for each study participant by populating HERMES with personal data for each person. This data has been gathered during the requirements analysis phase. Even though this data does not contain real-life data, it is personal in the sense that participants have already once experienced the moments stored in the system, which is crucial in order to evaluate such a personal system like HERMES. Even if this data is the data from one year ago, it provides a certain baseline for searching and retrieving information from the store, visualized by the HERMES system. This is the basis for the setup of the further tests. Once the study participant arrives in the lab he or she will receive an explanation about the HERMES prototype. Instructions in the form of tasks will be carried out by study participants to develop a feeling for the system and its functionalities, to evaluate user acceptance, usability and address potential improvements of the system. Tests will be carried out in the presence of a researcher to ensure that assistance for participants can be given at any time. Questionnaires will be kept consistent between the two test locations to compare results and determine differences based on cultural background. Additionally interviews with experts in the cognitive domain and care givers will be conducted. As described above, the prototype evaluation focuses on the first iteration of the applications that aggregate the information from the underlying sensory components. These applications provide the front-ends towards the user and allow the user to judge over user acceptance of the application as well as the underlying technologies through the lab and field trials. The evaluation targets are described in Table 5: Usability Evaluation of the three main front-ends (MyPast, MyFuture, PDA) MyPast Application

Usability User Acceptance (Personal system) Information Visualization (Audio/Video) Interface complexity

MyFuture Application Usability


/ 28

User Acceptance

PDA Application

Usability User Acceptance

Cognitive Game Evaluation of the cognitive game Cognitive Game Experience

Heuristic Evaluation Game Evaluation

Table 5: Aspects under Evaluation The evaluation-by-application-front-end makes it possible to evaluate the acceptability/quality of the underlying core components of the system without overburdening the user. These applications summarize all features that are offered for the first prototype by the underlying integrated system mapped in Table 2. The prototype evaluation focuses on the first iteration of the applications that aggregate the information from the underlying sensory components. These applications provide the front-ends towards the user and allow the user to judge over user acceptance of the application as well as the underlying technologies through the lab and field trials. In order to keep the total size of the evaluation manageable for the study participants, the whole study is broken up in three parts. 1. Focus on the evaluation of the MyPast, Myfuture and Cognitive Games 2. PDA Evaluation at home 3. User Acceptance and Information Visualization Evaluation This setup allow the user to ‘rest’ in between the three parts, but also allow the user to record some personalized information in their own environment in order to have additional personal material for each user in the system in order to create a second evaluation with respect to user acceptance based on more data. Also, the information visualization aspects become more challenging when more data is provided in the system. The focus of the usability tests carried out in Vienna will be on the user experience and accessibility of functionalities. That is due to the fact that speech technologies in HERMES are developed with a focus on Spanish. The following sections describe the evaluation of the different HERMES applications broken down to task level. Some of the tasks can not be used before the completion of the second HERMES prototype (e.g. emotion search).

5.2 Evaluation of myPast Last year the evaluation of HERMES MyPast front-end was carried out in INGEMA with 8 users. Based on this first evaluation the user interface has been altered drastically in order to support users’ further requirements. The following tasks describe the evaluation of the front-end.


/ 28

Starting exercises 1. Please open the application “MyPast”. 2. Please browse through the application for 1 minute and try to understand what you can do

with this application. Think out loud while doing it. Tasks 1. Please go back to the start page 2. Please use the system to find entries that took place between January and June 2008. (time

search) 3. Please find the first entry available in the system that took place at the doctor’s office. 4. Please find the entries associated with the grandchildren Mary and David (photo search).

Where was the last entry with David recorded? 5. Please find all entries that took place in the park. How many entries do you find in total? 6. Please find joyful entries between June 2008 and August 2008 where one of the children

Angie or Paul is involved. How many entries do you find? Note: Dates, persons and figures in the tasks are personalized and replaced with personal data relevant to the user in each case will be used to evaluate HERMES and therefore no fixed numbers can be specified.. After each task the following questions will be asked. • How well did the system support you in solving the task? • What would you personally change in the system in order to make it more supportive?

5.3 Evaluation of the MyFuture Front-End Starting exercises 1. Please open the application “MyFuture”. 2. Please browse through the application for 1 minute and try to understand what you can do

with this application. Think out loud while doing it. Tasks 1. Please imagine you have a meeting with a friend next week. You want to save the

appointment in HERMES and record a note for it. Can you try to do that? 2. Browse the entries in “MyFuture” and tell me what appointments you have set for the next

week. Are there notes added to the appointments? 3. Choose the appointment on the 12th of August, read the associated audio note and edit it. 4. Record a new note for the appointment at the 12th of August. 5. Check if you can find the entries that you stored using the HERMES PDA (once the PDA

gets near HERMES synchronization will happen automatically) 6. Spain-specific: Use the search interface for processed audio, i.e. search for single word,

search for stemmed word and use a Boolean-style search. 7. Spain-specific: Quiz to assess whether the user can retrieve person-specific information, i.e.

“During the conversation you’ve been asked <question>. What have you answered?” The user receives multiple choice answers.

After each task the following questions will be asked • How well did the system support you in solving the task?


/ 28

• What would you personally change in the system in order to make it more supportive? After all tasks with the MyPast interface, the following questions are asked: • Was it fun using the system? Why or why not? • What would you improve in this system? Why or why not? • Would you trust this system with your personal photos? • Which information inside your living room would you want to have recorded in such a

system? Why? • Which information would you NOT store in such system? Why? • If you would own such a system, would you show it to other people? Why or why not? • Would you browse the system together with other people? Why or why not? Evaluating interface complexity of the MyPast application Section 2.2 describes the impact interface complexity of systems can have and metrics to measure it. Section 2.2.1 describes additional metrics useful to measure interface complexity: • number of options presented on the interface • number of hierarchical levels present on the interface • amount of information being displayed and presented to user

(organization of information, easy access to information, adequacy of information levels) For the evaluation two types of interfaces are developed in conjunction to the functioning prototype that is presented to the user, for the HERMES MyPast application. These interfaces will be consistent regarding color schemes and symbols - the above described characteristics are varied. Test subjects are asked to complete a simulated scenario using the 3 different interfaces. After completion of each scenario test subjects will be asked to fill out a questionnaire (with 5 point Likert-Scale) and will be interviewed (based on the observation of test conductor) Performance results under the different interfaces, questionnaire and interview results provide us with an understanding about the proposed metrics influence on older adults as well as provide further input to improve the functional interface in the given context.

5.4 Evaluation of Cognitive Games For the evaluation of the cognitive games with users, EMG and GSR measures are taken throughout the evaluation of the games. These measures provide information about the arousal and emotional state of the user with respect to the interaction with the system, however controlled circumstances are required in order to evaluate. After the sensors for the EMG and GSR are attached, the game interaction mechanisms are explained to the user and the user is asked to start the game and play it for 10 minutes. Questions after playing the game: • Compared to the cognitive training you know, how well do you think this system can

support you with the training of your cognitive abilities? • What would you personally change in the system in order to make it more supportive? • Can you imagine performing your daily cognitive training with HERMES?


/ 28

5.5 Evaluation of the HERMES PDA For the evaluation of the PDA, a subset of four study participants will receive the PDA at the end of the evaluation of the cognitive games to take it home for one week. By giving the HERMES PDA to participants the participants can further content for the system and provide real life data about the usage of the PDA The following tasks are given as “take-home exercises” for participants: • Please use this PDA in the following week as often as you can for doing the following

things: o Whenever you have to write down an appointment use the PDA for it o Use the GPS functionality to add a location reminder o Use the possibility to record an audio note as an addition for an entry

• Whenever you experience troubles give us a call so we can support you • Write each day in the evening a text message (SMS). In the SMS please describe your

feelings and thoughts towards the HERMES device in a short form – no need for whole sentences. Additionally you can use the possibility of recording audio notes if you want to store lengthy comments and ideas regarding the HERMES system on the device.

• When the participant returns to the office after one week, the PDA will synchronize with the HERMES system. Participants are then asked to search for entries they made using the PDA in the synchronised system.

5.6 Information visualization After users return to the lab with the PDA, three different ways to visualize audio and video data are evaluated with regard to the perceived meaning, the perceived usefulness and the perceived ease of use in the given context with users – at their second visit in the office. Participants are confronted with the visualizations and asked to explain the perceived meaning. The perceived usefulness and the perceived ease of use in the context of HERMES are evaluated by means of questionnaire for the three variants of the visualized data.

5.7 User Acceptance Questionnaire At the end of the PDA study – as users return to the office for returning the PDA and interact one more time with the home system – they are presented with a questionnaire that aims to assess user acceptance based on the UTAUT model, which describes the following four constructs contributing to technological acceptance: • performance expectancy • effort expectancy • social influence • facilitating conditions Each of these constructs will be assessed in the questionnaire that the user will fill out after having been acquainted to all aspects of the HERMES system. Gender, age, experience, and voluntariness of use are affecting directly the four key constructs’ and will therefore be collected in conjunction in the evaluation.


/ 28

6. Evaluation of the second prototype Within the HERMES project we are interested in finding out how the interaction of older adults with the developed personal system takes place. To get insight in this we not only want to conduct tests in the lab, but rather deploy the system in its natural context, in the home of some potential users. Apart from conducting tests in the field and in the lab we will be in the position to have incorporated results from the first user trial. The second user trial is scheduled to take place between Month 31 to Month 34. This user trial will be performed with 12-16 participants in Austria and 8 extended homes in Spain. In this user trial usability, learnability, user experience and acceptability will be assessed based on the second prototype which is defined by the technology described in section 3.2. In broad lines, the evaluation will follow the same general procedure as the evaluation of the first prototype, with the large difference that the evaluation will partly take place in the real homes of people. As a second difference, the evaluation will include to a certain extent the ‘extended home’, which is defined as the person including the people in their direct surroundings, focusing on the social interaction and context of interaction with technology and how this context of use relates to the various aspects that are to be evaluated, i.e. usability, learnability, user experience and acceptability. The trial and assessment plan for the second prototype will largely depend on the shaping of the HERMES system determined by the first evaluation of the users. Depending on the way users prioritize the functionality and elements that are part of the first prototype, we will see if we need to re-scope these elements for the second prototype, and therefore adapt the trial plan according to these emerging needs in an iterative fashion. Where the measures will be largely the same, the interaction context differs substantially which will yield additional results, and in turn (due to user feedback on the first prototype) also the interface will be adapted and changed and therefore lead to different results but potentially also different methodologies required based on the initial reaction to the prototypes. To be able to adapt for these changing needs, we aim to iterate the D.7.1. deliverable to the extent that the testing and evaluation procedures described in this document to cover the first prototype evaluation in order to optimally fit the second prototype evaluation. With the second user trial scheduled for Month 31 (i.e. July 2010), the evaluation plan for the second field trial will be delivered in the Month 27 (March 2010).


/ 28

7. Data used in the Evaluation HERMES is a very personal system. It is intended for use in the private domain and as such we strive to carry out tests with personal data. It is unnecessary to say that results from the evaluation will have more meaning this way. We envision populating the HERMES database for each study participant with personal data from the requirements analysis we carried out in the first phase of the project. Also, some of the study participants will be given the HERMES PDA for a certain amount of time. After that we will use the data gathered for further evaluation purposes. To ensure the ethical soundness of the user evaluation, the ethical advisory board will review the deliverable D.7.1 as well as the informed consent form related to the evaluation. In a meeting of the ethical advisory board, comments and advice are discussed with the organizations conducting the field trials.


/ 28

References [1] Csikszentmihalyi, M. The Meaning of Things: Domestic Symbols and the Self.

Cambridge: Cambridge UP, 1981

[2] Venkatesh, V., Morris, M.G., Davis, F.D., and Davis, G.B. “User Acceptance of Information Technology: Toward a Unified View” MIS Quarterly, 27, 2003, 425-478.

[3] Dillon, A. (2001) User Acceptance of Information Technology. In W. Karwowski (ed). Encyclopedia of Human Factors and Ergonomics. London: Taylor and Francis.

[4] Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to theory and research. Reading, Mass. ; Don Mills, Ontario: Addison-Wesley Pub. Co.

[5] Rogers, Everett M. (1962). Diffusion of Innovations, Glencoe: Free Press, Ch. 7.

[6] ALAVI, M. and JOACHIMSTHALER, E. A. (1992). Revisiting DSS Implementation Research: A Meta-Analysis of the Literature and Suggestions for Researchers. MIS Quarterly. 1992; 16(1): 95-116.

[7] Nasib S. Gill, Few Important Considerations For Deriving Interface Complexity Metric For Component-Based Systems. Software engineering notes, 2004

[8] Martelli, S.,Nofrini,L., Vendruscolo,P.and Visani,A (2003) Criteria of interface evaluation for computer assisted surgery systems, International Journal of Medical Informatics, 72, 35-45.

[9] Coskun E. & Grabowski M. Impacts of User Interface Complexity on User Acceptance in Safety-Critical Systems. Proceedings of the Tenth Americas Conference on Information Systems, New York, New York, August 2004

[10] Dix, A., Finley, J., Abowd, G., and Beale, R. (1998) Human-Computer Interaction 2nd edition, Prentice-Hall,Upper SaddleRiver, NJ.

[11] ESA (2008) Essential Facts about the Computer and Videogame Industry, 2008 Sales, Demographic and Usage Data. Entertainment Software Association, http://www.theesa.com/facts/pdfs/ESA_EF_2008.pdf. Accessed 07 December 2008

[12] Köffel Ch. et al. Evaluating User Experiences in Games: Concepts and Methods, Springer, In Press

[13] Federoff MA (2002) Heuristics and usability guidelines for the creation and evaluation of fun in videogames. Master’s thesis, Department of Telecommunications, Indiana University

[14] Desurvire H, Caplan M, Toth JA (2004) Using heuristics to evaluate the playability of games. In: CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24-29, 2004). CHI '04. ACM, New York, NY, 1509-1512

[15] Hazlett, R.L. (2006). Measuring Emotional Valence during Interactive Experiences:

Boys at Video Game Play. Proceedings of CHI 2006 Conference on Human Factors in Computing Systems, Montreal, ACM Press, 1023-1028. DOI Link: http://doi.acm.org/10.1145/1124772.1124925


/ 28

[16] Potter Ralph K., George A. Kopp, Harriet C. Green, Visible Speech, D. Van Nostrand Co., NY, 1947

[17] Koenig, Walter K., H.K. Dunn, L.Y. Lacey, “The Sound Spectrograph,” in JASA, Vol. 18, p. 19-49.

[18] Donath, J., Karahalios K., Viegas Fernanda. Visualizing Conversation, MIT Media LAB, Proceedings of the Thirty-Second Annual Hawaii International Conference on Systems Sciences," 1999

[19] Regan L. Mandryk, M. Stella Atkins and Kori M. Inkpen, A continuous and objective evaluation of emotional experience with interactive play environments, in: CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems, Montr&\#233;al, Qu&\#233;bec, Canada, pages 1027--1036, ACM Press, 2006

[20] Partala T., Surakka V., and Vanhala T. Person-Independent Estimation of Emotional Experiences from Facial Expressions. IUI’05, 2005

[21] Richard L. Hazlett, Measuring emotional valence during interactive experiences: boys at video game play, in: CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems, Montr&\#233;al, Qu&\#233;bec, Canada, pages 1023--1026, ACM Press, 2006

[22] RD Ward and PH Marsden, Physiological responses to different WEB page designs (2003), in: International Journal of Human-Computer Studies, 59:1-2(199--212)

[23] IJsselsteijn, W.A., de Kort, Y.A.W., & Poels, K., The Game Experience Questionnaire:

Development of a self-report measure to assess player experiences of digital games, Manuscript in preparation

d.7.1 user evaluation planfp7-hermes.eu/uploads/media/deliverables/hermes_d... · criteria for...

Documents