Usability Test Plan for HolidayInn.com.cn

Holiday Inn Hotel Website for Chinese Speaking Users

Prepared for: Karen Bennett, Project Sponsor User Experience Manager at IHG

Cc: Dr. Carol Barnum, Project Advisor

Completed by: Team CBR Yufei Duan Yina Li Ying Li Qianying Liu Niven Sellars Michael Somer

Date: March 02, 2008

TABLE OF CONTENTS

PURPOSE......................................................................................................................................................................3

PROBLEM STATEMENT AND TEST OBJECTIVES...........................................................................................4

USER PROFILE FOR U.S.-BASED PARTICIPANTS............................................................................................6

METHODOLOGY AND TASKS/SCENARIOS.......................................................................................................7

NUMBER OF PARTICIPANTS........................................................................................................................................7LENGTH OF SESSIONS.................................................................................................................................................7SEARCH GROUND RULES............................................................................................................................................7TEST PROCEDURES.....................................................................................................................................................7

Overview/Briefing (10 minutes)............................................................................................................................7Scenario 1 (5 min).................................................................................................................................................7Scenario 2 (15 min)...............................................................................................................................................8Scenario 3 (5 min).................................................................................................................................................8Scenario 4 (5 min).................................................................................................................................................8Scenario 5 (10 min)...............................................................................................................................................8Scenario 6 (optional) (10 min)..............................................................................................................................8Closing (5-10 minutes)..........................................................................................................................................9

TESTING FACILITY................................................................................................................................................10

Evaluation Room.................................................................................................................................................10Control Room......................................................................................................................................................10Executive Viewing Room.....................................................................................................................................10

DATA COLLECTION METHODS.................................................................................................................................11Quantitative Data................................................................................................................................................11Qualitative Data..................................................................................................................................................11

PROJECT DELIVERABLES..........................................................................................................................................11

APPENDIX A: SCREENER QUESTIONNAIRE...................................................................................................13

APPENDIX B: PRE-TEST QUESTIONNAIRE.....................................................................................................14

APPENDIX C: POST-TASK QUESTIONNAIRES................................................................................................15

APPENDIX D: POST-TEST QUESTIONNAIRE………………………………………………………………….. 18

2

PURPOSE

The purpose of the Chinese HolidayInn.com.cn usability test is to collect feedback about how users use the Holiday Inn online hotel booking site in China, what problems they may encounter using the site and what improvements they would like to see to make it easier to book a hotel room.

Ms. Karen Bennett, the User Experience Manager of InterContinental Hotels Group, is the project sponsor. She would like to learn about the user’s experience on the Chinese web site, as compared to their experience with a competitor web site: www.elong.com If time permits, she would also like to know how their experience compares with www.holiday.com, the U.S. booking site.

This test plan describes: The problem statement and test objectives The types of users to recruit for testing The testing methodology and proposed tasks/scenarios The test setup in the Usability Center Plans for data collection and reporting Project deliverables The estimated project timeline Questionnaires and other materials to be used in testing, in appendices:

o Appendix A: Participant screening questionnaireo Appendix B: Pre-test questionnaireo Appendix C: Post-task questionnaireo Appendix D: Post-test questionnaire

3

PROBLEM STATEMENT AND TEST OBJECTIVES

This usability test of the Chinese Holiday Inn website will provide qualitative and quantitative data addressing IHG’s interests in understanding the Chinese user’s experience. We will assess the users’ experience with booking a room, as well as their overall satisfaction with the site. Tasks for testing www.HolidayInn.com.cn include:

The general feeling/layout of the site: Does the layout suggest the route first-time users will take to book a hotel?

The procedure of booking a hotel online: Basic search: Is it easy to use? Advanced search: Can users accomplish their goals on the advanced

search screen? Entering personal information: Does the website require reasonable and

suitable information for Chinese users? Do users understand all information requirements?

Language: Do users understand all the wording on the website? Are there any translation mistakes or misunderstandings?

Information in confirmation email: How quickly can users receive the confirmation email after booking a hotel online? Does the confirmation email contain enough information to suit users’ needs?

Perceived reliability of the site: Do users trust the website? Do they fill out the personal information readily?

Navigation: Can users find the most efficient navigation when they book a hotel or browse the website?

Satisfaction: Which aspects do users like and which aspects do they dislike?

These questions were devised using information supplied by our sponsor and our assessment of potential usability problems resulting from our heuristic evaluation of the site (submitted previously).

This usability study will be designed to allow us to extract mostly qualitative data. The focus of the study will be task-oriented and directed toward how the user subjectively responds to the issues listed above. Participants will be given a pre-test questionnaire to get information about their hotel booking experience and expectations, followed by scenarios that direct them to perform specific tasks. Post-task questionnaires after each scenario, and a post-test questionnaire are designed to obtain detailed and specific feedback about www.HolidayInn.com.cn features.

4

Users will be scheduled in one-hour sessions to include pre-test and post-test questionnaires. Users will be given 40 minutes to complete five scenarios. If participants complete the scenarios in less than 40 minutes, we have designed an extra scenario to obtain additional feedback.

The test will be conducted in a full-scale usability lab at Southern Polytechnic <www.usability.spsu.edu>. Logging and recording of sessions will be done using Morae and also with backup recordings on DVD. The participant computer will have

Microsoft’s Chinese character-based language installed.

5

USER PROFILE FOR U.S.-BASED PARTICIPANTS

To recruit prospective test participants who represent the site’s actual users, the project team identified two primary users groups and created two descriptions of those users. Personas for these two user profiles were submitted previously.

Since the situation is that of testing the Chinese website within the United States, we will use only one of the user groups/personas, which will allow us to recruit these users to participate in the study in the Usability Center.

The characteristics of the U.S.-based user group, along with other questions about prior experience, behaviors, and other criteria, will make up the screening questionnaire (Appendix A) for identifying suitable test participants.

A general description of the user characteristics is as follows:Must be able to speak and read Chinese as well as EnglishFirst language must be ChineseTravel purpose: either for business or for pleasureAge: 24-55Gender: female and maleMust have experience booking a hotel room online Must use the Internet at least 5 hours per weekMust not have any prior experience with the Chinese Holiday Inn website

METHODOLOGY AND TASKS/SCENARIOS

6

This section describes the testing methodology and scenarios we will use with six participants in one-hour sessions.

Number of ParticipantsWe plan to test a total of six participants: three who are traveling for business and three for pleasure. To ensure an adequate number of participants, we plan to recruit eight users total based on our screener; two out of the eight are back-up participants.

Length of Sessions The total length of each session will be an hour; including:

Welcome and pre-test questionnaire: 10 minutesTask Scenarios: 40 minutesPost-test questionnaire: 10 minutes

The estimated time for each scenario is noted below.

Test Procedures

Participants will not be allowed to use any resources that are not available on the Holiday Inn site. (IE: No use of Google or other outside sources.) This restriction will not be stated in the moderator’s introduction, so as not to encourage the participants to think about outside resources. However, if participants want to use these resources, the moderator will notify them that such action is not allowed for testing purposes.

Overview/Briefing (10 minutes)The moderator will welcome the participant and have him/her sign the video consent form. Next, the moderator will explain the facilities and ask the participant to think out loud. Also, the moderator will provide the participant with the pre-test questionnaire (Appendix B).

Scenario 1 (5 min)You and your spouse are thinking of going back to China for the Summer Olympics. You heard that the Holiday Inn has several locations in Beijing. You will visit the Chinese Holiday Inn website. (This will be minimized on the desktop.) Take a moment to look at the home page without clicking on anything. After you have familiarized yourself with the home page, tell us: What do you think you would do on the site to book a hotel room?

Scenario 2 (15 min)You are going to book a hotel room for the trip back to China. You want to reserve one hotel room with one bed for two people in Beijing. You will stay from July 20th to

7

July 25th. Here is the information you need to book the room:

First Name: Jing Last Name: Li Reserve for you and your spouse. Location: Beijing Check in date: July 20, 2008 Check out date: July 25, 2008 Credit Card Type: Visa Credit Card Number: 1234 2345 3456 4567 Credit Card Safe code: 890 Credit Card Expiration Date: 10/2009 Email: holiday@gmail.com

  Scenario 3 (5 min)You just answered a phone call from your relative who lives in Beijing. She wants you and your spouse to stay at her house during the visit. You accept the invitation, so you must now cancel your Holiday Inn reservation.

Scenario 4 (5 min)You have some relatives in Changchun, Jilin province. You want to visit them during your trip to China. See if there is a Holiday Inn available in Changchun from July 26-30. Scenario 5 (10 min)Go to the competitor site www.elong.com. See if a hotel room is available for you and your spouse between July 21 and 25 in Beijing. (Note: You are not going to actually book a hotel room on this website.) Look at the search results page and tell us:

What do you think of this site? What do you like about it? What do you dislike about it?

Scenario 6 (optional) (5 min)Go to the Holiday Inn U.S. website (www.holiday.inn.com).

Take a moment to look at the home page without clicking on anything.

Tell us what you think of this website, compared to the Holiday Inn China and eLong websites.

Closing (5-10 minutes)Moderator will discuss the experience with the participant and ask the participant to complete the post-test questionnaire (Appendix E).

8

TESTING FACILITY

The test will be conducted at the Southern Polytechnic Usability Center located in the Atrium Building on the campus, in rooms J-160-J163. The Usability Center includes three rooms:

Evaluation RoomThe evaluation room is where participants will complete tasks. The room has the following equipment:

Lenovo PC (Windows XP, campus network connection)Microsoft Chinese character-based language installed for this studyKeyboard and mouse16” Monitor3 wall-mounted cameras and a Logitec digital desktop cameraMicrophoneTelephone/intercom to control room

Control RoomThe control room is where the team will observe and record the test. There is a one way mirror between the evaluation room and the control room. The control room has the following equipment:

A logging computer with Morae logging softwareMonitors which display what the cameras are capturing in the evaluation roomHeadsets for communication among the team membersVideo recording and editing equipmentA picture-in-picture digital mixing boardA phone/intercom for the participant to call the help deskLighting controls

Executive Viewing RoomThe executive viewing room provides opportunities for additional observers from IHG to directly observe the test without interrupting the team. This room also serves as the place for team meetings, as well as the final presentation. The executive viewing room has the following equipment:

Lenovo PC (Windows XP and Microsoft Office 2007)2 desktop monitors, which display picture-in-picture viewing of the sessionA large screen with LCD projector to display picture-in-picture viewing of the sessionWhiteboard for notes during sessionsConference room table and chairsNumerous data ports for laptop internet accessWireless internet access

9

DATA COLLECTION METHODS

Data collection will be a combination of quantitative measurements and subjective observations. The quantitative and qualitative data measures are listed below.

Quantitative DataThe following data will be logged during testing sessions:

Time to find an available hotel Number of participants completing each task Number of problems encountered during each task Number of participants that fail to complete each task

From questionnaires, data will be collected regarding participants’ assessments of: The level of difficulty in booking a hotel room on the site The level of confidence using the site to book a room The level of reaction to the appeal of the site

Qualitative Data

Non-verbal and verbal feedback and observations will be collected, including: Facial expressions, such as:

o Confusiono Uncertaintyo Pleasure

Verbal comments when they think out loud, such as:o Complainingo Liking

Comments/questions, such as:o Asking questionso Calling for help

10

PROJECT DELIVERABLES

The findings will be presented to the sponsor and the course instructor in the following ways:

o a formal reporto a team presentation, includingo video highlights of the top positive and negative findingso electronic files of the session logso DVDs of the recorded sessions

These project deliverables will be presented and delivered on May 1.

11

APPENDIX A: SCREENER QUESTIONNAIRE

Name: (please print): _______________________________________________

Age: ____________ Gender: ___ Male

___ FemaleOccupation: ___________________

1. How often have you have booked a hotel online in the past year? Never (disqualify) 1-3 times More than 3 times

2. Who makes most of your hotel reservations? I book the hotel myself Someone books it for me (disqualify)

3. How often do you use the Internet? Less than 2 hours per day 2-5 hours per day More than 5 hours per day

4. What do you mainly use the Internet for? Circle all that apply Entertainment Information / Research E-Mail Social Networking Other______________________

5. How many nights have you spent in a hotel in the last six months? ____________

6. How many of these nights were for Business__________ Pleasure__________

7. What websites have you used the most to book hotel reservations?

8. Finish the following sentence: When it comes to providing my credit card number online:

I do not mind I will do it if absolutely necessary I don’t give out credit card information online

12

APPENDIX B: PRE-TEST QUESTIONNAIRE

Thank you for taking the time to participate in our usability test. Your participation will help to provide valuable feedback about your experience using a hotel’s website.

Before we begin, we would like you to fill out this brief questionnaire, so that we will have more information about your experience using the Internet and, more specifically, about your experience making travel arrangements.

Name: (please print): _____________________________________

1. What features impress you the most when you use a hotel’s website?a. b. c.

2. What features frustrate you the most when you use a hotel’s website? a. b. c.

3. When booking a hotel room, what information is most valuable to you:

When booking a business stay:

When booking a vacation stay:

4. Which of the following two typical booking activities do you prefer?

□ I use one or two preferred hotel websites that I’m familiar with □ I browse different hotel websites to have as much information as possible

13

APPENDIX C: POST-TASK QUESTIONNAIRES

Scenario 1: Look and feel/first impressions

Please rate the following: Very Easy

Easy Neither Easy nor Difficult

Somewhat Difficult

Very Difficult

Understanding the site navigationUnderstanding the site terminology (words, links)Understanding the first step you need to do to book a hotel room

Scenario 2: Book a room in Beijing for 2 people

Please rate the following:Very Easy

Easy Neither Easy nor Difficult

Somewhat Difficult

Very Difficult

Overall ease or difficulty of this taskUnderstanding the procedures of booking a hotel roomUnderstanding the search results page

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

Scenario 3: Cancel Reservation

14

Please rate the following:Very Easy

Easy NeitherEasy nor Difficult

Somewhat Difficult

Very Difficult

This task was . . .

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

  Scenario 4: Look for hotel in Changchun

Please rate the following:Very Easy

Easy Neither Easy nor Difficult

Somewhat Difficult

Very Difficult

This task was . . .

What aspects of the site made this task easy for you?

What aspects of the site made this task difficult for you?

Scenario 5: Competitor site (elong.com)Please rate the following:

Very Easy

Easy Neither Easy nor Difficult

Somewhat Difficult

Very Difficult

This task was . . .

What aspects of the site made this task easy for you, particularly as compared to Holiday Inn China’s website?

15

What aspects of the site made this task difficult for you, particularly as compared to Holiday Inn China’s website?

Scenario 6 (optional): Holiday Inn U.S. site

Please rate the following:Very Easy

Easy Neither Easy nor Difficult

Somewhat Difficult

Very Difficult

This task was . . .

What aspects of the site made this task easy for you, particularly as compared to Holiday Inn China’s website?

What aspects of the site made this task difficult for you, particularly as compared to Holiday Inn China’s website?

APPENDIX D: POST-TEST QUESTIONNAIRE

1. How easy or difficult was it to use the Holiday Inn China website? (Please circle the letter that best reflects your feelings.)

16

a. Very Easy.b. Moderately Easyc. Neither Easy nor Difficultd. Difficulte. Extremely Difficult

Please explain your answer in the box below.

2. Did you notice any on-screen instructions during the procedure of booking a hotel room on the Chinese Holiday Inn site? If yes, answer question 3. If no, skip to question 4.

3. If you noticed some instructions, what do you think of them?a. They helped me through the whole process.b. They were helpful when I needed it.c. I noticed them but I did not need to use any on-screen help.d. I read the instructions but they were hard to understand.e. I read the instructions, but they did not help at all.

Please explain your answer in the box below.

4. On a scale from 1 to 10, with 10 being the most confident, how confident do you feel after using the website that you will be able to book a hotel reservation with this website in the future?

17

Please write a number from 1 to 10: __________

Please explain your answer in the box below.

5. Rate you reaction to the look and appeal of the website? 1. Very appealing 2. Somewhat appealing3. Neutral4. Not very appealing5. Not appealing at all

Please explain your answer in the box below.

1. Were there any tasks that you found extremely difficult to perform? If so, what made them difficult?

18

We value and appreciate the time you have given us today. The following questions are optional. If you have time to answer them, we would greatly appreciate any feedback that you can provide us with.

2. Were there any aspects or features of the website that you found enjoyable and/or made completing a task easy?

3. If you have any additional comments about the website that you wish to share, please feel free to do so in the space below.

19