evaluation of a computer game menu interface during … · evaluation of a computer game menu...

Evaluation of a computergame menu interface during

the design process

Goran Lundin

February 24, 2008Master’s Thesis in Computing Science, 30 credits

Supervisor at CS-UmU: Hakan GullikssonExaminer: Per Lindstrom

Umea UniversityDepartment of Computing Science

SE-901 87 UMEASWEDEN

Abstract

The Company EA Digital Illusions is currently developing the next game in the Bat-tlefield franchise, this thesis is a part of that work. The next Battlefield game aimsat attracting a new, younger, and less experienced in computer games, target popula-tion. To attract these users it has been acknowledged that usability is important in thecomputer game. During this study the menu interface of the computer game has beendesigned. During the design process three studies have been run to ensure the usabilityof the system. First a Card sorting study was run, which after some design work wasfollowed by two usability studies.

This work has led to the design and creation of the interface as a Flash mock up andthe design will in time be implemented and included in the game.

Contents

1 Introduction 11.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Card sorting 32.1 The technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Pros and cons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 When to conduct a Card sort . . . . . . . . . . . . . . . . . . . . . . . . 42.4 Choosing participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.5 Preparing the test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.5.1 Card content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5.2 The cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.6 Conducting the test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.7 Analyzing the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.8 Web based Card sort - Optimal sort . . . . . . . . . . . . . . . . . . . . 7

3 Usability Testing 113.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3.1 Exploratory tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3.2 Assessment tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.3 Validation tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3.4 Comparison test . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4 Choosing participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.5 Preparing the test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.5.1 The test plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.5.2 Test material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.6 Conducting the test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.7 Analyzing results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.8 Developing recommendations . . . . . . . . . . . . . . . . . . . . . . . . 23

iii

iv CONTENTS

3.9 Morae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Exploring the Information Layout of a Game Menu 254.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.2 The study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2.1 Preparing the test . . . . . . . . . . . . . . . . . . . . . . . . . . 264.2.2 Choosing participants . . . . . . . . . . . . . . . . . . . . . . . . 274.2.3 Conducting the test . . . . . . . . . . . . . . . . . . . . . . . . . 274.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2.5 Analyzing the results . . . . . . . . . . . . . . . . . . . . . . . . . 284.2.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Evaluating the Design of a Game menu 315.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Common sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.2.1 Preparing the tests . . . . . . . . . . . . . . . . . . . . . . . . . . 335.2.2 Conducting the Studies . . . . . . . . . . . . . . . . . . . . . . . 34

5.3 The First Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.3.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3.2 Choosing participants . . . . . . . . . . . . . . . . . . . . . . . . 355.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3.4 Analyzing the results . . . . . . . . . . . . . . . . . . . . . . . . . 395.3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.4 Redesigning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.5 The Second Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.5.1 New design prior to the second study . . . . . . . . . . . . . . . 435.5.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445.5.3 Choosing participants . . . . . . . . . . . . . . . . . . . . . . . . 445.5.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.5.5 Analyzing the results . . . . . . . . . . . . . . . . . . . . . . . . . 485.5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.6 Redesigning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6 Conclusions 53

7 Acknowledgments 55

References 57

List of Figures

2.1 Design process diagram (parts of the figure have been removed) . . . . . 42.2 The Optimal Sort User Interface, no cards dragged into a category . . . 82.3 The Optimal Sort User Interface, one card dragged into a category . . . 8

3.1 The development life cycle including when each type of test is appropriate[13] 133.2 Curve showing usability problems found in comparison to number of users

tested . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.1 The interface main page after the first design iteration . . . . . . . . . . 325.2 The closet page of the appearance section . . . . . . . . . . . . . . . . . 325.3 Task score sorted by task, all participants included. 0 = completed with

ease, 1 = completed with difficulty, 2 = failed to complete . . . . . . . . 365.4 Time on task for all tasks and participants . . . . . . . . . . . . . . . . . 375.5 Mean completion time for all tasks . . . . . . . . . . . . . . . . . . . . . 385.6 Mouse clicks per task for all participants . . . . . . . . . . . . . . . . . . 385.7 Design of main page after the first usability study . . . . . . . . . . . . 425.8 My stuff appearance page after the first usability study . . . . . . . . . 435.9 Task score sorted by task, all participants included. 0 = completed with

ease, 1 = completed with some difficulty, 2 = completed with difficulty,3 = failed to complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.10 Mean task score sorted by task, all participants included. 0 = completedwith ease, 1 = completed with some difficulty, 2 = completed with diffi-culty, 3 = failed to complete . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.11 My Stuff Appearance section after the second study . . . . . . . . . . . 50

v

vi LIST OF FIGURES

List of Tables

2.1 Characteristics of card sorting . . . . . . . . . . . . . . . . . . . . . . . . 5

4.1 Which of the following age groups do you belong to? . . . . . . . . . . . 274.2 Hours of first-person-shooter games played a week . . . . . . . . . . . . 284.3 Visited communities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.1 Participant characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 355.2 Posttest questionnaire results . . . . . . . . . . . . . . . . . . . . . . . . 395.3 Participant characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 445.4 Posttest questionnaire results . . . . . . . . . . . . . . . . . . . . . . . . 47

vii

viii LIST OF TABLES

Chapter 1

Introduction

Digital Illusions Creative Entertainment (DICE) is one of the world´s most famous gamestudios[3] and the largest in Sweden. The company has developed games in differentgenres like pinball games and rally games. But the game that made the studio famousis Battlefield 1942, the game really put the studio on the map. Battlefield 1942 wasfollowed by titles like Battlefield 2 and Battlefield 2142. The Battlefield franchise hasmade the studio respected among gamers and game developers world wide.

The typical gamer that play a Battlefield game does so for a couple of hours a day andcould be considered a hardcore gamer. In an attempt to attract new groups of playersto the Battlefield franchise DICE is developing a new game directed at more casualgamers. To succeed in this it has been acknowledged that usability is an importantpart of the development. This thesis is a part of that work and aims at improving theusability of the user interfaces and the game. This will be done by conducting usabilitystudies on the interfaces of interest and from these results of the tests making changesand improvements to the user interfaces.

1.1 Problem Description

A part of the work to achieve high usability the development team at DICE want userstudies to be conducted and that the results of these studies will be made into a reportspresenting the findings and suggesting changes to the current design. The main focusof the study lies on usability testing and improving the menu system in the game.

1.2 Goals

The goal of this thesis is to develop the menus in the game. To insure high quality anda satisfactory user experience usability studies will be conducted and the results will bestudied and used to implement changes in the original design of the menus.

Since the aim of this game is to attract new players and players who previously havenot played games in this genre usability is of highest importance. The purpose of thiswork is to make the game interface so easy to use that the entire focus of the player ison the game play and enjoying the game.

1

2 Chapter 1. Introduction

Chapter 2

Card sorting

When designing a website, an application or any other system it is essential that thestructure and grouping of objects in the system is relevant and logical to the person whois going to use system. If the layout of the system is hard to understand and difficultto use it is likely that the system will not attract users.

Card sorting is a ”quick, inexpensive, and reliable method, which serves as input intoyour information design process.”[8]. It is a useful method when designing the overallstructure of a system, it could also generate suggestions for the navigation and menus.Card sorting may not provide the final solution to the structure of a system but, it cangive valuable hints and tips and help tackle questions that may arise throughout thedesign process. Card sorting may help in answering questions like[8]:

1. ”Do the users want to see the information grouped by subject, process, businessgroup, or information type?”

2. ”How similar are the needs of the different user groups?”

3. ”How different are their needs?”

4. ”How many potential main categories are there?”

5. ”What should those groups be called?”

The answers to all of these questions are important factors when deciding groupingsfor objects as well as labels for categories and menu options.

2.1 The technique

Donna Maurer defines Card sorting as ”Card sorting is a user-centered design methodfor increasing a system’s findability [8]. The process involves sorting a series of cards,each labeled with a piece of content or functionality, into groups that make sense tousers or participants.”[8]. The general idea of Card sorting is that the user is given aset of cards and on each of the cards a piece of information or function is written. Theinformation or function may be complemented by a short description on the card. Thegoal for the user is to sort these cards into groups that she feel are suitable. There aretwo different types of Card sorting, Open Card sorting and Closed Card sorting. Whenperforming an Open Card sort after that the user have organized the cards into groups

3

4 Chapter 2. Card sorting

it is up to the user to name the different groups with a name that the user feel is suitableconsidering the content of the group. In a Closed card sort the user is, already from thestart, given an initial set of groups and is asked to place the cards into the predefinedgroups. Open Card sort is most useful when designing a new system or adding a newstructure to an existing system. Closed Card sort is best utilized when adding newcontent to an existing structure or when checking the structure of an existing design[8].

2.2 Pros and cons

As with most other techniques Card sorting has it advantages and disadvantages. Onthe upside Card sort is an easy, cheap and quick to execute method. It is straightforward both for the organizer and the participant, it does not require lots of materialand large facilities, it is preferably conducted with pen and paper. The only resourcethat it can consume is a lot of the organizers time. It is a fast method in the sensethat many sorts can be conducted in a short period of time, which will generate lots ofdata. Card sorting is also to be considered an established method since it has been inuse for over ten years. One important and positive thing that Card sorting exploit isuser involvement. Since potential users or actual users of a system participates in thetest the resulting design decisions are made from actual user input rather than from gutfeeling, Card sorting also provides a good foundation for the structure of a system, itmay not provide the final solution but is a good guide.[8]

Figure 2.1: Design process diagram (parts of the figure have been removed)

One of the biggest downsides with Card sorting is that it does not consider user’stasks. It is only concerned with the structure of the information, if no consideration isgiven to the users tasks and how they are performed when designing the system, therisk is ending up with a system and information structure that is hard to use and doesnot support user tasks. Although the actual card sort is quick to perform the task ofanalyzing the results can be difficult and time consuming. On the downside is also thattest may only capture surface characteristics. If the user’s do not consider what a cardreally means and what it stands for it may be so that the sort is conducted more onsurface characteristics than the actual meaning of a card[8].

2.3 When to conduct a Card sort

Card sorting is a generative method in that there is not yet a design, the goal is to findout how different users think about different issues. It is most effectively used when

2.4. Choosing participants 5

designing a new site, a new area of a site or redesigning a site. One important thingto note is that Card sorting is not an evaluation technique and will not reveal what iswrong with a current site.[11]

As can be seen in figure 2.1 Card sorting is the first step in the design cycle tobe performed after the phase of Discovery and Research, have been excluded from thefigure. It should also be noted that Card sorting has to be complemented by othertechniques for evaluation of the design and making sure that final designs made coherewith the findings from the Card sort[8].

When designing most sites and systems there may be a benefit in using Card sortingto evaluate the structure of the site, although there might be difficulties when usingCard sort against certain sets of information[8].

As can be seen in table 2.1 there are different characteristics for different sites.Conducting Card sorts on sites with characteristics that fall under the last columnseveral Card sorts may have to be conducted and possibly matched with other types ofusability studies to reach a satisfying conclusion[8].

Table 2.1: Characteristics of card sortingEasy Challenging

Site size Small LargeType of con-tent

Homogeneous (e.g.,product catalogs,lists of services,directories of websites)

Heterogeneous(e.g., intranets,government websites)

Complexityof content

Participants under-stand most of thecontent

Complex or special-ist content

2.4 Choosing participants

When conducting a card sort choosing appropriate participants is as important as in anyother usability study. The most important factor is that the participants are representa-tive of the user group [8]. Card sorting may be done either in groups or with individuals.Maurer [8] recommends seven to ten participants when having individual participants,Nielsen [11] on the other hand recommend fifteen users when performing individual cardsorts. Using fifteen individual users will reach a correlation of 0.90, which is a reason-able level to stop. Increasing the participant number with fifteen users will only leadto an increase by 0.05, which in most cases isn’t much enough to defend the costs ofan increased number of participants[11]. When conducting a card test with groups therecommendation by Maurer [8] is fifteen participants divided into three groups.

One significant benefit with using groups is that they typically provide richer datasince they do not need to be prompted to think-aloud since they tend to discuss withone another. This in combination with the fact that a group can handle a larger numberof cards results in a richer data set from a group compared to individual[8].


2.5 Preparing the test

2.5.1 Card content

When preparing a card sort the first thing to do is to determine what information thatshould be included on the cards. The content on the cards could range from wholesections of a site or application to individual functions or pieces of information. It isimportant to be consistent with in chosen granularity since it can be difficult to groupcards with different levels of granularity. If using a granularity at the larger sectionsof a system it is important that the items within a group belong together. If there areambiguities concerning items in a group it is better to put them on separate cards andsee how the participants group them. It is also important to make sure that the differentcards have similar content since it can be difficult and produce unreliable results if thecontent is to varied[8].

One factor that also should be considered is the ambiguity on individual cards, sinceif interpreting a card is difficult the placement of that particular card may be misleadingor not it might placed at all since the meaning of the card isn’t clear[4].

2.5.2 The cards

When having decided what information should go on the cards it is time to prepare thecards. Each piece of information should go on an individual card. Deciding labels forthe cards is extremely important. They should be short enough to be quickly read butinformative enough so the participant can understand what the content is. The cards tobe used are preferably ordinary paper cards with one item on each card. If conductingan open card sort blank cards and a pen should also be provided so that the participantscan write down group names[8].

2.6 Conducting the test

During a Card sort the main task for the test monitor is to observe and lead the test. Themonitor should not interfere with the participants unless it is absolutely necessary. Itis important that all participants stay involved with the task. If one of the participantstries to take over the monitor should try to prompt the rest to take more initiative,likewise if a participant does not seem so involved then that person should gently beprompted to get more involved. It is important when prompting individual participantsor the whole group to try not to lead them in any direction. During the sort the monitorshould take notes to keep track of any insightful comments or questions that might havearisen.[8]

2.7 Analyzing the results

There are two ways that the analysis of a Card sort can be done, either by lookingfor broad patterns in the data or by using a spreadsheet into which the data is en-tered. When having a smaller number of cards it may be possible to see patterns bysimply laying out the cards on a table. Patterns can be seen through similar labels orgroupings[8].

When having a larger number of cards it can be advantageous to use a spreadsheetfor the data. When the data have been entered into the spreadsheet the work of trying

2.8. Web based Card sort - Optimal sort 7

to find patterns among the groupings and labels start. At this point in the analysis itis important to keep in mind the discussion that took place between participants whileperforming the sort as well as comments made by participants since this can lead toimportant insight about specific groupings and labels. Another thing that is importantis that at this stage the quest is not to find definitive answers but more insights andideas[8].

Independent on the type of analysis chosen patterns will emerge. These areas ofdifference provides insights such as[8]:

1. ”Content that participants haven’t understood well”

2. ”Content that could belong to more than one area”

3. ”Alternative paths to content”

4. ”How different types of participants see information”

It is important to let the analysis step take its time, and perhaps use several differentanalysis techniques to analyze the data to get more insight into different patterns.

2.8 Web based Card sort - Optimal sort

The most common way to perform a Card sort is to gather all of the participants in aroom where they can be monitored and supervised by the test monitor. These tests areperformed as described above with pen and paper. With computers and the evolutionof the Internet new ways of conducting Card sorts have emerged. Several different toolswith the same purpose have been developed to make it possible to perform Card sortson computers and over the Internet.

Optimal Sort[16] is a web based Card sorting tool meant to simplify the work. Sincethe program is web based it makes for a high degree of accessibility.

The base of Optimal Sort is the same as for an ordinary Card sort. It lets thedeveloper of the Card sort name as many cards as she likes. Optimal Sort has anadministrator interface where all administrative tasks are performed like entering cardsand creating questionnaires as well as analyzing the results. The cards are enteredinto a simple text field where each line represents a card. Optimal Sort also offers thepossibility to have a pretest questionnaire on the test web site. The test is limited tothree questions which is a downside with the program. It would be desirable to havethe possibility to add more questions since a three question questionnaire may not besufficient for all purposes. When editing the questionnaire the model is the same as forentering cards, a text field where each line represents an answer, it is also possible tochose type, single or multiple answers, for each question. Another feature that OptimalSort has is that it is possible to add an instruction that the user gets prior to the test.The instructions are accessible all the time during the test if the participant should feelinsecure on how to perform the sort. Optimal Sort render possible both open and closedcard sort, which one to use is selected by the administrator prior to the test.

The Optimal Sort user interface is straight forward and easy to understand. Oncethe test is started all of the cards are to the left of the screen without being categorized.As can be seen in figure 2.2 there are visible and clear instructions in the interface onhow the participant should act to begin the test.

As the user drags a card onto the surface a category emerge with the card in it,see figure 2.3. If the participant wants to add a card to a created category the new


Figure 2.2: The Optimal Sort User Interface, no cards dragged into a category

Figure 2.3: The Optimal Sort User Interface, one card dragged into a category

2.8. Web based Card sort - Optimal sort 9

card is dropped on the category. If the participant wants to create a new category shejust drop the card on an empty space and the new category will be created. As can beseen in figure 2.3 the instruction on the surface changes depending on the state of theprogram and on how far the participant has come in the card sort. At the top of eachcategory there is field into which the user can enter a label for the category for an opencard sort, in case of a closed sort categories would exist on the surface directly from thebeginning of the sort. At the bottom of the interface is a comments field where the usercan enter any comments regarding the test, motivations for decisions or issues regardingperforming the test.

When a participant is finished with the sort the results are stored in a database onthe web site. The administrator is then able to view and analyze the result data. Shehas the possibility to organize and filter the data depending on several different options.It is possible to view all categories created and the cards put in each category, it is alsopossible to view all the cards and see all categories a specific card was put in. Thereis also one view where the results are organized by participant. In this view it is alsopossible to filter participants dependent on their answers in the pretest questionnaire.One additional feature that Optimal Sort offers is the possibility to export the testresults as an excel document.

Chapter 3

Usability Testing

Usability testing is a term used quite freely to include many different types of tests toinsure the usability of a product. In this chapter the term usability testing will be usedto label tests which include participants that are representative of the target population.They evaluate a product to find usability issues and measure to which degree it meetsspecific usability criteria. This definition excludes test types like expert valuation andwalk-through s from usability testing[13].

Usability testing is a tool which is applicable to everything from large classical testswith many participants and complex test designs to small informal tests including onlya few participants and a simple and fast test design[13]. The emphasis in this chapterwill be on smaller less informal tests which are done many times during an iterativedevelopment process.

3.1 Goals

The overall goal of usability testing is to discover and correct usability issues with aproduct. The intent is to ensure products that are easy, fun and satisfying to use, theyshould also provide levels of functionality and utility that are in line with the targetpopulation’s demands[13].

In addition to these quite obvious goals of usability testing there are also otherssuch as increasing sales and the probability of resales. This could be the results sinceproducts that are usable, and hence attractive to use, attract a larger population of userswhich via word of mouth improve the reputation of the product and hence increase sales.Happy users also tend to remain with the product and purchase future releases of it.Usability testing also render it possible to give the product a competitive edge, usabilityhas today become a prime selling point[13].

3.2 Limitations

Although usability testing might be useful to, and support the development process of aproduct, it is not without its limitations. Testing does not ensure success nor usability ofa product. Whatever the size and formality of a test it can never 100 percent insure thatall issues with a product have been discovered and resolved as the product is released[13].

11

12 Chapter 3. Usability Testing

The setting up of a usability test is, independent of if it is conducted in a lab or inthe field, always to some extent going to be an artificial situation. The fact that it isa study might also in itself affect the results. Another thing to keep in mind is thatany choice of participants never can fully represent the user population since it is verydifficult to predict what the actual user population is going to look like after the producthas been released[13].

Despite the limitations that exist testing is worth while. Usability tests conductedwith care and at the appropriate time in the development cycle is a valuable tool fordiscovering usability issues with a product. All in all it is always better to test than notto test[13].

3.3 Methodology

The methodology of a usability test is less informal than for controlled scientific ex-periments. It is often hard to formulate adequate hypothesis and have a large enoughsample of participants as is required with classical experiments. When conducting ausability study it is better to go for a faster, smaller and less formal approach. Insteadof having a hypothesis developing a problem statement or test objectives, the method isto use a representative sample of the target population, observe a participant when us-ing the product being tested and continuously interrogate them, collect qualitative andquantitative data and lastly recommend improvements to the design of the product[13].

It should be noted as Cooper [2] has done that usability testing is a mean to evaluatea design, not an alternative interaction design. The perfect design will not be generatedfrom usability testing, it is a way to assess the effectiveness of existent design ideas andsmooth over the edges.

Rubin[13] propose four different types of test that can be conducted during differentphases of the development cycle, they are shown in figure 3.1, and are discussed in detailbelow.

Cooper[2] divides tests into formative tests and summative tests. He defines forma-tive tests as test that are conducted during the design cycle in the service of design.These type of tests are tests that are fairly quick and easy to perform. The most impor-tant factor with a formative test is that it is conducted during the design process whenthere is still time to change the design. Summative tests are tests on completed prod-ucts, these are conducted after the design process has ended when changes are difficultand often expensive to implement.[2]

Of the test types that are to be discussed below exploratory and assessment testsfall into the formative category and validation tests fall into the summative category.

3.3.1 Exploratory tests

An exploratory test should be conducted early in the development life cycle. It is mostoften used as a tool when the development team is working on the functional specificationor in the early stages of design.

The purpose of the exploratory test is to explore the effectiveness of early designconcepts in relation to the users mental model[13]. This model is the representationusers have of a product or system, the way they think it works based on previousexperience[12].

Exploratory tests usually involve quite extensive interaction between the participantand the test monitor when trying to determine the efficacy of early design concepts. One

3.3. Methodology 13

Figure 3.1: The development life cycle including when each type of test is appropriate[13]


way to answer the problem statements and questions asked prior to an exploratory testis through a mock up of the product or software. It is not necessary for the mock up tobe implemented in something like Flash[1]. It is sufficient to use static screens or a paperprototype as long as it displays the core functionality and features to explore. Duringan exploratory test the user performs representative tasks and if that is not possible shewalks through the product and answers questions from the test monitor[13].

The emphasis during an exploratory test is on high-level functions and features.The testing process includes a lot of interaction between the participant and the testmonitor. Cooper[2] states that this is one of the problems during exploratory testingsince it is easy for the test monitor to lead the participant in a certain direction and henceinfluence the participant and subsequently the findings. Often the participant is askedto think-aloud to constantly motivate and explain their actions and how they interpretthe product. Much of the information wanted is cognitive by nature so therefore anexploration of users thought process is important[13].

Independent of how an exploratory test is performed, i.e. either with a implementedmock up, using a paper prototype, through performing tasks or with a walk-through,the core feature is its emphasis on discussion and its focus on high level concepts andthought processes[13].

3.3.2 Assessment tests

The assessment test is probably the most common usability test conducted and also theeasiest and most straight forward for a novice to design and conduct. An assessmenttest is usually conducted early in or midway through the product development life cycle.It is usually performed after the fundamental or high-level design of the product hasbeen established[13].

The assessment test is used to expand the findings from the exploratory test. This isdone by evaluating lower level functionality and having the user perform full-blown tasks.If the purpose of the exploratory test is to investigate the skeleton then the assessmenttest can be said to explore the meat and flesh. Assuming that the conceptual modelof the product works well the assessment test evaluates specific tasks and tries to findspecific usability issues with the product[13].

When performing an assessment test, unlike the exploratory test, the user alwaysperforms tasks instead of only walking through pages and screens. The test monitor isalso less involved letting the user perform tasks more freely, exploring the product. Inaddition to the usability issues that might be found by observation from the test monitoralso quantitative measures like mouse and key clicks can be collected[13].

3.3.3 Validation tests

As the work with developing a new product progresses and nears the end it is time toconduct validation tests to certify the products usability. The goal for the validation testis to determine how the product compares to some predetermined usability standardsor benchmarks. The usability standard or benchmark could be project determined, ahistorical or company standard or even a competitors standard[13].

The intent of conducting a validation test is to make sure that the product meets theset standard prior to release and if it should be discovered that it does not clarify why.The standards usually come from the usability objectives decided early in the project,

3.4. Choosing participants 15

originating from market and user surveys, user interviews or just simply from educatedguesses by the development team[13].

The usability objectives are most commonly stated in terms of performance criterialike mouse clicks, time to perform a task or operation. The usability criteria could alsobe stated to achieve a certain ranking or rating from users[13].

A validation test is conducted in a similar way to an assessment test with some ex-ceptions. First of all the validation needs to have developed standards and benchmarks.When performing the tasks the participants are given tasks that require very little or nointeraction with the test monitor. Finally the collection of quantitative data is the pri-mary objective during a validation test, although reasons for substandard performanceare also identified[13].

3.3.4 Comparison test

The comparison test is not positioned at a specific point in the development life cycle.In early stages of the development the comparison test can be done via an exploratorytest to compare radically different interface styles and see which one works best withthe target population. In the middle of the design life cycle the comparison test can beused via an assessment test to compare the effectiveness of different elements. Towardsthe end the comparison test can be used to compare the developed product to othercompeting products. Typically the comparison test is used to determine which of thedifferent designs that is easier to use or learn, or to better understand the disadvantagesor advantages of different design[13].

Usually the comparison test involves side-by-side comparison of two alternative de-signs. It can be conducted as an exploratory test or it can be conducted as a moreclassical experiment with a test group and a control group. For exploratory compari-son tests the best results are usually achieved when comparing widely different designs.Rubin[13] concludes that this is due to two main reasons. First of all, having two widelydifferent designs to test forces the design team to widen their horizons and think ofnew radical ideas which might not have been explored if they just continued along apredictable pattern. Second, if designs are very different the participants really have tothink about what makes one design superior to the other.

3.4 Choosing participants

As in all types of usability studies the participants chosen are crucial to the credibilityof the results. The participants should be representative of the target population of theproduct being tested.[13]

Prior to selecting participants for a test it is important to create a user profile againstwhich potential users can be compared. This user profile should contain characteristicsthat are important for the participants to have. These characteristics include specificknowledge, demographic information and experience[13].

When developing the user profile it is important to base it on correct informationabout the product and on what knowledge that is necessary to use the product as wellas what characteristics it is likely that the end user will have. This information can beacquired from different sources. The first place to look is at the functional specificationof the product. Information about the end users is also likely to be found from acquiredmarket studies, which typically try to establish who the potential users are and whattheir knowledge base is[13].


When having created a user profile against which to match potential participants ofthe study it is necessary to decide how many participants to acquire. Nielsen [9] hascome to the conclusion that it enough with five participants and to run several studies.As shown in figure 3.2 the percentage of usability problems found when conducting a

Figure 3.2: Curve showing usability problems found in comparison to number of userstested

usability study increases rapidly up to approximately five participants. For more usersthe curve flattens. According to the Nielsen [9] the most important finding displayed infigure 3.2 is the difference in usability problems found between testing zero users andone user. Approximately one third of all usability problems are found just by testingone user, and the conclusion to be drawn from this is that it is always better to testthan not to test.

Nielsen recommends to test five users in a usability study[9]. With five users about85 percent of all usability problems will be discovered, which can be seen in figure 3.2,and proceeding with more participants is normally not defendable from an economicperspective. One situation where more than five users need to be tested is when theexpected target population is highly diverse. It is important to have representativeparticipants in the study that cover all characteristics of the user profile, if this cannotbe accomplished with only five users then more need to be tested.[9]

3.5 Preparing the test

Before conducting a user study it is important it is thoroughly prepared so that it isclear what to test and how. There are several documents that need to developed priorto testing. The first one is the test plan which serves as a blueprint for the entiretest. When the test plan has been developed a number of questionnaires and supportdocuments for conducting the test need to be developed.[13]

3.5. Preparing the test 17

3.5.1 The test plan

The test plan is the most important document when preparing a usability study. It isthe blueprint of the study and it addresses the how, when, where, who, why and whatof the study[13].

The test plan is the one document that contain all information about the test. Fur-thermore, the document keeps all interested parties briefed about the test, how it is tobe conducted and what is to be tested. The test plan should include the purpose of thetest, as well as high level reasons to why test is being conducted.

It should also declare a problem statements for the test, describe more exactly thequestions that the test will try to answer. The problem statement is probably the mostimportant part of the test plan and also the most difficult one to write in a clear andprecise manner. It is important that the problem statements are measurable. If they areto vague the tests might not answer the questions stated in the problem statements.[13]

The test plan should also describe the profile of a potential user of the product.Developing the user profile was discussed in more detail in a previous section.

How the test is going the be performed is another piece of information. This sectionshould describe which different parts that the test will contain and roughly what willhappen in each part, it should be clear to someone reading the test plan what is goingto happen in each part of the test. Developing a good test design is probably one of themore difficult tasks in running a usability test. The test design mainly depends on thetests problem statement[13].

In the test plan there should also be a task list which contain all the tasks that theparticipants are going to perform while testing the product. For each task in this listthere should be a brief description of the task explaining to someone reading the testplan what the user is going to do. For each task, if needed, a description should be givenof the requirements to be met for the participant to be able to perform the task. Thereshould also be a description of what represents a successful completion of the task[13].

One section in the test plan should also describe the testing environment and whatequipment is required to perform the test. The equipment to be described in this sectionis only the equipment that is going to be used by the participants, equipment used todocument and observe the test need not be described[13].

The role of the test monitor could also be a part of the test plan, it should describewhat the test monitor should do during the test and what his responsibilities are. Thissection is probably most important to people who are unfamiliar with the test and canbe used to give them some insight into what will happen during the test[13].

The evaluation measurements to be collected should also be stated in a section ofthe test plan. Listing the measures gives all interested parties a chance to make surethat they get the results they want from the test[13].

Last in the test plan it should also be stated how the findings of the test will be sum-marized and presented to interested parties involved with developing the product[13].

3.5.2 Test material

To be able to perform a usability study several other documents need to be preparedbefore the actual testing occasion. This is done to ensure that the testing runs alongsmoothly and as planned.


Orientation script

The orientation script is the document that describes what is going to happen duringthe testing session. It sets the tone of the test and should emphasize to the participantsthat it is not they who are being tested but the product. It is important that the toneof the script is friendly but still professional to make the participants feel at ease andcomfortable. It is suitable to limit the orientation script to one or two pages, otherwiseit may become tiering to listen to and the participant may lose focus. One importantthing when presenting the instructions to the participants is to read the orientationscript verbatim, this to ensure that all participants get exactly the same instructions.If not it may affect how they act during the test and consequently the test results[13].

The orientation script should include some different parts to cover all aspects of theinformation the participants may need. These should accomplish the following [13]:

1. Introduce all the persons that are involved in testing and that the user may see orcome in contact with.

2. Explain to the participant why they are there, what is to be tested and sufficientbackground to the product to enable the participants to perform the tasks.

3. Describe to the participant what all the equipment is that is going to be used andit is also important to tell the participant if the session is going to be recorded inany way.

4. Explain to the participants what is expected of them. Ask them to act as theynormally would, to ask questions, which may or may not be answered, and totake breaks if needed. It is important to keep a neutral tone when talking to theparticipants and not to say anything that might make them nervous, e.g. ”Mostpeople find this extremely easy”.

5. Assure the participants that it is not they who are being tested, emphasize thatit is the product and its performance that is the focus of the test and not theparticipants performance.

6. Explain any unusual protocols such as thinking-aloud to the participant before thetest starts and if necessary demonstrate.

7. Ask the participants if they have any questions concerning what was just presentedand if necessary clarify.

After that the orientation script have been read to the participant it is time to startthe test.

Background questionnaire

The background questionnaire is used to gather historical information about the partic-ipants, information that can be used during the analysis of the test results to see howdifferent backgrounds and experience might influence the behavior of a participant. Thebackground questionnaire is normally filled out just prior to the test, although if it isa long questionnaire it can be advantageous to send it to the participant to fill it outprior to the testing [13].

In addition the background questionnaire can be used to confirm that the correctparticipants show up. Since it is vital that the participants correctly represent the target

3.5. Preparing the test 19

population it is important to check this for the persons who showed up. The backgroundquestionnaire can also be useful for persons observing the test to better understand whythe participant act as they do[13].

When the background questionnaire is being developed there are some guidelines tofollow[13]:

1. All background information that might affect the results should be included in thequestionnaire.

2. To make the questionnaire easier to fill out and to analyze open-ended questionsshould be avoided.

3. A pilot using the questionnaire should be run to find any ambiguity.

A well designed background questionnaire can be very useful when analyzing theresults of a usability study.

Pretest questionnaire

The pretest questionnaire is, as is the background questionnaire, answered by the par-ticipant prior to the actual testing. It is used to gather the first impressions about aproducts ease of use prior to actually using it. The first impressions is important in set-ting the stage for the usage of the product. A product that looks easy to use might makethe users try a bit longer before giving up using it, therefore the pre-usage informationcan be important in understanding a participants behavior later on in the test.[13]

The research can also be used to see how usage affected the impressions of theproduct. This is done by giving the participant the exact same questionnaire after thetest as before. When comparing these results it can be seen if using the product increasesor decreases first impressions of the product[13].

A way to extend the use of the pretest questionnaire beyond only gathering firstimpressions is to actually ask the participants to define elements of the product that isconsidered to be self evident without having used the product. This questionnaire canalso be given to the participants after the test to see if using the product increased ordecreased their understanding of the product. Results form such a test can be used todiscover needs in changing terminology.[13]

Task scenarios

Task scenarios are representations of actual work that a typical user might perform whenusing the product. Expansions of the tasks in the task list developed for the test plan.The scenarios add context to the tasks to be performed and add motivations to why atask should be performed. It is also common to join several tasks into one scenario sincemany tasks may be coherent and that is the way they are usually performed[13].

For developing scenarios Rubin[13] has five key guidelines:

1. The scenarios should be realistic and provide complete motivations to why performthe task within the scenario.

2. The sequence of the scenarios should be in the order they are most likely to beperformed.


3. The scenarios which a participant is to perform should be matched against theirexperience.

4. Jargon and cues should be avoided, if a participant knows a word it is not consid-ered jargon. The scenarios should be written in such a way that they do not tellthe participant what to do.

5. Do not guide the participant through a scenario, state a clear goal and let theparticipant reach it on their own.

The scenarios can be read to the participant or be given to the participant to readthemselves. If the scenarios are long and complex it can be an advantage to read thescenarios to the participants since this gives the monitor the opportunity to interactwith the participants to clarify anything that is unclear. If the test is monitored fromanother room, and wanting to limit interaction with the participant, it is better tohave the participants read the scenarios themselves, not excluding the possibility for theparticipants to ask questions.[13]

One other thing that has to be considered is whether to present scenarios all at onceor one at a time. If all scenarios are presented at once there is the risk of overwhelmingthe participant and also that the participant get ahead of themselves, which may leadthem to act in a way that they otherwise would not have[13].

Posttest questionnaire

The posttest questionnaire is used to gather preference information from the participantsabout their feelings and opinions about the product being tested, this to acquire a betterunderstanding about the products strengths and weaknesses. The information gatheredfrom the posttest questionnaire is general and collected across the entire population ofparticipants. The questions are the same for all participants and are asked in exactlythe same way[13].

When developing the posttest questionnaire content is important, the questionsshould be precise, clear and to the point. The questions should be based on the problemstatements from the test plan, since these are the questions that the entire test aspire toanswer. The questions should preferably concern such things that cannot be observedduring the test like feelings, opinions and suggestions for improvement. The questionsshould also be as simple and clear as possible without any lengthy instruction, this toavoid any risk of ambiguity[13].

It should be noted that developing a questionnaire is not something that is done inthe last hour before the test. It takes time, requiring thorough consideration and testingof the questionnaire[13].

3.6 Conducting the test

When having finished the preparations of the test material it is time to conduct the test.This is the most difficult part of the process. When acting as a test monitor it is veryeasy to not just misinterpret what one sees but also to affect how the participant actand hence the outcome of the test[13].

When acting as a test monitor there are some guidelines one should follow to makethe testing run along as smooth as possible [13]..

3.6. Conducting the test 21

1. Act impartial, react to mistakes in the same way as to correct behavior.

2. Do not make the participants feel stupid or inadequate.

3. If a participant starts to fail a task remind them that it is not them being tested butthe product, and that failure helps you understanding how the product actuallyworks.

4. Encourage the participant to focus on their experience and not to think about howother participants performed.

To make the participants feel more at ease when participating in a usability studyit is good to start the testing with an easy task, as well as ending with an easy task sothe participants feel good about themselves when leaving [12].

The test monitor also have to be aware that his or her body language and voice mightaffect how the user acts. The monitor should always try to act as neutral as possible,keeping a neutral posture and tone of voice.[13]

If a participant has problems during the test and starts to struggle it is important forthe test monitor not to help the participant. A participant that is struggling provides theopportunity to see what happens when a person is struggling and how they recover[13].Nielsen[10] also states that it is by watching users struggle while using a product thatreliable results are acquired.

Before proceeding to the next task it is important for the test monitor to make surethat the participant has finished the previous task. If the test monitor jump in beforethe participants are sure they are finished that confirms to them that they are finishedwhich removes the moment of indecision prior to finishing the task. If the user is allowedto think one moment extra the user might start the task again and finish it incorrectly.The best way to avoid this problem is to ask the participant to signal when they arefinished[13].

If appropriate then the thinking aloud technique should be used. It is a techniquewhere the participants are asked to constantly explain what they are thinking, therebyproviding the test monitor with reasons to why a participant acts in a certain way[13].It is though important to use what the participant says in a correct way, the informationshould only be used as reasoning to why certain actions are taken and not as explanationsof problems or implications to design changes. This since it is how the user actually actsthat is important and discover flaws in the product, not how they think they act[10].

Probing and interacting with the participant should only be done when necessary.The frequency of interaction is also dependent on what type of test that is conducted.If the test is exploratory the amount of interaction is larger than if it is a validationtest[13]. Interaction to help a user should only be used as a last resort noting that theparticipant needed help and that the results might have been affected [12].

On the day of the test it is useful to follow a predetermined protocol to make thetest run as smooth as possible and to reduce any chance of ambiguity and difference inpresentation to different participants. Before the test the test monitor should preparementally on the task to come, and read through the problem statement to make sureshe know what to look for. It is also important to prepare oneself and ones attitude tomake the test session as comfortable and easy as possible.[13]

When the participant arrives it is important to have a game plan to follow. Theseare some important things to remember:

1. Make the participant feel comfortable, offer refreshments and make small talk.


2. Ask the participant what they think they are here to do, this do straighten outany mistakes.

3. Fill out background questionnaires, non disclosure agreements and such prior totesting.

When these steps have been carried out it is time to start testing, i.e. read theorientation script and set the stage for what is to be done during the session. The lastthing to do prior to testing is having the participant fill out any pretest questionnaires.When it is time to start the test monitor provides the participant with the scenarios,either by reading them or handing them out on a paper. During the test the monitorshould observe how the user acts and take notes of anything that is important for furtherdevelopment. After the test it is important to have the participant fill out all posttestquestionnaires to finish the data collection.[13]

Depending on the type of test a debriefing session could be conducted to discusshow the user acted and why [13]. It is however important to keep in mind that whata user say is not as reliable as how they actually act. Asking the user about designrecommendations should be avoided as well as having intricate discussions about howthey acted[10].

3.7 Analyzing results

After that the testing is finished it is time to analyze the data and to develop recom-mendations. Usually the process of analyzing data falls into two deliverables. The firstdeliverable is to be handed to the development team and the designers and is the pre-liminary deliverable. This provides the designers with something to work on directlyafter the test. The analysis for the written report is conducted directly after testing hasfinished and usually results in short written report or preferably an oral presentation.It is important to chose the preliminary recommendations with care so that the prelimi-naries aren’t things likely to be changed later. One option to avoid changing preliminaryfindings is just not to deliver any preliminaries. The problem with this option is thatif a designer observed the test they are going to act on their observations as they haveseem them. Therefore it is better to deliver some preliminaries before any presumptivechanges are made. The second deliverable is a complete report of all the findings fromthe test. The process of analyzing data for the complete report is a time consumingtask that usually takes between two and four weeks[13].

When starting to analyze the data the first step is to compile and summarize thedata. This is a step that usually begins already during the test session. During thisprocess all data need to be collected and put into one place and standardized for easieranalysis. This compilation preferably done right after the test or with in a day oftesting to have the results fresh in mind, hence making the process easier. After all testshave been run all compiled data should be summarized to a document documenting allparticipants[13].

If different types of performance data e.g. times in different tasks and task accuracy,have been collected the first task is to summarize these. When this is done it is time tosummarize the preference data, i.e. notes of observations, recordings made during thetest and results from questionnaires. When summarizing preference data it is importantto put the results into suitable categories that are appropriate for the specific test[13].

When all of the results have been summarized and categorized it is time to start toanalyze the data. The first thing to do is to identify the tasks that did not meet the

3.8. Developing recommendations 23

criterion for success and focus on them. When determining problematic tasks Rubin[13]generally labels tasks where less than 70 percent of the participants did not finish asproblematic and focuses on them. The 70 percent guideline is a useful tool for identifyingproblem areas with the product. In tests with few participants the problematic tasksare usually quite obvious[13].

When having identified problem areas and difficult tasks the work with concludingwhy these problems arose commence. It is important to get to the bottom of why aproblem arose, so it is worth spending some time on this task. Identifying the source oferror is not always an easy task and gets harder by the number of different components,screens or parts of the product the participant used before making the error. Usually thiscan be straightened out by talking to other persons watching the test, and by reviewingnotes and recordings from the test. When trying to identify the sources of error it isimportant to be focused on the task and try to not to develop recommendations for fixesbefore exploring all potential sources of error[13].

When all sources of error have been identified it is time to prioritize the problemsaccording to criticality. Rubin[13] defines criticality as the severity of the problemweighted with the probability that it will occur. It is important to rank the differentproblems to help the development team prioritize their work[13].

What is important to have in mind when analyzing results and especially reviewingresponses from participant surveys and different types of recordings is not to listen toomuch to the users but to watch what they do. Nielsen[10] says ”To design an easy-to-use interface, pay attention to what users do, not what they say. Self-reported claimsare unreliable, as are user speculations about future behavior”. This statement is veryimportant to keep in mind.

3.8 Developing recommendations

When all results from the tests are analyzed it is time to develop recommendations fordesign changes. This is the essence of the whole process of user testing. It is importantto have in mind when doing this that it is not always obvious which component that isresponsible for the problem. Just because the problem source has been identified thisdoes not mean that the solution is simple and at the source. It is actually very oftenquite subtle. Knowledge of human perception and on how humans work and interpretinformation are important for accurate recommendations[13].

When starting to develop recommendations it is wise to start at a high level andmake sure that design recommendations affecting a large portion of the product havebeen considered first, since those recommendations have the greatest impact on theproduct. When the broader issues have been addressed the smaller ones should beconsidered and written recommendations for. Discoveries made during a test is oftennot limited to things that can be corrected within the given time frame. The sameshould go for the recommendations, they should not be limited to design modificationsthat can be made within a near future. The recommendations should be divided into twosections, recommendations of changes that can be made in a near future and the onesthat cannot. The report to the development team should also contain a section coveringareas in which results are inconclusive and more research is needed. This is good as ithelps researchers to structure tests in the future. One should also not be afraid to pointout any limitations with the study as this only makes it more clear to the developmentteam what is covered and where further efforts are needed. The most important thing


when developing recommendations is to keep them as simple as possible, to be thoroughand leave no stone unturned.[13]

3.9 Morae

Morae is a software tool developed to be used when testing usability on web sites andsoftware products. It is used to identify usability issues and to pinpoint specific problemareas. Morae is divided into three different applications. There is the recording partthat is run on the computer that the participant is using, which records everything thatis happening on the screen, as well as, if a camera and microphone is connected to thecomputer, the participant and everything that is said. The second part of Morae is theObserver software, which is for people who want to observe the test from a distance.The separate Observer computers connect to the Recorder computer and thereby getthe live feed from everything that is happening on the test site. The Observer can alsobe used to make notes about the ongoing test as well as to mark the time line of thetest with important events and also logging when tasks are started and finished. Thelast part of the Morae software is the Manager. It is the part that is used to analyze theresults. In the Manager it is possible to play back recordings and analyze them step bystep. From the Manager graphs of results can be made to better visualize the results.Morae is also very portable since it can be installed on a laptop and brought out totest participants. As a tool during usability testing Morae is very useful and makes thetesting easier in all stages[15].

Chapter 4

Exploring the InformationLayout of a Game Menu

The user interface of a computer game consists of several different parts. Some are themenus that the user has to pass before starting to play the game. These pages normallycontain functions for customizing the game, choosing different game modes to play andof course the possibility to start the game. This new Battlefield game is no exception,as in all other computer games there is a menu in the game client in which the user cancustomize the game and choose different game modes.

When designing an information structure different methods can be adopted in thedesign process to best use the people designing it and making sure all aspects of thestructure is thought of. One important step in all design, interaction as well as infor-mation, is evaluating the design throughout the process. Evaluation is possible in allsteps of the design cycle from after the first information gathering steps to evaluating afinished product.

In this chapter the information design of the menu structure and ways to evaluatethe information structure is discussed.

4.1 Design

”Information design is the integrator that brings other disciplines together to createexcellent information solutions.”[6] With this Knemeyer [6] tries to explain that infor-mation design in it self is not an explicit discipline, it is a discipline that integrates othersto form a working whole. The part of information design that will be handled in this sec-tion is the one that is commonly referred to as Information Architecture, ”Informationarchitecture for the web is a discipline generally committed to structuring informationand task flow in a manner that optimally facilitates the user’s finding information orperforming tasks he or she wishes to.”[14]. Although this quote refers to InformationArchitecture for the Web it is applicable for Information Architecture in other contextsas well. Information Architecture is the sub discipline of Information Design that con-cerns organizing the information and flow between different parts of the system. It couldalso be defined as ”the art and science of organizing and labeling websites, intranets, online communities and software to support usability”[5]. The key concept to note in thisquote is that Information Architecture is very important to support usability, without

25

26 Chapter 4. Exploring the Information Layout of a Game Menu

a good structure for information and functions usability will be affected.When designing the information architecture for the Battlefield computer game, the

first step taken was to gather all information possible about the system to get a good andclear picture of which functionality and which information to display within the system.This step included reading design documents about the system as well as talking to thedifferent designers on the project to be sure not to miss anything. When a satisfiedresult was accomplished and all relevant information was gathered the first few steps inthe design of structure was taken.

The first step was to divide all the functions and pieces of information into groupsbased on coherence, e.g. information about keyboard configuration and the functionalityto change that configuration goes into the same group. When all different objects whereplaced in an appropriate category these categories where divided into subcategoriesto get a suitable size of the different subcategories. The constant goal was to createcategories and subcategories that could easily be comprehended simplifying where tolook for a certain piece of information.

When this step was finished it was time to make a user test. This user test is thefocus of this chapter of the report. In the sections below different methods in evaluatinginformation architecture through user participation will be presented and a study usingone of these methods will be presented.

4.2 The study

As a part in developing and designing the menus the contents have to be organized foreasy access and ease of use. Since this game is targeting, a for the Battlefield franchise,new group of players, it is important to get their input on layout and organization of themenus and of the game in general. When designing the menus the first step is to organizethem. As a part in this development a Card sort was conducted with participants bothfrom the targeted user group as well as with participants from more common Battlefieldplayers.

4.2.1 Preparing the test

The first step in preparing the test was to write down all possible actions and pieces ofinformation that was going to be in the menus. This was done based on an understandingon what functionality the menu system would contain. This was done through a seriesof iterations until finally a satisfying number of pieces of information and functionalityhad been agreed upon.

For this test I chose to use the web based tool optimal sort. This was done becauseit allows for a large number of participants in a short amount of time. A result of thechoice of using Optimal Sort was that all tests where conducted individually and notin groups as recommended by Maurer[8]. My aim was to get about 15-20 participantsfor the test, I was planning on having the test active for about one week to reach thedesired number of participants.

When having decided what cards to use, the pieces of functionality and informationto test, and settling with using Optimal Sort it was time to set the test up. In linewith how the Optimal Sort program works all the cards were entered on the web site,a total of 58 cards were included in the test, along with a questionnaire asking theparticipants some demographic questions concerning age, if they have used any of somecommunity sites on the Internet, and approximately how many hours they spend a week

4.2. The study 27

on playing first-person-shooter games. A brief instruction to how Optimal Sort worksand an introduction to the system and the features it contains was also included in thetest.

4.2.2 Choosing participants

Since the test contained information about the game that was classified as secret it wasnecessary that the participants had clearance to take part of that information. Thereforethe participants where recruited among the employees at DICE. This made it possibleto find participants from groups of players with different experience of computer gamesand first-person-shooter games. Although the majority of the employees at DICE haveto be considered quite experienced gamers there are those that have a less degree ofexperience with computer games. Despite that, this test turned to all employees andasked them to participate.

4.2.3 Conducting the test

To initiate the test an e-mail was sent to all employees at DICE asking them to partic-ipate. The expectation was to get a few answers, perhaps 5-10 for the total durationof the test time. When the test was launched it turned out that the number of willingparticipants far exceeded any anticipations. Within thirty minutes about 25 results hadbeen registered. Therefore the decision was taken to shut the test down after it hadonly been open for half an hour to avoid more participants than could be handled.

4.2.4 Results

When the Card test was closed 26 participants had completed the test. Out of these6 had not completed the test in a satisfying way, e.g. finished the test without sortingmore than 50 percent of the cards. These were discarded from the results. This left 20results to be analyzed, 5 more than recommended by both Nielsen[11] and Maurer[8] butthe decision was made to keep these test in spite of the extra work it would generate.

Of the 20 participants who’s results were taken into account the ages were distributedas in table 4.1 which shows that the majority of the respondents were between 25-35which is representative for a part of the expected target group. The other part ofthe expected target group is younger between 10-16, but it was impossible to get anyparticipants in those ages at DICE.

Table 4.1: Which of the following age groups do you belong to?Age groups Number

of partici-pants

Percentage

0-12 0 012-18 0 018-25 6 025-35 11 035- 3 0

Table 4.2 shows how many hours a week the participants played first-person-shooter(FPS)games, the majority only played 0-5 hours a week which is representative of the target


group for the game.

Table 4.2: Hours of first-person-shooter games played a weekHours Number

of partici-pants

Percentage

0-5 12 05-10 3 010-15 4 015-20 0 020- 0 0

It is believed that many of the users in the expected target group spend time ondifferent communities on the Internet. Table 4.3 shows that the majority of the par-ticipants in the test have spent time on one or more of the listed communities, but asignificant part have not.

Table 4.3: Visited communitiesCommunities Number

of partici-pants

Percentage

Habbo Hotel 1 0Myspace 2 0Facebook 12 0Xanga 0 0none 8 0

The participants created 125 different categories into which they placed the cards,it should be noted that many of the categories had similar names and hence similarcontent concerning the cards that where put in them.

4.2.5 Analyzing the results

When analyzing the results a spreadsheet tool developed by Donna Maurer[7] was used.The first step in analyzing the results with the spreadsheet was to enter all cards intoone tab in the document. The next step was to add a tab per participant in thetest and on that tab enter all the categories created by the user and which cards thatwhere in each category. Next the process of standardizing the categories begun. Theaim for that step was to analyze all the different categories that were created by theparticipants and analyze which cards that went into which categories and to try to createa standardized name for all the categories with similar cards in them. This work tooktime and it is difficult not to create too many categories. Putting too many categoriesin the same standard category also must be avoided. One part of this process was tocreate standardized labels for each standardized category and try to make that labelresemble the labels of the categories that were standardized. This process resulted in27 standardized categories. These categories where entered into a specific tab in thespreadsheet. This tab was then automatically populated with how many participantsthat used a standardized category, how many cards that were in that standardized

4.2. The study 29

category and how many unique cards that were in a category. These numbers showedwhich categories that were more popular and also to what extent they agreed on whichcards should go into one category. The less unique cards in a category the higher theagreement.

After an analysis of the standardized categories the categories with the highest agree-ment was picked out. It was also made sure that these categories covered all the cardsso no possible information or action was left out. This resulted in seven categories thatcovered all different cards. These categories differ somewhat from the original layoutcreated during the design phase but overall they are in agreement with the originaldesign. It was mainly two areas that where different from the original design.

4.2.6 Conclusions

Since the study came to the same result as the original design made prior to the studyit is reasonable that the original design satisfies and are in line with how a user thinksthe interface should be organized. Two areas differed from the original design. Theseareas are kept as originally designed since it is believed by the designer that the divisionmade by many of the tested users create a skewed division between areas and that thelayout will be more logical and work better if the original layout is kept. Wether thisassumption is correct will probably be discovered in future usability studies.

4.2.7 Discussion

Performing a card test is not at all as easy as it may seem. During the study severalproblems with the methods used and the layout of the test were discovered. The firstproblem that was discovered was that the instructions for the test and the backgroundinformation concerning the system was too vague. This test only covered functions thatwould be in the menu structure of the system and hence all categories created was tobe put into categories being situated within the menu structure. This was however notthe case, many participants created categories that they felt should be in the game, i.e.while playing the game, that is something that was beyond the scope of this test. Thisresulted in categories being created that could not be used and hence all the results fromthat participant had to be discarded. This problem hopefully could have been avoidedif the system background and the instructions of the test had been more clear.

Another problem that was discovered was that the text on some of the cards wheretoo vague. This was clearly discovered when analyzing the results, and was obvious intwo ways. Either the card was put in a category called ”do no understand” or it wasput in a category that, if the card was understood correctly, did not make sense withthe context of that category and the other cards that were put in it.

Another thing that also could be negative and should be considered when setting upa test is that the more cards the more time consuming it will be to conduct the test.Something that may lead to that the users get tired and put less focus on the task.

These aforementioned problems possibly arose because the test was conducted viaa website and not in a room with physical cards. Without test conductor, participantscould not ask clarifying questions about the cards, the scope of the test and how toperform the test. One advantage though that came from having the test on a websitewas that it was easy to reach a large group of possible participants. This may on theother hand not be only a good thing, Nielsen[11] recommends about fifteen participantsof an individual card sort. For this test about seventy participants begun conducting


the test although only about thirty finished it, this leads to the problem of selectingwhich results to use. Something that should have been done randomly, since the testwas quite extensive many participants only sorted about 30 percent of the cards andthe reported the test as finished. This lead to that the results to be used in the analysishad to be handpicked, something that is not desirable.

The issue of bias also arose when analyzing the results. Since the designer, who wasthe one who arranged the test and analyzed the results, prior to the test had made adesign of the information layout and structure of the system it became hard to analyzethe results and in particular create standardized categories without being influenced bythe given design. The conclusion from this is that this type of test should be done earlierin the design cycle, just after the requirements have been set and they have been brokendown into functions and pieces of information, not as in this case after a design havebeen made.

Overall the this type of test can be valuable in the design process as it gives anindication of how users think of the system and the organization of the system. But Ithink it is probably more suitable to perform these kinds of test with paper based cardsin real life and not on a website.

Chapter 5

Evaluating the Design of aGame menu

5.1 Design

After the initial information design phase it is time to design the more detailed structureof the separate pages and the interaction therein. When designing the different pagesit is important to try to be as consistent as possible so that the user of the system feelfamiliar with the interface when browsing. Having a consistent interface also makes iteasier for the user to learn how the interface works. Keeping the interaction style andlayout similar between pages increase recognition[12] and minimize the need of recall[12].The concept of recognition and trying to make the interface as consistent as possiblehave been the leading star during the design of the interface.

The design from the initial information design stage was used to divide the interfaceinto subsections and content in each subsection. Having this design as a foundationfor the later design work simplifies that since it is known what goes where and focuscan be put on making the interaction style as effective and usable as possible. Whendesigning the interface focus has been on making it as usable as possible for a quite wideand diverse target population. The goal for the system is to be usable for users of agesbetween fourteen to thirtyfive with computer game experience ranging from between afairly high degree experience to almost none. This poses a big challenge when designingthe interface. Picture 5.1 shows the main page of the interface after the first designiteration, figure 5.2 shows the loset page of the appearance section, on this page theuser can customize how the player looks.

The goal for this menu system has been to make it as simple as possible but stillhave fairly complex operations and possibilities within the interface. When starting thedesign most of the work was done with pen and paper to explore the possibilities ofdifferent layouts within the interface. The paper sketches where iterated several timesuntil a satisfactory design was found. When having completed the paper sketching anearly mock up of the interface was created in Flash.

Flash was used for several reasons, but mainly because it is a dynamic tool, in whichit is easy to create a correct layout, to implement different interaction styles and tomake the interface correctly represent as many features as possible. Having made themock ups in Flash allow them to be used during the usability testing that was conducted

31

32 Chapter 5. Evaluating the Design of a Game menu

Figure 5.1: The interface main page after the first design iteration

Figure 5.2: The closet page of the appearance section

5.2. Common sections 33

during the iterative design of the interface.When an initial high level design of the interface had been made it was decided it was

time to conduct the first study. Specifics on this study is described in the section calledThe First Study. The results from this study, as described below, was then used to alterand improve the design of the interface. After some further design work the interfacehad been implemented in more detail with more correct interactions and features thesecond study was run. It is described in the Second Study section. This study testeddeeper levels of the interface such as specific functions and interaction styles.

5.2 Common sections

Since there are parts of conducting a study that are similar independent of the study,these sections and the descriptions of this work have been summarized into commonsections for both studies.

5.2.1 Preparing the tests

Prior to running the tests several documents needed to be prepared in order to maketesting run along as smooth as possible. The first and most important document thatwas prepared was the test plan [13].

As a first step in writing the test plan the goals for the test was decided, theyprovided the foundation for how the test was to be conducted, which participants whereneeded, and what more documents needed to be prepared.

The first written document, that was used during testing, was the orientation script.This document covered the most important aspects discussed by Rubin [13]. It startedby describing how to welcome the participants to the test trying to make them feelcomfortable and at ease in the situation. After that a description of how the testing wasgoing to be conducted followed. Finally a short description of the system to be testedfollowed.

Based on the test plans description of the needed participants a background ques-tionnaire was written. This questionnaire included questions about the participantsbackground, concerning for example, computer experience, Internet experience and, ofcourse, computer and console game experience. The results of the background ques-tionnaire could be used in the analysis to find trends among different categorizes ofparticipants. The same background questionnaire was used for both tests.

For the first study a pretest questionnaire was written. This questionnaire was usedto ask the participants about their understanding of labels in the interface, prior tohaving any experience of them.

The last document, used as a part of testing, was the posttest questionnaire. Thistest included questions trying to answer and gather findings about the problem state-ments from the test plan document. The questions were mostly scale questions where theparticipant was given a statement consider. It also included some open ended questionswhere the participant was asked to list any parts of the interface they found difficult touse. For the first study the posttest questionnaire also included a section where the par-ticipants were asked to write descriptions of the same labels as in pretest questionnaire.This to be able to compare the participants understandings prior to and after using thesystem [13].

Prior to testing the tasks to be conducted were written on paper cards to be usedduring testing.


5.2.2 Conducting the Studies

After having prepared the documentation for the study the preparations for the actualtesting began. First of all Morae Recorder [15] was installed on a laptop on which thetests were to be run, and on the same machine a web camera was configured to beused for recording the test sessions. All questionnaires that were going to be used wereprinted.

The test was run in a conference room at DICE’s office. The room in which the testswere run was considered to have no effect on the test results since playing a computergame is not context sensitive.

When the participants first arrived at the test site they were welcomed and thankedfor participating in the test. The first thing they got to do was to answer the backgroundquestionnaire, this was done prior to hearing the orientation script so it would not influ-ence them when answering the questionnaire. When they had finished the questionnairethe testing began. First the orientation script was read to the participants. It was readword for word so that all participants would get exactly the same instructions and notbe biased in any way.

After reading the orientation script the participants of the first study answered thepretest questionnaire, the second test had no pretest questionnaire. The next step wasto start testing. During the test the participants were recorded. Both picture and soundas well as all actions taken on the screen were recorded, i.e. mouse movements, mouseclicks etc.

In the first study the tasks were made into more natural scenarios, as recommendedby Rubin[13], describing a goal to accomplish, on the road to which the participantsshould complete several tasks. These scenarios were read to the participants.

Test the test monitor observed the participant and took notes to remember problemsthat the participant encountered. The tests ran along smoothly for both study one andstudy two.

After that the actual testing was finished the participants answered the posttestquestionnaires, for the first test this was done on paper, but for the second test it wasincluded as a part of testing on the computer and run through Morae.

5.3 The First Study

The first usability study conducted was done fairly early in the design phase and wasaiming at evaluating the design of the high level structure of the menu interface. Highlevel structure in this context means that the menu and its sub menus are included.This first test was a combination of Rubin’s [13] exploratory test and assessment test.The test was conducted so that the participants were asked to perform full blown tasksto see to which section they would navigate to in order to perform the task. This to geta better understanding of the users mental model of this type of system.

During the test the participants were asked to think-aloud, this to be able to under-stand to how the user thought and why they took the actions they did. It was valuableduring the test to understand this to better understand where problems in the designexisted.

In the sense that the interface was in early stages of design and it was not possible toperform actual tasks the test was exploratory. The assessment part of the test resultedfrom performing the tasks without much interaction from the test monitor.

5.3. The First Study 35

5.3.1 Goals

For this test a number of goals or test objectives were formulated, these were:

1. Does the user understand the main section labels?

2. Does the user understand the sub section labels?

3. Can the user freely navigate between the different sections of the interface?

4. Do the screens reflect the user’s mental model concerning groupings of objects andsections?


For both tests the test plan stated the prerequisites the participants should fulfill to bea suitable candidate for participating in the test. Since the computer game is targetinga new user population it was important that the participants really matched. Dueto limitations in how participants could be acquired it was decided for the first testto recruit the participants from the employees at DICE. Although most of these aregamers suitable participants can be found within this group as well. Five participantswere selected in accordance with Nielsens [9] recommendations. Key characteristics forthe selected participants is common knowledge and experience of using computers, butslim or no experience of computer games and of FPS (first-person-shooter) games inparticular. The participants of this study were distributed as in the following table:

Table 5.1: Participant characteristicsGender Age

GroupComputerusage perday

Play com-putergames now

Frequency ofplay when play-ing/played

Participant 1 Female 25-35 8-10h Yes Monthly 0-2hParticipant 2 Female 25-35 8-10h No Monthly 0-2hParticipant 3 Male 25-35 8-10h Yes Weekly 0-2hParticipant 4 Male 13-18 0-2h Yes Weekly 0-2hParticipant 5 Male 25-35 2-4h Yes Monthly 4-6h

As can be seen in table 5.1 the participants include representatives of the entire targetpopulation, although with a large majority of the older group. Despite the skew distri-bution can be seen as a satisfying representation of the target population consideringthat the participants where recruited solely within DICE.

5.3.3 Results

The compilation and summarizing of data commenced already during testing phase butthe main part of the work came after the testing had been finished. The results thatwere used were divided into two different types, performance results and satisfactionresults. This two different types of data were then used during the analysis process, andare presented in detail below.


Performance results

The performance results are divided into to sections, Effectiveness and Efficiency. Theeffectiveness section covers results that are concerned with how successful participantswere at completing different tasks. It does not consider how the goals where achieved,only the extent to which they were. The measures included in the efficiency sectionrelates the effectiveness measures to how well the task were completed, for example howlong time it took to complete a task.

Effectiveness

Figure 5.3: Task score sorted by task, all participants included. 0 = completed withease, 1 = completed with difficulty, 2 = failed to complete

Figure 5.3 shows the task scores for all tasks and participants. As can be seen in thefigure task 3, 7, 10, 11, 13 and 16 were the tasks that caused problems for more thantwo participants. As a whole a majority of the tasks were finished without any majorproblems.

Efficiency

Figure 5.4 shows the time spent on each task by each participant. It clearly showsthat the tasks that generated the highest completion times where tasks, 3, 7, 10, 11 and13. This can be even more clearly seen in figure 5.5 which shows the mean times forall tasks. The mean time of task 4 is increased due to the long completion time of oneparticipant and is considered an anomaly.

Figure 5.6 shows the number of mouse clicks needed for each task. For each task nomore than 1-4 mouse clicks should be needed. The number of mouse clicks is in accordwith the results from mean time on each task and also with the task score. It showsthat the most difficult tasks, the ones with most mouse clicks, are 3, 5, 7 and 10. The


Figure 5.4: Time on task for all tasks and participants

fact that some results are much higher than others on tasks that previously have beenshown are not to be difficult can be explained with that participants tended to click inthe interface on parts of it that were not implemented, generating many mouse clicks.

Satisfaction results

The satisfaction results are more qualitative results than the performance results. Theseinclude answers from posttest questionnaires as well as comments made by the users.

Posttest questionnaire resultsThe first part of the posttest questionnaire was a section including Likert scale type

answers, the results from these are presented in table 5.2.The posttest questionnaire also included open ended questions where the users were

asked to state issues they thought existed with the interface. The following commentswhere gathered from these questions:

I thought the following aspects of navigating the interface where difficult

1. ”If the labels where to change place I still wouldn´t remember since I don´t readthe labels”

General comments

1. ”Icons instead of text in labels”

2. ”I think it looked and seemed easy”

3. ”It feels like a fast to navigate system where you fast/easy can begin to play andalso return after a match to finish”


Figure 5.5: Mean completion time for all tasks

Figure 5.6: Mouse clicks per task for all participants


4. ”In comparison with a game like Tiger woods its much easier and interesting tomake small changes , which is positive”

5. ”First reaction was that enter match was to be in the upper right corner, but ata second glance everything feels easy to find, probably more a habit. Fun with abit different looks”

Table 5.2: Posttest questionnaire resultsStronglydisagree

Disagree Neither Agree Stronglyagree

Overall I found thesystem easy to nav-igate

0 0 0 3 2

I felt I got a goodoverview of the in-terface

0 0 0 4 1

I found the labelseasy to understand

0 0 2 2 1

The items I waslooking for wherewere I expected

0 0 2 3 0


Due to limitations of what information is public about the computer game that thisthesis concerns, the analysis of certain results, unfortunately quite a large portion, havebeen excluded.

Task results analysis

The first task that posed a problem for the participants was task 10 Find how manypoints you have to the next level. Two out of five had problems completing the task andone participant failed to complete the task. The problems were to figure out where theinformation was in the system. It was not at all apparent that it was on the main page,so they solved it by trial and error. Of the two participants that completed the taskwithout any problems, the first had a strategy of ignoring the labels and just clickingaround to find information, therefore she read the information on each page. So whenshe entered the main page the first time she read all information that was on it andremembered that it was there. The second participant that completed the task withease had more experience on games and FPS games than the others and that personalready had a mental model that said the information should be on main. For the mainpage there exist no clear mental model on what’s on it and the label really says nothing,this makes the task difficult for first time or inexperienced users.

Task 11, Change your character, also posed problems to the participants. The buttonfor changing character was located on the top right of the screen above the menus in anarea that is always visible. The big issue here was that the button was not where theparticipants expected, most that had problems with this task looked for the button on


the main page besides the character there. It was difficult for the participants to locatean element that was above the menu and outside the ”area of focus”.

Task 13, Check which character you are logged in as, was also problematic for thesame reasons as for task 11. Since the information wanted was above the menu and thearea above the menu was fairly small it was difficult to locate the information there,again participants felt it should be beside the character on the main page.

The last problematic task for some users was task 16, Enter a match. The biggestissue with it was to recognize the button, it was not apparent enough, and the termi-nology on the button was not clear. This was most problematic for inexperienced users,the participants who had more experience of computer games found the button moreeasily and understood the terminology.

One finding that also came from the test was that users often were not entirely clearon which page they were. The labels in the menu were not clear enough and did notdraw enough attention to themselves to let the users know where they were.

What was also apparent when analyzing the results from the test was that the par-ticipants quickly learned how the hierarchical structure of the interface worked. Whenthey had learned it in one section it was easy for the participants to apply this knowledgeon other sections and they quickly built a mental model of how the system worked.

Posttest questionnaire analysis

When analyzing the posttest questionnaire the general opinion among the participantswas that the system was easy to navigate and it was also easy to get a good overview ofit. As for the labels the most participants felt that they were clear. Some participantsdid not think they caused a problem but that they were not obvious. The opinionswhere the same for locations of items on the structure.

In general the comments were positive, one participant did not care about the labels,she only clicked around the sections looking at the content without paying any attentionto the labels, therefore she suggested having icons in connection with the labels to havesomething to associate the information she found with. One other thing commentedwas the placement of the ”enter match” button. The participant felt it should be in theupper right corner, concluding to say its only a habit were it should be.

5.3.5 Conclusions

When looking at the results of the study and putting them in relation to the testobjectives stated prior to the study one can draw some important conclusions. First ofall the participants had some problems prior to the test comprehending the meaning ofsome of the main section labels. This meaning became clearer during the test and mostparticipants had a satisfactory understanding of the main section labels after completingthe test. As for the sub section labels these were easier for the participants to understandsince they had support from the main section labels when interpreting their meaning.

As for navigation in the interface none of the users experienced any problems. Theyfreely navigated between the menus and fully grasped the interaction style. As for thelast test objective it was a bit harder to measure and also had to be put in relation tohow well the users understood the labels. As a whole the groupings of objects mostlyfit the participants mental models. Most participants looked for objects in the correctsection assumed they understood the label. Summarizing the study it answered thetest objectives formulated prior to the study quite well and the results can be useful toformulate design suggestions.

5.4. Redesigning 41

Design suggestions

One of the more important design suggestions regards how to identify sub menus. Insome way it should be indicated in what sub-section a user is. Whether this is done witha label on each screen that says the same as the sub menu option, or if the content cancommunicate the information will be an issue of the implementation. This is necessaryfor first time users to get them over the learning threshold, and start with a positiveexperience of the system.

The main page is also something that should be redesigned. The label does notsay much to new or inexperienced users and does not say anything about what to findon that page. Hence either the label should be changed or possibly skipped and theinformation distributed onto other pages. That leaves a problem with what to have asa first page.

The button for changing character should be moved to a position where it is adjacentto the characters picture, since that felt like a more logical place for it. The same goesfor the information of the name of the character that is logged in. The problem thatmight arise is if the first page is skipped where to put this information.

The last design change that is proposed is to change the label of the ”enter match”button. The current label uses a specific terminology and is not that easy to under-stand for new or inexperienced users. Something like ”Play now” is probably easier tounderstand. One suggestions from a user comment is to in some way make the labelslook a bit different from each other to more easily identify a section on the look of thelabel rather than the text on it.

There were also some discoveries made that was not part of the test but came fromcomments made by users about the content of different pages. It became clear that thegroupings of objects have to be made more clear, what belongs to what, like that thecategory bar has to do with the items field and is not a place were items equipped onthe character are located.

5.4 Redesigning

With the findings from the first study and the implications of what needed to be changedat hand it was time to start to look at what changes to be made within the design ofthe interface.

The first change in the design was to change the label on the ”Enter match” buttonfrom ”enter match” to ”play now”. This to try to more clearly indicate what pressingthe button leads to. As a result of the study the organization of the interface was alsochanged, and the groupings among the menus and sub menus was changed. It was alsodecided to indicate the page shown by making the labels in the menu bars larger and ina different color of the active page than the other labels, this to make it stand out moreamong the labels.

For this test the statistics of the current player was only located on the first pageand it was apparent that this was not a satisfactory solution since it was difficult tofind, especially for inexperienced computer game users. Therefore the statistics bar wasredesigned and incorporated on all pages except the options pages.

The test also affected the future design of the interface, it gave suggestions for whatwas needed to fulfill when the content of the pages was designed to make it easier todistinguish between the different pages.


Figure 5.7 shows the interface main page after that the design changes have beenimplemented. The most significant changes made to the main page are that the nameof the character and the ”change character” button have been moved from the top ofthe screen to the top right of the statistics bar, below the menu. It is also worth notingthat the number of sections have been reduced from the original seven to five. The labelon the ”enter match” button has also been changed to ”play now” instead.

Figure 5.7: Design of main page after the first usability study

Figure 5.8 shows the appearance page after the redesign. The first thing to be notedis that the items and appearance sections have become sub sections under a new sectioncalled My Stuff. It can also be seen that the design of boxes for putting items in havechanged and they are now more logically grouped around the character and adjacent tothe part of the character which they represent.

The statistics bar has been added to all screens and can be seen on the appearancepage after the redesign. It now contains all the statistics that concern the player on allpages. The tab system has also been changed to better fit the size of the container belowit. It can also be seen that the jacket that the character has equipped is highlightedwith a yellow frame.

5.5 The Second Study

This second usability study on the games menu was conducted in the middle of thedevelopment cycle of the interface. After the first study was finished and design changessuggested, these changes were first of all implemented. The design of the system upuntil the first study only included the menu structure of the interface, now it was timeto start to design the interaction style and the layout of the specific screens in moredetail. The results of this design work can be seen in pictures 5.7 and 5.8.

This second test can be considered being an assessment test as described by Rubin[13]. During the test the participants were asked to perform tasks, which were possible

5.5. The Second Study 43

Figure 5.8: My stuff appearance page after the first usability study

to fully finish, which was not the case in the first study. As in the first study this secondstudy also tried to investigate how well the interface supported the users mental modelabout groupings of objects and functionality. In addition the test also investigated howwell the different interaction styles in the system worked, if the participants compre-hended how the system worked and how easy it was for them to learn how it worked.As during the first study the participants were asked to think aloud.

5.5.1 New design prior to the second study

The design of the interface aims at being as natural and easy to use as possible. Thishas affected the layout and interaction styles therein. As can be seen in picture 5.7 theinterface uses a navigation bar similar to the ones found on web sites and other softwareproducts. The main interaction style used, besides clicking on buttons, is drag anddrop. This means that users can drag objects from one part of the interface to anotherto perform a certain action, such as equipping the character with a jacket. This is oneof the actions that can be performed on the screen seen in picture 5.8.

When progressing in the game the players character can become eligible for the newobjects which will then be added to the characters inventory. This event is somethingthat need to be visualized in the interface. This is done via adding a highlight to allmenu labels which have a new object under them and also adding a highlight to theobject itself.

Most objects, like weapons, also have some characteristics which the user can displayby clicking on the object. This was first displayed with a tool tip on the object. Howeverthis was considered inhibiting when a user wanted to fast and easy compare differentobjects and was therefore changed to a static information box.

The overall goal for the interface is to be as consistent as possible, as many screensas possible should look and function the same, users should when entering a page for


the first time know how to interact with it based on their experience from previouslyvisited pages.

5.5.2 Goals

The goals or test objectives for the second study are the same as for the first study,with the addition of some new goals aiming at investigating the added design elementsdescribed in the prior section works. These test objectives were:

1. Does the user understand the main section labels?

2. Does the user understand the sub section labels?

3. Can the user freely navigate between the different sections of the interface?

4. Do the screens reflect the user’s mental model concerning groupings of objects andsections?

5. Does the user understand how to equip objects?

6. Does the user understand how to display information about objects?

7. Does the user understand the highlighting when acquiring a new object?


The participants of the second study were expected to meet the same demands of com-puter literacy and slim experience of computer games as for the first study. For thisstudy the participants were mostly recruited from outside of DICE to get participantsthat does not work with games on a daily basis. Also for this study five participantswere also recruited in accordance with Nielsens [9] recommendations. Out of these fiveonly one participant was recruited from within DICE. The characteristics of the userscan be seen in table 5.3.

Table 5.3: Participant characteristicsGender Age

GroupComputerusage perday

Play com-putergames now

Frequency ofplay when play-ing/played

Participant 1 Male 18-25 6-8h Yes Weekly 2-4hParticipant 2 Male 18-25 6-8h Yes Monthly 2-4hParticipant 3 Male 18-25 8-10h Yes Weekly 2-4hParticipant 4 Male 25-35 8-10h Yes Monthly 2-4hParticipant 5 Female 18-25 6-8h Yes Weekly 0-2h

The participants chosen for the second study do not cover the entire target popula-tion. They include only users from the older participant group. Apart from that theyfit the characteristics of the target population with a high usage of computers but quitefew hours of computer game usage.


5.5.4 Results

As for the first study the compilation and summarizing of results began during thetesting phase, but it was after the testing that the large part of the work was conducted.The results were divided into two types of results, performance and satisfaction results.Each set of results is discussed in detail below.

Performance results

One conclusion that was drawn from the execution of the first study was that efficiencymeasures are very difficult to use. This is especially so since the study uses the thinkaloud protocol, which means that the users are delayed when performing tasks and howmuch they talk affect timing. Therefore the efficiency scores for the second study havenot been included.

Effectiveness

Figure 5.9: Task score sorted by task, all participants included. 0 = completed withease, 1 = completed with some difficulty, 2 = completed with difficulty, 3 = failed tocomplete

As can be seen in figures 5.9 and 5.10 there were some tasks that caused moreproblems for the participants than others. Especially tasks 3, 7, 11 and 14 caused quitelarge problems for a majority of the participants. On tasks 7, 11 and 14 two or three ofthe participants failed to complete the task. Tasks 6, 8 and 9 also caused problems forthe participants. In general participants encountered more problems when performingthe tasks in the second study than in the first, this is possibly due to that the tasks weremore complex in the second study. It should be noted that the scores for the second


Figure 5.10: Mean task score sorted by task, all participants included. 0 = completedwith ease, 1 = completed with some difficulty, 2 = completed with difficulty, 3 = failedto complete


study had a finer granularity than the scores in the first study, having four steps wherethe first study had three.

Satisfaction results

Posttest questionnaire resultsAs for study one there was a posttest questionnaire asking the participants about

their thoughts about working with the interface. The posttest questionnaire had ques-tions with Likert scale type answers and open ended answers. The results from thequestionnaire can be seen in table 5.4.

Table 5.4: Posttest questionnaire resultsStronglydisagree

Disagree Neither Agree Stronglyagree

Overall I found thesystem easy to nav-igate

0 0 1 4 0

I felt I got a goodoverview of the in-terface

0 1 1 2 1

I found the labelseasy to understand

0 0 0 3 2

The items I waslooking for wherewere I expected

0 1 0 4 0

It was easy to equipitems

0 0 2 0 3

It was easy to getinformation aboutitems

0 1 0 1 3

I thought there wasmuch inconsistencyin the system

1 3 0 1 0

I needed to learnthings before I couldget going with thesystem

1 3 0 1 0

General comments

1. ”To top heavy, there are to much information on the top of the screen”

2. ”To equip something in a specific box was a bit hard to see. The yellow borderwasn’t so strong. Dimming the other boxes might be good”

3. ”The ’Play’ button does not look like a button, more like a label”

4. ”Looked a bit for the Play Now button (although it was pretty visible) I expectedit to be adjacent to mission related things”


5. ”It only took me a few minutes to feel comfortable, but the first minute or so Ifelt slight confusion”

6. ”I though the play now looked more like a company logo, less than a button, butwhen I started looking I found it fast. I would have liked to have it to the right,bottom or top”

7. ”Knowing when things would get equipped was difficult, since there was no high-light of where it would end up”


As for the first study there are limitations in what information that can be included inthe analysis section, therefore some tasks that posed problems for the participants maynot be addressed.

Task result analysis

The first task that posed a problem for a majority of the participants was task 7, Equipyour new jacket. This task began when the participants got a new jacket, they wherethen to follow the highlight to the correct section where the new jacket was and equip it.Most participants navigated to the correct section but did not succeed in equipping thenew jacket but equipped another jacket. This is a clear indication that the highlight didnot serve its purpose in guiding the users to a new item. From reviewing the recordingsit is quite obvious that the participants navigated to the correct section because ofreading the labels and not following the highlight. When about to equip the new jacketthe highlight did not guide the users at all. One participant actually thought that thehighlight indicated the jacket that was equipped.

Task 11, Equip a weapon, was also difficult for the participants. The purpose ofthis task was for the participants to equip a weapon to the equipment bar. On theequipment bar there is a limitation which has the effect that certain weapons, due tosome preconditions, can only be placed in certain slots. This was highlighted by lightingup the frame of the boxes where a weapon could be put. This highlight was not distinctenough and did not guide the participants.

The last task to pose a problem for the participants was task 14, Check how manypoints you have to your next level. Points is represented in the statistics bar by two num-bers, one post called ”Experience” which tells the user how many points their characterhas. Then there is the ”Next level” which represents the number of points needed forthe next level, so number of points to next level is ”Experience” subtracted with ”Nextlevel”. This caused a problem for a majority of the participants. Instead of interpreting”Next level” as the points at next level they interpreted it as points to next level. Itwas clear that participants either understood how to interpret it or they did not, noneof the participants hesitated that their interpretation was correct.

Posttest questionnaire analysis

In general the users found it quite easy to use the system, most of them thought thesystem was easy to navigate and they had no problems interpreting the labels. Amajority of the users thought that items were where they expected them to be, and thatthey were fairly easy to equip. Although one comment was made by a participant thatit was difficult to know where an item would end up when equipping it because that the

5.6. Redesigning 49

slots did not light up. One observation that was made during the test about equippingitems, was that it took some time for the users to learn how to equip items, althoughwhen they had learned the interaction style they quickly applicate it on other screensas well.

Most of the users felt the system had little inconsistency and that they had tolearn fairly little before being able to fully use the system. One thing that many usersmentioned in the open ended questions were that the ”Play now” button did not looklike a button but was confused for a logo and therefore not payed attention when lookingfor it.

In general the results from the second study posttest questionnaire was more diversethan for the first study, this could possibly be explained by that more and more complexfunctionality was included in the second study.

5.5.6 Conclusions

If putting the findings of the study besides the test objectives it become apparent thatthere are some issues with the current design of the interface. Many participants hadproblems when equipping items, the biggest problem was not in how to equip items butwhere they were allowed to be put. The limitation on certain items and where theycould be put was not sufficiently communicated through the interface.

Displaying information about objects was straight forward for all users and did notpose any problems.

When it comes to understanding the highlight trail towards new objects it was ap-parent that none of the users understood what it represented and hence did not useit. Overall there were no problems with navigating the interface and the participantsfound it straight forward and easy. The labels of the interface had been changed andrearranged since the last study and that increased the understanding of them and helpedthe navigation.

Design suggestions

One important design change that need to be made is in indicating where an item canbe placed when equipping it, it should be clearly indicated where it can go and whendragged over slots, which slot it is about to be placed in. It should also be indicated inwhich slots an item cannot be placed.

The highlight trail that leads to a new object in the characters inventory need to bemore clear about what it represents, especially at an item level.

The representation of points in the statistics bar need to be clarified so that it isclear what each number represents.

Finally the ”Play now” button needs to be redesigned to more resemble a buttonand not a logotype. Possibly it should also be relocated to the top left.

5.6 Redesigning

With the findings and design suggestions from the study finished redesign work of theinterface commenced, the changes can be seen in picture 5.11. The first thing that waschanged was to redesign the ”Play now” button so it looks more like a button, insteadof the square shape it had it got a round shape resembling a physical button.


The highlight trail that leads to a new item in the inventory was kept intact on thelabels but on the item the highlight of the item box was reinforced with a label on theitem itself indicating that it is a new item. The hope is that the label on the item willhelp users in understanding the previous trail and that users will be able to use thehighlight in a correct way.

Figure 5.11: My Stuff Appearance section after the second study

How the item slots looks like when equipping an item has also been redesigned. Theyellow frame that indicates where an item can be placed been reinforced along withdimming the slots in which an item cannot be placed. Everything to guide the users tothe correct slot for the item.

Finally the representation of points in the statistics bar was redesigned. Instead ofhaving a solely textual representation it have been complemented with a bar representinghow far the player has progressed between the current level, to the left, and the nextlevel, to the right. On the bar the current amount of points and how many that isneeded to go to the next level is displayed.

5.7 Discussion

Usability studies is an important tool in developing usable interfaces and systems andas Nielsen [9] notes it is always better to test one user than none. When performing anusability study it is important to be well prepared and to know what to do and how toact before, during and after the testing. It is extremely easy to affect a participant, in-tentionally or not and thereby have to discard the entire testing session. One conclusionthat can be drawn from my studies is that it should not be the designer who performsthe study. As a designer of a system one is all to biased to act as a test monitor aswell as analyzing the results. It is very easy to draw conclusions based upon how one,as a designer of the system, wants the system to work, not having an open mind andhence does not discover important issues and flaws with the system. But still, even as

5.7. Discussion 51

a designer of a system you always discover issues with parts of the system you thoughtworked well, but when put in front of a user does not work so well.

One thing that I have discovered when performing the studies is that it is extremelydifficult to act as a test monitor. Especially not helping the participants when they arestruggling is difficult. Before starting to act as a test monitor one should always watchan experienced test monitor to get tips on how best to act in certain situations that aresure to emerge and difficult to handle.

As aforementioned usability testing is an important and neglected part of the designprocess when developing a new product. One should not underestimate the importanceof running usability studies.

Chapter 6

Conclusions

When the work with this interface started there was no real design on paper, only aflow chart and a requirements for the system. The system has evolved in a pleasing wayand it would not have been what it is today if it were not for the studies conductedduring the design process. All studies have contributed in their stage of the developmentcycle, from the Card sort study, which gave insights into how actual users of a systemthought information and functionality should be grouped, to the Usability studies, whichevaluated actual mock up implementations of the interface and gave an important insightto how well the system supported users performing actual tasks.

The design process of a system and consequently the evaluation process of it issomething that never ends. It may be less extensive and have less possibility to affectthe design of the system as the development progress proceed. But it will never end noteven after the system has been launched. Systems are constantly being used, and henceevaluated by users even after launch. It is however important to conduct as much of thetesting of the system prior to launch when it is fairly easy to make changes.

As this report is being written the system is slowly moving from the design phase toearly stages of the development phase and it will in the future be put in front of actualusers on a day to day basis. It is only then it will be discovered if the design is successfuland if the studies conducted were sufficient and resolved all potential issues.

53

54 Chapter 6. Conclusions

Chapter 7

Acknowledgments

First of all I would like to thank Ben Cousins. He is the reason this thesis became areality since he is the one who hired me and let me write my thesis on DICE. Second Iwould like to thank my supervisor at Umea University, Hakan Gullikson, whom withoutthis thesis would have been much harder, and probably not as good. I would also liketo thank James Salt who is the Lead designer on the project and have helped me indeveloping this interface and discussing how it should be designed, he always have someconstructive input in stock. One person I also would like to thank is Robert Sammelinwho is a Concept artist at DICE and the one who have created the character that is inthe system mock up.

Last but most important I would like to thank my wife Karolina, without her supportwriting this thesis would have been much harder. Her taking care of our dogs andapartment all by herself made it possible for me to move to Stockholm for six months.So once again thank you my dear.

55

56 Chapter 7. Acknowledgments

References

[1] Adobe. Adobe - flash cs3 professional, interactive multimedia, interactive design.http://www.adobe.com/products/flash/.

[2] Robert Reimann Alan Cooper and David Cronin. About Face 3; The Essentials ofInteraction Design. Wiley Publishing, Inc, Indianapolis, Indiana, 2007.

[3] DICE. Dice company website. http://www.dice.se (Accessed 2007-09-21).

[4] Gerry Gaffney. What is card sorting?, 2000.http://www.infodesign.com.au/ftp/CardSort.pdf (Accessed 2007-09-05).

[5] Information Architecture Institute. What is information architecture, 2007.http://iainstitute.org/documents/learn/What is IA.pdf (Accessed 2007-10-01).

[6] Dirk Knemeyer. Information design: The understanding discipline, 2003.http://www.boxesandarrows.com/view/information design the understanding discipline(Accessed 2007-10-01).

[7] Donna Maurer. Instructions for use: Card sort analysis spreadsheet, 2007.http://www.rosenfeldmedia.com/books/downloads/cardsorting/cardsort analysis.pdf(Accessed 2007-08-30).

[8] Donna Maurer and Todd Warfel. Card sorting: a definitive guide,2004. http://www.boxesandarrows.com/view/card sorting a definitive guide (Ac-cessed 2007-08-29).

[9] Jakob Nielsen. Why you only need to test with 5 users, 2000.http://www.useit.com/alertbox/20000319.html.

[10] Jakob Nielsen. First rule of usability? don’t listen to users, 2001.http://www.useit.com/alertbox/20010805.html.

[11] Jakob Nielsen. Card sorting: How many users to test, 2004.http://www.useit.com/alertbox/20040719.html (Accessed 2007-08-30).

[12] Jenny Preece, Yvonne Rogers, and Helen Sharp. Interaction Design, beyond human-computer interaction. John Wiley and Sons, Inc, 2002.

[13] Jeffrey Rubin. Handbook of Usability Testing; How to, Plan, Design, and ConductEffective Tests. John Wiley and Sons, Inc, 1994.

[14] Lynn Stott. Information architecture: A rose by any other name..., 2004.http://www.boxesandarrows.com/view/information architecture a rose by any other name(Accessed 2007-10-01).

57

58 REFERENCES

[15] TechSmith. Techsmith morae, 2007. http://download.techsmith.com/morae/docs/whitepapers/morae datasheet.pdf(Accessed 2007-12-05).

[16] Optimal Usability. Optimalsort - what is it?, 2007.http://www.optimalsort.com/pages/what is it.html#demo (Accessed 2007-08-26).

evaluation of a computer game menu interface during … · evaluation of a computer game menu...

Documents