designing websites for esl learners: a usability testing study

35
207 CALICO Journal, 25 (2), p-p 207-240. © 2008 CALICO Journal Designing Websites for ESL Learners: A Usability Testing Study MIN LIU TOMOKO TRAPHAGAN JIN HUH YOUNG IHN KOH GILOK CHOI ALLISON MCGREGOR The University of Texas at Austin ABSTRACT The purpose of this paper is to report on a usability study for ESL websites conducted to gain insights from learners of English as a second language (ESL) as they interacted with specific sites. Five carefully selected ESL sites were tested by 10 different users generating a total of fifty testing sessions. Two major research questions guided the study: (a) what are intermediate-level ESL learners’ perceptions of the design features for the specific ESL websites? and (b) what design features contribute to learning from the ESL websites? Participant reactions to site specific learning tasks and evaluative questionnaires provided evidence of design features relevant to the ESL context. The findings address information, interface, and interaction design issues as well as highlight the importance of (a) organizing the site meaningfully, (b) recognizing the users’ prior knowledge to identify unambiguous labels and categories, (c) making the site easy to navigate, and (d) aligning learning activities or tasks with how ESL learners learn. Re- sults of the study provide implications of the usefulness of ESL websites, the effective- ness of site design features, and the degree to which sites meet ESL learners’ needs. KEYWORDS English as a Second Language (ESL), Website Design, Usability Testing, User-centered Design, Second Language Learning and Teaching INTRODUCTION The world wide web has become an established tool for language learning and teaching (Suss- er & Robb, 2004; Warschauer, 2005). Given the vast number of websites related to English as a Second Language (ESL) and the proliferation of their use, questions have been raised regarding the effectiveness of this evolving technology for learning English. Specifically, from an evaluation standpoint, Susser and Robb (2004) assert that “the vast quantity and variety of Internet resources for language study is welcome but leaves us with the problem of evalu- ating the quality of what is available” (p. 279). Investigating design effectiveness of ESL websites requires establishing good design criteria for the context of second language acquisition (SLA). One window into understanding the design factors is by obtaining information directly from users. Usability testing provides an opportunity for understanding users’ experience with a particular website. According to

Upload: phunglien

Post on 01-Jan-2017

223 views

Category:

Documents


1 download

TRANSCRIPT

207

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

CALICO Journal, 25 (2), p-p 207-240. © 2008 CALICO Journal

Designing Websites for ESL Learners: A Usability Testing Study

Min Liu

ToMoko Traphagan

Jin huh

Young ihn koh

giLok Choi

aLLison MCgregor

The University of Texas at Austin

ABSTRACT

The purpose of this paper is to report on a usability study for ESL websites conducted to gain insights from learners of English as a second language (ESL) as they interacted with specific sites. Five carefully selected ESL sites were tested by 10 different users generating a total of fifty testing sessions. Two major research questions guided the study: (a) what are intermediate-level ESL learners’ perceptions of the design features for the specific ESL websites? and (b) what design features contribute to learning from the ESL websites? Participant reactions to site specific learning tasks and evaluative questionnaires provided evidence of design features relevant to the ESL context. The findings address information, interface, and interaction design issues as well as highlight the importance of (a) organizing the site meaningfully, (b) recognizing the users’ prior knowledge to identify unambiguous labels and categories, (c) making the site easy to navigate, and (d) aligning learning activities or tasks with how ESL learners learn. Re-sults of the study provide implications of the usefulness of ESL websites, the effective-ness of site design features, and the degree to which sites meet ESL learners’ needs.

KEYWORDS

English as a Second Language (ESL), Website Design, Usability Testing, User-centered Design, Second Language Learning and Teaching

INTRODUCTION

The world wide web has become an established tool for language learning and teaching (Suss-er & Robb, 2004; Warschauer, 2005). Given the vast number of websites related to English as a Second Language (ESL) and the proliferation of their use, questions have been raised regarding the effectiveness of this evolving technology for learning English. Specifically, from an evaluation standpoint, Susser and Robb (2004) assert that “the vast quantity and variety of Internet resources for language study is welcome but leaves us with the problem of evalu-ating the quality of what is available” (p. 279).

Investigating design effectiveness of ESL websites requires establishing good design criteria for the context of second language acquisition (SLA). One window into understanding the design factors is by obtaining information directly from users. Usability testing provides an opportunity for understanding users’ experience with a particular website. According to

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

208

Nielsen, Snyder, Molich, and Farrell (2000, p. 7) usability testing “involves watching a person use the site in order to discover the ways the site aids or hinders people from reaching their goals.” Usability testing results not only highlight user preferences but also their behaviors while interacting with sites. The objective of this research is to contribute to design guidelines for ESL websites based on such directly observed user preferences and behaviors.

REVIEW OF THE LITERATURE

Websites can be useful resources in fostering English language learning. In particular, their capacity to offer authentic language tasks (e.g., reading authentic materials, participating in online chat, listening to native speaker recordings, and communicating with native speakers) and to boost students’ motivation holds promise for facilitating the acquisition of English. Some studies have shown positive effects of using web-based activities on the students’ con-versational, reading, and writing skills (Gu, 2002; Kung & Chuo, 2002; Lin, 2003). Other stud-ies have reported that web-enhanced activities fostered students’ development of in-depth cultural understanding while improving linguistic proficiency, as well as generating increased engagement in the learning activities (Hoecherl-Alden, 2000; Osuna & Meskill, 1998).

Literature on using websites in ESL classrooms, however, also suggests that simply providing technologically enhanced learning tools to students does not itself foster learning but that, instead, positive results require carefully designed materials based on a critical understanding of the relationship between the language-learning processes and technology (Warschauer, Turbee, & Roberts, 1996). According to Plass (1998), design should support the cognitive processes involved in linguistic and pragmatic skill development for SLA. His argu-ment is echoed by Peterson (1998, p. 350) who stated that “any general design rationale for the creation of web-based second language learning environments will be an integrative con-struct encompassing both a high-level theoretical view of how students learn and lower-level issues of interface design and site construction.” Therefore, an effective ESL web learning site should embody not only good design with technological enhancements but also connect to SLA theory.

Good design creates a positive experience for users and can lead to more desirable outcomes (Phyo, 2003). In human-computer interaction literature, much emphasis has been placed on the importance of user-centered design (Norman & Draper, 1986; Norman, 2002). Understanding how users think can help create websites that meet users’ goals and objec-tives (Krug, 2005). Design should address the three essential components of information, interface, and interaction. Information design is “the art of organizing data so that it is easy to find and use” the information on a site (Phyo, 2003, p. 2). Interface design refers to those elements that affect the appearance of a site (e.g., font, font size, colors, spacing, visuals, and icons) and the manipulation of these elements to both enhance the presentation of in-formation and make the site easy to navigate (Phyo, 2003). In other words, information and interface should be purposefully implemented so that they enrich the overall experience of using a site. Accordingly, the result of a well organized site is that users know at all times what they can do in the site, where they are, where they can go, and where they have been (Krug, 2005). Interaction design encompasses the elements that create a meaningful and enjoyable experience (Shedroff, 1994) for the purpose of enhancing the way people learn, work, communicate, and interact (Preece, Rogers, & Sharp, 2002). Clearly, the three design aspects are related to each other and, hence, many design features involve more than one aspect. Together, information, interface and interaction make up the basic tenets of a sound design and should, therefore, be incorporated in well designed ESL sites.

209

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Given the scarcity of empirical research addressing such design considerations for ESL sites, additional research is needed on how ESL learners actually use sites as a tool to facili-tate their learning. The objective of this ESL websites usability study was to gain evaluative information from ESL learners for the purpose of informing design practices. As stated above, usability testing involves observing target audience participants using sites so researchers can note how the sites help, confuse, or defeat users. Then, design principles drawn from the us-ers’ perspective (both impressionistic and behavioral feedback) gained from usability testing should help build effective and attractive websites (Preece et al., 2002). Two primary research questions guided the study:

1. What are intermediate-level ESL learners’ perceptions of the design fea-tures for specific ESL websites?

2. What design features contribute to learning from the ESL websites?

METHODOLOGY

Collection and Selection of ESL Websites for Usability Testing

This usability research study consisted of two phases: identifying potential ESL websites for usability testing and actual testing of five websites by ESL learners. Initially, a total of 80 websites were collected via solicitation from ESL instructors affiliated with a large university in the Southwest as well as Google searches for ESL-related websites. After reviewing each site’s goals, content, target skill levels, and media, a list of 37 websites was generated for poten-tial use in the study (see list of websites in Appendix A). A manageable number of candidate websites is recommended for evaluation (Susser & Robb, 2004), and according to Nielsen (n. d.), five evaluators are sufficient to locate most problems in a usability study. Given these recommendations, our goal was to select five sites from the list of 37 and test each site with 10 different ESL users to ensure both a manageable and rigorous research project.

To reduce the number of websites to five, we first excluded those websites that had broken links, charged a fee to use, or were part of other URLs on our list. Second, we elimi-nated sites that did not have a main learning component because our usability testing in-tended to evaluate websites from an ESL perspective in learning. At this point, the research team evaluated and rated each site using a 3-point scale based on how useful the site seemed to be. We retained the websites that received the highest ratings and two types of websites emerged: those that addressed more than one language skill and those whose primary focus was on grammar. Given this finding, we decided to focus our testing on the more comprehen-sive sites (sites addressing more than one language skill) and excluded those covering only English grammar. This resulted in a total of nine sites. The final selections were based on the type of media used, language skills covered, popularity, and appropriate activities for our in-termediate-level participants. The final set of five sites for the usability testing were

1. Stuff for English Learners (http://www.Stuff.co.uk/wicked.htm)2. English Club (http://www.englishclub.com)3. Grammar & Writing (http://grammar.ccc.commnet.edu/grammar)4. ESL Cafe (http://www.eslcafe.com)5. Study Zone (http://web2.uvcs.uvic.ca/elc/studyzone/index.htm)

Each site will be referred to by the name listed above in later sections of this paper. A brief description of each site follows.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

210

Website Descriptions

Stuff for English learners

The Stuff for English Learners (hereafter Stuff) site includes vocabulary, pronunciation, TOEFL, reading, and word games. It uses animation and interactivity supported by Shockwave and Flash technologies. For example, the Phrasal Verb Dispenser has a vocabulary activity for two-part verb combinations in which users first choose a topic (e.g. computers, emotions, etc.) from a side menu. Next, by clicking on arrows to change the verbs, particles, or meanings, users try to find the correct combination. Through another feature called the Phonetic Calcula-tor, learners hear English vowel sounds after clicking on the corresponding symbols based on the International Phonetic Alphabet.

English Club

The English Club site offers over 5,000 pages and provides numerous resources for both learners and teachers. These can be accessed through a tab menu across the top of the home page (Learning Center, ESL Clubhouse, Teacher’s Lounge, ESL Resources, and ESL World) or found through the site’s search function. Activities for learners include quizzes (some with automatic scoring), timed typing practice with automatic scoring, games, synchronous chat-ting, as well as English jokes, and articles for reading. The site includes lessons not only for general adult learners (from beginning to advanced levels), but also English for business and young learners.

Grammar & Writing

The Grammar & Writing site focuses on issues pertaining to writing, including the writing pro-cess, organization, and common problems for ESL learners such as primer style and sentence variety. The home page is organized around six themes (Word & Sentence Level; Paragraph Level; Essay & Research Paper Level; Ask Grammar, Quizzes, Search Devices; Peripherals & PowerPoint; and GrammarPoll, Guestbook, Awards) with pulldown menus to navigate to the information or activities included in each section. Lessons are mainly presented through text, but there are over 170 interactive quizzes and crossword puzzles with feedback created in JavaScript and CGI.

ESL Cafe

Users enter the ESL Cafe site by clicking on one of the following labels: Stuff for Students, Stuff for Teachers, or Stuff for Everyone. The Stuff for Students section includes Help Center, Hint-of-the-Day, Idioms, Phrasal Verbs, Pronunciation Power, Quizzes, Slang, and Student Fo-rums. Within this section, there are extensive lists of vocabulary (idioms, phrasal verbs, and slang) and over 40 interactive quizzes with automatic feedback/scoring on grammar, vocabu-lary, culture, reading, and writing.

Study Zone

The Study Zone site is designed for various levels of ESL learners: the lessons and practices are organized by five TOEFL score ranges (200, 330, 410, 490, and 570) with level specific lessons, grammar quizzes, reading, and vocabulary. At the 200 level, the vocabulary quiz-

211

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

zes are particularly extensive and guessing game activities are included. At the 490 and 570 levels, various activities such as timed reading, reading comprehension exercises, grammar exercises, crossword puzzles, and writing exercises are offered based on reading passages. The exercises have hints and/or immediate feedback.

Participants and Site Order

The participants in this usability study were intermediate-level ESL learners who volunteered to participate in the project in order to practice their English. Most were affiliated with a large research university in the southwestern part of the US and enrolled in ESL classes at the time of this research. Each participant was asked to test two sites with one hour allocated for each site. A short familiarity survey was given to the participants prior to the testing to determine whether they had had any previous experience using any of the websites being tested. Given their responses on the survey, the participants were randomly assigned to test a site with which they were not familiar. A total of 50 testing sessions were generated from 25 partici-pants. In other words, each of the five sites was tested by 10 different participants. We also counterbalanced the order in which the participants tested the sites to minimize any bias that could possibly be introduced through the site sequence.

Instruments

To address our research questions, participants’ information and feedback were collected through: (a) a demographic questionnaire, (c) an observation of how participants actually used the site while performing site related tasks, and 3) a feedback questionnaire evaluating the site. Construction of the instruments closely followed Nielsen et al.’s (2000) methodology and previous research on usability testing (Krug, 2005; Mayhew, 1999; Preece et al., 2002; Rubin, 1994) (see instruments used in Appendixes B and C).

Demographic questionnaire

The purpose of the demographic questionnaire was to collect participant information relevant to the usability testing of ESL websites. The questionnaire included three sections: (a) per-sonal information (e.g. age, gender), (b) level of computer and web experience, and (c) Eng-lish learning history including experience using the web for English training.

Site-related tasks

Based on Nielson et al.’s (2000) usability testing method, three types of tasks—first impres-sions, specific tasks, and open-ended tasks—were designed for each of the five websites tested. The research team developed these tasks to simulate activities learners would do in self-study. After each task, a think-aloud elicitation question was used to prompt verbal feed-back from the participants. Sample think-aloud questions included “Is it easy/difficult to find phrasal verbs in this site? Why? Do you like to learn this way (by trial and error)?” Each task is described in more detail below.

Task 1: First impressions. In order to elicit participants’ general first impression of the sites, free-form exploration was used for the initial task. The participants could look at any part of the website and were then asked to describe their first impression of the site.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

212

Task 2: Specific tasks. The aim of the specific tasks was to observe participants’ in-teraction with the site in semistructured tasks. Because the five websites covered different content, specific tasks were uniquely created for each site. For example, the following is a specific task used for English Club:

You want to improve your English speaking skill. Today, you have decided to study words and expressions for telephone English. Review the Telephone Terms section and answer the following questions:

1. Name one word or term that you newly learned from this site.2. How long did it take to find the Telephone Terms section?

Task 3: Open-ended tasks. We asked open-ended questions which required the partici-pants to browse the site in any way they chose and find information or express their opinions about the effectiveness of the activities. For example, the following are open-ended questions for Stuff:

Listen to a poem. Click Hints on Pronunciation for Foreigners and work on the activity for three minutes.

1. What did you learn from this activity about pronunciation?2. Do you think this information is useful?

For tasks 2 and 3, the topics were chosen carefully from different skill areas including grammar, listening, writing, reading, speaking, chatting, and games (see sample tasks in Ap-pendix B). One researcher created tasks for one site while other researchers from the team performed and evaluated the tasks. Modifications were made based on the feedback, and then two pilot-testing sessions were held to try out the tasks with two ESL participants from the target population of the study. The objectives of the pilot testing were to verify that (a) the tasks were understandable and doable, (b) the questionnaires made sense to the users, (c) the testing environment was functional, (d) the users understood the pretest introduction and instructions, and finally (e) the time allotted for each test was adequate. After pilot test-ing, several of the task explanations and survey questions were revised to make them more comprehensible. Also, to prevent user fatigue, the total number of tasks was reduced from five to four and the number of the required written responses was lessened.

Site feedback questionnaire

The purpose of the site feedback questionnaire was to assess the participants’ overall satis-faction with their website experience. First, based on a Likert scale of 1 (very unsatisfied) to 5 (very satisfied), each website was evaluated in nine aspects including ease of finding the information/activities, quality of the information/activities, and appearance of the site. In the second part of the questionnaire, participants were asked to rate how useful each site was to improve different English skills (i.e. speaking, listening, reading, writing, grammar, and cul-ture) on a scale of 1 (useless) to 5 (very useful). Finally, the third section included questions such as the likelihood of returning to the site and what they liked most or least about the site (See Appendix C for the questionnaire).

213

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Procedure and Testing Facility

The usability testing was conducted with one researcher as a moderator for each partici-pant. To ensure the consistency of the testing procedures, all the researchers followed the same testing protocol. When a participant arrived, the assigned researcher briefly explained the purpose of the test, the procedure, and instructions for each task. The participant then completed the demographic questionnaire. Afterwards, the participant was given the site fa-miliarity survey to identify previous exposure to any of the five selected sites. Based on this information, the researcher randomly assigned a site the participant had never seen or used before. Next, the participant went to the assigned website and completed the specific learning tasks described above. Each participant was observed while he or she navigated the site and performed the given tasks. During the observation, the researcher took notes and captured screen shots of the web page(s) in which the participant seemed to have difficulty in perform-ing the tasks.1 After each of the specific tasks, the researcher asked think-aloud elicitation questions. Finally, the participant was asked to complete the site feedback questionnaire.

The majority of the users, 23 out of 25, did their usability testing in one of the com-puter labs on the university campus. They used DELL PCs equipped with high-speed Internet connection with Internet Explorer and all plug-ins required to use the sites (i.e., Shockwave, Windows Media Player, and Flash Player). One user participated in this study in a university office with a Gateway PC laptop with the same software setup and Ethernet connection. The final participant completed the study at the home of one of the researchers using a PC desktop with the same software setup and high-speed cable connection.

RESULTS AND DISCUSSION

About the Participants

The ESL learners participating in this study ranged in age from 16 to 40, were intermediate-level in English-language learning, and equally represented males and females. All of them had experience in using computers, had access to the Internet, and daily surfed the web for various purposes. Three fourths of them had experience of using the web for studying English and reported doing so for about an average of 2 hours weekly (See Table 1).

Table 1Results of the Demographic Questionnairea

Age Gender Years of Eng-lish learning

Level of Englishb

Years of computer use

Computer experiencec

Web surfing experience

16-3072%

(n = 18)

male48%

(n = 12)

<= 541%

(n = 9)

intermediate91%

(n = 20)

2-512%

(n = 3)

basic52%

(n = 13)

basic60%

(n = 15)

31-4028%

(n = 7)

female52%

(n = 13)

> 559%

(n = 13)

intermediate high9%

(n = 2)

5-1060%

(n = 15)

experienced48%

(n = 12)

experienced40%

(n = 10)

advanced0%

(n = 0)

over 1028%

(n = 7)

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

214

Table 1 (continued)

Internet access

Hours of web use per day

Preference for using ESL webd

Experience in using web to study English

Hours of studying English using web

(per week)e

home76%

(n = 19)

< 252%

(n = 13)

low70%

(n = 14)

yes76%

(n = 19)

<= 268%

(n = 13)

home and work24%

(n = 6)

> 248%

(n = 12)

high30%

(n = 6)

no24%

(n = 6)

> 232%

(n = 6)aMissing responses were excluded. bThe Likert scale for this item ranged from 1 (Intermediate) through 5 (Advanced). c“Basic” represents levels 1 - 3 and “Experienced” represents levels 4 -5 on Item 9a of the questionnaire with 1 being “Not at All Experienced” and 5 being “Very Experienced” (see questionnaire in Appendix C).d“Low” represents levels 1-3 and “High” represents levels 4-5 on Item 14 of the questionnaire with 1 being “Not at All” and 5 being “Very Much.”eOnly respondents who use the web for studying answered this question.

Participants’ Responses in the Site Feedback Questionnaire

As shown in Table 2, the compiled ratings from the site feedback questionnaire indicated the highest general satisfaction with English Club followed by ESL Cafe. For these two sites, all items were rated 3 or higher on the 5-point scale (1-very unsatisfactory to 5-very satisfied). Participants reported viewing English Club and ESL Cafe positively in terms of information, interaction, and interface designs, as well as their value in helping them practice language skills. Participants also reported that they would be likely to use these sites again or recom-mend them to other people.

Table 2Results of the Site Feedback Questionnairea

Usability Aspect Stuff English Club Grammar & Writing

ESL Cafe Study Zone

Interfaceb 3.08 3.94 3.18 3.91 3.37

Interactionc 3.15 4.05 2.55 3.65 3.10

Informationd 2.80 4.05 3.35 3.90 3.55

Perceived Usefulness for Practicing Linguistic Skillse

2.66 3.57 2.55 3.29 2.87

Worth Visiting Againf 2.55 4.20 2.90 4.00 3.15aThe results of this questionnaire are based on a 5-point Likert scale.bInterface includes items 1, 3, 4, 5, and 9 of the questionnaire (see Appendix C). cInteraction includes items 6 and 7.dInformation includes items 2 and 8.eThis aspect includes item 10.fThis aspect includes items 11 and 12.

In examining how participants with different demographic characteristics rated their experiences using the five sites, we found that higher ratings were given by (a) females, (b) those with fewer years of English learning, (c) intermediate-level learners (rather than high-intermediate-level learners), (d) novice computer users, and (e) participants with no

215

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

experience in using the web to study English (see Tables 3 and 4 below). Other demographic differences were not noticeable.

The finding that female participants gave higher ratings than male participants is con-sistent with other research results indicating females tend to have more positive attitudes toward learning using computers than males (Hao & Liu, 2006). The results also indicated that less experienced English learners and computer users found these ESL sites more helpful than the more experienced users. Presumably, less experienced English learners were aided more or found more value in using the websites. Users with more computer and web experi-ence were perhaps more evaluative in their feedback indicating that those users with more technogy savvy may have different expectations than those with less experience.

Table 3Site Feedback Means Based on Demographic Questions

Demographic question Interface Interaction Information Linguistic skills

Worth visiting

Age group (n = 25) 16-30 (n = 18) 31-50 (n = 7)

3.513.45

3.243.46

3.493.64

3.043.04

3.303.54

Gender (n = 25) Male (n = 12) Female (n = 13)

2.673.27

2.573.06

2.713.29

2.292.94

2.343.10

Years of learning English (n = 22) <= 5 (n = 9) > 5 (n = 13)

3.803.33

3.563.15

3.813.37

3.372.93

3.843.15

Levels of English (n = 22) Intermediate (n = 20) Intermediate high (n = 2)

3.602.80

3.362.88

3.603.00

3.182.54

3.463.13

Years of computer use (n = 25) 2-5 (n = 3) 5-10 (n = 15) Over 10 (n = 7)

3.633.533.35

3.423.323.21

3.753.583.32

3.223.202.65

3.503.413.21

Computer experience (n = 25) Basic (n = 13) Experienced (n = 12)

3.583.39

3.403.19

3.693.35

3.192.86

3.383.35

Web surfing experience (n = 25) Basic (n = 15) Experienced (n = 10)

3.563.39

3.283.33

3.583.45

3.082.96

3.363.38

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

216

Table 4Site Feedback Based on Web-surfing Behaviors

Demographic question Interface Interaction Information Linguistic skills

Worth visiting

Internet access (n = 25) Home (n = 19) Home and work (n = 6)

3.423.74

3.223.54

3.513.58

3.043.05

3.233.79

Hours of web use per day (n = 25) <= 2 (n = 13) > 2 (n = 12)

3.593.38

3.373.23

3.693.35

3.232.87

3.543.19

Preference for using ESL websites (n = 20) Low (n = 14) High high (n = 6)

3.533.27

3.213.33

3.453.46

3.182.85

3.383.21

Experience in using web to study English (n = 25) Yes (n = 19) No (n = 6)

3.423.73

3.203.63

3.423.88

3.043.04

3.303.58

Hours of studying English using web (n = 19) <= 2 (n = 13) >2 (n = 6)

3.413.43

3.213.17

3.483.29

3.222.63

3.293.33

Findings from Site Task Performance Data1

Analyses of the participants’ reactions to the learning tasks took four steps. First, the obser-vation notes for each testing session were written up and compiled. Second, in order to make sense of the data in a consistent and systematic manner, we compiled observation notes from one site. The observation notes were organized into a table with three columns: (a) the participants’ reactions to the tasks as observed in their behaviors or their comments, (b) the researcher’s interpretation and description of what he or she observed, and (c) whether what was observed related to the information, interface, or interaction design of the site. As themes related to site design emerged, a template for organizing the data was established. Using this template, we organized the data for the other four sites and summarized the findings for each site. We attempted to extrapolate findings reflecting opinions of most participants and identi-fied design consideration across the sites. Finally, by drawing comparisons across all five sites, themes emerged. The following is a discussion of the results.

Overall site design issues

Organization of information. ESL websites often contain a rich amount of information. For users to access site content easily, organization in well thought out categories is imperative (Silver, 2005). Clearly presenting the site hierarchy or in layers of information is particularly important for the home page in which users attempt to understand how the information is organized in the site (Krug, 2005). For example, most participants favorably perceived the home page of ESL Cafe where the information was presented in three major categories (Stuff for Teachers, Stuff for Students, and Stuff for Everyone) and could identify how to enter the site through the navigation system (see Figure 1).

217

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Figure 1ESL cafe: Home Page

Similarly for Study Zone, the participants liked the idea of dividing the content of the website according to users’ English levels (see Figure 2).

Figure 2Study Zone: Home Page

They pointed out, however, that the TOEFL scores on the home page might not be meaningful to those users who have not taken a TOEFL test. Since there was no explicit expla-nation about the levels of the scores, some users might have a hard time understanding the significance of these numbers. Using more meaningful labels such as beginner, intermediate, and advanced together with (or without) TOEFL scores or providing a short explanation about the TOEFL scores would therefore have improved the usability of the site.

Participants found other aspects of ESL Cafe’s organization of information less clear.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

218

For example, on the Complete List page of the Idiom section, some participants stated that the alphabetical listing was not helpful and suggested listing the idioms by themes. Addition-ally, under Hint-of-the-Day, after clicking List All Entries, a long list of date, topic, number, and contributor’s name sorted by date was presented (see Figure 3).

Figure 3ESL Cafe: List All Entries Under Hint-of-the-Day

The participants were overwhelmed by the large amount of information and stated they could not easily find the information they were interested in. Similarly, on the Forum page, postings were listed without categorization, and some participants expressed frustra-tion in how much time it took to find information (see Figure 4). In all these cases, a theme-based categorization would have made the organization of information more transparent and accessible to the users.

Figure 4ESL Cafe: Forum Page

219

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Previous research on the organization of information has highlighted the importance of constructing a website’s content in a meaningful and relevant way from the users’ per-spectives (Preece et al., 2002). Our findings suggest that information on ESL websites can be meaningfully organized by category of user (e.g., teachers or students), skill level (e.g., beginner, intermediate, or advanced), task (e.g., grammar, reading, writing, listening, and speaking), or themes/topics.

Layout. Users have varying degrees of tolerance for visual noise, and some users are easily distracted by a complex layout (Krug, 2005). In general, participants in this study dis-liked busy layouts, those which included too much information or formats which required a lot of scrolling. Pages in the Grammar & Writing site, for example, tended to be long and wide because there were too many words on a page. The participants needed to scroll down to locate the information they were looking for (see Figure 5). As one user stated, “It was very difficult to find information …”.

Figure 5Grammar & Writing: Typical Layout of Content Page

Although there was an Index button on Grammar & Writing’s home page that led to an overview of all the topics covered in the site, it was placed near the bottom of the page. This potentially useful navigation tool, therefore, was not utilized by many participants.

Participants also disliked the long pages in ESL Cafe, which required a lot of scrolling (see Figures 3 and 4 above). In Study Zone, the vocabulary practice page was so long that us-ers had to scroll down repeatedly to view all the questions on the page. Participants suggested that the page could be designed with a different layout such as using pulldown menus or mul-tiple columns that could utilize the space on either side of the pictures. They also pointed out that the timer, which showed the remaining time for reading comprehension, was not visible after scrolling down in the window.

In general, pages crowded with words tended to overwhelm users, and long pages re-quiring a lot of scrolling made it difficult to find information quickly and easily. The challenge to present a rich amount of information in limited screen space necessitates efficient layout designs. For example, the reading comprehension section in Study Zone provided clearly defined task areas (Krug, 2005) in different frames so users could view the text passages, questions, and feedback to their answers simultaneously without feeling overwhelmed (see Figure 6). The participants particularly liked this feature.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

220

Figure 6Study Zone: Three Separate Frames

The participants also reported they could easily find information they were seeking in English Club, even with the large amount of information on the home page, because of the left-hand side navigation menu and a multiple-column layout (see Figure 7).

Figure 7English Club: Left-hand Side Navigation and Multiple-column Layout

Search function. A search engine is the first place where many web users go to find what they need (Krug, 2005; Silver, 2005). In English Club, all participants found the search engine function on the home page very helpful. However, it took a few participants some time to locate the search function (see Figure 8). The search box could have been displayed more prominently.

221

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Figure 8English Club: Search Engine Function

Use of pulldown menus. The use of pulldown menus are an effective way to save space when there is a lot of information to present in a website. However, they have a tradeoff: the items in the pulldown menus are not visible to users until they click on the pulldown menu or submenus in the pulldown menu (Krug, 2005). In the Grammar & Writing site, multiple pull-down menus were used as the main navigation tool, but participants could not see the items in the pulldown menus nor the hierarchy of the information without clicking on each one of the menus (see Figure 9).

Figure 9Grammar & Writing: Pulldown Menu

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

222

One user mentioned “[pulldown] menus in both the main and other pages are … very difficult and tedious [to use].” Pulldown menus can be useful, yet they should be used only for certain types of lists, such as alphabetized lists of known items (e.g., the 50 states) (Krug, 2005). Using them as the main navigation device is not effective as shown by the results of this study.

Home buttons and breadcrumbs. Breadcrumbs refer to a form of navigation that in-dicates the current location within the website by a list of pages from the current location up to the home page. Home buttons and breadcrumbs are among the most powerful navigation tools to indicate to users where they are in the site, especially for hierarchically organized sites (Silver, 2005). Breadcrumbs were not present or conspicuous in most sites we exam-ined. For example, in Grammar & Writing, navigating through content pages relied solely on the pulldown menus, and this made it hard for users to navigate back to the home page or other pages. The lack of site ID linked directly to the home page and the inconspicuous home button, placed at the bottom of each subpage (requiring users to scroll down to see it), in-creased participants’ frustration in navigation (see Figure 10).

Figure 10Grammar & Writing: Link to Home Page

In fact, two participants failed to find the Home buttons, and said, “I hope there is an icon that directly leads to the Home page somewhere because I had to click on the back but-ton of the browser several times.” This difficulty in navigation could have been reduced if the website had contained breadcrumbs.

The Study Zone site also provided go back buttons (to the previous or home page) as a main navigation tool, but these buttons were placed on the bottom of the page, and were hard for users to notice. Similarly, participants recommended a home button on Stuff and ESL Cafe to avoid getting lost. Such findings emphasized the need for using home buttons and breadcrumbs as essential navigation tools for sites containing a large amount of information with a complex hierarchical organization (Krug, 2005; Phyo, 2003).

Labels and word choices. Users decide which part of a website to click on based on the labels they see. The mapping between labels and the expected results from clicking them, therefore, should be clear to the users (Silver, 2005). As mentioned before, the TOEFL score

223

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

labels found on the home page of Study Zone could have been made more meaningful to us-ers if terms such as beginner, intermediate, and advanced had been used.

In the Idiom section of ESL Cafe, there were hyperlinks to Complete List, Meanings and Examples, and Idiom (see Figure 11).

Figure 11ESL Cafe: Labels

The Idiom hyperlink randomly brings up an idiom from the collection. However, only the word “Idiom” was hyperlinked, not the word “random” in front of it. One participant clicked on the link to Idiom without seeing “random” before it and got very confused to find one idiom presented to him. Also, another participant was very confused when he clicked on “E” on the Complete List page because the pulldown menu showed “F” instead of “E” (See Figure 12). In summary, when labels were not used properly, they misled and confused the participants.

Figure 12ESL Cafe: Menu of “E” Page

In the Grammar & Writing site, labels contained a number of grammatical terms, such as "conjunctions," "determiners," and "interjections," which were daunting to the intermedi-

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

224

ate-level ESL users. For example, “He [The participant] didn’t know the word plural and I [the moderator] had to explain it to him,” stated one moderator. Some participants experienced navigation problems simply because they were not familiar with the terms used in the menus and did not know where to click for the information they wanted, while the participants who happened to know such terms easily located their desired information. Although definitions of the terms were given at the beginning of each subpage, they did not appear to help the participants. This was perhaps because the participants were not likely to click on a menu item that they did not understand. Similarly, in using Stuff, some participants had trouble getting to the Phonetic Calculator page because they did not know what “phonetic” meant. Also in the Think of a Number activity in Stuff (see Figure 17 below), four participants failed to complete the task because they did not know the words “multiply” and “digits” used in the instructions.

Poor mapping diverts users’ attention from the task to the interface (Silver, 2005), thereby sacrificing learning efficiency. Labels and other language elements used in websites should be selected so that they are easily understandable by users. For ESL websites in par-ticular, users’ language proficiency and their prior knowledge should be taken into consider-ation in deciding which labels to use.

Color and font size. A home page is a preview into a website. Users often determine whether they want to look into the site further based on the feeling they receive from a home page. Choice of colors for a home page is critical in this respect. The yellow-based colorful de-sign of ESL Cafe (see Figure 1 above) was well received by most participants. One participant mentioned, “I like the interface design. They are colorful so that they make me interested in the things inside.”

In Stuff, five participants found the home page unattractive, and four of them specifi-cally attributed it to its dark background color, suggesting the need for the use of a lighter color (see Figures 13, 15, and 16 below). Similarly, participants perceived the grey back-ground color of the Study Zone site as gloomy. A dark background color makes pages not only unattractive but also makes the text hard to read (Vest, Crowson, & Pochran, 2005). Hence dark colors should be avoided for web page background.

Participants in general did no like the small font sizes used for some websites. For example, in English Club, a few participants complained that the home page was busy with numerous small-lettered links (see Figures 7 and 8 above). Similarly, in Study Zone, the small font size and narrow spacing in the Reading Comprehension section were mentioned as a sig-nificant design problem. Designers of these sites may want to consider increasing the size of the fonts.

Design of learning activities

Activity design and instruction. Activity designs for some sites were not intuitive, and partici-pants had difficulties in figuring out how to use the activities. This was particularly the case with Stuff. While exploring the site freely, four participants commented that they did not know how to do the activities. One participant, for example, said, “even though this website shows a new attempt to teach English using various multimedia tools, it is too difficult and compli-cated for me to figure out how to use each web page.” For example, in Phrasal Verb Drinks Dispenser (see Figure 13), the participants were supposed to change verbs, particles, and meanings until they found the corresponding combination of the three components using the arrows. Two participants simply clicked the buttons on the right to change the content areas and got lost. Another participant kept clicking the external links at the top of the page. Later,

225

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

this participant could not figure out how to check the right answers. There appeared to be too many operational steps for the users to perform the activity, and the operation was not intuitive enough. Although brief instruction was provided, many participants did not notice it because the instruction appeared below the activity box.

Figure 13Stuff: Phrasal Verb Drink Dispenser

In using ESL Cafe, a few participants got lost not knowing what they were supposed to do for the Hint of the Day activity. They explained that they did not know what the page was offering and/or how to use the page, specifically the search box. The title of the page, Hint of the Day, was not descriptive enough of what this section was about, nor did the search box or buttons clearly indicate what one was supposed to do on the page (see Figure 14). A brief description of what to do would be beneficial for this activity.

Figure 14ESL Cafe: Hint of the Day

In using Grammar & Writing, participants’ comments focused on navigation difficulties and the lack of instructions on how to use the website. For instance, one participant com-

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

226

plained, “I don’t know how to use … no introduction or instruction on how to get to the gram-mar part … it was not very clear to me.”

Ideally, the interface of an activity should be designed intuitively enough for users to easily navigate the site and complete the learning activities without instruction. If necessary, brief instruction should be provided in noticeable places for users to follow.

Animation and audio. One clear advantage of websites over textbooks or workbooks is their capacity to include multimedia such as graphics, animations, videos, and audios. In particular, it is desirable to provide audio in language learning websites since ample listening is considered important for language acquisition. In using ESL Cafe, one participant, who was very eager to learn how to pronounce the idioms presented in the site, said he sometimes had no clue how to pronounce the words without listening to them; there was no audio in ESL Cafe. The Pronunciation Calculator page in Stuff was one example that exploited animation and audio to facilitate language learning (see Figure 15).

Figure 15Stuff: Phonetic Calculator

In this activity, users could click the vowel buttons or combinations of buttons to play the vowels or diphthongs and also activate the animation that showed the lip movements to produce those sounds. Some participants mentioned that the animation was helpful to under-stand how to pronounce these sounds correctly. Unfortunately, the lips moved too fast and the animation was too small for some users to observe how the lips were moving. Many partici-pants mentioned that the audio in this activity was helpful. They said, “It was helpful because it gives actual pronunciation unlike book-type dictionaries” and “I could hear the difference between two sounds.” Also, for Hints on Pronunciation for Foreigners in Stuff (see Figure 16), one participant mentioned, “hearing the words helped me understand the words because I could compare some sounds that were similar to each other.”

While animation and audio seem to make activities look fun, users may not necessarily appreciate these features if they are not presented in a clearly defined learning framework. In the animation-based Think of a Number activity in Stuff, six participants said it was fun but that they did not consider it as helpful for learning. One participant said, “This is just for fun, not for learning English. If I have free time, I would do it just for fun, but not for studying.” Learning objectives should be defined before considering the inclusion of animation and audio, not vice versa. Also, it is important to consider loading time because a long loading time is likely to discourage users. Some activities in Stuff took a long time to load due to animations and audio.

227

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Figure 16Stuff: Hints on Pronunciation for Foreigners

Feedback. Complex activities may not be effective for learning because they demand a lot of time from users for relatively little learning. In the Phrasal Verb Drinks Dispenser activity in the Stuff site, participants were to change verbs, particles, and meanings until they arrived at the right combination of the three components (see Figure 13 above). All participants ex-cept for one stated that they disliked learning through this lengthy trial and error process. As one participant said, “Learning phrasal verbs this way is not easy, because you have to spend a lot of time, and it is boring. It would be better if they give you answers, after a couple of tries.” Another participant gave up on the task after spending a fairly long time on it and said she did not like this type of learning because finding a right combination was too difficult for her. She added, “If the website provided two of the three components and asked you to find the other one, it would have worked.” Still another participant said that one may not learn anything after spending so much time because the activity did not provide answers. Explor-atory learning needs to be designed with care. If users have to explore an activity too long without receiving any concrete feedback, they may lose interest in it or become frustrated. For Think of a Number activity in the Stuff site (see Figure 17), very few participants actually finished the task because the exercise required a long chain of responses, and making one mistake in the process resulted in entire failure in completing the task. If hints were provided for getting back to the right track after making a mistake, the activity may have been per-ceived as more useful by the participants.

Figure 17Stuff: Think of a Number

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

228

In all the above cases, task complexity could have been adjusted by incorporating some type of feedback. Potentially effective feedback for these activities could include provid-ing hints or answers on demand from users and providing a dictionary tool. Hints for Pronun-ciation for Foreigners in Stuff provided hints (see Figure 16 above), which alleviated some participants’ frustration when they could not find answers easily, although the participants still performed poorly due to the genre of the passage (poem) and the reading speed. Similarly, the Listen to Poetry activity in English Club allowed the participants to read the script along with the audio clip (see Figure 18), which was perceived as helpful. The participants, however, expressed that they would like to have an option of choosing whether they wanted to listen to the audio clip before actually looking at the script. When providing hints or answers, it is important to consider the ways for users to control the amount of help they receive, depend-ing on their language ability.

Figure 18English Club: Listen to Poetry

Many participants liked the Help Center in ESL Cafe in which learners could post ques-tions and receive answers. One participant explained that he was not good enough to com-municate with his teachers because of his pronunciation, but, if he used this online function, it would be much easier for him to ask his questions. A few participants, however, commented that they did not like to wait to receive answers via posting and preferred instead asking somebody they knew (e.g., their teachers) or finding answers promptly by using search tools. A few others expressed their desire to have quiz functions; one participant mentioned that information in ESL Cafe was good but not exciting and suggested including quizzes so that users could check their learning themselves. For Study Zone, participants stated that it would be more helpful if the website provided explanations for wrong answers.

When providing feedback, it is important to do so in a way not to discourage learners. One participant was annoyed with the feedback given in Phrasal Verb Drinks Dispenser in the Stuff site. Whenever she made a mistake, the program showed a pink object falling from the top with a dull colliding sound and displayed the word NO!, which she considered to be unpleasant (see Figure 19). Feedback should be provided to encourage learners to continue learning, and feedback to wrong answers should focus on the learners’ future improvement rather than their current mistake.

229

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Figure 19Stuff: Negative Feedback in Phrasal Verb Drinks Dispenser

SUMMARY AND CONCLUSIONS

Because learners’ ultimate goal in using ESL websites is to acquire knowledge and skills in English, their cognitive capacity should be oriented towards learning tasks rather than figuring out how to use the sites. In this respect, the usability of ESL websites is particularly impor-tant, and good design of a website increases its usability.

The findings of this usability study as revealed in the site feedback questionnaire (see Table 2 above) showed that overall the participants viewed the design of some sites (i.e., Eng-lish Club and ESL Cafe) more favorably than the others (i.e., Stuff and Grammar & Writing). The English Club site received the highest ratings of all five sites in terms of its information, interface, and interaction designs, and participants had the fewest problems in navigating the site while performing their assigned tasks as indicated in the site specific tasks discus-sion above. Such results indicate that English Club and ESL Cafe met ESL learners’ needs to a large extent. The results also showed that the participants were computer literate (see Table 3 above). All had web surfing experience and access to the Internet, and many of them used the web as a vehicle to learn English. This indicates a trend that as the web becomes a common tool at home and in classrooms, more ESL learners will rely on websites to develop their language skills. The design of ESL websites will become a significant factor to ensure their success and desirable learning outcomes. We have also found that more experienced computer and web users tended to have more critical views in evaluating websites (see Table 4 above). Creators of websites need to pay more attention to design practices as web technol-ogy matures and more people become experienced users.

As we observed our participants using the five sites, many usability issues surfaced. These issues, manifested less in some sites (e.g., English Club) but more in others (e.g., Stuff and Grammar & Writing), highlighted the importance of three design aspects: information, interface, and interaction. The insights we gained point to design features that are important to include in ESL sites to facilitate learning. Three main issues are important to consider for designing ESL sites.

First, ESL sites tend to carry a large amount of content because the development of language skills and knowledge requires exposure to a great amount of information. This com-plicates the design of content organization and navigation in multiple ways. In order for us-ers to locate desired information or exercises, content needs to be presented in an organized structure with meaningful categories recognizable to the users. Although one may be tempted to squeeze as much information as possible into a limited space, long pages crowded with a lot of information written in small fonts with narrow line spacing overwhelm users, and exces-sive scrolling hinders the visibility of information. Using common and established navigational devices such as search functions, links to the home page, and breadcrumbs will help users locate information quickly and enable them to identify where they are in the site and to move efficiently through the site.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

230

Second, users of ESL websites are trying to understand how to use the site in their second language. Special caution is required in deciding what words to use for labels and instructions. Technical terms should be avoided as much as possible, and labels and other language elements that are meaningful from a learner’s perspective should be used. Because users’ background knowledge and proficiency are likely to vary significantly, reference tools such as a glossary or links to dictionaries may also be useful.

Third, ESL websites often include learning activities such as quizzes, puzzles, and games. The design of learning activities should be guided by the learning objectives and SLA theory rather than the use of technology. Multimedia and feedback should be provided in ways to maximize learning.

Given the findings of this study, we recommend designers of ESL websites consider the following questions:

1. Is your site organized by level (e.g., beginner, intermediate, and advanced), skills (e.g., reading, writing, grammar, and speaking), themes or topics, and types of tasks (e.g., practice of irregular verbs)?

2. Are the pages easy to view in terms of page length, layout, text density, color, and font size?

3. How many clicks does it take to get to a specific place in your site, and can users find their way back to the main page easily? Should you add search functions, links to the home page, and/or breadcrumbs to your site?

4. What prior knowledge and level of English proficiency are assumed to make use of the labels and categories?

5. Will a learning activity or task on the site support SLA? Are the objectives of an activity and how to accomplish these objectives clearly communicated?

Websites are potentially powerful learning resources to a virtually limitless number of ESL learners. The design of ESL learning sites is a critical factor in how websites enhance their language-learning experience. Increasing the usability of an ESL website can lead to a more positive user experience and ultimately increase learning outcomes.

NOTE

1 After this article was accepted for publication, a number of the pages from the evaluated sites have been changed.

REFERENCES

Gu, P. (2002). Web-based project learning and EFL learners: A Chinese example. Teaching English with Technology: A Journal for Teachers of English, 2 (4). Retrieved October 24, 2005, from http://www.iatefl.org.pl/call/j_article10.htm

Hao, Y. W., & Liu, M. (2006). Students’ attitudes toward online interaction. Academic Exchange Quar-terly, 10 (4), 74-79.

Hoecherl-Alden, G. (2000). Turning professional: Content-based communication and the evolution of a cross-cultural language curriculum. Foreign Language Annals, 33 (6), 614-621.

231

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Krug, S. (2005). Don’t make me think: A common sense approach to web usability (2nd ed.). India-napolis, IN: New Riders Press.

Kung, S. C., & Chuo, T. W. (2002). Students’ perceptions of English learning through ESL/EFL websites. TESL-EJ, 6 (1). Retrieved July 10, 2006, from http://www-writing.berkeley.edu/tesl-ej/ej21/a2.html

Lin, A. (2003). An initial study on EFL learners’ attitude towards multimedia application in language learning. Teaching English with Technology: A Journal for Teachers of English, 3 (2). Retrieved October 24, 2005, from http://www.iatefl.org.pl/call/j_prev.htm

Mayhew, D. J. (1999). The usability engineering lifecycle: A practitioner’s handbook for user interface design. San Francisco: Morgan Kaufmann.

Nielsen, J. (n. d.). How to conduct a heuristic evaluation. Retrieved October 24, 2005, from http://www.useit.com/papers/heuristic_evaluation.html

Nielsen, J., Snyder, C., Molich, R., & Farrell, S. (2000). E-Commerce user experience: Methodology of the study. Retrieved October 24, 2005, from http://www.nngroup.com/reports/ecommerce/meth odology.html

Norman, D. A. (2002). The Design of everyday things. New York: Basic Books.

Norman, D. A., & Draper, S. (1986). User centered system design: New perspectives on human-com-puter interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.

Osuna, M., & Meskill, C. (1998). Using the world wide web to integrate Spanish language and culture: A pilot study. Language Learning & Technology, 1 (2), 71-92. Retrieved October 29, 2007, from http://llt.msu.edu/vol1num2/article4/default.html

Peterson, M. (1998). The virtual learning environment: The design of a website for language learning. Computer Assisted Language Learning, 11 (4), 349-361.

Phyo, A. (2003). Return on design: Smarter web design that works. Indianapolis, IN: New Riders.

Plass, J. L. (1998). Design and evaluation of the user interface of foreign language multimedia software: A cognitive approach. Language Learning & Technology, 2 (1), 35-45. Retrieved October 29, 2007, from http://llt.msu.edu/vol2num1/article2

Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: Beyond human-computer interaction. Hoboken, NJ: John Wiley & Sons.

Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. Hobo-ken, NJ: John Wiley & Sons.

Shedroff, N. (1994). Information interaction design: A unified field theory of design. Retrieved March 15, 2006, from http://www.nathan.com/thoughts/unified/index.html

Silver, M. (2005). Exploring interface design. Clifton Park, NY: Delmar Learning.

Susser, B., & Robb, T. N. (2004). Evaluation of ESL/EFL instructional web sites. In S. Fotos & C. Browne (Eds.), New perspective on CALL for second language classrooms (3rd ed.) (pp. 279-295). Mah-wah, NJ: Lawrence Erlbaum Associates.

Vest, J., Crowson, W., & Pochran, S. (2005). Exploring web design. Clifton Park, NY: Delmar Learning.

Warschauer, M. (2005). Socio-cultural Perspectives in CALL. In J. Egbert & G. M. Petrie (Eds.), CALL research perspectives (pp. 41-51). Mahwah, NJ: Lawrence Erbaum Associates.

Warschauer, M., Turbee, L., & Roberts, B. (1996). Computer learning networks and student empower-ment. System, 24 (1), 1-14.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

232

AppENDIx A

Websites on ESL

Websites Instruction/Resource

Target audience

Skill area(s) Media

01 http://www.stuff.co.uk/wicked.htm

instruction student comprehensive (gram-mar, vocabulary, writing, pronunciation, reading, listening, spelling)

text, audio, quizzes with feedback, graphics

02* http://doug-j.org/index.html

instruction/resource

student vocabulary, reading text, interac-tive quizzes

03 http://www.geocities.com/pccprep/qz.html

instruction/resource

student/teacher

grammar, vocabulary text, graph-ics, interactive quizzes

04 http://www.projec tx2002.org/frameset.html?interactive_english.htm

instruction student comprehensive (gram-mar, culture, pronuncia-tion, reading, vocabulary)

text, interac-tive quizzes

05 http://www.elfs.com/ELFsX.html

instruction student comprehensive (vocabu-lary, grammar, writing, pronunciation, reading, listening), TOEFL test preparation

text, interac-tive quizzes, audio, graph-ics

06 http://www.englishclub.com

instruction/resource

student/teacher

comprehensive (speak-ing, listening, reading, writing, vocabulary, grammar, pronunciation)

text, audio, graphics

07 http://www.manythings.org

instruction student comprehensive (vocabu-lary, grammar, reading, writing, listening, spell-ing, culture)

text, interac-tive quizzes, audio, graph-ics

08 http://iteslm.org/ESL/html

resource student/teacher

comprehensive text, audio, video

09 http://grammar.ccc.com mnet.edu/grammar/

instruction student writing, grammar text, graph-ics, interactive quizzes

10 http://www.study.com instruction/resources

student/teacher

listening, writing text, graphics, audio

11 http://towerofenglish.com

instruction/resources

student comprehensive text, graphics

12 http://www.eslcafe.com instruction/resources

student/teacher

comprehensive (gram-mar, vocabulary, idiom, phrasal verbs, words, slang, culture, reading, writing, punctuation)

text, graphics, sound, inter-active quizzes

13 http://web2.uvcs.uvic.ca/elc/studyzone/index.htm

instruction student comprehensive (reading, writing, grammar, vocab-ulary, puzzles)

text, graph-ics, interactive quizzes

14* http://lc.byuh.edu/CNN-N/CNN-N.html

instruction student listening, grammar, vo-cabulary

text, audio, interactive quizzes,

233

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

15 http://a4esl.org/q/h instruction student/teacher

grammar, vocabulary, games

text, interac-tive quizzes

16* http://deil.lang.uiuc.edu/web.pages/gram marsafari.html

instruction/resource

student grammar, reading text

17* http://www.hut.fi/~rvilmi/help/grammar_ help

instruction student grammar text, graph-ics, interactive quizzes

18 http://smccd.net/ac counts/sevas/esl/gram check/index.html

instruction student grammar, writing, listen-ing

text, graphics, audio, interac-tive quizzes

19 http://www.edunet.com/english/sounds/a-e.au

instruction/resources

student grammar text, audio, quizzes with feedback

20 http://owl.english/pur due.edu/handouts/esl/index.html

instruction/resources

student grammar text

21* http://ilcw.doshisha.ac.jp/users/kkitao/class/material/quiz

instruction student grammar text, quizzes with feedback

22 http://online.ohlone.cc.ca.us/~mlieu

instruction student grammar text, quizzes with feedback

23 http://www.esl-lab.com instruction student listening text, audio

24 http://www.americanac cent.com/analysis.html

instruction student pronunciation text

25 http://www.intranet.csupomona.edu/~jasanders/judi2.html

instruction student vocabulary text

26 http://www.geocities.com/athens/aege an/6720

instruction student idioms text

27 http://www-personal.umich.edu/~cellis/an tagonym.html

instruction/resource

student vocabulary text

28 http://www.intranet.csupomona.edu/~jasanders/judi2.html

instruction student slang text

29 http://www.aitech.ac.jp/ ~iteslj

resources teacher articles, teaching tech-niques, lesson plans related to ESL

text

30 http://www.eslpartyland.com

instruction/resources

student/teacher

grammar, idioms text, graphics

31 http://www.npr.org resources student/teacher

listening text, audio

32 http://www.toefl.org resources student general information on TOEFL test

text

33 http://esl.miningco.com resources/instruction

student comprehensive (gram-mar, vocabulary, reading, writing, pronunciation), exam preparation

text

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

234

34* http://lrs.ed.uiuc.edu/Impact

resource teacher reading, vocabulary text, audio

35* http://www.ql.umbc.edu/~kpokoy1/gram mar1.htm

resource/instruction

student/teacher

comprehensive text

36 http://www.eslbears.homestead.com/Index.html

resources/instruction

student/teacher

comprehensive (gram-mar, vocabulary, reading, writing, listening, games)

text, sound, quizzes with feedback

37* http://www.bbc.co.uk/worldservice/learnin genglish/compuspeak

resource student comprehensive text

* These URLs were no longer available at the time of the submission of this manuscript.

AppENDIx B

Tasks for http://www.stuff.co.uk/wicked.htm(One of the five sites tested in the study)

Task 1: Specific Questions1

1. Learn a phrasal verbYou decided to learn one phrasal verb (combinations of verbs and particles, for example, “give up”) every day.

(a) Using this site, find one phrasal verb that you did not know well before.

(b) How long did it take you to find the phrasal verbs on the Web site?

!!!STOP and tell the assistant that you’ve finished this task!!!

2. Learn PronunciationNow you are working on your pronunciation.

(a) Find out from the Web site how to pronounce [æ]and [ɘ]. Please pronounce the sounds to the observer.

(b) How long did it take for you to find the answer?

* The first task (Task 1: First impressions), in which users are asked to explore the site freely and tell their impression about the site, is not included on these task sheets for users because it was orally conducted. Therefore, on these task sheets, the second task is labeled as Task 1 (Specific Questions), and the third task is labeled as Task 2 (Open-Ended Tasks).

235

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

(c) In your dictionary it said that you can pronounce “amino” either as [æmi:nou] or as [ɘmi:nou]. Please pronounce the word to the observer in the two ways.

!!!STOP and tell the assistant that you’ve finished this task!!!

Task 2: Open-ended Tasks

1. Listen to a poem.Click “Hints on Pronunciation for foreigners” and work on the activity for 3 minutes.

(a) What did you learn from this activity about pronunciation?

(b) Do you think this information is useful?

!!!STOP and tell the assistant that you’ve finished this task!!!

2. Think of a numberClick “Think of a number” and go through the activity.

(a) Did you get the animal you were thinking of at the end?

(b) Was it fun?

!!!STOP and tell the assistant that you’ve finished this task!!!

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

236

AppENDIx CSurvey Instruments used in the Study

Background Questionnaire

User ID #

1. Demographic Information

1) Your name:

2) Your age:

16-20 21-25 26-30 31-35 36-40 41-50 Over 50

3) Your gender:

Male Female

4) Do you prefer using a Mac or PC ?

2. Computer Experience

5) Where do you access the Internet?

home work both other (specify )

6) How much do you use the Internet per day?

less than an hour 1-2 hours 2-4 hours

4-7 hours 8 hours or more

7) What do you generally use the Internet for (check all that apply)?

Study Research Education Entertainment News

Shopping Email Other

If other, please specify:

8) How long have you been using the computer?

less than 2 years 2-5 years 5-10 years

10-15 years more than 15 years

237

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

9a) Please rate your level of computer experience by drawing a circle.

Not experienced at all I-----------I-----------I-----------I-----------I very experienced 1 2 3 4 5

9b) Please, rate your web surfing experience.

Not experienced at all I-----------I-----------I-----------I-----------I very experienced 1 2 3 4 5

3. Website Usage for English

10) Have you used the Internet to study English? yes no

If yes, go to question 11. If no, this is the end of the questionnaire.

11) For what specific purpose do you use the Internet when you study English?

12) How long per week do you use the Internet when you study English?

less than an hour 1-2 hours 2-4 hours

4-7 hours 8 hours or more

13) Name 1 or 2 most frequently visited sites

14) Rate how much you like using the web to learn English.

Not at all I-----------I-----------I-----------I-----------I very much 1 2 3 4 5

15) How long have you been learning English?

less than 2 years 2-5 years 5-10 years

10-15 years more than 15 years

16) Rate your English skills.

intermediate I-----------I-----------I-----------I-----------I advanced 1 2 3 4 5

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

238

Site Feedback Questionnaire

User ID: Date: Tested site:

This questionnaire is for understanding how you feel about the site you tested. Please rate the site by circling the appropriate number. As you evaluate other sites, you may change ratings for this site later.

Very Unsatisfied Neutral Very Satisfied

1. Ease of finding the information/activities Comment:

1 2 3 4 5

2. Quality of information/activities Comment:

1 2 3 4 5

3. Ease of reading the text Comment:

1 2 3 4 5

4. Appearance of site, including colors and graphics Comment:

1 2 3 4 5

5. Speed of pages displaying Comment:

1 2 3 4 5

6. Fun, entertainment value Comment:

1 2 3 4 5

7. Overall learning experience Comment:

1 2 3 4 5

8. Ease of understanding the instruction for activities Comment:

1 2 3 4 5

9. Ease of moving around the site without getting lost Comment:

1 2 3 4 5

10. Mark how useful this site is for improving the English skills you practiced.

Check the skills that you practiced Useless Neutral Very Useful

Speaking 1 2 3 4 5

Listening 1 2 3 4 5

Reading 1 2 3 4 5

Writing 1 2 3 4 5

Grammar 1 2 3 4 5

Culture 1 2 3 4 5

11. How likely are you to return to this site on your own?

No way -2 -1 0 1 2 I’ll probably return the next time I sit down at my computer.

239

CALICO Journal, 25 (2) M. Liu, T. Traphagan, J. Huh, Y. I. Koh, G. Choi, and A. McGregor

Explain why you are or are not likely to return to this site.

12. Would you recommend using this site to your friends who are learning English?

No way -2 -1 0 1 2 I’ll definitely recommend this site.

Explain why you would or you would not recommend this site.

13. What do you like best about this site?

14. What do you like least about this site?

AUTHORS’ BIODATA

Min Liu, Ed.D., is Professor in the Instructional Technology (IT) Program at the University of Texas at Austin. She is also the Coordinator and Graduate Advisor for the IT Program. She has been teaching courses on new media design, production, and research for over 15 years. Her teaching and research interests center on the impact of new media technology on learning and the design of engaging and interactive learning environments for all age groups. She also has strong interests in technology use for second language learning. She publishes widely in leading instructional technology research journals and serves on a number of editorial/manu-script review boards. She is active in professional organizations such as AACE, AERA, and ISTE. She has directed and managed both CD-ROM and web-based development projects, including award winning ones.

Tomoko Traphagan is a research associate in the Division of Instructional Innovation and Assessment at the University of Texas at Austin. She has been involved in the research and evaluation of various instructional technology programs at the university, including webcast-ing, Second Life, digital media services, course management systems, web-based inquiry-based learning, and usability of instructional websites. Her research interests lie in the impact of technology on students’ learning behaviors and performance.

Jin Huh is Assistant Professor in the Department of English at the Korea Military Academy. He is a graduate of the Foreign Language Program, Curriculum and Instruction at the University of Texas at Austin. His areas of interest include teaching foreign language pronunciation, ac-cent modification, multimedia-aided language teaching, and teaching writing.

CALICO Journal, 25 (2) Designing Websites for ESL Learners: A Usability Testing Study

240

Young-Ihn Koh recently received her Ph.D. degree from the Foreign Language Education Pro-gram at the University of Texas at Austin and is now a part-time lecturer at Dongduk Women’s University and Hanyang University in Seoul, Korea. Her research interests include teaching and learning with technology, especially the use of computer-mediated communication with young and adult ESL learners, teaching culture, and bilingual children’s codeswitching pat-terns.

Gilok Choi is a Ph.D. student in School of Information at the University of Texas at Austin. Her area of specialization is in human computer interaction (HCI), user interface design, and usability studies and methods.

Alison McGregor is a graduate from the Educational Psychology Department at the University of Texas at Austin. Her research interests include technology in language learning, skill devel-opment, and prosodic features of American English. She has worked in the ESL field for over 10 years and has taught English in Taiwan, Japan, and the United States. She specializes in accent modification for nonnative English speakers.

AUTHORS’ ADDRESSES

Min LiuThe University of Texas at AustinInstructional Technology ProgramDepartment of Curriculum and Instruction1 University Station D5700Austin, TX 78712Phone: 512 471 5211Fax: 512 471 [email protected]

Tomoko TraphaganThe University of Texas at Austin Division of Instructional Innovation and Assessment, Continuing and Innovative EducationP.O.Box 72462616 Wichita StreetAustin, TX 78713-7246Email: [email protected]

Click below to find more Mipaper at www.lcis.com.tw

Mipaper at www.lcis.com.tw