how self-reflection exercises influence user comprehension: a usability study report

SCHOOL OF LANGUAGES, LITERATURE, CULTURE AND COMMUNICATION

ASSIGNMENT SUBMISSION FORM

Student Name: Michael Halloran

Student ID Number: 13035657

Course of Study: Technical Writing (Distance Learning) Graduate Certificate

Year: 2013

Lecturer Name: Yvonne Cleary

Module Code: TW5221

Date of Submission: 29 November 2013

I, Michael Halloran, declare that the attached essay/project is entirely my own work, in my own words, and that all sources used in researching it are fully acknowledged and all quotations properly identified

How Self-Reflection Exercises Influence User Comprehension: A Usability Study Report

Michael Halloran

Student Number: 13035657

Course Name: Graduate Certificate in Technical Communication

Module Code: TW5221

Supervisor: Dr. Yvonne Cleary, University of Limerick

Date of Submission: 29 / 11/ 2013

TW5221 Michael Halloran - 13035657

i

Abstract

This research report details a usability study of an online grammar course for teachers of

English as a foreign language. The purpose of the study was to investigate if participants’

understanding of course concepts improved after completing self-reflection exercises.

Ten teachers divided into two teams acted as test participants. All participants did the

first unit of the course. However, the test intervention required one team to complete all

embedded self-reflection exercises while the other team ignored them. Afterwards, the

researcher tested each participant on grammar concepts in a post-course exam of ten

questions. The researcher used these exam results to carry out a t-test of two independent

means in order to evaluate the test hypothesis.

This report includes an extensive literature review, a methodology section, a

discussion section, and three appendices.

KEY WORDS: Constructivism, e-learning, hypothesis test, online course, self-reflection,

usability test.

Acknowledgements

I want to thank Prof. Philip Rubens for helping me to refine this research project. I also want

to thank Dr. Yvonne Cleary for her advice, encouragement, and mentorship over the course

of TW5221. I want to thank the participants for setting aside time to take part in the test.

Finally, I want to thank Liam Halloran and Bryna Greenlaw for proofreading the final report.


ii

Contents

Abstract ....................................................................................................................................... i

Acknowledgements ..................................................................................................................... i

Section 1: Introduction ............................................................................................................... 1

Section 2: Literature review ....................................................................................................... 2

Section 2.1: Overview ........................................................................................................... 2

Section 2.2: Theories of education and e-learning: behaviourism, cognitivism,

constructivism and connectivism ........................................................................................... 2

Section 2.3: Usability, user experience and e-learning .......................................................... 5

Section 2.4: Data analysis of usability tests (hypothesis testing) .......................................... 8

Section 2.5: Conclusion ......................................................................................................... 8

Section 3: Methodology ............................................................................................................. 9

Section 3.1: Hypothesis ......................................................................................................... 9

Section 3.2: Quantitative usability testing – an empirical research method .......................... 9

Section 3.2.1: Sample selection: choosing participants ..................................................... 9

Section 3.2.2: Control conditions..................................................................................... 10

Section 3.2.3: Ethical considerations ............................................................................... 10

Section 3.2.4: Procedure ................................................................................................. 10

Section 3.2.5: Statistical data analysis ............................................................................. 11

Section 4: Results and discussion ............................................................................................ 12

Section 4.1: Testing the difference between two means ...................................................... 12

Section 4.2: Discussion of research questions ..................................................................... 13

Section 4.3: Possible reasons for test results ....................................................................... 13

Section 4.4: Conclusion and recommendations ................................................................... 14

Section 5: References ............................................................................................................... 15

Appendix 1: Exam sheet .......................................................................................................... 18

Appendix 2: Instruction sheet .................................................................................................. 19


iii

Appendix 3: Research ethics committee consent form ............................................................ 20

List of Figures

Figure 1. The “User Experience Honeycomb” (Morville 2004). ............................................... 5

Figure 2. Post-course exam results. ......................................................................................... 12


1

Section 1: Introduction

This is a report on a usability study of e-learning interfaces and whether self-reflection

exercises should be included in online courses. The usability study was quantitative in that it

gathered numeric data from user testing.

The purpose of the usability study was to test the impact of embedded self-reflection

exercises on users of online courses and whether self-reflection is useful in helping users

understand concepts they come across.

The course that I tested was Grammar for Teachers: Language Awareness. Teachers

at the university where I work did the course as part of their professional development.

However, I tended to ignore the self-reflection exercises when I did the course myself and I

wanted to know why.

Through this study, I wanted to find out why the designer included the self-reflection

exercises in the course, whether the exercises were really necessary, and whether they raised

or lowered student attrition rates. However, these questions proved to be subjective and not

easily tested. Therefore, Professor Philip Rubens helped me refine my ideas into a

quantitative study based on a post-course exam of ten questions (see Appendix 1). This report

details the background, the methodology of data collection, the control conditions, the ethical

considerations, the results of the study, and a discussion of the data. Ten participants

volunteered to do the first unit of the course as well as the exam. The number of participants,

the scope and the length of the study were limited by time, location, and resources.

The layout of this report is as follows: Section 2 is a review of literature dealing with

learning theory, usability testing and hypothesis testing. Section 3 deals with the test

hypothesis and the methodology used to carry out the usability study. Finally, Section 4

presents the results of the study and offers some recommendations.


2

Section 2: Literature review

This review looks at some of the literature that has been written on the theory and usability of

e-learning programs. It also includes a short description of hypothesis testing as outlined by

Hughes and Hayhoe (2008, pp.64-71).

Section 2.1: Overview E-learning is becoming a major organizational use of the internet (Zaharias and

Poylymenakou 2009, p.76; Angelino et al 2007, p.2). Blended learning, already ubiquitous in

universities, is penetrating second-level education; e-learning programs are also being used

for corporate and professional development programs.

Koohang et al (2009, p.91) state that constructivist learning theory “focuses on

knowledge construction based on learners’ previous experience” and is therefore “a good fit

for e-learning because it ensures learning among learners.” Ally (2001, p.31) states that

“learners should be given time and the opportunity to reflect”; embedded questions will

encourage learners “to reflect on and process information in a relevant and meaningful

manner.” Alley suggests getting students to generate a learning journal. Koohang et al (2009,

p.95) agree by stating that reflection activities “will encourage the learner to be responsible

for his or her own learning.” But what are the other theories behind e-learning?

Section 2.2: Theories of education and e-learning: behaviourism, cognitivism, constructivism and connectivism Early online courses were designed based on behaviourism (Ally 2011, p.19). This school of

thought focuses on learners’ observable and measurable behaviours (Ally 2011, p.19). A

behaviourist online course would:

Inform students of course outcomes so they can set expectations.

Have regular tests as an imbedded part of course design.

Introduce learning material in a sequenced way.

Encourage students to provide feedback.

Cognitivism looks at learning from an information processing point of view (Ally 2011,

p.20). A cognitivist online course would:

Place important information in the centre of the screen for reading.


3

Highlight critical information with headings and clear formatting.

Tell learners why they should take the course.

Match the difficulty level to the learner’s cognitive level.

Jakob Nielsen in an interview with elearningpost (Nichani 2001) touches on an aspect of

cognitivism:

You need to keep all the content fresh in learners mind [sic]…For example, response time. Even after a

few seconds you always forget what was the track or sequence you were following…It is important that

your brain keeps the context.

(Nichani 2001 para. 4.)

Ally also says a cognitivist course would encourage students to use their existing knowledge

to help them make sense of the new information (Ally 2011, p.24).

Some other aspects of this approach include:

Chunking information to make it more memorable (Miller 1956).

Varied learning strategies to accommodate different kinds of learners.

Varied modes of information delivery: textual, visual and verbal.

Learner motivation strategies: intrinsic motivation and extrinsic motivation.

Metacognition: make a student aware of their learning capabilities.

Assignments that have real-life application and information.

As mentioned above, constructivism sees the learner as active. Stimuli are received from the

outside but it is the learner who actively creates the knowledge (Ally 2011, p.30). A

constructivist online course would:

Give learners meaningful activities in practical situations.

Provide first-hand information, without the contextual influence of an instructor, so

that students can personalise the information themselves.

Encourage cooperative learning.

Provide guided discovery activities.

Use embedded questions (or a learning journal) to encourage learner reflection.

Provide a high-level of interactivity.

Finally, connectivism is “a theory for the digital age, where individuals learn and work in a

networked environment” (Ally 2011, p.34). Ally sketches some general guidelines based on

this theory. Learners need to:


4

Be autonomous and independent; appropriate use of the internet is encouraged.

Unlearn old information and models in favour of the most up-to-date information and

models; learners need to identify the most important information.

Be active in a network of learning and acquire knowledge on an ongoing basis.

Must be allowed to connect with others around the world in order to share knowledge

and opinions.

Gather information from many resources to reflect the networked world and the

diversity of thinking within it.

Ally states that “further work needs to be done on how this theory can be used by educators

to design learning materials” (Ally 2011, p.38).

Finally, Ally suggests these different theories can be used to deal with different

aspects of a course:

Behavourism to teach facts.

Cognitivism to teach principles and processes.

Constructivism to teach real-life applications of learning.

Anderson (2011) provides a framework of how people learn:

Knowledge-centred learning give access to a vast selection of content and activities

but quality information is highlighted and filtered by the community of users.

Assessment-centred learning is based on “formative and summative assessment by

self, peer and teachers” (Anderson 2011, p.66).

Learner-centred learning changes in response to group and learner models and content

is changed based on student and teacher use.

Community-centred learning uses many formats for collaborative and individual

interaction.

In relation to knowledge-based and community-centred learning, Nielsen discusses usability,

design and aesthetics of good discussion forums for learners:

I actually believe much more in discussion groups than I believe in chat rooms as ways of allowing

students to interact…real-time chat effectively becomes very thin and not nearly as valuable as

discussion groups where people can think a little bit before they post and the instructor can moderate it

which a also good.

(Nichani 2001, para. 10)


5

Section 2.3: Usability, user experience and e-learning What is usability? Nielsen (2012, What – Definition of Usability, para. 1) defines usability as

“a quality attribute that assesses how easy user interfaces are to use” and defines it by five

qualities:

Learnability.

Efficiency.

Memorability.

Errors (the quantity and quality of errors a user makes).

Satisfaction.

Utility (does the interface do what the user needs?).

User experience (UX) is related to usability in that it “focuses on having a deep

understanding of users, what they need, what they value, their abilities, and also their

limitations” (U.S. Department of Health & Human Services 2013a, para. 1). Morville (2004)

uses a honeycomb to illustrate the facets of user experience:

Figure 1. The “User Experience Honeycomb” (Morville 2004).

The most basic way to improve usability is user-testing (Nielsen 2012, How to Improve

Usability, para. 1) and this process is three-fold:

Get representative users to test the interface.

Ask users to do representative tasks.


6

Observe users and take notes on their experience.

The U.S Department of Health & Human Services (2013b) outlines some other evaluation

methods:

Focus groups: moderated discussion involving five to ten participants.

Card sort testing: participants organise topics into categories that make sense to them.

Wireframing: creating a two-dimensional illustration of a page’s interface.

First click testing: examines what a test participant would click on first on the

interface in order to complete their intended task.

Satisfaction surveys.

The U.S Department of Health & Human Services goes on to discuss what the researcher

should do after gathering data from one of the above methods:

Evaluate the usability of the website.

Recommend improvements.

Implement recommendations.

Re-test the site to measure the effectiveness of your changes.

While these methods can help researchers test usability, there are recognised usability

principles for interaction design, often referred to as heuristics (Nielsen 1995a). Heuristic

evaluation is part of an iterative design process and usual involves a team of evaluators.

Nielsen recommends the use of five evaluators, but three at the least (Nielsen 1995b, para. 2).

Jefferies and Desurvire (1992, p.39) found that just one evaluator was the least powerful

evaluating technique when they experimented with different usability tests.

Heuristic evaluation does not provide a systematic way to generate fixes, but rather

aims to solve design issues by reference to established usability principles (Nielsen 1995b,

para. 12). Nielsen (1995a) provides a list of “10 Usability Heuristics for User Interface

Design”:

Visibility of system status.

Match between system and the real world.

User control and freedom.

Consistency and standards.

Error prevention.


7

Recognition rather than recall.

Flexibility and efficiency of use.

Aesthetic and minimalist design.

Help users recognize, diagnose, and recover from errors.

Help and documentation.

Do these principles apply to e-learning? A team at The University of Georgia found that

Nielsen’s list needed to be augmented (Benson et al 2002). They evaluated an e-learning

program designed for the American Red Cross. They created a “protocol for e-learning

heuristic evaluation” and fifteen usability and instructional design heuristics for the

evaluation of e-learning programs. Their augmented evaluation heuristics included Nielsen’s

original ten and five new principles:

Learning Design.

Media Integration.

Instructional Assessment.

Resources.

Feedback.

Zaharias and Poylymenakou (2009) suggest that a usability evaluation method for e-learning

needs to place motivation above functionality. They split e-learning usability attributes in

two: usability and instructional design. Under usability they include, navigation learnability,

accessibility, consistency and visual design. Under instructional design they include,

interactivity/engagement, content and resources, media use, learning strategies design,

feedback, instructional assessment and learner guidance and support (Zaharias and

Poylymenakou 2009, p.80). This ultimately feeds into the most important part of e-learning,

motivation by students: attention, relevance, confidence and satisfaction. Intrinsic motivation

can be characterised as the drive arising within the self to carry out an activity whose reward

is derived from enjoyment of the activity itself (Zaharias and Poylymenakou 2009, p.80) The

learning interface needs to encourage intrinsic learning motivation. Ally says that extrinsic

motivation should also be used, citing Keller’s ARCS model (Attention, Relevance

Confidence and Satisfaction) (Keller 1987; Ally 2011, p.28).


8

Section 2.4: Data analysis of usability tests (hypothesis testing) To test and analyse a usability intervention (the independent variable), a researcher will use

two groups: a test group and a control group (Hughes and Hayhoe 2008, p.64). The results of

the test are known as the dependent variable. The research analyses the average results of the

test group and control group to see if there is a statistically significant difference. Therefore,

it is important to include these factors when stating a test hypothesis.

In order to test the hypothesis Hughes and Hayhoe (2008, p.65; p.71) suggest using

Microsoft Excel to conduct a t-test of two independent means:

State the test hypothesis, including the independent variable, the dependent variable

and the expected direction.

Recast the test hypothesis as a null hypothesis.

Collect the data through the researchers chosen method.

Enter data into a spreadsheet.

Use the function COUNT to tally the sample sizes of the test and control groups.

Use the function AVERAGE to calculate the mean.

Use the function STDEV to calculate the standard deviation for each group. Standard

deviation is “an indicator of the variation of the data in the sample…[it] is helpful for

envisioning how widely the data vary from the average” (Hughes and Hayhoe 2008,

p.63).

Use the function TTEST to calculate the probability (the p value) that the results

could be caused by differences in the samples, rather than the intervention.

Hughes and Hayhoe (2008, p71) state that “typically, you can reject the null if the p

value is less than 0.1.

Finally, if you can reject the null, then accept the test hypothesis.

Section 2.5: Conclusion This literature review has looked at the theories behind e-learning course design and how to

test the usability of such courses.

Online courses will become more central to general education and connectivism, as

mentioned by Ally (2011), might be a very large target for research and potential

applications.


9

Section 3: Methodology

Grammar for Teachers: Language Awareness is a course designed for future, inexperienced

and experienced teachers of English as a foreign language. To successfully complete the

course users must “reflect upon [their] own knowledge of the English language…participate

in discussion forums with other teachers…keep a learner journal” (Cambridge University

Press and UCLES 2013). All of these components suggest a strong constructivist basis for the

course.

Section 3.1: Hypothesis Does self-reflection actually help students process information and become better learners? I

set about proving the following hypothesis:

In online courses, users will better understand course concepts if they complete self-

reflection exercises.

The goal of testing this hypothesis was to establish whether designers should include self-

reflection exercises in online courses. To reach this goal, I proposed these questions:

Do users understand course concepts better after they self-reflect?

Do users misinterpret course concepts if they do not self-reflect?

Is there any noticeable difference between users who self-reflect and those who don’t?

Section 3.2: Quantitative usability testing – an empirical research method In order to generate data suitable for a t-test of two independent means, I chose the following

testing methodology.

Section 3.2.1: Sample selection: choosing participants

The course is aimed at teachers of any level of experience. I chose ten participants, male and

female, between the ages of twenty-five and sixty-five. All participants were teachers at

different stages in their careers; none had attempted the course before. Based on Nielsen’s

(1995b, para. 2) recommendation of using five usability evaluators, I divided the participants

randomly into two groups of five. I will refer to these two groups as Team A (the test group)

and Team B (the control group).


10

Section 3.2.2: Control conditions

The usability study took place in a large, air-conditioned and brightly-lit university

classroom. I observed one participant at a time. I spaced out the participant slots over five

days between 8.30am and 4.30pm depending on the participants’ availability. Cambridge

English Teacher’s suggested completion time of the whole course was five hours which is too

long for most people to put time aside to complete. Instead, I chose “Unit 1: Nouns and

pronouns” to act as a representative example of the whole course. I found that registration,

Unit 1 itself, and the post-course exam took forty-five minutes for me to complete. I took this

timing into account when informing participants about the length of the test. Finally, I

provided all participants with the following items:

A pencil and eraser.

A Lenovo G480 laptop with wireless mouse and internet access.

A pre-course task sheet (see Appendix 2).

A post-course exam sheet (see Appendix 1).

A bottle of water.

Section 3.2.3: Ethical considerations

The FAHSS Ethics Committee at the University of Limerick approved this quantitative

research project. The approval included conducting pre-/post-surveys as part of a usability

study. I answered “No” to all questions on the approval form checklist. Participants signed a

consent form (Appendix 3) but will remain anonymous. I did not make any audio or video

recordings. The anonymous exam sheet is the only record.

Section 3.2.4: Procedure

Participants sat at a desk with the laptop. They signed the consent form and received the task

sheet. After they complete the task sheet, I directed them to complete ‘Unit 1 – Nouns and

Pronouns’ using the onscreen directions only. I emphasised that they must read everything

and watch every video. However, I implemented the following intervention:

Team A members must attempt all self-reflection exercises.

Team B members must ignore all the self-reflection exercises.

I observed all participants to ensure they followed this intervention protocol. I prompted them

when they strayed from the instructions or provided them with assistance when they asked.


11

Registration and unit completion took forty to eighty minutes depending on the participant.

After they complete the unit, I removed the laptop and task sheet and gave the participants a

pencil, eraser and exam sheet. The exam consists of ten questions which tested participants’

comprehension of the course concepts. Participants had to attempt all questions. The exam

took less than five minutes to complete per participant.

Section 3.2.5: Statistical data analysis

The exam sheet consisted of ten questions and was based on concepts covered in the course.

The first five questions required one answer. The second five questions required three

answers. This added up to a potential total score of 20 out of twenty 20. I pilot tested the

exam sheet on two colleagues. This was to see whether the questions were readable and easy

to understand. While these two colleagues did sign the consent form, I did not keep their

results for the record as this was simply a proofreading exercise.

After the ten participants of the actual test had finished, I tallied up their exam scores

and followed Hughes and Hayhoe’s (2008, p.71) hypothesis testing method as I outlined in

the literature review.


12

Section 4: Results and discussion

Below are the results of the analysis. In section 4.2 I restate my initial goal questions and

discuss the answers.

Section 4.1: Testing the difference between two means

Hypothesis: In online courses, users will better understand course concepts if they

complete self-reflection exercises.

Null Hypothesis: In online courses, users will better understand course concepts if

they do not complete self-reflection exercises.

When assigning arguments in the function TTEST (probability) in Microsoft Excel, I chose a

one-tailed (directional) test. Under the type argument, I chose “3” as this was a “nonpaired

test where the variances were not equal (Hughes and Hayhoe 2008, p. 69). See Figure 2

below for details of the data analysis.

Post-Course Exam Results

Team A Team B

8 18

17 12

15 14

16 14

11 15

n 5 5

mean 13.4 14.6

SD 3.78153 2.19089

p 0.28018

n= total sample size

SD = standard deviation

p= probability

Figure 2. Post-course exam results.

By reviewing the means of both groups we can see that the average score for Team A was

13.4 out of 20 while for Team B it was 14.6 out of 20. Already we see that the hypothesis is

not accepted based on the raw averages. If the probability is less than 0.1 than the null

hypothesis can be rejected. However, the p value is 0.28018, meaning the null hypothesis


13

cannot be rejected. Therefore, based on the available data: the test hypothesis is not

accepted.

Section 4.2: Discussion of research questions Based on the data, I will now answer my research questions as outlined in Section 3.1:

Do users understand course concepts better after they self-reflect?

No, Team A participants had a lower total average than Team B.

Do users misinterpret course concepts if they do not self-reflect?

No, Team B had a higher average than Team A and one Team B participant scored

the highest of all the participants, 18 out of 20.

Is there any noticeable difference between users who self-reflect and those who don’t?

No, there is only a 9% difference between the two groups’ averages.

Section 4.3: Possible reasons for test results Does this all mean that self-reflection exercises are redundant and that we should accept the

null hypothesis? Possibly. However, this would require us to reject a huge part of

constructivist theory. Therefore, it is more likely that the test itself was flawed. There are

several possible reasons why these results occurred:

The sample size was not big enough.

Unit 1 is an eighth of the total course. The test may not have been representative

enough of the whole course. A test of two units may have been better.

The exam sheet questions should be re-worded because two participants

misunderstood questions 2 and 6.

Unit 1 may not have been challenging enough. Nouns and pronouns are basic

concepts for experienced teachers.

Some participants may not have used the self-reflection exercises to their full

potential. Some wrote very long passages while others wrote only a couple of lines.

Participants may not have read the questions thoroughly. The participant who scored

18 out of 20 read every part of the online unit aloud. This participant also read the

exam questions aloud twice.

The exam questions may not have accurately reflected the self-reflection exercises.


14

In hindsight, the hypothesis is flawed. A better hypothesis might be worded to include

the dependent variable; in this case, the results of the post-course exam:

Users will have stronger recall of online course concepts in a post-task exam if they

complete embedded self-reflection exercises.

Allowing participants to access the online journal of their self-reflections may have

altered their test results.

Allowing participants to discuss the reflections in the course forum could also have

altered the results as well.

The course self-reflection exercises may have been poorly designed themselves; they

may have discouraged participants from really considering the course concepts.

Section 4.4: Conclusion and recommendations In this report, I have presented a literature review of e-learning theory and usability, the

methodology I used to test my hypothesis, and finally, an analysis of the resulting data. In

Section 3.1, I stated that “the goal of testing this hypothesis was to establish whether

designers should include self-reflection exercises in online courses.” Considering my test

results, the answers to my research questions, as well as the possible faults with the test itself

(as mentioned in Section 4.3), I have to state that the test results are inconclusive.

It is possible that self-reflection exercises do not aid learners. However, self-reflection

is an integral part of constructivist learning theory, and constructivism is central to

contemporary e-learning design theory. Therefore, it is essential that further research is

carried out in this area. I would recommend another study is carried out on Grammar for

Teachers: Language Awareness, using a larger sample size and a larger representative section

of the course. The future researcher should also take into account all the other faults I

mentioned in Section 4.3 and address them before conducting the research.


15

Section 5: References

Ally, M. (2011) ‘Foundation of Educational Theory for Online Learning’, in Anderson, T.,

ed., The Theory and Practice of Online Learning, 2nd

ed., Edmonton: AU Press, 15-44.

Anderson, T. (2011) ‘Towards a Theory of Online Learning’ in Anderson, T., ed., The

Theory and Practice of Online Learning, 2nd

ed., Edmonton: AU Press, 45-74.

Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, S. and

Reeves, T.C. (2002) ‘Usability and Instructional Design Heuristics for E-Learning

Evaluation’, in Barker, P. and Rebelsky, S., eds., Proceedings of World Conference on

Educational Multimedia, Hypermedia and Telecommunications 2002, Chesapeake, VA:

AACE, 1615-1621.

Cambridge University Press and UCLES 2013 (2013) ‘Grammar for Teachers: Language

Awareness’, Cambridge English Teacher available:

http://www.cambridgeenglishteacher.org/courses/details/18606 [accessed 2 Nov 2013].

David, A. and Glore, P. (2010) ‘The Impact of Design and Aesthetics on Usability,

Credibility, and Learning in an Online Environment’, Online Journal of Distance Learning

[online], 13(4), available:

http://www.westga.edu/~distance/ojdla/winter134/david_glore134.html [accessed 13 Oct

2013].

Hughes, M. and Hayhoe, G. (2007) A Research Primer for Technical Communication:

Methods, Exemplars, and Analyses, New York: Lawrence Erlbaum Associates.

Jeffries, R. and Desurvire, H. (1992) ‘Usability testing vs. heuristic evaluation: was there a

contest?’, SIGCHI Bulletin, 24 (4), 39-41.

Keller, J. (1987) ‘Development and use of the ARCS model of instructional design’ in

Journal of Instructional Development, 10(3), 2-10.


16

Koohang, A., Riley, L. and Smith, T. (2009) ‘E-learning and Constructivism: From Theory to

Application’ in Interdisciplinary Journal of E-Learning and Learning Objects [online], 5, 91-

109, available: http://ijklo.org/Volume5/IJELLOv5p091-109Koohang655.pdf [accessed 2

Nov 2013].

Miller, G.A. (1956) ‘The magical number seven, plus or minus two: Some limitations on our

capacity for processing information’ in Psychological Review, 63, 81-97.

Morville, P. (2004) ‘User Experience Design’, Semantic Studios [online], 21 June, available:

http://semanticstudios.com/publications/semantics/000029.php [accessed 24 Nov 2013].

Nielsen, J. (1995a) ‘10 Usability Heuristics for User Interface Design’, Nielsen Norman

Group [online], 1 January, available: http://www.nngroup.com/articles/ten-usability-

heuristics/ [accessed 13 Oct 2013].

Nielsen, J. (1995b) ‘How to Conduct a Heuristic Evaluation’, Nielsen Norman Group

[online], 1 January, available: http://www.nngroup.com/articles/how-to-conduct-a-heuristic-

evaluation/ [accessed 13 Oct 2013].

Nielsen, J. (2001) ‘First Rule of Usability? Don't Listen to Users’, Nielsen Norman Group

[online], 5 Aug, available: http://www.nngroup.com/articles/first-rule-of-usability-dont-

listen-to-users/ [accessed 13 Oct 2013].

Nielsen, J. (2012) ‘Usability 101: Introduction to Usability’, Nielsen Norman Group [online]

4 January, available: http://www.nngroup.com/articles/usability-101-introduction-to-

usability/ [accessed 13 Oct 2013].

Nichani, M., ed. (2001) ‘Jakob Nielsen on e-learning’, elearningpost [online], 16 January,

available:

http://www.elearningpost.com/articles/archives/jakob_nielsen_on_e_learning/ [accessed 13

Oct 2013].


17

U.S. Department of Health & Human Services (2013b) ‘Usability Evaluation Basics’

Usability.gov [online], available at: http://www.usability.gov/what-and-why/usability-

evaluation.html [24 Nov 2013].

U.S. Department of Health & Human Services (2013a) ‘User Experience Basics’,

Usability.gov [online], available: http://www.usability.gov/what-and-why/user-

experience.html [accessed 23 Nov 2013].

Zaharias, P. and Poylymenakou, A. (2009) ‘Developing a Usability Evaluation Method for e-

Learning Applications: Beyond Functional Usability’, International Journal of Human-

Computer Interaction, 25(1), 75-98.


18

Appendix 1: Exam sheet EXAM SHEET Questions 1-5 are worth one point each 1. What is an alternative name for

“word class”?

______________________________________

2. Give one example of an abstract,

countable, common noun

______________________________________

3. Complete the sentence: ____________________is a word that is used to

show a sudden expression of emotion.

4. Complete the sentence: ____________________is a word that connects

words, phrases and clauses in a sentence.

5. Complete the sentence:

Concrete nouns can be seen, touched or

____________________.

Questions 6-10 are worth three points each

6. Give three examples of determiners

1. 2. 3.

7. Give three examples of demonstrative pronouns

1. 2. 3.

8. Give three examples of ordinal number quantifiers

1. 2. 3.

9. Give three examples of collective nouns

1. 2. 3.

10. Give three examples of indefinite pronouns

1. 2. 3.

Total Score: /20


19

Appendix 2: Instruction sheet

Instructions

1. Open Google Chrome.

2. In the search bar type “www.cambridgeenglishteacher.org” and press

return.

3. In the top-right corner click “register.”

4. Fill in the registration form with your details (N.B. only use numbers and/or

letters in your password).

5. Click “create account.”

6. Open your email inbox and select the message titled “validate email

address.”

7. Follow the link in the message.

8. Sign into your new account.

9. Click “courses” in the navigation bar.

10. Scroll down to “Grammar for Teachers: Language Awareness” and click

“open.”

11. Select “Unit 1 – Nouns and Pronouns” and begin the course.


20

Appendix 3: Research ethics committee consent form

FACULTY OF ARTS, HUMANITIES AND SOCIAL SCIENCES

RESEARCH ETHICS COMMITTEE

CONSENT FORM

Consent Section:

I, the undersigned, declare that I am willing to take part in research for the project entitled

TW5221 Usability Study.

I declare that I have been fully briefed on the nature of this study and my role in it and

have been given the opportunity to ask questions before agreeing to participate.

The nature of my participation has been explained to me and I have full knowledge of

how the information collected will be used.

I am also aware that my participation in this study may be audio recorded and I agree

to this. However, should I feel uncomfortable at any time I can request that the

recording equipment be switched off. I am entitled to copies of all recordings made

and am fully informed as to what will happen to these recordings once the study is

completed.

I fully understand that there is no obligation on me to participate in this study.

I fully understand that I am free to withdraw my participation at any time without

having to explain or give a reason.

I am also entitled to full confidentiality in terms of my participation and personal

details.

______________________________________ __________________________

Signature of participant Date

how self-reflection exercises influence user comprehension: a usability study report

Documents