ABSTRACT

Mayank chadha[Email address]

The aims of this project to investigate and record usability testing of a mobile application that uses dimensions of context to receive electronic updates from the university’s virtual learning environment. Previously,a user study of two RSS based information retrieval widgets to support mobile learning within a higher education environment was implemented and results of which were published in [1]. Created and deployed for implementation on a GPS enabled Nokia devices, the user is presented with arbitrary information from the virtual learning environment based upon either their schedule or location. The study aimed to identify whether integrating location context awareness can override the utility of temporal based mobile electronic alerts. The major function of both the mobile applications is to disseminate information surrounding course updates and deliver these in a context of either time or space, aiming to identify if there is precedence between these two dimensions of context. Results from both quantitative and qualitative data indicated that both applications were evenly helpful for receiving information and supported the way in which the students organized their learning. The other outcome of interest was that both the space and time applications were deemed equally unobtrusive in their personal space, although utilizing location information was deemed a less comfortable form of ambient information.Presently, a third dimension of context (activity) was included and further usability testing is required.

The aim of this project is to carry out a study with particular focus on usability testing of the mobile application developed

Introduction

At this present moment in time mobile devices are the extreme source of delivering information

to its users rather it be simply browsing the internet for multimedia and social networking or for

navigation from one route to the other. The abundance in mobile technologies in the today’s

world has let the user to explore vast amount of facilities from news to shopping, from

entertainment to your important email updates, at fingertips. Coupled with the wireless

connectivity, mobile devices can provide some powerful applications to its user. And the nature

of it being portable and on the move, has the ability of outweighing the capability of present

computer systems. The use of mobile devices for delivering learning tools is an attractive

concept. Termed as mobile learning (m-learning), this new technology allows people to

participate in learning activities without being tied to a fixed location and provides users with

convenient and flexible access to learning resources ‘’anytime, anywhere’’[1].

With the invention of GPS in present day mobile systems, a user’s location can be set as update

points. Context-aware dimensions such as location, schedule and activity mobile application can

now offer automated updates to the users which in case of students could help in their learning

environment. M-learning is fast growing area where most of the learning institutions already

have implied and integrated their learning environment to mobile based platforms. These m-

learning sources can increase the effectiveness and quality of learning in a student friendly space.

In order to provide a better learning source in mobile education the applications have to be

highly focused on collaborative, communicative, constructive and supportive activities which

can enhance a students learning experience. Hence the Usability of the application plays a really

crucial role. By usability of an application means the effectiveness and providing a high quality

system to the learners. The modes and methods on how to test if a system is usable or not will be

explained further in details in coming topics.

1

1.1 Project Aims

The aims of the project is to investigate the usability of context-aware mobile application in the

university’s learning environment. Previously, a study was made using the temporal and spatial

dimensions of context where the users were to receive automated RSS updates over their mobile

devices depending on the location and schedule selected by the user. Recently a third dimension

Activity has been introduced and further usability testing is required.

1.2 Report Structure

The report will be structured in following way:

Chapter 2 will include all the explanations of terminologies used in the development of the

application all the background research and the systems which were related to the development

process in some ways.

Chapter 3 will include the design and implementation of the procedure being followed in the

usability testing of the application and will give an insight to area to look at while performing the

test.

Chapter 4 will include the evaluation process and usability testing results of the study. This will

also include the insights to the feedback from the test users from this study.

Chapter 5 will demonstrate how it is like to use the system from a user’s point of view. This will

include the screenshots of the application working on an Android phone.

Chapter 6 will evaluate the overall findings, conclusions, scope of improvements and will

discuss if the end results meets the aims of the project along with summed up outcomes of the

project.

2

2. Background & Related Work

In this part of the report I will focusing on the previous research been done and all the existing

technologies which are relevant to the project in some ways.

2.1 Enabling Technologies

2.1.1 Android operating system

Android operating system is a mobile operating system that makes the use of java based

programming language knows as Android programming language in the application

development. It is fairly new and one of the top leading mobile operating system in the market

till date. It was first distributed in 2007 and is led by Google Inc. According to Canalys (a

leading business analysis company), Android was the bestselling mobile operating system

platform in the last quarter of 2010 with over 200 million Android devises in use by November

2011 [5]. A research shows that in second quarter of 2012, 105 million units of Android based

smartphones were sold, capturing the 68% of overall market shares till Q2 2012 [6]. Within in

the Android community there are Android software developer which are working on the

development of mobile applications in order to extend the functionality of the devices. Being an

open source programming language (meaning you can actually see the coding bits of the

programs) it serves as the best platform for application developers to add the newly custom made

features to an already available mobile application and tailor them according to their desired

needs. The main advantage of the android programming is that it is free to use, ie you don’t need

to purchase an authority license to build a mobile application. Keeping this in mind and the fact

that how popular it is amongst students, the m-learning application has been designed on

Android operating system and the features, and various other artefacts associated with the

working, usability will be discussed in the upcoming sections.

3

2.1.2 Eclipse IDE

Eclipse is a multilingual Integrated Development Environment (IDE) comprising of a base

workspace and extensible plugin systems to customizing the environment. It is basically written

in java and has a well-established support for programming in Android. It features the drag and

drop GUI’s and is able to use all the packages provided in Android SDK. Eclipse allows it

developers to create a virtual GUI so that they can work and test their applications on desired

screen sizes and variable memory just like any Android smartphone without actually having to

transfer the application to a mobile phone.

Again, Eclipse IDE is an open source project which is run by the community Eclipse foundation

that ‘develop open platforms and products’. Primarily it is a java based platform but with the use

of various optional plug-ins the use can be extended to other programming languages such as

C/C++, Fortran, PHP, COBOL, JavaScript, Python etc [7].

2.1.3 Android SDK

Android Software Development Kit (Android SDK) comprises of a set of tools which are needed

in the development of the application. It includes an emulator, libraries, debugger,

documentation, sample source codes and tutorials. Android SDK is the main source of building

any kind of Android application [8].

2.2 Context Awareness

The understanding and definition of context awareness has been a research field spanning

decades, but has its potential realized with the arrival of mobile devices. The term context-

awareness has commonly been used for two different kinds of application approaches: to capture

context so that it can later be used as a cue for information retrieval or more commonly, to user

context to adapt device behavior to correspond to the manner of its usage (Dourish 2004) [9]. In

addition to these two cases, context-awareness can be used for providing information

4

presentation to its user (Dey 2001) [10]. There haven’t been one specific definition to describe

the term but some attempts are presented.

Schilit et al. (1994) [11] introduced the term context-aware computing which

‘adapts according to its location of use, the collection of nearby people and objects, as well as

changes to those objects over time’.

In addition to this, they proposed three general categories for describing the context, user

context, physical context & computing context. The user context here describes the general

situation from the user’s point of view such as user’s activity and social factors. Physical context

comprises information of the physical environment. Computing context includes the use of

available computing & application resources such as device connectivity.

As stated by Dey and Abowd (2000) [12],

‘Any information that can be used to characterize the situation of an entity. An

Entity is a person, place, or object that is considered relevant to the interaction

between a user and an application, including the user and applications

themselves’.

In regard to this they state that a context-aware may support three kinds of features: 1)

presentation of services and information to its users, 2) automatic execution of a service and, 3)

tagging of context to information for its later retrieval. Thus, according to the selected definition

of context one needs to figure out on how to choose relevant factors, in order to make sure that

they are taken into consideration in the application development process.

Brown et al [13] described the concept of context as ‘the formal or informal setting in which a

situation occurs’, it can include many aspects or dimensions, such as environment, user goals,

tasks and activities. The diversity in the meaning of context can be summarized in Table 1.

5

2.2.1 Using Context-Awareness in Mobile Applications

The use of context-awareness is not only limited to mobile applications, but has a scope and

many other kind of applications, for instance recording of the audio and videos in the lecture

rooms (Abowd 1999) [14], Keeping track of the patients in hospital beds (Bardram 2004) [15].

However in scope of this project the discussion of this is limited and is more focused towards the

use in mobile applications.

Location being the highly usable context-awareness source has its use in many different kind of

mobile applications. With just the user’s point of location one can find really useful information

in sense of time management and social navigation. Location awareness forms a significant area

for research studies as it has a huge potential to offer to the present generation of mobile users.

Discount offers nearby shopping/eating places depending upon the user’s present location,

navigation aid systems, city/museum tour guide systems, lunch menu choices in nearby

6

restaurants, store opening hours and various location based advertising systems works on the use

of location based contexts in mobile application systems.

Apart from Location based context awareness systems, time based context awareness mobile

applications also provides a big range of facilities according to the user’s time domain

information. The user’s time zone, the device time the user is using is used to trigger temporal

based features of the mobile applications. Some of the common examples would be cinema time

applications which tell its users the next available cinema timings depending on their current

device/time zones. Another simple example would be a shopping store official mobile

application which tells it users whether the shop is open/closed depending on the time they

switch to the mobile application.

There are several kinds of mobile sensors within the mobile phones which allow the application

to make use of the user’s spatial and temporal sensitive information in order to deliver these

information. The mobile application will only perform correctly if it has the access to use the

device’s application programming interfaces. The permission is demanded by the application to

make use of the device API which is only asked once. An example of this is shown below with

the help of a mobile application screenshot for asking its user to gain full access to the device’s

API [4].

7

Ofcourse, this permission is asked only once during the installation of the mobile application.

After this the application gains the full access to user and device’s information in order to

maintain a seamless information flow to and fro from the server to its user.

Figure 1 shows the available sensors in a modern mobile device which could be used to

determine the user context.

8

One of our own university’s mobile application for instance, Pocket Lancaster is a perfect

example of how this process works. This mobile application uses both the time based and

location based context-awareness features in order to tell its users the upcoming events in or

around the campus. It also features the nearest point of interest places whether it be a coffee shop

or a lecture room locations based on the user’s current location. Some working screenshots of the

mobile application are shown below to make it more clear how this mobile application uses the

both features within the same module [16].

9

10

2.3 What is Usability?

Usability is an inseparable part and a building block of any type of application development

process. In the hoard of tightening market competition, application developers and competing

amongst each other developing similar featuring applications. How to choose which one amongst

all is better? Usability is the key to provide a shear strategy to outstand one application amongst

others. Usability reflects to the easiness and efficiency of use in order to provide full user

satisfaction with application.

According to ISO 13404 on Human-Centered Design Process for Interactive Systems defines the

Usability as:

‘Extent to which a product can be used by specified users to achieve specified goals with

effectiveness, efficiency and satisfaction in a specified context of use’.

Thus an attempt of employing the usability criteria with the use of context awareness is a

relevant approach. Figure 1 presents the usability framework described by ISO/DIS 9241-11,

which describes the relation between usability measures and goals.

11

2.3.1 Why do we need it?

According to Nielson (1994), Usability can be best defined with the five attributes Learnability,

Efficiency, Memorability, Errors, and Satisfaction.

Learnability here emphasizes on the ease of learning of the various aspects the application has to

offer to its user. The system should be easy to learn so that the user can rapidly start learning the

system. Efficiency relates to the system being efficient to use, so that once the user has learnt the

system a high level of productivity can accumulated in less time. Memorability signifies that the

system should be easy to remember, so that the casual user is able to return to the system after

some period of not having used it, without having to learn everything all over again. Errors

12

relates to the system having a low error rate, so that the user makes few errors while using the

system. Even though if the users do end up making some errors, they could easily recover from

them and be sure to not make those errors once again. Satisfaction, being the highly prioritized

one signifies that the user is completely satisfied with the use and ease of the system. In other

words the system should be pleasant to use.

Preece et al (2002) [17] sets a good illustration of the usability goals and user experience goal

putting up together.

13

2.3.2 Usability in M-Learning

A learning environment will be accepted or not from the users, according to its practical

acceptability and usefulness that include its usability and utilization [18]. The usability of a

learning environment is an important attribute as it influences the students by providing them a

good use of learning space available by making full use of it. In order to enhance the students

learning approaches mobile learning environment plays a big role in modern day mobile

generation. Most of the mobile phones in the present time are capable of providing support for

vast amount of application and providing learning through the mobile devices is indeed an

interesting concept. The integration of mobile devices with learning applications provides a

better way of learning for the students by removing the limitations of time and place in learning

activities which increases the student’s interest in learning. Being able to see your lecture notes

on the go over a mobile phone, being able to be up to date with all the classroom announcements,

discussions over the mobile application enhances the student’s experience in learning and the

usability of such kinds of applications provides a higher user satisfaction level and helps them to

remember content of their learning environment by minimizing errors which eventually leads to

an efficient and effective way of learning.

There are some potential benefits of usable learning environment [19] 1) More efficient user

tasks leads to better and more effective mode of learning activities, 2) More effective learning

activities leads to better understanding of the concept and successful learning, 3) Increased

satisfaction level of the students with the environment leads to a better way of learning once

students starts to enjoy the interaction with the environment, 4) Increased satisfaction and

effectiveness leads to the reuse of the learning environment, 5) Reduced errors enhances the user

satisfaction and their interest towards the education and hence provides even better mode of

effectiveness. 6) Higher satisfaction with the learning environment encourages the student

towards learning.

14

3 Design &Implementation

In this section we will be discussing the designed nature of the test, implementation procedure

and the process I would be following for the usability testing of the mobile application developed

for the study. Testing the mobile application on the real world users is the best way to test the

usability of any kind of application. It gives a clear cut view on, how other users for whom the

mobile application is actually have been developed for, finds the system to use.

Several questions are raised in the process of developing any type of application. As the

application is based for all types of individual users irrespective of the fact that they have past

experience of using such type of system or not. 1) How to make an easy to use, a simple straight

forward approach, and more useful and productive type of mobile learning environment? 2) How

to provide users with the experience that fits according to their style of learning and adapts itself

in a way the user want it to be? 3) How to make the mobile learning environment that could fit

according the user’s background knowledge and objectives? 4) How to improve the overall

usability of the system? This goes through the insight of the usability testing approach I

followed.

Figure 1 shows the framework followed for the usability testing process.

15

3.1 Testing Strategy

The general aim for the testing part was to conduct the user testing on a set of users, where users

are a mixed group of students from the university within different courses. The testing strategy

was to aim a specific mix of students where their past experience with the mobile devices and

using the different sort of mobile applications had no part to play for the test. The test was to

focus highly on the qualitative analysis of the application rather than the quantitative analysis

due to the nature of the mobile application. The mobile application was a pretty straight forward

functioning app where the users were given RSS based alerts depending upon the student choice

of context-awareness attribute. Also as suggested by Nielson (1994) ‘think aloud protocol’ might

not work always, as it greatly depends on the testing goals and the nature of application to be

16

tested. The quantitative results may not be always relevant and can be misleading because of the

individual differences amongst the individuals. Quantitative studies are based on statistical

analysis that as Nielson states are often false, biased, misleading or overly narrow. Better to

emphasize on the insights of qualitative research. A group of users with various backgrounds

could offer a qualitative feedback in different approaches which improves the scope of

improvement could be done in the system. Even though quantitative analysis could be

misguiding I did not want to completely ignore this aspect in my project. Prior to the testing I

evaluated the quantitative approach towards the basic functionality of the mobile application, and

results of both quantitative and qualitative criteria will be discussed in the upcoming sections.

3.2 Test Setup

The testing procedure was initially aimed to be performed with 30 students within the university,

but due to the lack of people who volunteered for the testing and the time available I have

evaluated all the testing results on 12 users. The test users for this study were all the students of

Lancaster University, studying different disciplines and variable degree year. The users consisted

of 12 users where 7 were males and 5 were females. Not surprising, all 12 of the 12 users had

some past experience with the mobile devices and applications. 8 out of 12 students had known

about some forms of mobile learning including VLE and iLancaster. The test will be conducted

in a laboratory setup where the users will be given an Android supporting smartphone with the

preinstalled applications designed for this project. Prior to the beginning of the test, users will be

given some certain set of instruction stating the nature, purpose and main goal of the usability

test. During the test the users will be asked to perform certain tasks which in turn will make them

familiar with the application, how it works and all the available features it could avail to the

students. Once the test has finished the users will be given a couple of questionnaires, one asking

them about the satisfaction with the nature of this test and the other one regarding their feedback

on the application, stating their satisfaction level with the use of application, and describing some

positive or negative aspects about the mobile application. The test will generally be about for 20

minutes including the preparation, instructing, conducting the test itself and gathering the

17

following feedback questionnaire. Which will then later be considered for the usability test

evaluation purposes stating different aspects of the mobile application.

3.3 Laboratory test

The test was conducted in the Engineering Building, here at Lancaster University around the

normal hours of the evening after the college hours, it was made sure that none of the users were

in a hurry, or were getting late for something, or were unintentionally there. The test was

arranged at the agreed time and the day available according to the free time of the students. The

users were given 3 Android based Samsung Galaxy S3 phones running the current android

version of 4.2.2 along with a detailed written set of instructions stating that the aim of this test is

to check the usability of the mobile application designed for the m-learning purposes. The users

were told specifically that I haven’t designed this mobile application, so they are free to share

their thoughts about the application, its use, the scope in improvements, and the graphical user

interface of the system. No camera recording systems were used as I didn’t think it was

necessary according to the nature of application being tested. Also, users were allowed to work

on ‘Think aloud protocol’, ie, they could share their gestures in words if they need to. The main

aim for this test was to test their reaction towards the concept of the mobile application and what

it can actually offer them in their study life style.

Following are the tasks which were asked to perform during the test:

1) Switch off the mobile phone.

2) Switch on the mobile phone.

3) Press the Menu button.

4) Navigate through the Menu.

5) Locate the mobile application.

6) Run the Mobile Application by clicking on the Icon.

18

7) Explore through it.

8) Locate the settings menu.

9) Navigate through the settings menu.

10) Explore through it by switching the available setting buttons (on/off).

11) Exit the application.

Once the user has finished all these listed tasks. They were asked to fill the testing procedure

evaluation questionnaire (ASQ). The After Scenario Questionnaire (ASQ) is 3 item questionnaire

which was designed by the IBM to determine the user satisfaction after the completion of the

task. It basically consists of the user satisfaction level from the completion of the tasks they were

asked to perform, the time consumed in the task was appropriate and the support information

provided for the tasks. A screenshot below shows the outlook of the ASQ.

It is designed using a 7 point scale, where 1 denotes the strongly agree and 7 denotes strongly

disagree. 25 & 26 describe the psychometric evaluation can be summed up to a one scale

denoting the overall satisfaction score of the task.

19

An overall evaluation questionnaire was also performed and evaluated after the ASQ, consisted

of application quality and performance based questions. The Computer System Usability

Questionnaire (CSUQ) also developed by IBM consisted of 19 questions which assesses the user

satisfaction on the overall system usability. It was divided into 3 subscales, questions 1 to 8 were

classified to figure out the System Usefulness, questions 9 to 15 were scaled to figure out the

Information Quality of the overall system, and questions 16 to 18 were then scaled to the

Interface Quality. The last question in the CSUQ was aimed for the overall satisfaction. Similar

to the ASQ, CSUQ was based on a 7 point scale with, 1 being the strongly agree and 7 being the

strongly disagree. Additional comments were also designed in order to point out any thoughts

user has to share about any aspect of the mobile application. In the end user were asked to state

some positive and some negative features they think could be improved also helped the

evaluation of the project. A screenshot of the CSUQ questionnaire is also featured on the next

page of the report.

Figure 1. Computer System Usability Questionnaire used for the study

20

4 The System in Operation

This section will feature how the mobile application works from a user’s point of view. As this

project aims at how these context aware mobile application could help a user in their learning

style, the coming parts will demonstrate how the mobile application could be beneficial for a

student to be up to date with their VLE environment.

Figure 1(a) shows the main user interface for the RSS based mobile application where all the

new uploads to the Moodle VLE are displayed. New lecture notes, coursework uploads, any new

added discussions appears in a form of a thread as shown in figure 1(b).

Figure 1(a) & 1(b)

21

The updates are displayed in a descending order of the dates they have been added. The newest

upload will be displayed on the top. These added updates can be further scrolled up or down if a

user wants to look for some specific added content from a past date.

Once the user clicks on the content of their choice, the thread opens in a new tab displaying the

detailed content with the upload URL where the content have been added. Every time an update

is added to the system the user receives a pop-up dialogue saying that there is an update since

you checked the mobile device. When the user clicks on the pop up feature it then redirects the

user to the newly added content displaying the detailed view of the topic along with the URL

address. This feature is showed in figure 2(a) and 2(b).

22

Figure 2(a) & 2(b)

Figure 3(a) and 3(b) displayed below are the layouts of setting menu screen of the activity based

context-aware mobile application and temporal based context-aware mobile application

respectively. The setting option can be found on the top right side corner of the menu screen

denoted by the setting icon in the menu screen and the icon on the top left side corner of the

screen is for the main RSS feed title screen which was discussed earlier in Figure 1(a).

For the Activity based application figure 3(a), user has the options for selecting when they want

these updates to be delivered to them. Three types of these optional features are included in the

application. 1) Permit the alerts when walking, as the application has access to the user’s device

API which was discussed in previous section, so when a user is walking for instance let’s say

around the campus, the inbuilt GPS sensor of the mobile device detects the change in the

23

location of the user and triggers this feature, the application will then check if there are any

updates which are to be delivered. Of course this is when this feature is opted ‘On’ by the user.

Similarly, the second feature when switched on by the user, the application will check for the

updates when a user is on the move (travelling or driving). With the help of suitable third party

mobile applications available on the Android market this feature is feasible with the voice

dictating application. That means, if a user is driving and this feature is opted ‘On’ and there is

an application on voice dictating already installed on the user’s mobile device. He will be

delivered the alert using the voice automated alert system which indeed is a very useful feature

of this mobile application.

Figure 3(a) & 3(b)

24

For the Temporal based Context aware mobile application figure 3(b), A user has choice to input

how many times per day they want the mobile application to check for the updates. For Example

if a user inputs 4 times a day, the application will then automatically check for the available

updates 4 times in a day and then deliver them to the user with help of RSS feeds.

One location based context aware mobile application also designed in the same way which

allows the user to receive the RSS updates depending on the location of the object points such as

learning zone, library etc. The application checks for the updates once the student reach a

specific area point within the campus. So a user can choose amongst the three temporal, activity

and spatial based context aware application whichever matches their study style. Earlier a

comparison between temporal and spatial context aware mobile application has been done and

the results were stated by Crane et al 2011. I will be evaluating the usability analysis and the

results in the next chapter.

25

5 Evaluation and Results

This section of the report discusses the results obtained from the questionnaires which were

performed in the task evaluation process and the overall system usability testing in order to

provide a better view for developing mobile applications. This will help to further state the

usability of the application and will outline the features which could be improved in the system.

5.1 Black Box Testing

This method is used to make sure all the basic features provided by the application work fine

when they are executed. This technique helps to figure out if there is any erroneous outcomes or

unexpected result when a task is performed in the application.

5.1.1 Main Screen

Test No. Test Expected Outcome Result

1 Click the application icon Launches the

applicationPASS

2 Click on an update Displays the content PASS

3 Click on settings icon Displays the available

options

PASS

4 Exit the application Application Exits PASS

26

5.1.2 Settings Screen

Test No. Test Expected outcome Result

1 Change the walking option in

settings

Toggles On/Off PASS

2 Change the driving option in

settings

Toggles On/Off PASS

3 Change the resting option in

settings

Toggles

On/Off

PASS

5.2 Black box testing results

According to the black box testing results the application performs all the basic feature it

provides effectively. That means the application does not crash or perform any unexpected

results. The application was easy to install and uninstall and had no problems so far.

5.3 User Testing Results

The results that I received narrowly indicated that:

1) They have used similar features in the application.

2) It doesn’t take that much of time for them to understand the use of application.

3) They think it could help them in their time management skills.

27

4) The system was really easy to use and straight forward.

5) They are willing to use it in future

6) They are comfortable with the concept that how context aware features uses their

location, device time and personal data. As according to them for most of the application

presently in the market they use these kind of features.

According to the task evaluation survey most of the users were satisfied with the purpose and the

nature of the task. They were able to understand how the application can be useful in their

learning space. The users gave the ASQ rating of 2.66 overall satisfaction from the task

evaluation process. They were confused in the beginning of the test, about the nature of test

being in laboratory based set-up. But they understood by the end of the evaluation process how it

could help the system to improve its existing features.

5.4 Computer system usability questionnaire results

The CSUQ questionnaire was used to evaluate the usability of the system overall. Table 1 shows

the average rating given by the 12 test users on each of the sub categories of the system usability

discussed in earlier chapter. The score ranges from 1 to 7 of the individual users but the average

rating given by the user is summed of in the table below.

Overall Usability 2.24

System Usefulness 2.72

Information Quality 2.77

Interface Quality 2.93

Table 1: Users average usability ratings

28

Overall the application received a positive response from its users. Most of the users have agreed

that the application is useful and easy to use. Table 1 shows that user’s opinion varied somewhat

in terms of the usability but most of the users have given the application a rating between 1 to 3

on a 7 point scale.

Figure 1 User’s overall satisfaction level

5.4.1 System Usefulness

Most of the users think that system is really useful. It has the ability to help the users in their

time management skills as well as being updated with their courses/modules over the mobile

phone. 10 out of 12 students have rated the application that it is easy to use the application. 11

out of 12 students rated the application as they were comfortable using it. 8 out of 12 users think

that they can use this application in their work space. Remaining users said that they couldn’t

navigate to the URL listed in any of the threads. To access those updated content over the phone

29

was not possible as the URL listed in the menu wasn’t hyperlinked. In order to access the newly

uploaded content they had to type the whole URL given in the RSS feed manually over the web

browser of the phone in order to access what the actual content was. Figure 2 shows the overall

rating given by the users.

Figure 2 User’s feedback on System Usefulness

5.4.2 Information Quality

This category shows the quality level of information provided within the application. This

includes the data content within each RSS based message. The users have given the CSUQ rating

of 2.77 out of 7. There were no unexpected and irrelevant data within the application users

commented. However, some users think that the application could have been integrated with the

Lancaster University’s own Moodle environment iLancaster. Some users commented that

application could have provided the download link of the added content instead which I think

30

was an interesting aspect. Most of the users were satisfied with information quality. Figure 3 and

Figure 4 shows the variable rating given by the users under this field. Majority of users think that

application as such did not need any specific help section as it was fairly easy to understand and

really simple to operate.

Figure 3 Information Quality (1)

31

Figure 4 Information Quality (2)

5.4.3 Interface Quality

Under this category users rated the mobile application on an average of 2.93. Most of the users

think the application had a really simplistic user interface, which was really easy to learn, to use

and to remember. They got familiar with the system right after they used the application. They

understood all the features within the mobile application and appreciated the simplicity of the

application. Most of the user said the application did not have any type of unnecessary features

which were not used. Navigation through the interface was fairly simple. However one of the

user didn’t like the user interface. They think the user interface could be improved by adding

more features to it. 9 out of the 12 users somewhat liked interface of the application as they think

adding extra features increases the complexity of the program. Although the distribution of

people finding the user interface pleasing is quite distributed but overall they intend to ask for a

simple structured mobile application.

32

Figure 5 shows the Interface Quality of the application

5.4.4 User Feedback

Apart from the questionnaire some user feedback were also received in the evaluation process.

Majority of the users came up with the most popular positive attributes of the mobile application

1) Useful

2) Simple

3) Helpful

4) Usable

5) Effective

6) Collaborative

33

Some negative aspects according to some users point towards the information content, lack of

integration with the built in web browser of the mobile phone as they could not access the URL

provided with the data content with in the RSS feed, and improving the user interface by adding

more features.

5.5 Summary

This chapter has explored the testing and the evaluation of the mobile learning application. The

usability test conducted earlier in this project deduced the overall usability of the mobile

application for this project. The Black box testing signifies that the application and all the

features included within the application were working appropriately. There were no unnecessary

features in the application which were not used during the testing procedures. The usability

testing performed specifically for this task was focused more on the laboratory setting as user’s

full time and focus were recommended while they share their views on the aspects of the mobile

application. The testing method showed different aspects of the application from a user’s point of

view. Some of the attributes were highlighted during the evaluation procedure. Some of them

were positive and some of them were negative. In order to present a better view on the usability

both the quantitative analysis of the system along with qualitative data are needed. However it

was apparent that there was a scope in improving the user interface and the information content

according to some users, but then it’s the prototype of the application. Relevant features and

additional plugins could be added after the pilot model of application.

34

6 Conclusion

This section of the report we will revisit the initial stated aim and all the work that was done in

the project, unimplemented work that was supposed to be done and the scope of improving the

system as an overall. This section mainly focuses on whether the initial aim of this project

described earlier in the beginning of this report have met.

6.1 Summary

The report has explored the fact that how mobile devices has become a vital part in a student’s

life. Every system is now at the verge of being mobile or it already has almost. Mobile devices

can serve a big difference by exploiting the areas of their learning environment on the go. How a

user can still be updated with all the learning content and lectures and all the important

discussions which cannot be missed now can be available to the students wherever they are. This

project explored the usability of mobile learning application in order to provide a student a good

learning environment on their mobile device. It also explored some similar environments which

made this study possible or are somehow related in the development procedure of the mobile

application. In really simple words the student at whom this study and application were based,

were asked how an environment like this could help them? By sharing their thoughts on the

mobile application, they can create a scenario for the developers who can actually bring their

ideas into the mobile application. This project insights on some of those user point of view based

scenarios in order to provide a better scope of features within the application.

6.2 Review of the Aim

The overall aim of the project which were stated in the introduction to this report.

To investigate the usability of the context-aware application in learning

environment.

35

Three dimensions of the context-awareness were exploited in mobile application designed for

this study. Spatial dimension, temporal dimension, and Activity based context-awareness. All

three came out to be equally important and equally useful for the student based on the student’s

choice and nature of their learning style. The mobile application was very easy to use, simple to

understand and efficient to use. Users who were in fact the university’s own students found that

the application can be used effectively and can provide positive outcomes in their time

management skills. The mobile application was based on the Android platform which was a

good plus point as some of the users were using the same platform based mobile phones in their

life style. Most of the users who were the test users of this study have used M-learning

applications which were similar to the one I used in this study in some ways, which indicates that

the concept of bringing learning to mobile indeed is a great idea. Users showed great interest in

sharing their thoughts on bringing learning content on their mobile. The study shows that they

can come up with ideas which could help in making the system more systematic and more

efficient to use. However there were some features they think could be improved which will be

covered in the coming sections.

6.3 Problem Encountered

In the beginning of this project I did experience some of the problems, which were then fixed

after the discussion with my supervisor. I was given clear guidance on how to look at the whole

scenario and scope of the usability testing.

During the project, gathering the test users for the study was difficult as it was really unlikely for

them to be all free at the same time for the test. So that created a bit of delay in the evaluation

procedure. But then I did eventually was able to collect all users at one place at the same time.

Performing the Test itself was somewhat challenging, as I was the only one conducting,

instructing the users at the same time but then it was a good experience.

36

Running the designed applications on the Eclipse software was not possible to install on various

computers in the Lab. So I decided to gather as many Android phones I could to perform the

usability testing which was then performed 3 mobile phones.

6.4 Unimplemented feature

The test initially was aimed to be performed on 30 users but due to lack of time and gathering 30

test users and conducting test amongst all of them was very difficult for me. Which was then

discussed with my supervisor and we decided to perform the test on 12 users instead.

6.5 Improvements and possible future work

In this section we will be discussing about the scope of all improvements which came into the

picture during this study.

6.5.1 Extension of the mobile app with the mobile inbuilt browser

One improvement that could be made is adding a web browser support in the application. For

example once a user get an automated RSS update, he receives the update as the date, time, the

topic in which the upload has been made and the web moodle URL. But that web URL is not

hyperlinked. In other words If a user can simply click on the given URL he or she should

automatically be able to redirect the URL to the mobile device’s inbuilt browser. A snapshot of

the update received by a user is display below.

37

Figure 1 A snapshot of a RSS Update.

The URL in the update is inaccessible. So if a user who is already mobile (limitation of time and

availability space) get an update about a new lecture note being added he cannot just simply click

on the URL. The user have to type the whole URL manually to a web browser in case he or she

wants to see what the actual content/discussion is.

38

6.5.2 Integration of the application with iLancaster

The university’s m-learning environment application is really popular amongst student. It has

number of features that are available within the mobile application including library, bus

timetable etc etc. According to one newly added feature in the iLancaster mobile app. A student

can now access their moodle virtual learning environment within the app itself. The application

which was designed for this study if possible could be integrated with the iLancaster application.

So that once the user clicks on the URL provided by the RSS feed, simply by clicking on that

link he or she should be able to access the content. A screen shot of the iLancaster’s Moodle

environment is displayed below.

39

6.5.3 Improving User Interface

From the usability test performed for the study, the results shows that there is still scope for

enhancing the user interface of the application. The usable mobile app should be pleasing to look

at as well as satisfying to use. The GUI could be enhanced with the help of custom background

or simply by adding some graphical icons for the application. The user interface could be

categorized is such a way that all the newly alerts about the lecture/course notes should go in one

section and all the discussions should go under the thread of discussion. It will eventually

increase the usability of the overall system. This leaves the space for the future work and gives

an insight to the possible work which could be done to enhance the user’s experience in m-

learning.

40

6.6 Lessons Learned

This Project has had a lot to teach me. The main things which I have learnt while working on this

project is the importance of time management skills. I gained supervising experience while

conducting the usability test and has encouraged me to do something like this in the coming

future. I got to improve my organization skills which was one of the key feature while doing this

project on the whole. I have gained some experience with the basic Android programming as

well in order to carry this study.

The main lesson I have learnt will be key to success in developing any type of system, whether it

be a mobile application or any type of system in general. Learning about the possibilities of

improvement in a mobile application I have gained some really valuable core concepts on how to

make any type of system usable by testing it on the real world users.

6.7 Overall Conclusion

This project has fairly completed its goal to investigate the usability of a mobile application in

learning and its usability amongst student. However I was able to produce some of the features

that could improve the overall usability of the application from the study. This will help in

improving the overall functionality of the mobile application in the future.

This project has been a lot to be learnt from and has given me insights on how to make a system

even more efficient. Testing user’s focused mobile application on the same area of users creates

new scope for feasible environment which could have been there in the application.

41

References

[1] Nielsen J,. Usability 101: Introduction to Usability. (2007)

[2] Nielsen J., & Mack L.R., Usability Inspection Method, John Wiley & Sons Inc. 1994, New

York.

[3] Nielsen J., Usability Engineering 1994.

[4]L.Crane, P. Benachour, P.Coulton, “A User Study of Spatial and

Temporal Context Aware Technologies to support Mobile Virtual

Learning Environments”. [MLearn2011 Conf. Beijing, China,

pp.335-342, 2011]

[5] Google’s Android becomes the worlds leading smart phone platform (Canalys)

http://www.canalys.com/newsroom/google%E2%80%99s-android-becomes-world

%E2%80%99s-leading-smart-phone-platform

[6] IDC Press Release http://www.idc.com/getdoc.jsp?containerId=prUS23638712

[7] Eclipse IDE http://en.wikipedia.org/wiki/Eclipse_(software)

[8] Android SDK Wikipediahttp://en.wikipedia.org/wiki/Android_software_development

[9]Dourish P (2004) What We Talk When We Talk About Context. Personal and Ubiquitous

Computing 8: 19-30.

[10]Dey AK (2001) Understanding and Using Context. Personal and Ubiquitous Computing 5:

4-7.

Springer-Verlag London Ltd

[11] Schilit B, Adams N, Gold R, Tso MM, Want R (1993) The ParcTab Mobile Computing

System. In

42

Proceedings of Fourth Workshop on Workstation Operating Systems, Oct 1993: 34-39.

[12]Dey AK & Abowd GD (2000a) CybreMinder: A Context-Aware System for Supporting

Reminders. In Proceedings of Second International Symposium on Handheld and Ubiquitous

Computing, HUC 2000: 172-186.

[13] Brown LM, Brewster SA & Purchase HC (2006) Multidimensional Tactons for Non-visual

Information Presentation in Mobile Devices. In Proceedings of MobileHCI’06: 231-238.

[14] Abowd G (1999) Classroom 2000: An Experiment with the Instrumentation of a Living

Educational Environment. IBM Systems Journal, Special issue on Pervasive Computing 38(4):

508-530.

[15] Bardram JE (2004) Application of Context-Aware Computing in Hospital Work – Examples

and

Design Principles. In Proceedings of ACM Symposium on Applied Computing: 1574 – 1579.

[16] Pocket Lancaster http://pocket.lusu.co.uk/

[17] Preece J, Rogers Y & Sharp H (2002) Interaction Design. Beyond Human-Computer

Interaction.

John Wiley & Sons Inc.

[18] Evans K M & Finck N., Interview with J. Nielsen, The web professional online magazine.

[19] Fetaji M http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4588461&tag=1

[20] Monahan T. Usability Testing of Collaborative & Interactive Iniversity on a mobile device

43