seminar thesis in e-government systems - unifr.ch€¦ · seminar thesis in e-government systems ....
TRANSCRIPT
Seminar Thesis in E-Government Systems
University of Fribourg, Department of Informatics
eGovernment
Evaluation of SmartParticipation Platform.
STUDENT NAMES: José Alberto Mancera Andrade STUDENT NUMBERS: 10-801-207 COURSE NAME: Electronic Business and Recommender Systems
DEPARTMENT: Department of Informatics
SUPERVISOR: ASSISTANT:
Prof. Dr. Andreas Meier Luis Terán
DATE OF SUBMISSION:
29-02-2016
II
Table of contents
List of Figures ............................................................................................................. V
1. Introduction ............................................................................................................. 1
1.1 Problem statement ............................................................................................ 1
1.2 Research Objectives and Methodology ............................................................. 2
1.2.1 Research Questions ................................................................................... 2
1.2.2 Objectives and Output of the thesis ............................................................ 2
1.2.3 Research Methodology ............................................................................... 2
1.3 Timetable .......................................................................................................... 3
1.4 Addressees ....................................................................................................... 3
2. User Centered Design (UCD) ................................................................................. 4
2.1 UCD Design Process ........................................................................................ 5
2.2 Elements involved in Human Computer Interaction (HCI) ................................. 6
3. SmartParticipation Evaluation Design .................................................................... 7
3.1 Main Evaluation Methodologies ........................................................................ 7
3.1.1 Usability ...................................................................................................... 7
3.1.1.1 User Segmentation .................................................................................. 7
3.1.1.2 User Tasks Assignment ........................................................................... 8
3.1.1.3 Questionnaire .......................................................................................... 8
3.1.2 Heuristics .................................................................................................... 9
3.1.2.1 Design Principles Definitions ................................................................. 10
3.1.2.2 Severity Errors ....................................................................................... 10
4. Smart Participation Evaluation .............................................................................. 11
4.1 Heuristic Evaluation ........................................................................................ 12
4.1.1. - Main Page ............................................................................................. 12
4.1.2. – Participate or Article List ....................................................................... 13
4.1.3. – Show Article Top .................................................................................. 14
4.1.4. – Show Article Down ............................................................................... 15
III
4.1.5. – Article Statistics .................................................................................... 16
4.1.6. – Login ..................................................................................................... 17
4.1.7. – Forgot Password .................................................................................. 18
4.1.8. – Register ................................................................................................ 19
4.1.9. – Search .................................................................................................. 20
4.1.10. – User Profile ......................................................................................... 22
4.1.11. – User Profile Update ............................................................................ 22
4.1.12. – User Settings ...................................................................................... 23
4.1.13. – Feedback Page .................................................................................. 24
4.1.14. – Recommendations .............................................................................. 25
4.1.15. – New Article ......................................................................................... 26
4.2 Usability Evaluation ......................................................................................... 30
4.2.1. – Usability Evaluation Matrices Results ................................................... 30
4.2.1. – Qualitative Analysis .............................................................................. 33
5. Evaluation Analysis and Recommendations Resume .......................................... 34
5.1 Main Problems Detected ................................................................................. 34
5.2 Recommendations .......................................................................................... 36
6. Low Level Prototype ............................................................................................. 38
6.1 SmartParticipation Web Pages ....................................................................... 38
6.1.1. - Main Page ............................................................................................. 38
6.1.2. – About .................................................................................................... 40
6.1.3. – Participate or Article list ........................................................................ 41
6.1.4. – Show Article .......................................................................................... 42
6.1.5. – Login ..................................................................................................... 44
6.1.6. – Register ................................................................................................ 45
6.1.7. – Search .................................................................................................. 45
6.1.8. – Feedback Page .................................................................................... 46
6.1.9. – User Profile ........................................................................................... 47
IV
6.1.10. – New Article ......................................................................................... 48
6.1.11. – Recommendations .............................................................................. 48
7. Conclusions .......................................................................................................... 50
8. Future Work .......................................................................................................... 51
9. References ........................................................................................................... 52
V
List of Figures
Figure 1.- Timetable Table .......................................................................................................... 3
Figure 2- User Centered Design Cycle ......................................................................................... 4
Figure 3.- User Centered Design Process .................................................................................... 5
Figure 4.- Human Computer Interaction Aspects ....................................................................... 6
Figure 5.- User's Evaluation Tasks .............................................................................................. 8
Figure 6.- Question 1 .................................................................................................................. 9
Figure 7.- Question 2 .................................................................................................................. 9
Figure 8.-Common Design Principles .......................................................................................... 9
Figure 9.- Severity Error Scale ................................................................................................... 11
Figure 10.- SmartParticipation Evaluation Process .................................................................. 11
Figure 11.- Main Page Heuristic Evaluation ............................................................................. 12
Figure 12.- Article List Heuristic Evaluation .............................................................................. 13
Figure 13.- Article Top Heuristic Evaluation ............................................................................. 14
Figure 14.- Article Down Heuristic Evaluation .......................................................................... 15
Figure 15.- Article Statistics Heuristic Evaluation ..................................................................... 16
Figure 16.- Login Heuristic Evaluation ...................................................................................... 17
Figure 17.- Forgot Password Heuristic Evaluation ................................................................... 18
Figure 18.- Register Heuristic Evaluation ................................................................................. 20
Figure 19.- Search Heuristic Evaluation .................................................................................... 21
Figure 20.- User Profile and User Profile Update Heuristic Evaluation .................................... 23
Figure 21.- User Settings Heuristic Evaluation ......................................................................... 24
Figure 22.- Feedback Page Heuristic Evaluation ...................................................................... 24
Figure 23.- Recommendations Heuristic Evaluation ................................................................ 25
Figure 24.- New Article Heuristic Evaluation ............................................................................ 29
Figure 25.- Easy Task Analysis .................................................................................................. 31
Figure 26.- Medium Task Analysis ............................................................................................ 32
Figure 27.- Hard Tasks analysis ................................................................................................ 32
Figure 28.- Comfort and Experience Level ................................................................................ 33
Figure 29. - Main Problems Detected ....................................................................................... 36
Figure 30.- Recommendations.................................................................................................. 37
VI
Figure 31. - Main Page ............................................................................................................. 38
Figure 32.- Main Page with Video ............................................................................................ 39
Figure 33.- About Page ............................................................................................................. 40
Figure 34.- Participate Page ..................................................................................................... 41
Figure 35.- Review Article ......................................................................................................... 42
Figure 36.- Article with Negative Comment ............................................................................. 43
Figure 37.- Sing In Window ...................................................................................................... 44
Figure 38.- Forgot Password Mini Window .............................................................................. 44
Figure 39.- Sing Up ................................................................................................................... 45
Figure 40.- Search Mini Window .............................................................................................. 45
Figure 41.- Search Results ........................................................................................................ 46
Figure 42.- Feedback Window .................................................................................................. 46
Figure 43.- User Profile ............................................................................................................. 47
Figure 44.- New Article Creation .............................................................................................. 48
Figure 45.- Recommendations Page......................................................................................... 49
1
1. Introduction
1.1 Problem statement
Overview
The relationship between politicians and citizens has been always in constant
interaction since the creation of governments. Most of the time, citizens find difficult
to feel identified with a certain political party or candidate and in certain cases
politicians lose the visibility of the topics or problems that society demand to be
attended.
SmartParticipation Web Platform emerges as a potential solution to cope with these
problems. It supports open discussions and political community building through
dynamic profiling.
The goal of this seminar thesis is to evaluate the platform from the user centered
design perspective and present a deep analysis that will help to provide not only a
better diagnostics from the usability and performance but also specific
recommendations to improve the system in a future version.
2
1.2 Research Objectives and Methodology
1.2.1 Research Questions
The next group of questions is the guideline of our study, each of them is answered
in sequence during the evolution of the document.
1. What is the design impact from the user perspective using the platform?
2. What are the required parameters, elements or models to consider in order to
evaluate the platform?
3. What are the User interaction issues in the platform?
4. What are the potential recommendations or improvements needed to create a
better version of the platform from the user centered design perspective?
1.2.2 Objectives and Output of the thesis
The goal of this seminar thesis is to evaluate the SmartParticipation platform from the
user centered design perspective and present a deep analysis which includes a
customized selection of parameters, elements or models that play a main role in the
evaluation of the platform. In addition the study includes an analysis of the user
interface and interaction issues which eventually will help to provide specific
recommendations and improvements for future versions of the platform.
1.2.3 Research Methodology
In order to answer the questions of this case of study, our analysis will be based on
the SmartParticipation platform, user data, previous studies in the use of
eGovernment systems and recommender systems. Furthermore, selected textbooks
will be taken into account, research papers, similar study cases or examples provided
by seminar supervisor.
3
1.3 Timetable
10-07-2015 Acceptance of working title
10-11-2015 Submission of the proposal
October 2015 Continue literature Research and reading.
Writing Chapter 1, 2 and 3
November 2015 Writing Chapter 4,5,6 Draft of the paper
Finishing the report
Revision and Correction
22.12.2015 Midterm Appointment
27.02.2016 Submission of the thesis report
29.02.2016 Presentation of the thesis report Figure 1.- Timetable Table
1.4 Addressees
The target audience of this paper is focused mainly on students in the fields of
information management, marketing, political sciences and professionals who are
involved in the field of eGovernment. The results of this seminar should provide the
parties mentioned above not only valuable knowledge in order to better understand,
analyze and improve the quality and design of Voting Advice Applications, but also a
better understanding of interface influence and the impacts that User Centered
Design plays in the design of such platforms as SmartParticipation.
4
2. User Centered Design (UCD)
In order to get a better understanding and interpretation of the findings of this
seminar study, it is necessary to understand the basic notions of User Centered
Design (UCD) and some terminology that is used later in the result interpretation.
UCD emerges as field, which answers some questions related to a specific platform
or web service such as: What can I easily build on this platform? What can I create
from the available tools? Or what do I as a programmer find interesting?
UCD is a discipline concerned with the Design, Implementation and Evaluation of a
platform or web application that humans use or interact. The figure 2 shows these
three main interactions that are inside of a main cycle, which is in constant movement
with the aim to improve the platform.
Figure 2- User Centered Design Cycle
UCD is vastly used nowadays to understand user experience and interaction with the
system in the way to identify important elements that help designers to improve the
platform in a new version.
5
2.1 UCD Design Process
The UCD design process is shown in the figure 3 and it contains the phases that a
conceptual web product has to walkthrough in order to become a complete
application that fulfil the final user expectations.
Articulate
User and task descriptions
• Questionnaires• Interviews• Focus groups
• Questionnaires• Diary studies• Ethnography• Interviews• Observation• Focus groups
Brainstorm
Throw-away prototypes
• Participatoryinteraction• Walk-through• Paper prototypes• Sketching• Storyboards• observation
• User involvement• Focus groups• Representations and metaphors
Refine
Testable prototypes
• Heuristics• Usability• WOz
• GUI design• Interface guidelines• Style guides
Complete
Complete systems
• Field Testing
Goa
lM
etho
ds a
nd to
ols
Prod
uct /
Res
ult
Figure 3.- User Centered Design Process
There are four main goals in the UCD design process:
• Articulate: The idea is to find the problem to be solved and list the desires
and characteristics that the final users have by many methods in order to get
the types of users and general task descriptions that in a way our future
system should fulfill.
• Brainstorm: Here the goal is to collect ideas from users, programmers and
start to generate the first approach of the platform. It is not necessary to start
to build the application, here paper prototypes are an option for fast
implementation and practical to receive some feedback from the final users or
even for the designers.
• Refine: In this part, the designers proceed to generate a first attempt for a
prototype in low, moderate or high level implementation. In this stage the
prototype should be tested by users or other techniques/methods and receive
some feedback that may affect again the prototype design.
6
• Complete: Here is the phase of launching the product of platform as a formal
service and measure the success or go back to a previous stage to continue
its refinement.
Based on these definitions, this seminar paper is focused mainly on the Brainstorm
and Refine phase for the SmartParticipation Platform. The seminar does not attempt
to achieve a full functional prototype.
2.2 Elements involved in Human Computer Interaction (HCI)
In addition to the UCD design process, defining the important aspects involved in HCI
are the key to support the seminar work analysis, the figure 4 shows the main
elements related to the Human Computer Interaction.
Figure 4.- Human Computer Interaction Aspects
Cognitive Process: Focus to understand the way that we user thinks and the role of
his attention, perception, memory, reasoning and problem solving.
Physiology: Understand the consequences and implications to see how the body
interacts with the system in terms of vision, hearing, touch/feeling and movement.
Emotions: Refers to moods, sentiments and emotions involved in the use of the
system or application and its effects.
Society: Focus mainly in how the environment of a person or people who surround
him affect or influence his decisions.
These elements will be referred later in our analysis when the evaluation of the
system takes place. For the moment is important just to keep in mind their meanings.
HCI
Cognitive Process
Emotions
Society
Physology
7
3. SmartParticipation Evaluation Design
As it was mentioned previously, SmartParticipation is already a full functioning
platform and categorized as a complete system, nevertheless it has not been
analyzed from the UCD design process. In order to perform an evaluation, it is
needed to bring the current system (SmartParticipation) back to the refine and
brainstorm phases of our model and perform two main evaluation techniques:
Heuristics and Usability. Based on the results of these techniques, a potential
brainstorm stage will be integrated in the analysis in order to get additional features
that may be integrated in a new low prototype fidelity version.
3.1 Main Evaluation Methodologies
Considering the purpose or the platform and the time to perform the evaluation,
Usability and Heuristics methodologies are used with the aim to provide a first
diagnostic of the system. The combination of both methodologies bring a powerful
result in the sense that one is focused in the users and the other in the functionality
design.
3.1.1 Usability
This method is conducted with real users by three specific techniques at the same
time: observations, walk-through and think a loud method. These techniques will
show user’s abilities, problems performing tasks in the system and level of
satisfaction or frustration.
3.1.1.1 User Segmentation
The Usability evaluation requires to segment users in groups in order to understand
the level of involvement with the platform. In this case the study expects to bring
generous feedback because the platform is working as a complete system and the
user can test the real functionality. For the purposes of our study we will select a
minimum of 8 participants and prepare an evaluation where all the subjects are
assigned to all tasks (within subjects) in different order, based on the characteristics
of system, we will choose users who are not familiar with e-Government platforms.
8
3.1.1.2 User Tasks Assignment
In order to conduct the usability evaluation mentioned above there is needed to
design a series of tasks that all the types of users have to perform. The proposed
tasks for the SmartParticipation system are shown in figure 5.
Figure 5.- User's Evaluation Tasks
The performing of these tasks by the users will provide important feedback
information with respect the usability of the system
3.1.1.3 Questionnaire
The usability evaluation also considers a questionnaire, which consists of two
questions, one at the beginning of the test (Figure 6) and a second one (Figure 7) at
the end of the usability evaluation. The scale used is from 1 to 5 where 1 represents
beginner or not at all comfortable and 5 experienced and very comfortable.
These questions have the aim to help to know better the user experience with the
platform in order to obtain a qualitative analysis that can be useful to get some
insiders about how particular users perceive the usability of the platform.
Easy
Explore main page and
understand the main purpose of
the platform
MediumSee articles
Create an account
Review and rate an article
Fullfill feedback Formulary
Follow a tutorial for recommendations
Hard
Modify User Profile
Create a new article with description and images
Lost password, retrieve a new one
9
Figure 6.- Question 1
Figure 7.- Question 2
3.1.2 Heuristics
This evaluation method has the advantage that there are not end users required and
it involves a systematic analytical inspection to see if the SmartParticipation interface
complies to design guidelines. This methodology applies common design principles
from D.Norman and from ergonomic principles Bastien/Scapin in references [2] and
[3]. The principles are shown in figure 8.
Figure 8.-Common Design Principles
Visibility
Feedback
Visible Constraints
Mapping
Consistency
Affordance
Idioms and Propulation Steriotypes
10
3.1.2.1 Design Principles Definitions
1. - Visibility: It is focused to see if the functions of the system are visible in a way
that the user can observe that these functions are relevant, otherwise invisible
automatic controls can make it more difficult to use.
2. - Feedback: It verifies if the system has simple interactions with the user such as
sound, highlight, animation, tactile or combination of these. Using it in the right way
can also provide the necessary visibility for user interaction and inciting the user for
the next step. (i.e. After pressing a button, the user receives a screen notification that
something has happened).
3. - Visible Constraints: It detects the action restrictions that the user can do in a
system and analyses if the design prevent the user from selecting incorrect options or
it helps the user to reduce the input error. (i.e. Entering a date in a system).
4. - Mapping: It verifies the natural relationship between controls and displays, it
ensures that the user knows for instance, what is the button that send an email or
delete an element.
5. - Consistency: It is related to see if the interface have similar operations and use
similar elements for similar tasks, for example that the publish article button is always
on the right and it does not change position depending of the different webpage. The
main benefit is that consistency helps to make interfaces easy to learn and use.
6. - Affordance: Refers to an attribute or an object that allows people to know how to
use it (suggest how one could use it, “to give a clue”). In graphical interfaces it
measures the intuitively level of the interface.
7.- Idioms and Population Stereotypes: It deals with interface idioms, which
represents arbitrary behaviors (i.e. red means danger, green means safe) and
population stereotypes varies in different cultures like presenting temperatures in
Fahrenheit or Celsius depending of the region where the website is consulted.
3.1.2.2 Severity Errors
Based on the heuristic design properties, each of them will be weighted during the
evaluation and the values that correspond to the severity of the error are defined from
11
1 to 5, where 1 is acceptable and 5 very critical and it has to be fixed as soon as
possible. (See figure 9)
Severity Level Description
1 I do not agree that this is a usability problem at all
2 Cosmetic problem only: no need to be fixed unless extra time is available
on the project.
3 Minor usability problem: fixing this should be given low priority.
4 Major usability problem: important to fix, should be given high priority.
5 Usability catastrophe: imperative to fix this before product can be released.
Figure 9.- Severity Error Scale
In the next section, these principles will provide useful recommendations during the
Heuristic analysis.
4. Smart Participation Evaluation In this section the two evaluation methodologies are applied to the SmartParticipation
platform, the chapter is composed by the application of both methodologies with their
respective comments and later there will be a general analysis section.
The complete evaluation process is shown in the figure 10, where we can see the
order of each evaluation.
HEURISTICS USABILITY Results
Figure 10.- SmartParticipation Evaluation Process
In the figure 10, heuristics is used in the first instance to recognize immediately errors
that can affect directly human computer interaction with the system and later be able
to create a low level prototype (graphical interface) that can be tested with real users
in the usability test.
12
4.1 Heuristic Evaluation
The evaluation will be performed in every page of the SmartParticipation platform
with an evaluation matrix that contains the design principles and their respective
weights from figure 9.
4.1.1. - Main Page
The main webpage provides a clear explanation of the purpose of the system,
nevertheless it is necessary to re-arrange the information in a new version, where it
looks like more as a commercial platform.
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem and just need to re arrange the
information
Feedback 1 With the interaction with links or buttons there is always a progress
icon
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept
Idioms 3 The Interface has too much information that confuses the user and
it does not provide a clear purpose of the platform.
Affordance 1 The buttons with names make clear their function Figure 11.- Main Page Heuristic Evaluation
13
4.1.2. – Participate or Article List
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem and just need to re arrange the table
Feedback 1 With the interaction with links or buttons there is always a progress
icon
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept
Idioms 2 The Interface speaks the user language but it does not explains the
units used for the popularity section.
Affordance 1 The buttons with names make clear their function Figure 12.- Article List Heuristic Evaluation
14
4.1.3. – Show Article Top
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem and the presentation of the picture
could be more centered
Feedback 1 With the interaction with links or buttons there is always a progress
icon
Mapping 1 There is no problem knowing the function of the buttons
Consistency 2 The arrange of menus is kept but the colors of the buttons change
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function Figure 13.- Article Top Heuristic Evaluation
15
4.1.4. – Show Article Down
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change with other sections
Feedback 1 With the interaction with links or buttons there is always a progress
icon
Mapping 1 There is no problem knowing the function of the buttons
Consistency 2 The arrange of menus is kept but the colors of the buttons change
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function Figure 14.- Article Down Heuristic Evaluation
16
4.1.5. – Article Statistics
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change with other sections and
modification of the green section
Feedback 1 With the interaction with links or buttons there is always a progress
icon
Mapping 1 There is no problem knowing the function of the buttons
Consistency 2 The arrange of menus is kept but the colors of the buttons change
Idioms 1 The Interface speaks the user language but there is no explanation
of details of the statistics.
Affordance 1 The buttons with names make clear their function Figure 15.- Article Statistics Heuristic Evaluation
17
4.1.6. – Login
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the size of the login
textfields.
Feedback 1 With the interaction with links or buttons there is always a progress
icon and also a message for errors.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function. Figure 16.- Login Heuristic Evaluation
18
4.1.7. – Forgot Password
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the size of the textfields.
Feedback 1 With the interaction with links or buttons there is always a progress
icon and also a message for errors.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function Figure 17.- Forgot Password Heuristic Evaluation
19
4.1.8. – Register
20
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the size of the textfields.
Feedback 1 With the interaction with links or buttons there is always a progress
icon and also a message for errors.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function Figure 18.- Register Heuristic Evaluation
4.1.9. – Search
21
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the colors of the text fields.
Feedback 4 With the interaction with links or buttons there is always a progress icon, a message for errors and no action by pushing search button for articles and when the text field is empty
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept but no consistency in the articles
search with error messages
Idioms 2 The Interface speaks the user language but statistics need to
clarify units
Affordance 1 The buttons with names make clear their function Figure 19.- Search Heuristic Evaluation
22
4.1.10. – User Profile
4.1.11. – User Profile Update
23
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the size of the text fields.
Feedback 1 With the interaction with links or buttons there is always a progress
icon.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is maintained
Idioms 1 The Interface speaks the user language but too much text
sometimes that confuse the user.
Affordance 1 The buttons with names make clear their function
Constraints 1 User does not make mistake by the scale between discussion and
opinion. Figure 20.- User Profile and User Profile Update Heuristic Evaluation
4.1.12. – User Settings
24
Principle Weight Analysis Result
Visibility 2 It is just a cosmetic problem to change the size of the text fields.
Feedback 1 With the interaction with links or buttons there is always a progress
and provide feedback for bad input information icon.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function
Constraints 1 There is a verification of the input data in the textfield Figure 21.- User Settings Heuristic Evaluation
4.1.13. – Feedback Page
Principle Weight Analysis Result
Visibility 1 No cosmetic problem
Feedback 1 With the interaction with links or buttons there is always a progress
icon.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is maintained.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function
Constraints 1 User does not make mistake by the scales between awful and
awesome Figure 22.- Feedback Page Heuristic Evaluation
25
4.1.14. – Recommendations
Principle Weight Analysis Result
Visibility 2 Cosmetic problem for the colors and size of submenus
Feedback 1 With the interaction with links or buttons there is always a progress
icon and tutorial well explained.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is kept.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function
Constraints 1 User does not make mistake by the scales between awful and
awesome Figure 23.- Recommendations Heuristic Evaluation
26
4.1.15. – New Article
27
28
29
Principle Weight Analysis Result
Visibility 2 Cosmetic problem for the colors and size of submenus
Feedback 1 With the interaction with links or buttons there is always a progress
icon and tutorial well explained.
Mapping 1 There is no problem knowing the function of the buttons
Consistency 1 The arrange of menus is consistent.
Idioms 1 The Interface speaks the user language
Affordance 1 The buttons with names make clear their function
Constraints 1 User does not make mistake by the scales or rating articles. Figure 24.- New Article Heuristic Evaluation
Based on all these properties among all the different menus, and options in the
SmartParticipation Platform. It was possible to find several minor improvements, the
platform originally has a good functioning and purpose to test some recommender
algorithms to the users and it achieves properly its purpose. Nevertheless, the
heuristic evaluation gives some recommendations through its properties in order to
improve the design in a new prototype from the perspective of a web designer.
It is important to remark that the heuristic evaluation was performed by one person,
so there may be still some unidentified improvements to the platform, which will be
complemented by a usability evaluation in the next section.
30
4.2 Usability Evaluation
As it was explained in section 3.1.1, a series of tasks will be performed from easy to
hard difficulty level and the evaluator must observe and not interfere with the logic of
the user while the user is performing the task. As previously mentioned the usability
evaluation was performed with 8 users in a within modality (all the users perform all
the tasks), using the think aloud method, in order that the users speak their thoughts
while doing the assigned task.
The users come from different educational backgrounds no related to computer
science, they are in an age range between 27-30 years old and their anonymity in
this study is preserved. In addition, this study does not make any distinction between
male and females.
Participants will be given instructions on what they are to do and they will be
encouraged to complete task as quickly as possible. They must not have a mobile
device that may distract them during gameplay, and will be monitored to ensure
proper task order. At the beginning and end the user will answer a short two
questions questionnaire in order to measure qualitative analysis.
The usability evaluation in this seminar is considered as a complement of the
heuristic analysis, which will provide us a deeper idea of the user decision process,
understanding or attitude, during the interaction of the user with the platform.
The next subsections resume the final findings derived from the transcript and
observation analysis of the 8 users.
4.2.1. – Usability Evaluation Matrices Results
The next tables’ matrices show the task, the time to complete the task in average by
the users and the analysis results.
Easy Task with Time Analysis Results
Explore main page and understand the main purpose of the platform.
The user got confused with the main interface, it was hard to
find the small box that describes the main purpose of the
system.
The main webpage does not talk a homogeneous language for
the users who are not familiar with computer science terms.
It is hard to guess the order of start to read the information, the
31
Average time to
complete the task
among users: 1.04 min
size of fonts do not express a clear hierarchy order where to
start to read.
Some users had to go to other secondary webpages to guess
what the system was about
The main interface does not reflect an intuitive way to tell the
users what is the platform about.
Users expressed that they spend already a lot of time and they
are not willing to continue to explore the webpage. Figure 25.- Easy Task Analysis
Medium Tasks with Time Analysis Results
Review Articles
Average time to complete
the task among users:
2.14 min
The user does not understand the meaning of the
recommendation numbers and it confuses him/her.
The statistics are not clear and units are not specified, in a
certain way the user would prefer a nomenclature box
The images should be bigger
The appearance of the green colors in the background give
the sensation to the user of being navigating in an old
fashion interface
Some users do not find disturbing the green color in the
statistics background but they would like to know the
meaning of the values.
Create an account
Average time to complete
the task among users:
2.07 min
The interface does not speak the language of the user, it is
not clear where is the personal data, instead there is a big
explanation of algorithms that user may not understand.
The amount of information presented at the first time of an
account creation is very extensive and confusing for the
user.
It is not specified the symbols allowed in the new password
or the minimum length of the password.
Review and rate an article
Average time to complete
the task among users:
The user is not sure about the functionality of the rate scale
purpose.
User took time to understand that in order to rate an article,
it should select the value and confirm with a button.
The article can be rated several times, the user it is not sure
if he can only rate one.
32
2.40 min The platform does not have a constraint to indicate if article
rate can be done only one time or several times.
Section to provide positive or negative commentaries is
tedious and difficult for the user to split his thoughts, he
would like to write ideally only one big commentary,
therefore it discourages him to provide comments.
Follow a tutorial for recommendations Average time to complete
the task among users:
2.59 min
It was very hard to find the tutorial, the button is confused as
a red title with no action.
The first description of the tutorial confuses the user when it
mention “columns”, the user does not understand to which
column the tutorial is referring.
The different colors in the tutorial section are helpful and
were a good insight for the users to realize that these were
the columns that the tutorial was talking about. Figure 26.- Medium Task Analysis
Hard Tasks with Time Analysis Results
Modify user profile
Average time to
complete the task
among users: 3.09 min
The slider does not show the values, so the user does not
know what is the value selected.
User gets confuse sometimes for the order or where to find the
personal details section.
It is not trivial to find all the options.
Once the values in the profile are changed, after the change,
the webpage redirect to another section, which causes a
consistency problem in the user.
Create a new article with description and images Average time to
complete the task
among users: 5.19 min
The user could not find how to add pictures.
Too many options with the article editor that make it confusing
The editor interface is not intuitive
In order to add the picture most of the users had to create an
article with only text and later re-edit it and attempt again to
add the pictures.
Figure 27.- Hard Tasks analysis
33
4.2.1. – Qualitative Analysis
The qualitative analysis was measured based upon two questions that were asked
before and after the experiment. (Figure 6 and Figure 7). The chart below (Figure 28)
shows the results of the user’s expertise in forum participation level and the
perceived comfort level of the experience of the platform by the end of the
experiment. As we can see, the expertise level of the participants that performed this
experiment was not high (an average of 1.5 out of 5), which shows they rarely use
forum participation platforms. The results also show that users expressed a low level
of comfort (an average of 2.375 out of 5) during experiment.
Figure 28.- Comfort and Experience Level
0
0.5
1
1.5
2
2.5
3
3.5
1 2 3 4 5 6 7 8
COM
FORT
LEV
EL A
ND
EXPE
RIEN
CE (1
-5)
PARTICIPANTS
User Experience Level and Comfort
Expertise Comfort
34
5. Evaluation Analysis and Recommendations Resume As it was shown in the previous section, the results of both evaluation methods show
that in general the main purpose of the system works as expected but there is a good
opportunity for improvements in the user interaction aspect of the platform. As it was
analyzed, users found difficult to navigate in all the menus of the system. In the next
subsection a series of main detected problems are shown and later a series of
recommendations are proposed in order to improve the human computer interaction
(HCI) of the SmartParticipation platform.
5.1 Main Problems Detected
The figure 29 provides a list of the main problems detected based on the heuristics
and usability methods. It is important to mention that the next problems detected are
presented in an abstract way based on the results of two evaluation methods in order
to present the information in a common terminology that the web designers and final
users understand. However in order to create a new version of the platform, it is
necessary that the web designer take in consideration the results of the sections 4.1
and 4.2.
Property Description of the problem by evaluation method
Visibility Heuristics: Cosmetic problems in the menus, contrast of colors, size of
fonts, submenu arrangement and information presentation
that cause navigation confusion for users.
Usability: The buttons and their arrangement causes confusion in the
article creation, revision, tutorial and profile modifications.
Feedback Heuristics: There is no feedback of the button search article in the search
section when it is empty and if we type any kind of input.
Usability: The user could not understand that the article search was not
returning a value because the user was typing incorrect
inputs.
Constraints Heuristics: There is no feedback of the button search article in the search
35
section when it is empty and if we type any kind of input.
Usability: The user could not understand that the article search was not
returning a value because the user was typing incorrect
inputs.
The user made several errors creating an account in the
password section, it was not specified the password syntax
requirements.
Modifying profile the slider does not show the values, so the
user does not know what is the value selected.
Mapping Heuristics: Although each button has a clear specification of the
functionality, sometimes their position creates confusion.
Usability: Understood the functionality of certain buttons such as login or
username or password but sometimes it got confused if he
has to push a button to update his profile for instance.
Consistency Heuristics: Although most of the buttons are clear in their function, there
are some that have some different color that could create
confusion with the user.
Usability: The user got confused with the colors and size of some
buttons, especially the tutorial button which was so big that
was confused with a title.
Affordance Heuristics: There is in principle no problem with the affordance of the site,
except for the tutorial part of the recommendation and the
main page that offers too much information in a unstructured
way.
Usability: Users experience problems for the tutorial section, it was not
intuitive as expected in some sections, such as the tutorial or
update the user’s profile
Idioms and user language
Heuristics: The main page, profile and tutorial speak sometimes a
technical language or provide some unnecessary information
36
to the user such as the technologies used to develop the web
service or the calculation of the recommender algorithm.
Usability: The user got confused by reading numerous times the
instructions of the tutorial and also to discover the main
purpose of the platform at the main page.
The statistics did not offered to the users a valued since they
do not have an explanation of the units or their purpose. Figure 29. - Main Problems Detected
5.2 Recommendations
The table in the figure 30 considers the problems detected during the evaluation and
provides a general skeleton recommendation for each section. It is important to
consider that a web developer who is willing to program these recommendations,
should follow the figure 30. Nevertheless, the details in the design improvements of
each section should be consulted in the results of the usability and heuristic
evaluation from sections 4.1 and 4.2.
Section Recommendation
Main Page Make it simpler and create an about page to add all the credits of
the authors and extra information
Participate or Article list
The presentation of the articles should be based on their image
preview and in principle no statistics preview but if the popularity
score must be displayed, then a small legend should explain the
units or the meaning of the number.
Article page In this part statistics should have a legend or a figure that the user
can map directly the meaning of the numbers and all the article
should be displayed in one page. In addition positive or negative
comments should be added at the end.
Login This should be instead of a new page, a discrete sub window in
the current page in order that the user can authenticate without
changing the current web page.
37
Forgot Password It should open a submenu or redirect to a new page that shows
several options to remember your password.
Register This should be a submenu or a new webpage where the user can
introduce the minimum amount of information, in order to avoid
that he will get desperate and lose the interest.
Search The option should be map with a universal symbol and display a
submenu that allows the user search by user or article.
User Profile This section is critical, processes as update or user settings
should be visible in one single page without menus in order to
avoid confusion and the personal information should be shown
first and later advanced options.
Feedback Page This option should be available with or without login in order to
encourage user’s participation and should be visible from the
main page.
New Article It is crucial to simplify the way that the photos and posts can be
created and most of the users prefer a simple editor, therefore
adding photos or text to the article should be easier without
embedded script editors.
Recommendations The tutorial button should be mapped in an icon and the
information should be presented in a graphical way, rather than
text. Figure 30.- Recommendations
Based on the evaluation and analysis results of the platform, it is possible to take a
further step and consider all the changes and generate a low level prototype that
could take into account the most important recommendations.
38
6. Low Level Prototype The aim of the low level prototype is to present a further idea, skeleton version of the
SmartParticipation layout or interface, this will help to solve the most critical usability
problems and provide a simple graphical idea to a web designer who would like to
implement a more functional and complete prototype.
6.1 SmartParticipation Web Pages
Based on the previous evaluations, the new prototype considers less sections and
present the information in a more compact way, with the aim that the user feels more
comfortable with the interface interaction.
6.1.1. - Main Page
The new redesigned main page has the aim to present the minimum of information
that the user cannot get confuse by a bomb of information as we can observe in the
figure 31.
Figure 31. - Main Page
39
In case that the user would like to know more about how SmartParticipation works, it
is necessary to click in the button “How it Works” which display a CSS script that
open a video mini window in the main page as we can observe in the figure 32.
Figure 32.- Main Page with Video
40
6.1.2. – About
In order to eliminate the confusion of the different information of the web main page,
it is important to add the objective, authors and technologies used to create the site
in a different web page with the less text possible and more graphics. A potential
proposal of these characteristics is presented in the figure 33.
Figure 33.- About Page
41
6.1.3. – Participate or Article list
The participate webpage allows to visualize the articles by their representative image,
with a small title at the bottom of each picture with a little description of the meaning
of the numbers that are located below each image. A button “create article” is also
available but it will require to sign in. (Figure 34)
Figure 34.- Participate Page
42
6.1.4. – Show Article
Simplicity should be a must, users during the usability test showed to spend more
time navigating through articles and see statistics. The new proposal design of this
section involves a design all in one with sequential order of the information. (Figure
35). When the user rate the article over 50 then the page suggest to write a positive
comment and when it does in the opposite, it suggest a negative comment. (Figure
36).
Figure 35.- Review Article
43
Figure 36.- Article with Negative Comment
44
6.1.5. – Login
In the previous version sign in was performed in an additional page, based on the
user’s behavior, most of them does not like to change web page while they want to
sing in, because they may be doing something already in the current page. That is
why a CSS in HTML5 can solve the problem with a micro window as it is presented in
figure 37.
Figure 37.- Sing In Window
The micro window also count with the options of “Forgot Password”, which in case to
be pressed a new micro window appears (Figure 38).
Figure 38.- Forgot Password Mini Window
45
6.1.6. – Register
Following the figure 36, if we are new users, we can Sign Up and then a new mini
window appears to start the registration process. (Figure 39)
Figure 39.- Sing Up
6.1.7. – Search
The search mini window (Figure 40) allows the user to select the type of search and
keep the user in the same web page.
Figure 40.- Search Mini Window
The user is redirected to a new search page in case that he clicks on the button
“Search” and have a display menu of information.(Figure 41)
46
Figure 41.- Search Results
6.1.8. – Feedback Page
The feedback can be accesses with or without login and it is also a mini window
(Figure 42), this window disappears automatically after the “submit” button is pushed.
Figure 42.- Feedback Window
47
6.1.9. – User Profile
This template for user profile solves the problem of having two different pages
describing the user, here the user can access first to their personal data and later to
its recommender options. The button “Update” is always visible in case that the user
would like to make changes. (Figure 43)
Figure 43.- User Profile
48
6.1.10. – New Article
The new version or the article creation is to make easier to the user to update the
information quickly without an embedded text editor that confuses the users in order
to add or remove media elements.(Figure 44)
Figure 44.- New Article Creation
6.1.11. – Recommendations
The new recommendations page considers to organize the information in main
categories:
Popular Articles
Recommended Articles
Recommended Users
Finally, if the user gets lost, it can click on the “Icon question mark” in order to start a
tutorial that can explain the meaning of each of the sections.(Figure 45)
49
Figure 45.- Recommendations Page
Before finishing this section, it is important to mention that the low prototype
presented in this seminar is a general guideline skeleton. The web designer who
would like to improve this prototype can move forward from this template model and
add more details or consider more characteristics such as text field verifications or
simple notification errors from the usability and heuristic evaluations.
50
7. Conclusions
After an extensive analysis and evaluation of the SmartParticipation platform, it was
possible to identify different opportunity design elements in the platform that needed
to be changed. The heuristics and usability evaluation provided important points of
improvements in order to inspire a low level prototype redesign.
Although the heuristic evaluation provides important feedback, it is always crucial to
consider also evaluations that consider users. Final user’s utilization and feedback
are vital to improve a system and it remains as a main advice for future evaluations of
the SmartParticipation platform.
The users who participating during this study were convinced that the idea of having
a platform of discussing topics and articles, represents an interesting service for
them, nevertheless the presentation of the information and the facility navigate in the
platform play an important role to keep users interested and willing to participate.
Finally, the user centered design cycle can always be used to find potential
improvements in current web platforms and avoid that they become obsolete, during
this seminar the challenge was to apply new concepts from User Centered Design
into a system, find the correct users and finally translate the improvements creating a
low level prototype that can provide examples of a desired new version of the
SmartParticipation Platform.
51
8. Future Work
After the creation of the second prototype based on the results of the evaluations,
there are some further steps that are necessary to bring the second low level
prototype to production.
Development of a SmartParticipation App for iOS or Android
Second heuristic evaluation considering more evaluators, rather than one.
Creation of a high or medium level prototype of the proposed new system, which
implies add all the functionalities of the original version.
Make a new usability evaluation based on the second high level prototype version
of SmartParticipation and eventually compare the same tasks between the
original platform and new prototype in order to see the differences y make
quantitative analysis.
Make usability evaluation considering behavior between male and female users or
by different age segments.
Integrate social networks and security guidelines to keep the integrity of the site.
Such as verification of data input in the text fields and SSL encryption for the data
transmission between the local computer and the server.
52
9. References
[1]. - Teran, Luís. Smart Participation: A Fuzzy-Based Recommender System for
Political Community-Building. Ph.D. Thesis of the University of Fribourg, Department
of Informatics, 2014.
[2].- Scapin, DL, Bastien JMC, Ergonomic Criteria for evaluating the ergonomic
quality of interactive systems, Behaviour & Information technology, 16, 1997, pp.220
– 231
[3].- Bastien, JMC, Scapin, DL, Leulier C., The ergonomic criteria and the ISO 9241-
10 Dialogue Principles: A comparison in an evaluation task, Interacting with
computers, 11(3), 1999, pp.299 – 322.
[4].- Human Computer Interaction 3nd ed – Dix, A., Finlay, J., Abowd, G.D., and
Beale. R. Prentice Hall, 2004.
[5].-Interaction Design, 2nd ed – Sharp, H., Preece, J and Y. Rogers. John Wiley and
Sons Ltd. West Sussex, UK. 2007.
[6].-Handbook of Human Computer Interaction – Jacko, J.A. and Sears, A. Lawrence
Erlbaum Associates, New Jersey, USA. 2003.