modeling visual search time and its application to design virtual...
TRANSCRIPT
MODELING VISUAL SEARCH TIME AND ITSAPPLICATION TO DESIGN VIRTUAL KEYBOARDS
Pradipta Kumar Saha
MODELING VISUAL SEARCH TIME AND ITSAPPLICATION TO DESIGN VIRTUAL KEYBOARDS
Thesis submitted to theIndian Institute of Technology Kharagpur
for award of the degree
of
Master of Science (by Research)
by
Pradipta Kumar Saha
Under the guidance of
Dr. Debasis Samanta
School of Information TechnologyIndian Institute of Technology Kharagpur
Kharagpur - 721 302, IndiaJuly 2013
c⃝2013 Pradipta Kumar Saha. All rights reserved.
CERTIFICATE
This is to certify that the thesis entitled Modeling Visual Search Time and itsApplication to Design Virtual Keyboards, submitted by Pradipta Kumar Sahato Indian Institute of Technology Kharagpur, is a record of bona fide research work undermy supervision and I consider it worthy of consideration for the award of the degree ofMaster of Science (by Research) of the Institute.
Date: 26/04/2013
Dr. Debasis SamantaAssociate ProfessorSchool of Information TechnologyIndian Institute of Technology KharagpurKharagpur - 721 302, India
DECLARATION
I certify that
a. The work contained in the thesis is original and has been done by myself underthe general supervision of my supervisor.
b. The work has not been submitted to any other Institute for any degree or diploma.
c. I have followed the guidelines provided by the Institute in writing the thesis.
d. I have conformed to the norms and guidelines given in the Ethical Code of Conductof the Institute.
e. Whenever I have used materials (data, theoretical analysis, and text) from othersources, I have given due credit to them by citing them in the text of the thesisand giving their details in the references.
f. Whenever I have quoted written materials from other sources, I have put themunder quotation marks and given due credit to the sources by citing them andgiving required details in the references.
Pradipta Kumar Saha
Dedicated to My Parents
ACKNOWLEDGMENT
During this period of my postgraduate study there are many people whose guidance,support, encouragement and sacrifice has made me indebted for my whole life. I takethis opportunity to express my sincere thanks and gratitude to all these people.
First and foremost I would like to express my deepest gratitude to my revered supervi-sor Dr. Debasis Samanta for his invaluable guidance, and his encouragement throughoutmy work. His guidance and support is far beyond duty. His constant motivation, sup-port and infectious enthusiasm have guided me towards the successful completion of mywork. My interactions with him have been of immense help in defining my research goalsand in identifying ways to achieve them. His encouraging words have often pushed meto put in my best possible efforts. Above all, the complete belief that he has entrustedupon me and has instilled a great sense of confidence and purpose in my mind, which Iam sure, will stand me in good stead throughout my career.
It gives me immense pleasure to thank the heads of the department Prof. J. Mukhopad-hyay and Prof. R. Mall for the world class infrastructure provided in the department tothe students. My sincere thanks to all of my departmental academic committee membersProf. A. Gupta, Prof. C. R. Mandal, Prof. S. Sural, Dr. S. K. Ghosh, Dr. K. S. Rao,Dr. S. Misra for their valuable suggestions during my research. I sincerely rememberthe support of office staffs Mithun Da, Soma Di, Malay Da, Vinod Da, Pratap Da andothers. I am also grateful to all members of School of Information Technology.
There is no way this thesis could ever have been completed without the inspiration,encouragement, criticism and patience of Sayan Da and Jayeeta.
I wish to convey my special thanks to my old friends Chandan, Bodhi and Aparnafor their constant support and help during the various stages of my work. I am greatlyindebted to many of my friends for their constant inspiration. The support of my labmates namely Sushantada, Debasish, Rajkumar, Col. Ranjit Singh, Prasenjit, Shobhanamadam, Ashalata madam, Shankar, Prasenjit, Barsha, Narendra, Jaswasi, Jainath, Gau-rang, Anant, Soumitri is invaluable. I would also like to express my thanks to Soumyajit,Barikda, Sajalda, Ranjan, Pushpitadi, Aditi, Soumya, Arindamda, Nirnay, Sudhamay,Gautamda, Subarao, Kanchan, Partha, Saptarshi, Subhomoy, Soumyadip and manymore. It is a great fun and source of ideas and energy to have friends like Sayan,Manoj, Pradipta, Soumalya, Santa, Jayeeta, Indira and many more during my stay at
vii
IIT Kharagpur. I also thanks all the family members of contemporary fellow neigh-bor, in particular Amit, Ashokeda, Kakali boudi and Anki for making the stay at IITKharagpur, ever memorable.
Nothing would have been possible without the moral support of my parents whohave been the pillars of strength in all my endeavors. I am always deeply indebtedto them for all that they have given me. I also thank all the other members of myfamily including my brother and sisters for their love, affection and timely help. Finally,to thank my wife Sayani, I really have no words to express my gratitude for all hersupport, encouragement, understanding and sacrifice, without which it would have beenimpossible for me to finish this work.
viii
Abstract
Performing user based evaluation of any user interface is practically an expensive andtedious job. To alleviate this issue, researchers advocate automatic design and evalua-tion procedure which includes modeling of different activities with respect to differentinterface design parameters. Visual search is one of such important activity. Visualsearch is a significant cognitive subtask of a sighted user in graphical user interfacebased interaction. Indeed, time required to perform visual search task is treated as animportant parameter while evaluating usability and user friendliness of an interface. Tothe best of our knowledge, work on modeling of visual search time is not reported. Wemay note that, visual search time depends on many visual search features such as size,shape, position etc. of the objects space of an interface. This thesis aims to address theabove mentioned research concern. More precisely, the main objective of our researchis to identify different visual search parameters and then to develop a computationalmodel to predict the visual search time for a given interface. So far the user interfaceis concerned, we limit our investigation to virtual keyboard, which is a graphical userinterface to compose texts. In the following, we summarize the major works carried outin this thesis.
Initially, we have studied visual features reported in different literatures which in-fluence visual search task. As impact of many of these features is subjective to userand beyond the scope of measuring them quantitatively, we have identified the visualfeatures which do not vary from user to user and also relevant in the context of a vir-tual keyboard interface. Next, we have conducted an empirical user evaluation on eachfeature to substantiate their degree of influences and come up with the features whichsignificantly influence the visual search time. Then, we have performed several user-based evaluations to accumulate knowledge about user performances with variation ofidentified features. To accomplish the modeling task, we have followed three modelingapproaches namely linear regression, non-linear regression and Support Vector Machine-based regression. Finally, the proposed model along with Fitts’-digraph model have beenused as two objectives to design an optimized virtual keyboard with respect to lesservisual search time and mouse movement time.
From our investigation, we observe that there are four visual search features in vir-tual keyboard interface which significantly influence the visual search time of a user.The computational model developed using SVM-based regression method outperformsother methods. It is also observed that considering visual search time into account, wecan develop a virtual keyboard with 12.37% improvement in text entry rate.
Keywords: visual search time, virtual keyboard, interface design, computational mod-eling, automatic design, design evaluation, human computer interaction
ix
Contents
Certificate i
Declaration iii
Dedication v
Acknowledgment vii
Abstract ix
Contents xi
List of Figures xv
List of Tables xvii
List of Symbols and Abbreviations xix
1 Introduction 11.1 Visual Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Visual Search Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Scope and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Related Work 92.1 Visual Search Features Identification . . . . . . . . . . . . . . . . . . . . . 102.2 Modeling Techniques for Prediction . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Modeling of Visual Search Activity . . . . . . . . . . . . . . . . . . 132.2.2 Support Vector Regression-based Modeling . . . . . . . . . . . . . 15
2.3 Model-based Approaches to Design Virtual Keyboard . . . . . . . . . . . 17
xi
Contents
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 Identification and Analysis of Visual Search Features 253.1 Listing of Visual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2.2 Interface Used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.4 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4 Modeling of Visual Search Time 494.1 Training Data Set Generation . . . . . . . . . . . . . . . . . . . . . . . . . 504.2 Model to Predict Visual Search Time . . . . . . . . . . . . . . . . . . . . . 51
4.2.1 Linear Regression with Multiple Features . . . . . . . . . . . . . . 524.2.2 Non-linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . 544.2.3 Support Vector Regression . . . . . . . . . . . . . . . . . . . . . . 554.2.4 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.3 Validation of the Proposed Model . . . . . . . . . . . . . . . . . . . . . . . 614.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5 Virtual Keyboard Layout Optimization 655.1 Proposed Virtual Keyboard Design Approach . . . . . . . . . . . . . . . . 66
5.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.1.2 Virtual Keyboard Design using Multiobjective Optimization . . . . 68
5.2 Movement Time Optimized Virtual Keyboard Design . . . . . . . . . . . . 725.3 Empirical Study to Evaluate Designs . . . . . . . . . . . . . . . . . . . . . 765.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6 Summary and Conclusion 796.1 Contribution of Our Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.2 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.3 Future Scope of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Publications 85
xii
Contents
References 87
xiii
List of Figures
2.1 Some well known virtual keyboard layouts . . . . . . . . . . . . . . . . . . 21(a) Opti keyboard layout [66] . . . . . . . . . . . . . . . . . . . . . . . . 21(b) Metropolis keyboard layout [106] . . . . . . . . . . . . . . . . . . . . 21(c) GAG I keyboard layout [79] . . . . . . . . . . . . . . . . . . . . . . 21(d) GAG II keyboard layout [79] . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Quasi-Qwerty keyboard layout [7] . . . . . . . . . . . . . . . . . . . . . . . 232.3 Sath keyboard layouts [20] . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
(a) Sath-Rectangular keyboard layout . . . . . . . . . . . . . . . . . . . 23(b) Sath-Trapezoidal keyboard layout . . . . . . . . . . . . . . . . . . . . 23
3.1 Virtual keyboard layouts used in experiments . . . . . . . . . . . . . . . . 31(a) Opti keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 31(b) Avro keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 31(c) iLipi-H keyboard layout with inflexion window . . . . . . . . . . . . 31
3.2 A sample layout with target object . . . . . . . . . . . . . . . . . . . . . . 343.3 Effects of different font sizes on visual search time . . . . . . . . . . . . . 36
(a) Avro keyboard with 12pt font size . . . . . . . . . . . . . . . . . . . 36(b) Visual search time for different font size . . . . . . . . . . . . . . . . 36
3.4 Effects of similar shaped characters on visual search time . . . . . . . . . 383.5 Effects of spacing between keys on visual search time . . . . . . . . . . . . 39
(a) Opti keyboard with 50% spacing between keys . . . . . . . . . . . . 39(b) Visual search time for different spacing between keys . . . . . . . . . 39
3.6 Effects of varying number of items on visual search time . . . . . . . . . . 40(a) iLiPi-H keyboard having 67 items . . . . . . . . . . . . . . . . . . . 40(b) Visual search time for different number of items . . . . . . . . . . . 40
3.7 Effects of search field size on visual search time . . . . . . . . . . . . . . . 41(a) Avro keyboard occupying 20% of screen area . . . . . . . . . . . . . 41
xv
List of Figures
(b) Visual search time for different search field size . . . . . . . . . . . . 413.8 Effects of different position of keys on visual search time . . . . . . . . . . 43
(a) iLiPi-H keyboard divided into 9 blocks . . . . . . . . . . . . . . . . 43(b) Result for different position of keys . . . . . . . . . . . . . . . . . . . 43
3.9 Effects of ordering of characters on visual search time . . . . . . . . . . . 443.10 Opti keyboard with two group . . . . . . . . . . . . . . . . . . . . . . . . 443.11 Effects of grouping and group size on visual search time . . . . . . . . . . 45
(a) Visual search time for different number of grouping . . . . . . . . . 45(b) Visual search time for varying group size . . . . . . . . . . . . . . . 45
4.1 Virtual keyboard layouts used for validation . . . . . . . . . . . . . . . . . 61(a) Fitaly keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . . 61(b) Guruji keyboard layout . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.1 Flowchart of virtual keyboard design using NSGA-II algorithm . . . . . . 705.2 Mouse movement and visual search time optimized keyboard layout . . . 725.3 Flowchart of virtual keyboard design using Genetic Algorithm . . . . . . . 745.4 Crossover operation in Genetic Algorithm . . . . . . . . . . . . . . . . . . 755.5 Keyboard layout minimizing mouse movement time . . . . . . . . . . . . . 75
xvi
List of Tables
3.1 Description of participants . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2 Similar character set of Bengali language . . . . . . . . . . . . . . . . . . . 373.3 Summary of statistical analysis for different features . . . . . . . . . . . . 47
4.1 Description of participants . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2 Summary of Collected Data . . . . . . . . . . . . . . . . . . . . . . . . . . 514.3 Different non-linear model with corresponding R2 . . . . . . . . . . . . . . 544.4 Different approaches with corresponding R2 . . . . . . . . . . . . . . . . . 604.5 Description of participants considered for validation . . . . . . . . . . . . 62
xvii
List of Symbols and Abbreviations
List of Symbols
p Pearson Correlation Coefficient
µ Mean
σ Standard Deviation
hθ(x) Hypothesis or Function
α Learning Rate
x Independent Variable
y Dependent Variable
x(i)j jth feature of ith training set
R2 Coefficient of Determination
D Training Dataset
ξ Slack Variable
ε Estimation Precision
αi Lagrange Multiplier
K Kernel
List of Abbreviations
AAM Area Activation Model
ANOVA Analysis of Variance
CPS Character Per Second
DF Degree of Freedom
EPIC Executive Process-Interactive Control
FD Fitts-Digraph
xix
List of Symbols and Abbreviations
FIT Feature Integration Theory
GA Genetic Algorithm
GOMS Goals, Operators, Methods and Selection
GS Guided Search
HCI Human Computer Interaction
KLM Keystroke Level Model
KS Key Size
MOGA Multi-objective Genetic Algorithm
MSE Mean Square Error
MT Movement Time
NC Number of Characters
NPGA Niched Pareto Genetic Algorithm
NSGA Non-dominated Sorting Genetic Algorithm
NSGA-II Non-dominated Sorting Genetic Algorithm - II
PAES Pareto-Archived Evolution Strategy
PC Position of Character
RBF Radial Basis Function
RT Reaction Time
SD Standard Deviation
SK Space between Keys
SPEA Strength Pareto Evolutionary Algorithm
SVM Support Vector Machine
SVR Support Vector Regression
TBS Target-Background Similarity
UCIE Understanding Cognitive Information Engineering
UI User Interface
VEGA Vector Evaluated Genetic Algorithm
VST Visual Search Time
WBGA Weight-based Genetic Algorithm
WPM Words Per Minute
xx
Chapter 1
Introduction
Of present day technologies, computer application has been spread in every sphere of life
and it has become a challenge to the user interface designer to design an interface which
has appeal to all category of users. As a consequence, interaction styles have changed
from command mode to direct manipulation mode [25]. With direct manipulation style of
interaction, a user can perform any task much more comfortably as it offers clear visibility
of intermediate and final results. While accessing the interface, the main concern of user
turns to find proper objects visually which are responsible for performing a task. As a
consequence, for a sighted user, visual search contemplates to be a significant cognitive
subtask which can be involved with performing any task in current user interfaces.
The objective of this thesis is to design a methodology to evaluate visual search tasks
automatically. More specifically, we plan to develop a computational model to predict
the average visual search time given an object space. In the domain of user interface
design, many object spaces are possible and each object space demands its own treatment
so far the perception task is concerned. Considering this, we limit the investigation only
to virtual keyboard [66], which is a graphical user interface to compose texts. The rest
of the chapter is organized as follows. Section 1.1 provides an overview of visual search
task. Details about visual search time and different features which influence the same
1
1. Introduction
are discussed in Section 1.2. The scope and objective of our work are presented in
Section 1.3. Finally, Section 1.4 presents the organization of this thesis.
1.1 Visual Search
The visual search processes that people use in Human Computer Interaction (HCI)
tasks have a substantial effect on the time and likelihood of finding the information
they required. Users encounter many challenges in finding the information they seek
when visually searching. Visual search is an intriguing human activity to study because
it requires complex interplay among three processes namely perceptual, cognitive and
motor [13]. Most important factor for visual search is eye movement. Information that
is perceived through eyes will vary as a function of the orientation of eyes. Visual acuity
is higher at the center of people’s field of vision known as fovea [4]. Cognitive processes
allow one to analyze the information perceived through the eyes. Motor process results in
actions generated by the cognitive process, that is, taking decisions once the information
is processed.
Visually searching for information is often fastest and most useful way of finding
information in a variety of user interfaces. Functionality such as web search or the Find
command present in many operating systems can be used to find items on a computer
screen quickly. However, there are many instances in which visual search is more useful.
Few of these scenarios are listed in the following.
• Searching among many similar results, where it is difficult to specify a search query
to locate the desired target (e.g. examining web search engine results).
• When an application does not include a ‘Find’ command (e.g. in video games).
• When the exact target is not known by the user (e.g. looking for items that match
some vague concept or goal).
2
1.2. Visual Search Time
In these cases, fast eye movements are necessary where many visual objects can be
evaluated by the user simultaneously, and target(s) can be located if exists. The visual
search process that people use has a substantial effect on the time and likelihood of
finding the information they seek. In fact, users encounter many challenges in finding
the information they seek when visually searching. In other words, visual search is
affected by many factors of the layout such as grouping, color, spacing, text size etc.
and also the strategies used like item-by-item, using labels or not, following the columns
or not etc. It has been observed from experiments that intensity of the object feature
often takes the decision of selecting target object from the distractor in the object space.
The more similar objects in the visual search space can be distinguished by more number
of features [93]. Human may first scan the object space, collect feature information from
object(s) in a parallel process then search for the required object with the special unique
feature in a serial process [98].
1.2 Visual Search Time
Visual search is at the heart of visual information searching. It is a subtask in each and
every visualization based interaction with computers. So the time required to perform a
visual search task is treated as an important parameter while evaluating usability, user
friendliness of an interface or any object space. Visual search time of a given object
space can be measured by either user-based evaluation or model-based evaluation.
The user-based evaluation requires to perform some tasks by users with different
perceptual and cognitive capabilities on an interface or an object space. Depending
on the users’ feedback, the average visual search time can be estimated and similarly
usability of the interface can be determined. This is a time consuming, tedious job and a
sufficient number of users are required for every interface to estimate visual search time
accurately. On the other hand, the model-based evaluation simulates users’ behavior
while using an interface. A mathematical model can be designed for predicting the
3
1. Introduction
performance of user interface in terms of visual search time, depending upon several
parameters of the user interface. Once a model is developed for a standard interface, it
can be used for measuring usability of the similar interfaces. In user-based evaluation,
users who are not familiar with the task may not get interest in accessing the interface
which causes poor results. In contrast, a computational model uses different features
to estimate average visual search time and thus provides both time and cost effective
solution as well as requires less users’ effort.
So, the model-based evaluation or the computational model can be considered as
User Interface (UI) prototyping tool that can produce quantitative predictions of how
users will behave, when the prototype is ultimately implemented. Thus, it provides a
rapid and inexpensive way to explore a large variety of UI ideas, compare them and
narrow down the options to a handful of designs to be empirically tested with users.
One can rapidly analyze competitor’s products as part of a competitive analysis and
compare new ideas with an existing version of the system to ensure that the new design
is better than the old one.
The most important contribution of computational cognitive models in the field of
HCI is that the models provide the science base that is needed for interface analysis
tools. Projects such as CogTool [46,90] and CORE/X-PERT [91] are at the forefront of
tools that utilize cognitive modeling to predict user interaction based on a description of
the interface and task. These tools provide theoretically-grounded predictions of human
performance in a range of tasks without requiring that the analyst, that is, the person
using cognitive models, be knowledgeable in cognitive, perceptual, and motored theories
embedded in the tool. Designers of application interfaces could use such tools to evaluate
their visual layouts, reducing the need for more expensive human user testing early in
the development cycle. Potential usability problems in interfaces could be identified
early, before time consuming user testing.
4
1.3. Scope and Objectives
A visual search time prediction model useful for specific application has been
proposed previously [37, 42]. But it is fixed for an object space and varies with only
the number of objects in the object space. The method has not considered any other
features of the objects in object space. So, the challenge still exists for addressing the
issues as well as developing a predictive model for calculating visual search time for
different interfaces.
1.3 Scope and Objectives
Estimating visual search time helps interface designers to evaluate their design with
minimum effort. But most of the work reported till date is mainly about describing
the visual search methodologies. This work proposes to address this limitation. We
propose to develop a computational model to predict the average visual search time
given an object space. In the domain of user interface design, many object spaces are
possible and each of them demands their own treatment so far the perception task is
concerned. Considering this, we limit the investigation only to virtual keyboard, which is
a graphical user interface to compose texts. Nowadays, modeling different tasks involved
in the context of text composition like messaging, chatting etc. draws more attention
with increasing use of PDAs, mobiles etc. Virtual keyboard contains different visual
features which affect text entry rate. Earlier, a model to predict visual search time has
been proposed by Hick and Hyman [37, 42]. This model only considers the number of
keys present in the keyboard and lacks in acquiring other visual search features like
shape, size, grouping, ordering etc. which also influence visual search time in finding a
key in the keyboard. With this perspective, in this dissertation, the plan of our work is
as follows.
• Identification and analysis of visual search features: All features of an
interface do not contribute equally for predicting the visual search time for it.
Some features vary with user, while some features are not applicable for a specific
5
1. Introduction
interface. Hence it is required to find out the set of features applicable for virtual
keyboard.
• Proposing a model to predict visual search time: After identifying the visual
search features, a computational model needs to be developed to predict visual
search time. This type of modeling requires knowledge about user performance on
various combination of identified feature values. So, several user-based experiments
are required to collect the knowledge to develop prediction model. The accuracy
of the model can be verified through users’ experiment.
• Application of the proposed model to design virtual keyboards: The
developed model then can be used to design an optimum virtual keyboard layout.
The layout can be optimized in terms of minimal mouse movement time as well as
minimal visual search time.
1.4 Organization of the Thesis
The thesis contains six chapters including this introductory chapter. This chapter
discusses about the motivation of the thesis work and main objectives of the work.
It gives a brief description of visual search methodologies and their types, importance
of visual search in interface design, different cognitive modeling and visual search time
modeling.
Chapter 2: Related Work
This chapter includes the state of the art for visual search task and provides an overview
of computational modeling. Also, we survey the existing model-based approaches of
virtual keyboard design.
Chapter 3: Identification and Analysis of Visual Search Features
In this chapter, we describe our approach to identify significant visual search features in
6
1.4. Organization of the Thesis
the context of virtual keyboard interface. This chapter includes user study on variation
of different visual search features and identification of significant features related to
virtual keyboard design parameters.
Chapter 4: Modeling of Visual Search Time
The proposed computational model to predict visual search time of virtual keyboard
interface, based on the identified features, is discussed in this chapter. This chapter also
includes details of data collection of user performance for different variation of multiple
visual search features, which is required to develop the model, performance of different
modeling approaches on the collected data and validation of the proposed model.
Chapter 5: Virtual Keyboard Layout Optimization
In this chapter, we describe our approach to design virtual keyboard layout based on
two objectives: minimization of mouse movement time and minimization of visual search
time. To judge the effectiveness of our optimized design, we design another virtual
keyboard with respect to minimum mouse movement only. The evaluation of these
designs through different user experiments are also discussed in this chapter.
Chapter 6: Summary and Conclusion
In this chapter, we summarize our research findings and major contributions of the
work. This thesis considers some assumptions and we discuss the threats to validity of
our claims subjective to these assumptions. Finally, we highlight extension of our work
and scope of future work.
7
Chapter 2
Related Work
In this chapter, a survey of literature related to the contributions made in this
dissertation is reported. This dissertation aims the modeling of visual search time and
application of the model in virtual keyboard design. Visual search is a perceptual task
of locating intended object visually from the object space. Thus, the search performance
as well as the search time required for the task depend on various features or factors
of the object space. So, first we discuss about various work on visual search feature
identification. Then, we include a survey on different modeling approaches. The state of
art scenarios in application of different mathematical models towards virtual keyboard
layout design are also illustrated in this chapter.
The rest of the chapter is organized as follows. Section 2.1 presents a survey of
state of art approaches in identification of different visual search influencing parameters.
Existing predictive models are discussed in Section 2.2. Section 2.3 describes several
work on model-based approaches to design virtual keyboard. Finally, the chapter is
summarized in Section 2.4.
9
2. Related Work
2.1 Visual Search Features Identification
Visual search task is known to be governed by the various features present in the search
space. Thus, identification of these features is required prior to modeling of the search
task or time. An outline of related literatures in the field of visual search task influencing
feature identification is reported in this section.
Feature Integration Theory (FIT) [93] states that people shift attention serially from
one object to the next, deciding for each whether it is the target or not. According to FIT,
attention must be directed serially to each stimulus in a display whenever conjunctions
of more than one separable feature are needed to distinguish the possible objects
presented. It has been assumed that the visual scene is initially coded along a number of
separable dimensions such as, color, orientation, spatial frequency, brightness, direction
of movement and features are registered early, automatically, and in parallel across the
visual field, while objects are identified separately and only at a later stage, which
requires focused attention. This process is said to be necessary when conjunctions of
object features (color, shape, size, orientation etc.) differentiate targets from distractors,
for example, searching for a red X among green X’s and red O’s (conjunction search).
According to FIT, attention is necessary for the correct perception of conjunctions,
although unattended features are also perceived next to conscious perception as without
any attention. However, these results could also be the result of inefficient parallel search
processes. This type of theories are supported by a variety of evidence presented a simple
parallel model that reproduced the finding of feature search times independent of number
of objects in the search display and conjunction searches times linearly dependent on
number of objects.
Guided Search (GS) [95,97,98] is a computational model of how visual features, such
as color and orientation, direct visual attention. Guided Search predicts that the order
in which objects are visually searched is affected by four scenarios, the “strength” of
objects’ visual features (e.g. their blueness, yellowness, steepness, and shallowness), the
10
2.1. Visual Search Features Identification
differences between objects, the spatial distance between objects, the similarity to the
target, and the distance of objects from the center of gaze (i.e. the eccentricity).
Active vision is the notion or collection of theory that asserts eye movements are
central to visual processes, including visual search [24]. It asserts that understanding
where and when the eyes move, and how information gathered during eye movements
is utilized, are critical for understanding vision and in particular visual search. The
reported literature reveals that no model of visual search provides answers to all of the
questions of active vision. However, every question of active vision is addressed by at
least one model.
According to Nagy et al. [71], several experiments are designed to investigate the
effects of target and distractor heterogeneity on the threshold for detection of a color
target in a search task. In their first two experiments, stimuli are chosen so that the
target and distractor stimuli varied along one Cardinal axis in color space, while the
target differed from distractors along another Cardinal axis. Varying stimuli along a
Cardinal axis other than the Cardinal axis that differentiates target and distractors can
impair performance in visual search tasks. Further experiments show that the presence
of heterogeneous distractors has little or no effect on thresholds when location or color
cues indicated that these stimuli are irrelevant to the task. Results suggest that the
effect of heterogeneity in these experiments is attentional in nature rather than sensory.
Neidar and Zeilensky [73] judge the effects of target-background similarity on visual
search. In their experiments, users search for targets among distractors under varying set
size and target-background similarity (TBS) conditions. Manual errors and reaction time
increased with TBS, although search slopes did not significantly differ. Eye movement
analysis reveal that the majority of fixations fell on discrete distractors rather than on
the target-similar background, even under high TBS conditions. These data suggest a
biased search process, salient patterns segmented from a background are preferred while
more target-similar unsegmented regions of the background are relatively neglected.
11
2. Related Work
Research suggests that neither visual nor verbal working memory have a major
impact on the fundamental visual search processes. Visual and verbal working memories
are limited capacity, temporary stores for visual and verbal information. These two
memories show little overlap in functionality [60]. Research has shown that when verbal
or visual working memory is occupied, visual search remains efficient [99]. When people
visually search for a shape while performing a task that is presumed to occupy visual
working memory, the rate at which people visually searched was unaffected [99]. A
similar result is found when visually searching for a word while verbal working memory
is filled [58, 59]. These findings do not mean that working memory does not affect
visual search tasks at all. In general, for each modality, people can recall four things
on average [62]. If the visual search task requires storing multiple items in memory
before search is terminated, limitations on working memory could require that the user
terminate search early or forget items for later use. However, the use of spatial memory,
especially for where one has previously fixated, does appear to affect visual search.
Research has shown that when spatial memory is occupied, visual search efficiency is
reduced [74]. A memory for previously fixated locations is also suggested by other
research. A study of the visual search in “Where’s Waldo?” scenes, in which a cartoon
figure is hidden within complex scenes, found that saccades tend to be directed away
from the locations of previous fixations [50].
2.2 Modeling Techniques for Prediction
One objective of our work is to develop a computational model to predict visual search
time. To the best of our literature survey, there is no prior work dealing with this
problem. However, there are a large volume of work to develop model related to visual
activities of human which are may not be so relevant to our work but have some
coherence. We also report few techniques proposed to develop predictive models in
different application domains.
12
2.2. Modeling Techniques for Prediction
2.2.1 Modeling of Visual Search Activity
When people use visual search, their eyes are moved and independent shifts of attention
are not used [23,24]. Since different information is available depending on the orientation
of the eyes [6, 24], the movements of the eyes as well as head and body movements
are important for models of visual search behavior. This is especially true due to the
increasing size of computer displays and the increasing ubiquity of computing interfaces.
A variety of models have been developed to predict visual search behavior. Some models
have been developed specifically to predict the performance in a narrow domain, such
as graph perception. Others have been developed to predict the effects of specific visual
features in a broad range of visual search tasks.
The Area Activation Model (AAM) [77] is also a computational model of how visual
features control visual attention. The characteristics of AAM and GS are common in
many ways, but differ in at least one important way. The AAM assumes that all objects
near the center of gaze are searched in parallel and GS assumes that objects are searched
serially.
Barbur, Forsyth, and Wooding [4] propose a computational model to predict eye
movements in visual search. The model uses a hierarchical set of rules to predict where
people’s gaze will be deployed. Like the AAM, Barbur, et al.’s model assumes that all
objects near the center of gaze are searched in parallel. It differs from the GS and AAM
in that eccentricity is the only visual feature that determines where the gaze moves next.
Understanding Cognitive Information Engineering (UCIE) is a computer model of
human reasoning about graphs and tables [61]. UCIE is based on GOMS (Goals,
Operators, Methods and Selection Rules [45], an engineering model for predicting task
execution time. UCIE extends the applicability of GOMS for visually searching task
with a model. The time to perceive objects, eye movements, and a limited memory for
information provide constraints for the simulation of how people scan graphs and tables
to answer questions about the graph or table.
13
2. Related Work
Executive Process-Interactive Control (EPIC) is a framework for building
computational models of tasks that lends itself well to building models of visual
search [49]. EPIC provides a set of perceptual, motor, and cognitive constraints based
on a variety of psychological literature. Models of visual search built within EPIC tend
to explain visual search as the product of cognitive strategies, perceptual constraints,
and motor constraints.
An eye tracking study, conducted to examine the search strategies of users, and
a revised model-based on the results of the eye tracking study have been done by
FleetWood and Bryne [27]. The revised model incorporates EMMA [80] and changes
in search strategy. Findings indicate key environmental influences of icon search,
particularly set size and icon quality, evaluate the vision module in the underlying
cognitive architecture, and provide some illumination on the strategies of users.
Computational cognitive models are computer programs that simulate aspects of
people’s perceptual, motor, and cognitive processes. Cognitive modeling is used in
two ways; in a post hoc fashion to help explain the behavior of people performing
a task and in a priori fashion to predict how people will perform a task. The
most important contribution of computational cognitive models to the field of Human
Computer Interaction (HCI) is that the modeling provides a science base that is
badly needed for predictive interface-analysis tools. Projects such as CogTool [46] and
CORE/X-PRT [91] are at the forefront of the development of tools that utilize cognitive
modeling to predict user interaction based on a description of the interface and task.
These tools provide theoretically grounded predictions of human performance for a range
of tasks without requiring that the analyst be knowledgeable in the cognitive, perceptual,
and motor theories embedded in the tool.
Designers of device and application interfaces may be able to utilize such tools to
evaluate their visual layouts, identify potential usability problems early in the design
cycle, and reduce the need for more human user testing early in the development cycle.
14
2.2. Modeling Techniques for Prediction
Predicting people’s visual interaction is one aspect of user behavior that research with
interface analysis tools is trying to improve. To this end, a recent version of CogTool [90]
incorporates modeling work based on an early summary of the work [35]. However,
interface analysis tools such as CogTool and CORE/X-PRT do not yet fully account
for human vision, as in where the eyes move and what they do and do not see. A
partial account of visual information processing is provided by EMMA [80], which is an
extension to the ACT-R [3] modeling framework underlying CogTool. EMMA provides a
simulation of the eyes including where the eyes move and how quickly visual information
is processed. But this falls short of a complete account of active vision; automated
interface analysis tools do not yet simulate active vision.
Building cognitive models to explain users’ behavior in a post hoc fashion has a rich
history. In explanatory modeling, human data is collected and models are built to explain
the observed behavior. Such explanatory cognitive models have been used to understand
web link navigation behavior [29], driving behavior [81], and time interval estimation [88].
Explanatory models are useful in their own right, to expand our understanding of user
behavior, but are also useful for informing a priori predictive models. For example, the
explanatory modeling of driving behavior [81] has been used to inform the development
of a predictive model of driver behavior while utilizing a cell phone [47].
2.2.2 Support Vector Regression-based Modeling
Support Vector Regression (SVR) has been introduced to solve regression and prediction
problems. SVR is a method of estimating function which establish relationship between
independent variables and dependent variable [94]. Presently, SVR is widely used in
various predictive task like financial forecasting [92], stock market prediction [102],
exchange rate prediction [39] etc.
Cao and Tay [11] deals with the application of SVR in financial time series forecasting.
This work analyzed the feasibility of applying SVR in financial forecasting by comparing
15
2. Related Work
it with the multilayer back-propagation neural network and the regularized radial basis
function (RBF) neural network. They have experimentally investigated variability in
performance of SVR with respect to the free parameters. This paper proposes the
concept of adaptive parameters by incorporating the non-stationarity of financial time
series into SVR. Authors have observed that, SVR with adaptive parameters can achieve
both higher generalization performance and use fewer support vectors than the standard
SVR in financial forecasting.
Yang et al. [102] used SVR to financial prediction problems. The financial data
are usually noisy and the associated risk is time-varying. Therefore, the SVR model
described in the paper, is an extension of the standard SVR which incorporates margins
adaptation. By varying the margins of the SVR, authors reflect the change in volatility
of the financial data. The effect of asymmetrical margins is analyzed to allow for the
reduction of the downside risk. The paper shows that the use of standard deviation to
calculate a variable margin gives a good predictive result.
The work by W. C. Hong [39] presents a hybrid support vector machine (SVM) model
to exploit the unique strength of the linear and nonlinear SVM models in forecasting
exchange rate. The parameters of both the linear and nonlinear SVM models are
determined by Genetic Algorithm. Authors have employed numerical examples from
existing literatures to compare the performance of the proposed model.
Travel time is a fundamental measure in transportation and accurate prediction is
crucial to the development of intelligent transportation systems and advanced traveler
information systems. Wu et al. [101] used SVR for travel time prediction and compared
its results to other baseline travel time prediction methods using real highway traffic
data. Since support vector machines have greater generalization ability and guarantee
global minima for given training data, SVR performed well for time series analysis.
Compared to other baseline predictors, the result showed that the SVR predictor can
significantly reduce both relative mean errors and root mean squared errors of predicted
16
2.3. Model-based Approaches to Design Virtual Keyboard
travel times.
2.3 Model-based Approaches to Design Virtual Keyboard
In digital devices, text can be entered usually by two ways, through QWERTY hardware
keyboard and virtual keyboard. One of our objectives is to design virtual keyboard
taking an estimation of visual search time into account. In this section, we first discuss
about various mathematical models used in keyboard layout design followed by a brief
description of design principle of some virtual keyboards. Also, we summarize existing
optimization approaches to design virtual keyboard, reported in recent literature.
In our review of existing literatures, we found two different performance evaluation
models for soft keyboards, namely KLM/GOMS models and Fitts-Digraph model.
KLM/GOMS Models: Keystroke Level Model (KLM) [12] and its successor Goals,
Operators, Methods and Selection (GOMS) rules [13] are two predictive user modeling
techniques widely used in HCI [44,45]. KLM/GOMS models allow designers to perform
quantitative analysis of system behavior from a description of the system, making it
possible to identify design problems or compare alternate designs. The work described
in [52–54] use KLM analysis to develop models for keyboard evaluation.
A KLM/GOMS analysis consists of the following stages.
• A prototype of the system is conceptualized or designed.
• A task is described using the corresponding modeling language. The description
consists of the steps to be taken to carry out the given task. For KLM analysis,
the description is a linear sequence of basic steps known as primitives. A hierarchy
of goals is used to describe the task in GOMS. Six primitives along with their
empirically estimated values are used in KLM/GOMS analysis.
• A quantitative analysis of the task description is performed using values of the
primitives to evaluate keyboard layouts.
17
2. Related Work
In typing through virtual keyboard, a task consists of inputting a string of characters.
However, performance of a user may vary depending on the nature such as number,
frequency of the characters present in the task. Since it is not possible to evaluate
user performance for all possible combinations, there is always a possibility of getting
wrong performance prediction because of task dependent approaches in KLM or GOMS.
Also in KLM/GOMS analysis techniques, it is necessary to construct task descriptions
for each design to compare among a set of alternate designs. However, construction of
task description is a tedious and time consuming process, as it needs to be performed
manually. For a large design space like keyboard layout design, it is impossible
to construct such task descriptions for all alternate designs. On the other hand,
performance models, those are easy to automate, will be more useful to evaluate and
compare among alternate designs for a large design space.
Fitts-Digraph Model: To address the limitations of KLM/GOMS based approach, two
qualities namely task independence and easy to automate are desirable in the models.
Another model having these characteristics is reported in the literature. The model
developed by Soukoreff and MacKenzie [86] is known as Fitts-Digraph model [64,67,105].
The Fitts-Digraph (FD) model [86] is developed primarily to evaluate soft keyboards used
on small sized mobile devices. In the model, it is assumed that a user selects character
keys from the keyboard interface with single finger or stylus movement. Following
components have been considered to develop the model.
• Visual search time: To select a character key from an interface, a user first needs to
locate the key on the interface. The time to locate a key is called the reaction time
or visual search time and is denoted by RT . In the FD model, RT is calculated
using the Hick-Hyman law [37, 42], as shown in Eqn. 2.1. In this equation, a and
b are empirically decided constants and N is the total number of character keys
18
2.3. Model-based Approaches to Design Virtual Keyboard
present on the interface.
RT = a + b × log2N (2.1)
• Movement time: In addition to locating a character key, the user also requires
finger or stylus movement to select the key. The movement time, that is, time to
make a finger or manual movement from one key to another, is calculated using the
Fitts’ law [26,63], shown in Eqn. 2.2. In the equation, MTij is the movement time
from the ith character to jth character, a′ and b′ are empirically decided constants,
dij is the Euclidean distance between the ith and jth keys and wj is the width of
the jth key.
MTij = a′ + b′ × log2(dij/wj + 1) (2.2)
• Digraph probability: In the FD model, the probability of occurrence of character
pairs or digraphs are considered. The digraph probability is calculated from a
text corpus by Eqn. 2.3, where fij is the digraph frequency and Pij is the digraph
probability of characters i and j. K is the total number of characters present in
the corpus.
Pij = fij/K∑
i=1
K∑j=1
fij (2.3)
Using Pij and MTij , mean movement time (MT ) for a given layout is calculated with
Eqn. 2.4, where N is the number of keys present on the interface.
MT =N∑
i=1
N∑j=1
(MTij × Pij) (2.4)
Using Eqn. 2.1 and Eqn. 2.4, text entry rate of a soft keyboard is calculated in characters
per second (CPS) as shown in Eqn. 2.5.
Text Entry Rate (CPS) = 1RT + MT
(2.5)
19
2. Related Work
Other than the models discussed above, researchers have followed several alternate
approaches to design various keyboard layouts. A brief description of those approaches
are discussed in the following.
Getschow et al. [31] introduced one algorithmic approach to virtual keyboard design.
In this article they have discussed about a simple assignment procedure called greedy
algorithm that place alphabets in the most easily accessible positions according to
each character frequency rank order. However, the greedy algorithm ignores many
arrangements that could be substantially better because it does not consider the letter
placement with respect to each other.
To improve the keyboard layout for single-finger typing Lewis et al. [56] developed
another layout. In this paper, they have introduced a network model of English character
to determine the layout design. Here, first a symmetrical matrix of the relative frequency
of unordered English-language digram is created and then a minimally connected network
generated by analyzing the matrix through Pathfinder network-definition program. The
target of this approach is to minimize movement time by reducing the distance between
strongly associated letters.
MacKenzie and Zhang [66] developed a model of the psychomotor act of key tapping
to predict user performance in text typing through virtual keyboards. This model has
several components such as, linguistic data, Fitts’ law, a shortest-path model, and a
key-repeat time measure. The model generates a theoretical text entry rate (in words
per minute) for any layout of virtual keyboard. It thus allowed to evaluate alternate
designs on paper before proceeding to an empirical evaluation. Following this model,
they have proposed one optimized virtual keyboard, Opti (Fig. 2.1(a)), where the key
arrangement of the keyboard is finalized by trial-and-error method.
Zhai et al. [106] minimized the energy, or tension, of a keyboard layout through a
well-known optimization method-the Metropolis algorithm, shown in Fig. 2.1(b). The
Metropolis algorithm is a Monte-Carlo method widely used in searching for the minimum
20
2.3. Model-based Approaches to Design Virtual Keyboard
energy state in statistical physics [69]. Considering the value of Fitts-Digraph model as
Fitts-Digraph energy, the problem of designing a high performance keyboard is equivalent
to searching for the structure of a molecule (the keyboard) at a stable low energy state
determined by the interactions among all the atoms (keys). Applying this approach,
they designed and implemented a software system that did a random walk in the virtual
keyboard design space. In each step of the walk, the algorithm picked a key and moved
it in a random direction by a random amount to reach a new configuration. The level of
Fitts’ energy in the new configuration is then evaluated. The Metropolis function decides
whether the new configuration is kept as the starting position for the next iteration or
not.
(a) Opti keyboard layout [66]
B K D G
C A N I M Q
F L E Space S Y X
J H T O P V
R U W Z
(b) Metropolis keyboard layout [106]
(c) GAG I keyboard layout [79] (d) GAG II keyboard layout [79]
Figure 2.1: Some well known virtual keyboard layouts
21
2. Related Work
Raynal et al. [79] use the concept of genetic algorithm [32] to solve the virtual
keyboard optimization problem in English language. They consider the average mouse
movement time as fitness function and try to maximize text entry speed only. They
took two existing layouts Metropolis and Opti, as inputs for the algorithm and obtain
the best solution or key arrangement are called GAG I and GAG II keyboard, as shown
in Fig. 2.1(c) and Fig. 2.1(d) correspondingly.
Majority of the works on virtual keyboard design have attempted to maximize text
entry rate by arranging the key positions. However, there exists few approaches to
design virtual keyboard considering optimization of multiple objectives, discussed in the
following.
Eggers et al. [21] design the keyboard by optimizing six ergonomic criteria, tapping
load distribution, number of keystrokes, hand alternation, finger alternation, finger
posture, and hit direction using Ant Colony optimization algorithm [17]. Deshwal et
al. [16] use Eggers criteria as the performance indexes and developed an optimal Hindi
keyboard using Genetic Algorithm. But those ergonomic criteria are not applicable to
virtual keyboard design.
Yin et al. [104] optimize ergonomic criteria and disambiguation/prediction
effectiveness simultaneously using Cyber Swam method [103] for keyboard designing.
Their keyboard also outperform the QWERTY and Dvorak [65] keyboard. Quasi-
Qwerty [7] keyboard layout, shown in Fig. 2.2, is designed for optimizing both average
motor movement time to enhance text entry rate and designing layout close to QWERTY
to reduce initial visual search time. The keyboard achieves 12% faster text entry rate
than QWERTY keyboard. Though Quasi-Qwerty is slower than Atomik [65], it requires
less initial text entry time.
Dunlop et al. [20] use multiobjective optimization to design keyboard layout for
touchscreen phone based on three design metrics: minimizing finger travel distance,
a new metric to maximize the quality of spell correction by reducing tap ambiguity
22
2.4. Summary
Figure 2.2: Quasi-Qwerty keyboard layout [7]
and maximizing familiarity through similarity function with the standard QWERTY
layout. Using the methodology, they design two keyboards namely Sath-Rectangular
(Fig. 2.3(a)) and Sath-Trapezoidal (Fig. 2.3(b)).
(a) Sath-Rectangular keyboard layout (b) Sath-Trapezoidal keyboard layout
Figure 2.3: Sath keyboard layouts [20]
2.4 Summary
In this chapter, we have discussed various approaches related with identifying different
parameters which influence visual search task of an interface. Proper identification of
those parameters helps designers to further find out the degree of influence of those
parameters on individual interface. These approaches also support designers to come
up with effective visual search task easing interaction techniques. Our survey reveals
that the visual search parameters do not affect every interface in the same way but vary
according to degree of influence. As an example, search strategy to find a key in virtual
keyboard not always be same but varies from one instance to other randomly. So, in case
23
2. Related Work
of virtual keyboard, fixed search strategy cannot be taken as a visual search influencing
parameters.
Many ways to quantify visual search task till date based on design parameters have
been reported. However, our literature survey does not find any work on modeling visual
search time based on virtual keyboard influencing design parameters. Moreover, there is
also an important need to identify the impact of each parameter on visual search time.
There exists significant work on model-based approaches in designing virtual
keyboards. We observe that Fitts’-Digraph model has been widely used for designing
as well as optimizing virtual keyboard layout. Two components are useful in developing
this model namely visual search time to find a key on a keyboard and mouse movement
time to move the pointer onto desired location. The model also needs unigram and
bigram frequencies extracted from corpus in corresponding language. Apart from these
strong mathematical models, researchers have followed several optimization approaches
to design various keyboard layouts. Single objective optimizations in context of virtual
keyboards usually produce solutions maintaining mouse movement minimized layouts.
Moreover, we also analyze several multiobjective virtual keyboard design principles
for English keyboard, which optimize text entry rate along with tap ambiguity etc.
However, none of the approaches have considered visual search time as an objective
to design virtual keyboard. In the next chapters, all these limitations in the existing
works have been addressed to develop visual search time prediction model and design a
multiobjective virtual keyboard to increase text entry rate and decrease time to search
a key. In order to develop the model, it is necessary to identify the virtual keyboard
design parameters which have significant influence on visual search. Identification of
visual search time influencing virtual keyboard design parameters is discussed in the
next chapter.
24
Chapter 3
Identification and Analysis of
Visual Search Features
Towards the automatic evaluation of visual search time of a user interface, we have
studied visual features reported in different literatures which influence visual search task.
All visual features not necessarily contribute equally in visual perception. Some features
may not be applicable for a particular interface as well as the effect of few features varies
from a user to another user. So, to model visual search task, a categorization of visual
features is required. We aim the categorization with respect to applicability on virtual
keyboard interface which is one of the popular text entry mechanisms. Further, the
features which may be applicable in virtual keyboard interface design may not influence
visual perception equally and also may depend on user. In order to judge the user
dependency and influence, several experiments with users have been carried out.
This chapter consists of five sections. Section 3.1 lists and describes various visual
search features reported in different literatures. Experiments and experimental details
to identify the significance of each features is presented in Section 3.2. The results of
these experiments and corresponding conclusion are discussed in Section 3.3 and 3.4,
respectively. Finally, Section 3.5 summarizes this chapter.
25
3. Identification and Analysis of Visual Search Features
3.1 Listing of Visual Features
Visual search activity is known to be governed by many visual search features. The
following visual search features, which user interface designers usually refer, are listed
below with a brief description of each.
• Size: The task of finding an object among distractors differs with the size of the
same [8, 14]. For example, if an interface contains objects of different sizes, then
visual search time is not necessarily same for all objects.
• Shape: User interface designers believe that shape of an object is a factor in visual
perception [8,14]. When a user searches for an object, it is usually compared with
objects in the search space [4, 76]. In other words, the target objects need to be
compared with a set of similar objects, where shape of the objects controls the
time of matching.
• Spacing: Placing of objects in a design space affects the visual search activity [22].
It is true that if objects are placed sparsely then it demands a different visual search
time than when the objects are placed densely.
• Orientation: Orientation of an object influences the visual search task and as a
result, visual search time varies with objects’ orientation [14]. Like, time to find a
straight line among straight lines with similar orientation is not same compared to
the time to locate the same straight line from a set of straight lines with different
orientation.
• Total number of items: Usually objects of an interface visible at a time are
compared randomly against the target till the target is located [4, 76]. Thus the
total number of items present in a visual search space controls the finding of an
object in the search space [37,41,42].
26
3.1. Listing of Visual Features
• Number of distractors: A distractor is an object which is not the target at
an instance. So, in a visual search space all different items except the target are
considered as distractors [8, 18,36].
• Types of distractors: The heterogeneity among distractors like variation by size,
shape, orientation etc. contributes moderately in visual search task [8, 19, 22, 27,
71,73,76,97].
• Size of a search field: User interface designers observe that size of a search field
is also a factor in visual perception [8, 14, 76]. If an interface have a significant
number of items, distributed evenly and visible at an instance then searching for
an item usually become random [10,36,96]. Although, search time does not solely
depend on size of the search space, but also varies in parallel with density of the
object within the unit space of the interface [68].
• Place of a target: Placing of target in a proper position within an interface
often helps user to find it in a short time [96]. Usually, interface designers and
psychologists observe that user’s focus is more concentrated for some portion or
zone of the interface [49,61,77]. So, finding any objects residing at or around those
areas often becomes easy.
• Ordering: It has been observed that ordering of objects affect visual search
task [2, 10]. Like, the time required to find an item in a ordered list differs from
the time required for an unordered list.
• Grouping: In any interface, the object can be found more quickly if it is placed
apart from the crowd [41]. Also, it is comparatively easy to find an individual
object from a group containing specific feature based objects rather than from the
total interface consisting of several constraints.
27
3. Identification and Analysis of Visual Search Features
• Size of a group: Size of a group indicates the number of items or objects in a
particular group. Visual search time to locate an object depends on the size of a
group [37,42].
• Labeling of a group: Labeling of groups within an interface also helps user to
identify them more specifically [2]. As an example, it would be easier to access the
virtual keyboard if labeling can be done in each group like vowel group, consonant
group, numeric group etc.
• Color of a text and background: Color property of an object may influence
visual search time of the object [73]. If a target is easily discriminable from its
distractor due to color difference then the search time required to find the target
visually become different [8, 14].
• Contrast of a background: Visual search time is also affected by background
contrast. Greater background contrast helps more to identify the target, that is,
visual search time becomes lesser [73]. For an example, effort required to read
a black colored text with white background and yellow colored text with white
background is not equal [71–73].
• Contrast of distractors: When there is a notable difference in contrast between
target and distractors then the time to identify the target differs [68, 71, 73]. For
example, from a set of similar shaped objects, a red colored object (target) can be
easily identified when other objects (distractor) are green colored.
• Ageing and memory: Older adults face more difficulty than younger adults
to identify and locate a target defined by a conjunction of features among
heterogeneous distractors [55]. It has been observed that with practice, people
can learn to find a target from randomized object set almost as quickly as from
ordered object set [10,41].
28
3.2. Experiments
• Search strategy: Visual search time directly depends on the search strategy
followed while searching, like top-down searching, bottom-up searching, item-by-
item searching, randomized searching etc. [8, 14].
3.2 Experiments
Since all visual features does not contribute equally in visual perception [93, 97], a
categorization of visual features is required. We aim the categorization in the context of
virtual keyboard interface. The preliminary analysis of those features reveals that some
features are not applicable in the context of virtual keyboard.
Orientation: In a virtual keyboard, orientation of all characters or keys present in the
keyboard layout is similar [65]. So, we should not consider this feature while modeling
visual search task in the context of virtual keyboard.
Search strategy: As keyboard contains a large number of keys as well as users get
acquainted after accessing the layout for some time, so it is not required for the user
to search all keys within the interface following any search strategy. The searching is
accomplished in random fashion usually [95,98].
Distractor type: Although, some keys like Backspace, Enter, Shift etc. are larger
in size than other keys in keyboard interface, they do not affect on visual search task
significantly. After being familiarize with the keyboard interface, users can find those
buttons quickly not because of their different size and shape than distractor, but from
their natural tendency (e.g. after certain time, user will guess the buttons correctly from
their fixed position) [95,98].
Varying spacing: We have observed that, in a virtual keyboard interface both the
horizontal distance and the vertical distance between keys remain same throughout the
keyboard layout. The variation of horizontal distance and vertical distance do not
consider as a good virtual keyboard design methodology. If varies, the design lacks
user comfort [65].
29
3. Identification and Analysis of Visual Search Features
Color : While accessing the graphical user interface, color feature retains impression
in user’s mind. In contrast, the feature behaves differently from user to user. So, the
color choice of background of virtual keyboard, button color, font color etc. may attract
one user but may distract another [73]. So, it is unlikely that a particular combination
of colors in the keyboard may satisfy all the users.
Further, the features which may be applicable in virtual keyboard interface design
may not influence visual perception equally and also may depend on user. In order to
identify the user dependency and influence, several experiments with users have been
carried out. In the following, we discuss about experimental setup, interfaces used to
perform the experiments, details of participants considered in experiment, experimental
procedure and outcome of the experiments.
3.2.1 Experimental Setup
All experiments have been conducted using 2.4GHz Pentium Core2Duo machine with
a 17" color monitor with 1280 × 1024 resolution. The developed interfaces for these
experiments are written in C# using Visual Studio 2008. All key press events are
recorded automatically and stored in a log file. The mouse positions are also stored in
separate log file using a separate window hook program written in C#. All experiments
are done in Windows 7 environment.
3.2.2 Interface Used
Three virtual keyboard interfaces have been used namely Opti, a virtual keyboard layout
for English language [66], Avro, a virtual keyboard layout for Bengali language developed
by OmricornLab [75] and iLipi-H, a virtual keyboard layout for composition of text in
Hindi proposed by [82]. Figure 3.1 shows the layout of each interface.
Opti (a frequency-based keyboard layout): It is one of the optimized virtual keyboard
layouts for the English language. Figure 3.1(a) shows the Opti layout as described by
30
3.2. Experiments
(a) Opti keyboard layout
50
61
72
83
94
a i u e o
k j
t h
Tab ---> <--- Backspace
ZWJ ZWNJ
S P A C E Enter
(b) Avro keyboard layout
(c) iLipi-H keyboard layout with inflexion window
Figure 3.1: Virtual keyboard layouts used in experiments
MacKenzie and Zhang. The keyboard layout was optimized for speed using trial and
error, Fitts’ law, and bi-gram frequencies of characters.
Avro (an alphabetical keyboard layout): This keyboard uses the alphabetical
arrangement of Bengali alphabets. The consonants are divided into two sub parts, as
shown in Fig. 3.1(b). All vowels are present in a row at the bottom of the layout.
31
3. Identification and Analysis of Visual Search Features
iLiPi-H (a multizonal, frequency and inflexion window based keyboard layout): In this
keyboard layout, Hindi characters are spatially arranged in layered zones depending on
their frequencies of occurrences (Fig. 3.1(c)). The high frequency characters are placed
in a central zone, the next higher frequency characters are placed in its outer zone
surrounded by the central zone and so on. In addition to this, inflexions are dynamically
appeared through an inflexion window for each consonant.
3.2.3 Participants
To evaluate the design with participants, we include 21 participants in our experiments.
Out of them, 12, 8 and 15 participants have tested the Bengali, Hindi and English
language keyboard interfaces, respectively. If a participant is familiar with multiple
language then the participant may have considered for multiple interface. Participants
are chosen from different educational backgrounds and with various level of computer
proficiency. The participants are having English and native language familiarity in terms
of reading and writing. However, some businessmen and housewives do not have any
exposure in English. There are 9 female and 12 male participants belong to different
age groups with average age is 27.52 years (SD = 3.61). The detail of the participants
is shown in Table 3.1.
Table 3.1: Description of participants
Participant’s profileNumber ofparticipantsOccupation Age
group Education Computerproficiency
Student 13–27
Post Graduate Level 1 5
Under Graduate Level 1 6
Higher Secondary Level 2 2
Office staff member 31–48Graduate Level 1 2
Higher Secondary Level 3 2
Business person 34–53Graduate Level 2 2
Higher Secondary Level 3 2
32
3.2. Experiments
We decide computer proficiency on the basis of expertise in four different tasks.
• Total length of time a participant has worked with computer.
• Number of application softwares known by participant.
• Length of daily work-time with computer.
• Number of messages typed in computer in their mother languages.
In view of the above, we categorize the participants at three different levels.
• Level 1: Having more than 3 years of experience with computer, know operating
systems, programming and application software.
• Level 2: Having 1−3 years experience and mainly familiar with application software
such as document processing, email and Internet browsing.
• Level 3: Having less than 1 year experience and poor in computer related task.
3.2.4 Experimental Procedure
In our experiments, four to seven different experimental sessions are carried out by each
participant. Each session includes several trials having two parts: stimulus presentation
(target) and layout presentation (interface). The goal of each trial is to locate the target
within the interface. An experimental trial begins by presenting a target followed by
a virtual keyboard interface till the goal is achieved. A screenshot of the layout with
target is shown in Fig. 3.2. We have used three different fonts namely Times New Roman,
Vrinda and Mangal to display English, Bengali and Hindi characters respectively, and
the font remain same throughout all experiments.
The target is displayed until the participant either hit the Spacebar or clicks the
mouse, indicating he/she is ready to proceed. Once the participant is ready, a virtual
keyboard interface is displayed. Participants are instructed to either press the Spacebar
33
3. Identification and Analysis of Visual Search Features
Figure 3.2: A sample layout with target object
or click the mouse again as soon as they locate the target within the interface. After
that, the participant is required to move the mouse to the identified target and click the
same. A trial is considered to be finished when these steps are completed. The visual
search time of each trial has noted programmatically and stored into log file. Next, a
fresh trial is started in the same way with a different target. A session continues till all
characters are considered in trials or the participant does not willing to continue further.
The cursor position is programmatically controlled and locked on the initial screen
that is where the target is displayed first. Cursor is programmatically kept hidden on
the interface until a participant finds the target within the interface, so he/she is not
getting distracted by the cursor position. For remaining of a trial, the mouse position
has not been controlled. It has been reported that a participant may get acquainted
with the interface while accessing over a long time [96]. This result lesser visual search
time compared to the situation where the participant is not familiar with the interface.
The difference becomes higher over the time. Although this effect depends solely on user
and affects intelligent users mostly, still we have tried to reduce this. In our experiments,
the chance of selecting any of the three interfaces for a particular trial is equal.
By analyzing the log files, we have calculated the center value for each variation of
a feature and plotted them into graph. Note that both median and mean can be used
to calculate the center value, but median reduces the average of the absolute deviations
34
3.3. Experimental Results
whereas mean makes it biased towards the extreme value. Hence, we have used median to
calculate the center value. We have also conducted analysis of variance1, that is, ANOVA
test to study the significance of each feature on visual search time. The ANOVA tests are
conducted using Statistical Package for the Social Sciences, also called SPSS tool [43].
3.3 Experimental Results
In this section, we discuss about the outcome of the experiments for each visual search
features.
Size: Size is a primitive property of all kind of objects. So, in a virtual keyboard
interface size may refers to size of the entire interface, size of a button or size of the text
appears on button, that is, font size of the text. Here, we have focused our experiments
on font size and kept other sizes or features unchanged throughout these experiments.
A set of experiments to observe the influence of size on visual search time has been
performed. For each experiment, a keyboard layout has been chosen, different font sizes
have been applied on that keyboard keeping all other features unaltered. The font size
is varied from 6pt to 16pt with an increment of 2pt.
From our experiments, we have observed that characters are almost unreadable for
font size less than 5pt, so 6pt is considered as minimum font size. Figure 3.3(a) shows
instance of Avro keyboard layout with 12pt font size. The analysis of experimental results
is shown graphically in Fig. 3.3(b). We have also calculated median of search time for
different font sizes and keyboards and observed that visual search time is higher at lower
font size. ANOVA test reveals that there is a significant difference between performance
of different font sizes (F5,217 = 18.12, p < 0.05).
Shape: Shape of an object is defined by the geometric properties of the object. In
context of virtual keyboard interface, the shape may refer to the virtual keyboard itself,
the buttons of the interface or the characters included in the interface. We have limited1Analysis of variance, http://en.wikipedia.org/wiki/Analysis_of_variance
35
3. Identification and Analysis of Visual Search Features
(a) Avro keyboard with 12pt font size
(b) Visual search time for different font size
Figure 3.3: Effects of different font sizes on visual search time
our experiments only to shape characteristics of the characters. To achieve this target,
first we have identified and grouped the similar character of a language. Here, we have
considered Bengali language for our experiments.
36
3.3. Experimental Results
Table 3.2 shows the similar character set of Bengali language. Later, when a user is
asked to find a character which belongs to a group of that set, one or more characters
from that group may present in the interface. We have analyzed the experimental results
and depicted the same in Fig. 3.4. From the analysis, it has been substantiated that the
presence of similar shaped characters in virtual keyboard interface affects visual search
time, although the effect is very less. The increment in visual search time due to presence
of four similar shaped characters over single character is 4.21%. ANOVA test indicates
that there is no significant difference among the performance of similar shape characters
on visual search time (F3,94 = 2.301, p > 0.05).
Table 3.2: Similar character set of Bengali language
Similar characters Base character
Group 1 A, Aa, t tGroup 2 U, Ŕ, D, Ĺ DGroup 3 E, Ţ, Č EGroup 4 O, Ů OGroup 5 k, b, r, C bGroup 6 T, Z, Ľ ZGroup 7 J, y, P J
Spacing: Spacing between the keys is also considered as an important visual feature.
In the context of a virtual keyboard interface, spacing between the keys refers to the
horizontal or vertical gap between the key buttons. The experiments are performed
on varying space between keys keeping other features constant. Also, for a particular
instance, we maintain equal distance between all keys. A set of experiments to observe
the influence of spacing on visual search time has been performed. In these experiments,
the spacing between the keys is varied from no spacing to 100% of the button width with
a step of 25%. Figure 3.5(a) shows Opti keyboard with 50% of button width as spacing
between keys. It has been observed that spacing between the keys affects visual search
time.
37
3. Identification and Analysis of Visual Search Features
Figure 3.4: Effects of similar shaped characters on visual search time
From the experiments, we have observed that visual search time is lesser when the
distance between two keys is around 25% of the key size. Figure 3.5(b) depicts results
collected from the experiments which are conducted on different conditions on three
keyboard interfaces. From the analysis of ANOVA, we have concluded that the mean of
performance of users for different spacing are significantly different (F4,205 = 15.643, p <
0.05).
Number of items: Many items present in the interface may distract the concentration
of user while searching for specific item and as a result, search time increases. To
study the effect of number of items present in an interface, the virtual keyboard
layouts are required to be modified to have different number of characters. In these
experiments, we have considered 7 different variations among number of items, that is,
6, 10, 15, 25, 40, 56, 67. An instance of iLiPi-H keyboard with 67 characters is shown in
Fig. 3.6(a).
It has been observed that number of items within the interface influences visual search
time. It is also observed that some users search the item within interface jumbled up
38
3.3. Experimental Results
(a) Opti keyboard with 50% spacing between keys
(b) Visual search time for different spacing between keys
Figure 3.5: Effects of spacing between keys on visual search time
with several objects efficiently but others may not perform the same. As a consequence,
the result may vary from user to user for unchanged features except number of items. We
have observed that usually visual search time increased with the increasing number of
items present in the interfaces. Experimental results in different scenario are plotted in a
graph which is shown in Fig. 3.6(b). The ANOVA test reveals that there is a significant
difference between the mean of visual search time as determined by different number of
39
3. Identification and Analysis of Visual Search Features
(a) iLiPi-H keyboard having 67 items
(b) Visual search time for different number of items
Figure 3.6: Effects of varying number of items on visual search time
items on keyboard (F6,266 = 41.673, p < 0.05).
Number of distractor: In any keyboard interface, number of distractors is almost
equivalent with number of items present in it. So, no other experiment has been
performed, as results would have been similar.
Search field size: The size of the area where user is intended to search the item may
affect the visual search time. To measure the effectiveness of search field size on visual
search task, we conduct experiments. In these experiments, search field size, that is,
40
3.3. Experimental Results
interface size has been varied from 10% to 50% of the screen area with an increment of
10%. Avro keyboard occupying 20% of screen is shown in Fig. 3.7(a).
(a) Avro keyboard occupying 20% of screen area
(b) Visual search time for different search field size
Figure 3.7: Effects of search field size on visual search time
The result signifies that, if all features remain unchanged except search field size,
visual search time increases while search field size is less than 20% or greater than 40%
41
3. Identification and Analysis of Visual Search Features
of the screen area. Figure 3.7(b) depicts the results collected from the experiments
which are conducted with different screen sizes on three keyboard interfaces. We have
performed ANOVA test on visual search time for different search field size. From the
analysis we have concluded that the mean of visual search time for different search field
size are not significantly different (F4,87 = 2.352, p > 0.05).
Position: Positioning of objects in the interface also influences visual search time.
Similarly, in a virtual keyboard interface, different key positions also affect the visual
search time. There are 30, 61 and 66 different key positions possible for Opti, Avro,
iLiPi-H keyboard interfaces, respectively. As it is not possible to observe the alteration
of visual search time for each position and accommodate all the results pictorially, we
divide the keyboard layout into 9 different blocks as shown in Fig. 3.8(a).
We have calculated visual search time for each block and plotted in a graph, shown in
Fig. 3.8(b). From the graph, we can observe that visual search time varies for different
position of characters. A reported measure analysis of variance reveals that, there is
a significant difference between mean of visual search time for nine different blocks
(F8,314 = 11.29, p < 0.05).
Ordering: Ordering of objects within interface is also considered to be an influential
factor in determining visual search time. We have considered alphabetical, frequency-
based and random ordering of keys in our experiments. The experimental result
establishes the fact that random ordering of the objects results in more searching time
than other orientations. We have noticed that different kinds of ordering influence users
much in finding keys from the interface. If all features are same, then it has been found
that alphabetic arrangement helps a user more in finding a key from the keyboard.
The other frequency-based arrangement takes less time than highest-valued random
arrangement. The observed effect of these different arrangements of keys on visual search
time is shown in Fig. 3.9. The ANOVA test on experimental results reveals that there
is a significant difference between the mean of visual search time for different ordering
42
3.3. Experimental Results
(a) iLiPi-H keyboard divided into 9 blocks
(b) Result for different position of keys
Figure 3.8: Effects of different position of keys on visual search time
of keys (F2,24 = 28.59, p < 0.05).
Grouping and group size: An object can be found out quickly in an interface if it
belongs within a particular group of objects and group size is not large. To study the
effect of grouping and group size on visual search time, we have modified keyboard
layouts maintaining similar type of characters, that is, consonant, vowel, numeral etc.
in a group. As an effect, the layout contains a maximum of 7 groups with varying group
size. Figure 3.10 shows an Opti keyboard layout modified to organize characters into
two groups with different group size.
43
3. Identification and Analysis of Visual Search Features
Figure 3.9: Effects of ordering of characters on visual search time
Figure 3.10: Opti keyboard with two group
The experimental result establishes the fact that a moderate number of groups, each
having a minimum number of objects, facilitate more in obtaining lesser visual search
time. The results are graphically depicted in Fig. 3.11(a) and Fig. 3.11(b). ANOVA
analysis reveals that, the mean of visual search time as determined for different grouping
and group size are not equal. The observed value of ANOVA for different group is
44
3.4. Observation
F6,30 = 12.49, p < 0.05 and grouping is F6,79 = 27.426, p < 0.05.
(a) Visual search time for different number of grouping
(b) Visual search time for varying group size
Figure 3.11: Effects of grouping and group size on visual search time
3.4 Observation
We have performed experiments on varying sizes of object. From the result, we can
observe the tendency of visual search time growth with respect to different text size
45
3. Identification and Analysis of Visual Search Features
and fixed fonts in three different languages. The outcome shows that for all keyboards,
visual search time is pretty high for small sized fonts (like 6pt). The visual search time
decreases with increasing font size up to a moderate value (12pt). But when the size
of object is high (14pt or more), the curve corresponding to each language keyboard
grows up. It means that, users get acquainted with text size belonging to certain range.
Beyond that, the visual search task for finding keys becomes time expensive for human.
From the effect of space between object in a virtual keyboard, it can be observed
that visual search time varies significantly with variation of space between object. Visual
search time is less when space between objects around 25% of the object size. The number
of items in a virtual keyboard also contributes in visual search time and corresponding
effect is also significant. We have observed that visual search time gets higher and
increases almost linearly when number of items in virtual keyboard raises. On the other
hand, proportion of screen area occupied by virtual keyboard does not affect significantly
in visual search time. Visual search time is almost constant irrespective of proportion of
screen area occupied. The statistical analysis of our observed data for different features
are summarized in Table 3.3. Here DFn, DFd and F value represent degree of freedom
between groups, degree of freedom within groups and ratio of mean square error between
groups and mean square error within groups, respectively.
Table 3.3 establishes the fact that there are 6 features, which contribute more in visual
search time in the context of virtual keyboard. The features are:
• Size of elements
• Space between elements
• Number of elements
• Position of elements
Thus, we would limit our investigation of developing a model which will compute average
visual search time of a virtual keyboard interface in terms of identified features only.
46
3.5. Summary
Table 3.3: Summary of statistical analysis for different features
Feature Name DFn DFd F value Significant(p ≤ 0.05)
Size of elements 5 217 18.12 Yes
Shape of elements 3 94 2.301 No
Space between elements 4 205 15.643 Yes
Number of elements 6 266 41.673 Yes
Search field size 4 87 2.352 No
Position of elements 8 314 11.29 Yes
Ordering of elements 2 24 28.59 No
Grouping of elements 6 30 12.49 No
Group size 6 79 27.426 No
3.5 Summary
This chapter presents the effect of different visual search features on visual search
time. The visual search features related to virtual keyboard design parameters, which
significantly influence visual search task are identified. To accomplish the task, first, we
have listed the features reported in various literatures which influence visual search time.
Next, we have performed several experiments of virtual keyboard interfaces with users of
different expertise level. From the statistical analysis of results of these experiments, we
have identified four features namely size, space, number and position of characters, which
significantly influence visual search task while composing text through the interface. The
identified features influence many other cognitive task performed in the context of virtual
keyboard. Further, these features would be helpful while developing a computational
model of visual search time for virtual keyboard, which is presented in the next chapter.
47
Chapter 4
Modeling of Visual Search Time
A computational model of visual search time estimates the average search time of a
keyboard layout from various keyboard design parameters. Thus, the model is helpful in
evaluation and automatic design keyboard layouts. Earlier, we have discussed about the
identification of virtual keyboard design parameters which make significant influence on
visual search in finding a key. In this chapter, we develop a predictive model of visual
search time based on those identified parameters. As user is an important concern
in assessing the interface, we perform several user-based evaluations to accumulate
data of user performances with variation of different features. These collected data
are then used to generate the model. The experimental setup, interfaces used and
experimental procedure we have followed are already discussed in Chapter 3. Then,
using the experimental data, we have built three predictive models based on three
different regression approaches namely Linear regression, Non-linear regression and
Support Vector Regression (SVR). We have analyzed the fitness of each modeling
approach against the gathered data. Analysis substantiates that SVR based approach
predicts with higher accuracy than other two approaches. In order to validate this
modeling approach, we have also conducted another set of user evaluation and judged
the performance by comparing model predicted value with user data.
This chapter consists of four sections. Section 4.1 details about experiments to
49
4. Modeling of Visual Search Time
collect knowledge about user performance on variation of different visual search features.
Different modeling approaches exercised to develop a visual search time predictive model
are discussed in Section 4.2. Then, Section 4.3 describes validation of our proposed
model. Finally, Section 4.4 summarizes this chapter.
4.1 Training Data Set Generation
In Chapter 3, we have identified four visual search features which significantly influence
the visual search task of virtual keyboard. We now vary only these features and
accumulate data involving a large users’ pool. These data constitute training data
set for the visual search time prediction model we have planned to develop. Our
approach to gather training data set is discussed in this section. We have followed
same experimental setup, interfaces and experimental procedure for these experiments,
as discussed previously in Chapter 3. However, alongwith the previously mentioned
participants we have included another set of participants for our experiments. Details
of the participants are given in Table 4.1.
Table 4.1: Description of participants
Participant’s profileNumber ofparticipantsOccupation Age
group Education Computerproficiency
Student 13–27
Post Graduate Level 1 5
Under Graduate Level 1 8
Higher Secondary Level 2 3
Secondary Level 2 2
Office staff member 31–48Graduate Level 1 4
Higher Secondary Level 3 3
Business person 34–53Graduate Level 2 4
Higher Secondary Level 3 3
Aged person 55–62Graduate Level 2 2
Under Graduate Level 3 3
Housewife 35–45Graduate Level 1 3
Secondary Level 3 2
50
4.2. Model to Predict Visual Search Time
All experimental data are stored programmatically into log files. Analyzing log
files, we have removed the erroneous data, that is, when the presented target and the
participant’s identified target are different. From the analysis we have also observed
that, sometimes participants get distracted due to environmental noise and take large
amount of time to locate the target. Thus data with visual search time greater than
µ + 2 × σ are filtered out from modeling. Here, µ and σ represent mean and standard
deviation, respectively. Summary of the collected data is tabulated in Table 4.2. Finally,
28734 data have been considered for the modeling task.
Table 4.2: Summary of Collected Data
Total collected data 30882
Mean (µ) 6.34
Standard deviation (σ) 3.81
Outlier data 2148
Data for modeling 28734
4.2 Model to Predict Visual Search Time
We have followed three different approaches toward developing predictive models of
visual search time. These approaches are linear regression, non-linear regression
and Support Vector Machine for regression (SVR). In linear regression with multiple
features, we have analyzed two different approaches; normal equation [70] and gradient
descent [70]. Whereas, non-linear regression has been analyzed through some of the
standard models [5]. SVR has been trained using the Radial Basis Function (RBF)
Kernel [94] and cross validation [94] has been performed. In depth description and
performance analysis of these approaches are given in the following.
51
4. Modeling of Visual Search Time
4.2.1 Linear Regression with Multiple Features
Linear regression attempts to model the relationship between two or more explanatory
variables and a response variable by fitting a linear equation to observed data [70].
Every value of the independent variable x is associated with a value of the dependent
variable y. Linear regression with multiple features is a generalization of linear regression
considering more than one independent variable. The hypothesis hθ(x) given by basic
model of linear regression is shown in Eqn. 4.1.
hθ(x) = θ0 + θ1x1 + θ2x2 + · · · + θnxn (4.1)
where x1, x2, . . . , xn are the multiple features or independent variable, θ0, θ1, θ2, . . . , θn
are the model parameters and hθ(x) is the hypothesis or function. In linear regression,
data are modeled using linear predictor functions, and model parameters are estimated
from the data.
Our training data contain four features. So, x1, x2, x3 and x4 represent the size of
objects, space between objects, number of objects and position of object in keyboard,
respectively. In our work, we have explored two different approaches of linear regression,
normal equation [70] and gradient descent [70]. The developed programs for both
approaches are executed using GNU Octave [78].
Normal Equation: The value of θ for the above hypothesis in normal equation
approach can be computed using Eqn 4.2.
θ = (XT X)−1XT y (4.2)
where X is a 28734 × 5 matrix having 1 in first column which represents the feature
vectors and y is a 28734 × 1 matrix representing visual search time for corresponding
feature vector of X. We compute the predicted visual search time based on this model.
The observed value of θ are θ0 = 6.93, θ1 = −6.19, θ2 = −12.83, θ3 = 17.54 and θ4 = 1.17
52
4.2. Model to Predict Visual Search Time
and the observed value of R2, coefficient of determination1, for this model is 0.39.
Gradient Descent: Gradient descent, also known as the steepest descent, is a first-
order optimization algorithm [70]. Here, the steps to be taken are proportional to the
negative of the gradient (or the approximate gradient) of the function at current point.
The objective of this approach is to minimize the error (J(θ)) which can be computed
using Eqn. 4.3.
J(θ) = 12m
m∑i=1
(hθ(x(i)) − y(i)
)2(4.3)
Here, m is number of data in training set, that is, 28734. x(i) represents ith training set,
hθ(x(i) is the predicted visual search time for x(i) and y(i) is the observed visual search
time for x(i). One way to do this is using the batch gradient descent algorithm. In this
approach, each θj is simultaneously updated as shown in Eqn. 4.4.
θj = θj − α1m
m∑i=1
(hθ(x(i)) − y(i)
)x
(i)j (simultaneously update θj for all j) (4.4)
where, x(i)j indicates jth feature in ith training set and α is the learning rate which
controls convergence of the algorithm. The higher value of α increases the convergence
rate of the algorithm but sometimes may create non-convergence situation. Contrary,
smaller value of α assures convergence of the algorithm but increases number of iteration
and in turn total computation time.
First, input data is normalized using z-score2 normalization. Then, we compute the
cost function (J(θ)) for different values of α. We observe that our problem converges
for α ≤ 0.1 with the training data. The observed value of θ are θ0 = 4.59, θ1 = −11.38,
θ2 = −14.08, θ3 = 18.93 and θ4 = 4.79 for α = 0.01. The computed R2 of gradient
descent approach for these values is 0.42.
1Coefficient of determination, http://en.wikipedia.org/wiki/Coefficient_of_determination2z-score indicates, by how many standard deviations an observation is above or below the
mean. It is derived by subtracting the mean from an individual observation and then dividingthe difference by the standard deviation.i.e. z = x−µ
σ , here µ is mean and σ is standard deviation.
53
4. Modeling of Visual Search Time
4.2.2 Non-linear Regression
Non-linear regression is a form of regression analysis in which observational data are
modeled by a function which is a non-linear combination of parameters and depends
on one or more independent variables [5]. Unlike linear regression, which is restricted
to estimating linear models, non-linear regression can estimate models with non-linear
relationship between independent and dependent variables.
In order to compute non-linear regression one need to specify a non-linear model
which states the relationship between independent and dependent variables. The initial
values of model parameters are also need to be specified. Here, we have computed with
five standard models. The computations are performed using SPSS tool [43]. Details of
each model along with R2 are shown in Table 4.3. In this table, b1, b2, b3, b4 represent the
various model parameters and x represents the combined value of independent variables.
Here, we have followed the linear regression model to combine independent variables into
a single value.
Table 4.3: Different non-linear model with corresponding R2
Model Definition R2
Asymptotic Regression [5] f(x) = b1 − b2 × b3x 0.49
Gauss [5] f(x) = b1 × (1 − b3 × exp−b2x2) 0.53
Log-Modified [5] f(x) = (b1 + b3x)b2 0.59
Ratio of Quadratics [5] f(x) = (b1+b2x+b3x2)(b4x2) 0.68
Richards [5] f(x) = b1
(1+b3×exp−b2x)1
b40.62
From the Table 4.3, we observe that Ratio of Quadratics model performs better than
others with our input dataset. The values of constants for this models are b1 = 479.15,
b2 = 276.47, b3 = 92.54 and b4 = 22.68.
54
4.2. Model to Predict Visual Search Time
4.2.3 Support Vector Regression
Support Vector Machine (SVM) was developed by Vladimir Naumovich Vapnik [94]
to solve the classification problem. Recently, SVM have been successfully extended to
regression and density estimation problems [9]. SVM based regression or Support Vector
Regression (SVR) is a method to estimate a function that maps from an input vector to
a real number based on training data. Margin maximization and kernel trick are used
in SVR for nonlinear mapping [85].
In SVR, the training dataset (D) for regression is represented as follows,
D = {(X1, y1), (X2, y2), ..., (Xm, ym)} (4.5)
where Xi is a n-dimensional vector of independent variables and yi is the real number,
that is, dependent variable or observation for each Xi. The SVR function F (X) makes
a mapping from an input vector Xi to the target yi as shown in Eqn. 4.6.
F (X) ⇒ w · X + b (4.6)
where w is the weight vector and b is the bias. The goal is to estimate these
parameters w and b of the function that give the best fit of the data. A SVR function
F (X) approximates all pairs (Xi, yi) while maintaining the differences between estimated
values and real values under ε precision, that is, yi−w·Xi−b ≤ ε or w·Xi+b−yi ≤ ε [85].
Thus, the margin of this estimation is identified as,
margin = 1∥w∥
(4.7)
Consequently, by minimizing ∥w∥2 to maximize the margin, the training of SVR
55
4. Modeling of Visual Search Time
becomes a constrained optimization problem as follows.
minimize : L(w) = 12
∥w∥ (4.8)
subject to : yi − w · Xi − b ≤ ε
w · Xi + b − yi ≤ ε
This solution assumes that the input data do not contain any errors. To allow some
errors to deal with noise in the training data, the soft margin SVR uses slack variables
ξ and ξ̂. Then, the optimization problem can be revised as shown in Eqn. 4.9.
minimize : L(w, ξ) = 12
∥w∥ + C∑
i
(ξ2i + ξ̂2
i ), C > 0 (4.9)
subject to : yi − w · Xi − b ≤ ε + ξi, ∀(Xi, yi) ∈ D
w · Xi + b − yi ≤ ε + ξ̂i, ∀(Xi, yi) ∈ D
ξi, ξ̂i > 0
The constant C > 0 is the trade-off parameter between the margin size and the amount
of errors. The slack variables ξ and ξ̂ deal with infeasible constraints of the optimization
problem by imposing the penalty to the excess deviations which are larger than ε. To
solve the optimization problem, we can construct a Lagrange function from the objective
function with Lagrange multipliers [85] as follows,
minimize: L(w, ξ) = 12
∥w∥2 + C∑
i
(ξi + ξ̂i) −∑
i
(ηiξi + η̂iξ̂i)
−∑
i
αi(ε + ηi − yi + w · Xi + b)
−∑
i
α̂i(ε + η̂i − yi + w · Xi + b) (4.10)
subjected to: η, η̂i ≥ 0 and α, α̂i ≥ 0
where ηi, η̂i, αi, α̂i are the Lagrange multipliers. The optimization problem with
56
4.2. Model to Predict Visual Search Time
inequality constraints can be changed to following dual optimization problem.
maximize: L (α) =∑
i
yi (αi − α̂i) − ε∑
i
(αi + α̂i)
−12∑
i
∑j
(αi − α̂i) (αi − α̂i) XiXj (4.11)
subjected to:∑
i
(αi − α̂i) = 0, 0 ≤ α, α̂i ≤ C
Accordingly, the SVR function F(X) becomes the following function.
F (X) ⇒∑
i
(αi − α̂i) XiX + b (4.12)
Here, Xi is the ith training data and X represents the set of all training data. This
function can map the training vectors to target real values with allowing some errors
but it does not support the nonlinear mapping between independent and dependent
variables [85]. So, the kernel is applied by replacing the inner product of two vectors
Xi, Xj with a kernel function K(Xi, Xj) [94]. The transformed feature space is usually
high dimensional, and the SVR function in this space becomes nonlinear in the original
input space. Using the kernel function K, the inner product in the transformed feature
space can be computed as fast as the inner product Xi · Xj in the original input space.
The linear optimization function of Eqn. 4.11 is changed by using kernel function as
shown in Eqn. 4.13.
maximize: L (α) =∑
i
yi (αi − α̂i) − ε∑
i
(αi + α̂i)
−12∑
i
∑j
(αi − α̂i) (αi − α̂i) K (Xi, Xj) (4.13)
subjected to:∑
i
(αi − α̂i) = 0
α̂i ≥ 0, αi ≥ 0, 0 ≤ α, α̂i ≤ C
57
4. Modeling of Visual Search Time
Finally, the SVR function F(X) becomes the following using the kernel function [85].
F (X) ⇒∑
i
(α̂i − αi) K (Xi, X) + b (4.14)
The basic steps in SVR are consists of the following.
• Scaling of collected data.
• Selection of kernel.
• Cross validation to find out best value of the parameter(s).
• Train with entire training set using best value(s).
• Testing of developed model.
Data scaling: Scaling of data before using those in SVR is important [94]. The main
advantage of scaling is that it reduces the dominance of higher numeric values on smaller
values. It also helps to reduce numerical complexities during computation. Kernel values
usually depend on the inner products of feature vectors. Thus, large attribute values
might cause numerical complexities. It is preferred to use same scaling method to scale
both training and testing data. We use z-score normalization to scale the collected data.
Kernel selection: In our work, we use Gaussian Radial Basis Function (RBF) as SVR
kernel. The RBF kernel nonlinearly maps samples into a higher dimensional space. The
RBF kernel is represented as
K(Xi, Xj) = eγ∥Xi−Xj∥2 (4.15)
So it, unlike the linear kernel, can handle the case when the relation between class labels
and attributes is nonlinear. Further, the linear kernel is a special case of RBF since the
linear kernel with a penalty parameter C̃ has the same performance as the RBF kernel
with some parameters (C, γ). The polynomial kernel has more hyperparameters, which
inuences the complexity of model, than the RBF kernel.
58
4.2. Model to Predict Visual Search Time
Cross validation: There are two parameters for SVR with RBF kernel; C and γ. It
is not known beforehand which C and γ values are best for a given problem. Thus,
parameter search is required to identify the best values of C and γ so that the model
can accurately predict unknown data. Note that, it is not useful to achieve high training
accuracy. Thus, the strategy is to separate the data set into two parts, of which one
is considered as unknown or testing set. The prediction accuracy obtained from the
unknown set more precisely reflects the performance on an independent data set. This
procedure is known as cross-validation.
The cross-validation procedure can prevent the overfitting problem. In k−fold cross-
validation, we first divide the training set into k subsets of equal size. Sequentially, one
subset is tested using the model trained on the remaining k − 1 subsets. Thus, each
instance of the whole training set is predicted once so the cross-validation accuracy is
the percentage of data which are correctly predicted.
To find the best values of C and γ through cross validation, grid search is performed.
We test with various pairs of C and γ values and then choose the one with the best
cross-validation accuracy. The values of C varies from 2−5 to 2−15 with step 22 and γ
varies from 2−15 to 23 with step 22. We select 1000 data, as sample, form the collected
data to decide best values of C and γ. We perform 5 − fold cross validation to choose
the best C and γ. We obtained best values of C and γ as 2.4 and 0.02, respectively.
Training with SVR: The collected data are divided into two parts. We use 80% of
collected data for training with SVR. The best values of C and γ, required for training,
are already decided through cross validation as mentioned earlier. Now, we use the 80%
training data to train the SVR and generate the final model.
Testing of the model: We have developed a model through SVR with RBF kernel
using 80% of the collected data. Testing of the developed model is required to judge the
accuracy of the model. Thus, we have tested model with remaining 20% of the collected
59
4. Modeling of Visual Search Time
data. The mean square error computed with test data for this approach is 1.17. The
computed R2, using Eqn. 4.16 [70], of this approach is 0.91. Here, V ar(y) indicates
variance of y.
R2 = 1 − MSE
V ar(y)(4.16)
4.2.4 Observation
We have developed three visual search time predictive model with three approaches
namely linear regression, nonlinear regression and Support Vector Regression. We have
tested the developed model and computed coefficient of determination that is R2, for
each approach as shown in Table 4.4.
Table 4.4: Different approaches with corresponding R2
Approach Definition R2
LinearRegression
Normal Equation θ = (XT × X)−1 × XT × y 0.39
Gradient Descent J(θ) = 12m
∑m
i=1
(hθ(x(i)) − y(i)
)20.42
Non-linearRegression
Asymptotic Regression f(x) = b1 − b2 × b3x 0.49
Gauss f(x) = b1 × (1 − b3 × exp−b2x2 ) 0.53
Log-Modified f(x) = (b1 + b3x)b2 0.59
Ratio of Quadratics f(x) = (b1+b2x+b3x2)(b4x2) 0.68
Richards f(x) = b1
(1+b3×exp−b2x)1
b40.62
Support VectorRegression
Regression usingRBF kernel
F (x) ⇒∑
i(α̂i − αi)K(xi, x) + b
0.91K(xi, xj) = eγ∥xi−xj ∥2
From the Table 4.4 we can observe that Support Vector Regression based modeling
approach performs better than other modeling approaches for our collected data.
60
4.3. Validation of the Proposed Model
4.3 Validation of the Proposed Model
We have experimented with 3 different interfaces and 42 participants to collect data
for modeling. We have modeled the collected data with three different approaches and
observed that SVR performs better than other with R2 as 0.91. However, the accuracy of
this approach can better be established if similar type of result are observed for different
interfaces and participants. In order to achieve this, an empirical study has been carried
out to determine the efficacy of the proposed model. In the study, performances predicted
by the proposed models for a set of virtual keyboards are compared with the observed
user performances. The experimental setup we have followed in the empirical study is
same as discussed earlier. We have used two new interfaces namely Fitaly, a virtual
keyboard for English and Guruji, a Hindi virtual keyboard layout attached with Guruji
search engine for these experiments.
Fitaly (a frequency based keyboard layout): The Fitaly one-finger keyboard [105] is
designed to optimize mouse movements during the text entry with one finger, a stylus or
a pen. Figure 4.1(a) shows the interface of the Fitaly keyboard. This keyboard’s name
is taken from the letter sequence along the second row of keys.
(a) Fitaly keyboard layout
(b) Guruji keyboard layout
Figure 4.1: Virtual keyboard layouts used for validation
Guruji (an alphabetical keyboard layout): A virtual keyboard is a part of Guruji
search engine interface [33], which allows users to enter queries using mouse as the input
61
4. Modeling of Visual Search Time
device. The keyboard layout in Hindi is shown in Fig. 4.1(b). The first row comprises
of Hindi vowels completely, in alphabetical order. The rest of the keyboard layout is
divided into two parts. The first part comprises of first twenty five consonants in order.
Second part is made up of the remaining consonants.
Here, we include 11 new participants in our experiments for evaluating the proposed
model. These participants are not considered in earlier experiments. The details of the
participants are shown in Table 4.5.
Table 4.5: Description of participants considered for validation
Participant’s profileNumber ofparticipantsOccupation Age
group Education Computerproficiency
Student 13–29Post Graduate Level 1 3
Under Graduate Level 1 1
Higher Secondary Level 2 2
Office staff member 31–48Graduate Level 1 2
Higher Secondary Level 3 1
Housewife 35–45 Graduate Level 1 2
We have observed visual search time for different combination of feature values from
user experiments. The corresponding visual search time for individual combination is
also calculated from the proposed model. The observed mean square error and R2
between predicted value and user performance are found as 1.31 and 0.88, respectively.
4.4 Summary
In the context of virtual keyboard interface, developing a visual search time prediction
model remains a serious problem. This chapter addresses the problem and proposes a
computational model to predict visual search time of a virtual keyboard layout from
different keyboard layout design parameter. Prior to modeling, we have performed
some experiments with user to collect knowledge about user performance on variation
of different visual search features. Then we have exercised three different modeling
62
4.4. Summary
approaches namely linear regression, non-linear regression and support vector machine
for regression. By analyzing these model performances we observe that support
vector machine for regression based modeling approach performs better than other two
approaches. Finally, to validate the developed model, we perform another set of user
based experiments with new participants and interfaces and compared user performance
with model predicted values. The proposed model is not only useful to predict visual
search time of a virtual keyboard layout, it can also be applied to design an efficient
and visual search time minimized virtual keyboard layouts. In the next chapter, we
discuss our approach to design optimum virtual keyboard taking our visual search time
prediction model into account.
63
Chapter 5
Virtual Keyboard Layout
Optimization
It is evident that visual search time to find a key greatly influence the text entry rate while
composing text through virtual keyboard. Previously, researchers have concentrated
on enhancing text entry rate by optimizing mouse movement only. However, the
development of a virtual keyboard with minimum mouse movement time as well as visual
search time incurred is desirable. In order to account visual search time in the design, a
predictive model has been developed using SVR approach, as described in Chapter 4. It
is difficult to evaluate designs on the basis of visual search time calculated by proposed
model alone, as the text entry rate significantly depends on mouse movement time
also. Hence, in order to arrive at a design methodology for virtual keyboard design,
both these measures has to be taken into account. In this chapter, we propose an
approach to design a virtual keyboard optimizing both mouse movement time and visual
search time. To judge the effectiveness of our multiobjective design approach, we design
another virtual keyboard using genetic algorithm (GA) considering mouse movement
minimization only. To inspect the efficacy of the proposed multiobjective optimized
design, we have evaluated both keyboard layouts with users and calculated text entry
65
5. Virtual Keyboard Layout Optimization
rate. The achieved text entry rate through optimized keyboard is higher than other
keyboard’s text entry rate.
This chapter consists of five sections. Our proposed approach of designing virtual
keyboard is discussed in Section 5.1. Section 5.2 describes the alternate traditional
design approach using GA. An empirical study has been carried out to show the validity
of the proposed design approach, discussed in Section 5.3. Finally, Section 5.4 contains
the summary of the chapter contents.
5.1 Proposed Virtual Keyboard Design Approach
In this work, we propose an approach to design a Bengali virtual keyboard which achieves
higher text entry rate from proper combination of different virtual keyboard design
parameters. Moreover, we consider the single tap virtual keyboard only, defining the
following assumptions:
1. All keys are of equal size and square shape.
2. Equal amount of horizontal and vertical spaces in between all adjacent keys.
3. Each key contains only one character.
4. Keyboard layout remains static over the time.
We design a multiobjective optimized Bengali virtual keyboard layout based on two
optimality metrics; mouse movement minimization and visual search time minimization.
The average movement time is measured from Fitts-digraph model [105] and visual
search time is computed by our proposed model discussed in previous chapter.
5.1.1 Problem Statement
The text entry rate in character per second (CPS), performance measurement of a virtual
keyboard, is a function of both average movement time (MT ) and visual search time
66
5.1. Proposed Virtual Keyboard Design Approach
(V ST ). So, the problem statement of maximization of text entry rate with respect to
both objectives is shown in Eqn. 5.1.
max CPS = f(MT, V ST
)(5.1)
Mean Movement Time (MT ): Mean Movement Time MT according to Fitts-digraph
model [86,105] is calculated by summing up movement time (MTij) between all digram
multiplied by corresponding digram probability (Pij), as represent in Eqn. 5.2. Here N
is the number of character present in keyboard layout.
MT =N∑
i=1
N∑j=1
(MTij × Pij) (5.2)
Movement time (MTij) from ith key to jth key is calculated from Fitts’s law [63] as shown
in Eqn. 5.3, where a and b are empirically determined constants, Dij is the Euclidean
distance between the center of both keys and Wj is width of the target key (jth key).
MTij = a + b log2
(Dij
Wj+ 1
)(5.3)
Pij , the digraph probability of occurrence of jth character after ith character, is calculated
using Eqn 5.4. Here, we analyze the Bengali Wikipedia corpus, to compute digram
probabilities of occurrences.
Pij = fij
N∑i=1
N∑j=1
fij
(5.4)
Visual Search Time (VST:) Our proposed visual search time model described in
Chapter 4 is used as a measurement of VST, which is expressed in terms of virtual
keyboard design parameters. In other words, V ST is a function of four design parameters
namely key size (KS), space between keys (SK), number of characters in the layout
67
5. Virtual Keyboard Layout Optimization
(NC) and position of the character (PC), that is,
V ST = F (KS, SK, NC, PC) (5.5)
and it is computed by combining the equations shown in Eqn. 4.14 and 4.15.
Here, we consider only square shaped keys and in the design space, font size of any
jth key is varied from 8pt to 14pt, with an increment of 2pt. Please note that in a
particular keyboard we use same font size for all keys of the keyboard. We consider
equal amount of horizontal and vertical space for any two adjacent keys in a particular
keyboard interface. In our design space, space between any two adjacent keys namely
ith key to jth key is varied 20% to 35% of the key size, with an increment of 5%. The
number of character remains same as 53, for Bengali, through out all designs and the
position of character is determined from the layout.
5.1.2 Virtual Keyboard Design using Multiobjective Optimization
There are many approaches to solve a multiobjective problem using GA like
Vector Evaluated Genetic Algorithm (VEGA) [83], Multiobjective Genetic Algorithm
(MOGA) [28], Niched Pareto Genetic Algorithm (NPGA) [40], Weight-based Genetic
Algorithm (WBGA) [34], Non-dominated Sorting Genetic Algorithm (NSGA) [87],
improved NSGA (NSGA-II) [15], Strength Pareto Evolutionary Algorithm (SPEA) [107],
Pareto-Archived Evolution Strategy (PAES) [51] etc. In our work, we use NSGA-
II algorithm, which is treated as a fast (computational complexity O(MN2); M is
number of objectives and N is number of initial population) and elitist multiobjective
GA proposed by Dev et al. [15]. It has been reported that, compared to other elitist
multiobjective GAs (PAES, SPEA etc.), it has better diversity preservation [15,48]. So,
it can compete with them in the context of converging in the true Pareto-optimal front.
Moreover, traditional approaches (MOGA, NSGA) use the concept of fitness sharing
by niching. The main problem with sharing is that it requires the specification of a
68
5.1. Proposed Virtual Keyboard Design Approach
sharing parameter and performance of the sharing function method depends largely on
the chosen sharing parameter. NSGA-II replaces the sharing function approach with a
crowded-comparison approach that eliminates difficulties. Detailed procedure of NSGA-
II is explained below.
NSGA-II algorithm follows the usual crossover and mutation operators, but selection
operator work differently from simple GA [15]. Selection is done with the help
of fast non-dominated sorting, crowded distance estimation and crowded-comparison
operator [15]. We optimize key arrangement and other virtual keyboard design
parameters by employing NSGA-II algorithm, to minimize mouse movement time along
with minimization of visual search time. Each step of our virtual keyboard design
approach using NSGA-II is illustrated in Fig 5.1.
Here, we generate 20 (N) chromosomes of each 57 bits in initial population (P ).
Initial 53 bits of a chromosome use real encoding to represent the key arrangement. The
last 4 bits of a chromosome encoded in binary form, where each of 2 bits represent key
size and space between keys, respectively.
Here, single point substring crossover technique [38] with randomly selected crossover
point in between 20 to 30 is applied in the real coded portion. Similarly, for binary
portion, we also use the single point binary crossover technique.
For mutation of real coded portion, we randomly select two point and swap each
other. Similarly, for binary portion each bit of the chromosome is mutated 0 to 1 and
vice-verse with mutation probability 0.1. The objective functions are calculated from
Fitts-digraph model [105] and our proposed model discussed in the previous chapter.
We perform ranking and crowding distance operations as described in NSGA-II
algorithm [15] to select individual for next generation. In fast non-dominated sorting,
rank is assigned to each solution according to its non-domination level [15]. Any solution
of a particular rank is not better with respect to other solutions of the same rank.
After ranking, we calculate crowding distance for each solution of all non-domination
69
5. Virtual Keyboard Layout Optimization
Figure 5.1: Flowchart of virtual keyboard design using NSGA-II algorithm
levels to get the density estimation of solutions. The crowding distance is calculated
by measuring the average distance of two nearest solution on either side for each of the
70
5.1. Proposed Virtual Keyboard Design Approach
objectives. Finally, the overall crowding distance of each solution is computed as the
sum of distance values corresponding to each objective. We use crowded comparison
operator [15] to select the solutions toward a uniformly spread-out Pareto-optimal front.
Here, if two solutions are belong to different non-domination ranks, then we select the
solution with lower rank. Also, we select the solution with lesser crowded distance, if
both solutions are in the same front.
We select 20 individuals by performing ranking and crowding distance operations
and consider them as initial population for next generation. We execute the process
iteratively up to 100 generations to find the set of optimal solutions. Pareto-optimal set
of 20 non-dominated solutions are obtained at the end of 100 generations.
Then, a Fuzzy-based approach [1] is applied to choose the single best compromise
solution from the non-dominated solutions. The ith objective function of a solution in
the non-dominated set, Fi, is represented by a membership function µi as defined in
Eqn. 5.6.
µi =
1 Fi ≤ F mini
F maxi −Fi
F maxi −F min
iF max
i < Fi < F mini
0 Fi ≥ F maxi
(5.6)
Here, the maximum and minimum values of ith objective function is denoted by F imax
and F imin, respectively. For each non-dominated solution j, the normalized membership
function µj is calculated as shown in Eqn. 5.7, where n is the number of objective
functions considered and m is the number of non-dominated solutions. Here, the value
of m and n are 2 and 20 respectively.
µj =
n∑i=1
µji
m∑j=1
n∑i=1
µji
(5.7)
71
5. Virtual Keyboard Layout Optimization
The solution having the maximum value of µj is the best compromise solution for all
objectives. Here, we use the design parameter values obtained from the best compromise
solution to design our proposed multiobjective optimized virtual keyboard. Here, values
of key size and space between keys are 12pt and 30%, respectively.
The implementation has been done using MATLAB tool in Windows 7 environment
in a PC having Pentium Core2Duo processor with 2.4 GHz clock speed. The keyboard
layout designed using the obtained feature values is depicted in Fig. 5.2.
e a
iu
o
s
Figure 5.2: Mouse movement and visual search time optimized keyboard layout
5.2 Movement Time Optimized Virtual Keyboard Design
To measure the effectiveness of optimized keyboard design, we design another keyboard
layout considering mouse movement time minimization only. We apply Genetic
Algorithm (GA) to arrange the characters in such a way that it leads to minimum mouse
movement time. Mean mouse movement time (MT ), calculated from Fitts-digraph
model [105], is used as fitness function to minimize mouse movement time (Eqn. 5.2).
We optimize key arrangement of virtual keyboard layout to minimize mouse
movement time using GA. We kept other design parameters like size, spacing etc. as
obtained from our NSGA-II based approach. As Space character is extensively used
in text composition, we place it at the central position of the keyboard having double
72
5.2. Movement Time Optimized Virtual Keyboard Design
width of any character button and exclude it from considering in key arrangement. The
arrangement of characters on keys is targeted to possess minimum number of mouse
movements. We apply a genetic algorithm-based method to find an optimal arrangement
of characters on keys. The flowchart of the approach is depicted in Fig. 5.3.
We follow the real coded ordered GA [100] to solve the above mentioned problem. We
define a chromosome which is of 53 characters length. To decide an initial candidate, we
choose a random arrangement of characters. We decide 20 such random arrangements
for each candidate and consider as the initial population in our work.
After generating the initial population, we calculate the cost values based on the
objective function for each candidate and arrange them in ascending order. We consider
rank-based selection method [84] to generate offspring for the next generation. We
divide the candidate parents into two groups; one group contains 10 individuals with
higher cost values and the rest contains in another group. Ten individuals of first
group are considered as population of next generation. However, to decide remaining 10
population, we randomly take two parents from the two different groups and perform
mating between them.
We follow the substring crossover technique [38] which prefers that a part of the
first parent should be copied in the offspring and the rest should be taken in the same
order as they appear in second parent. Let us consider any two chromosomes A and B
corresponding to two candidate parents in the current population. We consider P1 and
P2, as the crossover points in the chromosomes. The crossover mechanism copies a part
of the length |P2 − P1| from chromosome A and paste it into the child chromosome C in
between P1 and P2, both inclusive. For the rest of the parts in chromosome C, we fill
with characters in the order as they appear in the parent chromosome B. We perform
crossover operation with crossover probability 0.9. The crossover procedure we have
followed is illustrated in Fig. 5.4.
Repeating the above procedure, we construct offspring of strength 20, the population
73
5. Virtual Keyboard Layout Optimization
Figure 5.3: Flowchart of virtual keyboard design using Genetic Algorithm
size we have decided in each iteration. Next, we consider a mutation operation which
consists of swapping the locations of two characters with respect to a zone. We have
74
5.2. Movement Time Optimized Virtual Keyboard Design
……. …….
……. …….
……. …….
A
B
C
P1 P2
Figure 5.4: Crossover operation in Genetic Algorithm
carried out the mutations in the resulting chromosomes with mutation probability 0.1.
The process is executed iteratively up to 100 generations for finding the optimal
solution. After 100 iteration, chromosome corresponding to the highest rank value
is selected as the solution. We design a keyboard layout following the optimization
approach which is shown in Fig. 5.5.
e o
a
u
is
Figure 5.5: Keyboard layout minimizing mouse movement time
75
5. Virtual Keyboard Layout Optimization
5.3 Empirical Study to Evaluate Designs
We have performed an empirical study to ensure the effectiveness of our design
approaches. In this study, we compare user performance in terms of text entry rate
during text composition with those designed keyboard layouts.
We have considered 10 new participants of varying educational background having
Level 1 and Level 2 computer proficiency, as discussed previously in Chapter 3, for the
experiments. We use a text corpus of 50 Bengali sentences considering the occurrence
of almost all characters of Bengali. This corpus is created by selecting several portion
of Bengali wikipedia page.
In each experiment, each user is given with a keyboard interface to type the selected
text corpus freely. During experiments, the composed text through each keyboard are
stored into log files separately for each participant. Then, by analyzing the log files, we
calculate text entry rate in terms of words per minute as shown in Eqn. 5.8.
Text Entry Rate (WPM) = |T | − 1S
× 60w
(5.8)
Here, |T | is the length of text entered by user that is the number of characters entered.
S represents the time taken for entering text by a user, in seconds. It is measured from
the entry of the first character to the last. w denotes the average length of words for a
language, 5.11 for Bengali.
The text entry rate achieved from keyboard layout designed through NSGA-II based
approach is 6.72 words per minute. However, observed text entry rate from keyboard
layout designed through GA-based approach is 5.98 words per minute. So, the achieved
text entry rate of NSGA-II based approach is 12.37% higher than the keyboard layout
designed through GA-based approach.
76
5.4. Summary
5.4 Summary
In this chapter, we propose the design of a multiobjective optimized Bengali virtual
keyboard interface using NSGA-II to minimize mouse movement time as well as visual
search time. To judge the efficacy of the optimized keyboard, we design another alternate
virtual keyboard through traditional approach that is using GA, minimizing mouse
movement only. Then, an empirical study has been carried out with the existing virtual
keyboards on the basis of text entry rate. The study reveals that our proposed keyboard
performs better than other with respect to text entry rate.
77
Chapter 6
Summary and Conclusion
The main objective of our research is to develop a computational model for predicting
visual search time of a virtual keyboard layout, as well as we achieve some improvement
in keyboard layout design. The outcomes of our research are discussed in Chapter 3 to
Chapter 5. This chapter summarizes the major contributions of our work and future
scope of extending the research.
This dissertation contributes in modeling visual search time and demonstrates its
application toward designing virtual keyboard. Virtual keyboard, a popular text
composition interface contains different visual features which significantly affect the text
entry rate. Existing Hick-Hyman Law [37,42] to predict visual search time considers only
the number of keys present in the keyboard and lacks in acquiring other visual search
features like shape, size, grouping, ordering etc. This work addresses this limitation.
We develop a computational model to predict the average visual search time given an
object space. In the domain of user interface design, many object spaces are possible
and each of them demands their own treatment so far the perception task is concerned.
Considering this, we limit the investigation only to virtual keyboard which is a graphical
user interface to compose texts.
User interface designers advocate many visual features to be incorporated in designs
so that a user can interact with the interfaces in a better way. Impacts of many of those
79
6. Summary and Conclusion
features are subjective to users and beyond the scope of measuring them quantitatively.
It is therefore an issue to identify the visual features which are users’ specific and which
are quantitatively measurable or analyzable. In this work, we propose a methodology to
identify all such features. Our approach includes listing of visual features which might be
important in visual perception and then identification of the significant visual features
considering degree of influence. We use three virtual keyboard interfaces namely Opti, a
frequency based layout for English, Avro, an alphabetical layout for Bengali and iLipi-H
a multizonal, frequency and inflexion window based layout for composing text in Hindi,
to perform this experiment. To identify visual search time parameters which significantly
influence the performance of text entry rate, we have conducted several experiments
with 32 users; the required data are logged into file and statistically analyzed through
ANOVA test. This work establishes the fact that there are 4 out of 18 features, which
contribute more in visual search time in context of virtual keyboard. The features are
size of elements, space between elements, number of elements, and position of elements.
We study several mathematical models based on the identified features to estimate
visual search time of a virtual keyboard. To accomplish this task, we have gathered user
accomplished visual search times on different combination of feature values through
experiment. The proposed modeling is carried out by three different approaches
namely linear regression, non-linear regression and, support vector regression. In
linear regression with multiple features, we have analyzed two different approaches,
normal equation and gradient descent. Non-linear regression has been analyzed through
5 standard models among which Ratio of Quadratics model performs better than
others with our test data. Finally, visual search time model based on support vector
regression has been developed. This model is able to predict visual search time for a
virtual keyboard with different combination of design parameter values. The models
are validated with both in domain and out domain data on the basis of statistical
performance metric like mean square error (MSE) and R2. The experimental results
80
6.1. Contribution of Our Work
conclude that the proposed model performs better than linear and nonlinear models.
The accuracy of our proposed model can better be judged by comparing its performance
with different interfaces and participants. So, an empirical study has been carried out
to measure the efficacy of the proposed model. In this study, performances predicted
by the proposed model for a set of virtual keyboards are compared with the observed
user performances. Here, we have used two new layouts namely Fitaly, an English
one-finger keyboard optimizing mouse movements during the text and Guruji, a Hindi
keyboard layout attached with Guruji search engine for the experiments. We include
15 participants to evaluate the proposed model. The observed mean square error
and computed R2 between predicted value and user performance are 1.31 and 0.88,
respectively.
We design a multi objective optimized virtual keyboard using NSGA-II to minimize
mouse movement time as well as visual search time. The mouse movement time and
visual search time are measured using Fitts-digraph model and our proposed visual
search time model, respectively. To quantify the effectiveness of the proposed keyboard,
we design another virtual keyboard by employing GA to minimize mouse movement time
only. We evaluate the proposed keyboard layout with 10 different users and achieve
text entry rate of 6.72 words per minute compared to 5.98 words per minute for GA
based approach. In other words, the achieved text entry rate is 12.37% higher than the
keyboard layout designed through traditional approach.
6.1 Contribution of Our Work
Several contributions have been made in the domain of developing a visual search time
predictive model of virtual keyboards and designing multiobjective optimized virtual
keyboard. These contributions are listed below.
Visual search features identification: Several visual feature influencing design
parameters, reported in various literatures, are listed. Then, empirical study with
81
6. Summary and Conclusion
users and ANOVA tests have been carried out on those parameters which establish
that there are four parameters which have significant influence on visual search time.
The parameters are size of elements, space between elements, number of elements and
position of elements.
Visual search time model: A computational model is developed to predict visual
search time of a virtual keyboard based on the identified features. We have exercised
various mathematical approaches to develop the predictive model. We find that support
vector regression based model outperform over other models. The experiments with users
conclude that the observed mean square error and computed R2 between predicted value
and user performance result are 1.31 and 0.88, respectively.
Keyboard layout optimization: We design a virtual keyboard based on
multiobjective optimization using NSGA-II algorithm based on two optimality metrics,
minimizing mouse movement time as well as visual search time influenced by design
parameters. The mouse movement time and visual search time are measured using Fitts-
digraph model and our proposed visual search time model, respectively. The proposed
approach achieves 12.37% higher text entry rate than the keyboard layout designed
through traditional approach.
6.2 Threats to Validity
Relating to our experiments and experimental results, we would like to point out their
validity and limitations.
External validity: We have followed several user-based experiment to identify
significant visual search features as well as generate training data set. For the
experiments, we have considered 10 to 44 users with various educational backgrounds.
Further it is assumed that every users perform experiments maintaining high
concentration level. The results of the user evaluation is therefore subject to the
82
6.2. Threats to Validity
limitation on number of users involved.
Internal validity: We have not considered few visual search features in the process
of identifying significant visual search features. However, experiment can be extended
by considering features like search strategy, background-foreground color etc. In the
present work, our experiments have been restricted to desktop environments, only. It
would be another interesting matter to validate the results with small display devices
such as cell phone, iPod etc. We have considered three keyboard interfaces for our
experiments. Some other keyboard layouts like iLeap [89], Lipik [57], Gate2Home [30]
etc. can considered into evaluation.
Construct validity: The visual search time predictive model is developed through
Support Vector Regression. However, other soft computing approaches such as Simulated
Annealing (SA), Ant Colony Optimization (ACO), Particle Swarm Optimization (PSO),
Artificial Neural Network - Genetic Algorithm (ANN-GA) can be considered for
modeling. The optimal arrangement of character placing in the layout is governed
by the factors: digraph probability and NSGA-II algorithm. We have considered the
Wikipedia corpus to calculate digraph probabilities for languages aimed in this work.
It has been verified that size of the corpus are comparable to the size of other sources
such as daily newspapers. As an alternative to NSGA-II, multiobjective optimization
problem solving could be carried out with optimization methodology such as Niched
Pareto Genetic Algorithm (NPGA), Strength Pareto Evolutionary Algorithm (SPEA),
Pareto-Archived Evolution Strategy (PAES) etc. We have chosen NSGA-II because of
its better fitting with the objective function, higher convergence rate and hence faster
computation.
83
6. Summary and Conclusion
6.3 Future Scope of Work
• Proposed visual search time model can be extended for mobile devices. It will help
in developing virtual keyboard for small display devices.
• Specialized virtual keyboard can be designed for physically challenged persons.
• The proposed modeling approach can be extended to other type of interfaces such
as menu driven interface, gesture-based interface, tangible interface etc.
84
Publications out of this work
• P. K. Saha, D. Samanta, S. Sarcar and M. K. Sharma. Analysis of Visual SearchFeatures. International Journal of Human Factors Modelling and Simulation,InderScience, Vol. 3, No. 1, pages 66 − 89, 2012.
• P. K. Saha and D. Samanta. A Computational Model of Visual SearchTime, Human-centric Computing and Information Sciences, SpringerOpen (UnderRevision).
• M. K. Sharma, D. Samanta, P. K. Saha and S. Sarcar. Visual Clue: An Approachto Predict and Highlight Next Character. In Proceedings of the 4th InternationalConference on Intelligent Human Computer Interaction (IHCI), Kharagpur, India,December 27 − 29, 2012. (IEEE Xplore)
• D. Samanta, S. Ghosh, S. Dey, M. K. Sharma, S. Sarcar, P. K. Saha and S.Maiti. Development of Multimodal User Interface to Internet for Common People.In Proceedings of the 4th International Conference on Intelligent Human ComputerInteraction (IHCI), Kharagpur, India, December 27 − 29, 2012. (IEEE Xplore)
• M. K. Sharma, S. Dey, P. K. Saha, and D. Samanta. Parameters Effecting thePredictive Virtual Keyboard. In Proceedings of the IEEE Students’ TechnologySymposium, pages 268 − 275, Kharagpur, India, April 3 − 4, 2010. (IEEE Xplore)
• S. Sarcar, S. Ghosh, P. K. Saha, and D. Samanta. Virtual Keyboard Design:State of the Arts and Research Issues. In Proceedings of the IEEE Students’Technology Symposium, pages 289 − 299, Kharagpur, India, April 3 − 4, 2010.(IEEE Xplore)
85
References
[1] M. A. Abido. Multiobjective Evolutionary Algorithms for Electric Power DispatchProblem. IEEE Transactions on Evolutionary Computation, 10(3):315 – 329, 2006.
[2] D. Anderson, E. Anderson, N. Lesh, J. Marks, B. Mirtich, D. Ratajczak, andK. Ryall. Human-Guided Simple Search. In Proceedings of the 17th NationalConference on Artificial Intelligence, pages 209–216, Austin, Texas, Jul 31–Aug 22000.
[3] J. R. Anderson, M. Matessa, and C. Lebiere. ACT-R: A Theory of Higher LevelCognition and Its Relation to Visual Attention. Human Computer Interaction,12(4):439 – 462, 1997.
[4] J. L. Barbur, P. M. Forsyth, and D. S. Wooding. Visual Search, chapter EyeMovements and Search Performance, pages 253–264. Taylor & Francis, Inc.,Bristol, PA, USA, 1993.
[5] D. M. Bates and D. G. Watts. Nonlinear Regression Analysis and Its Applications.John Wiley & Sons, 2007.
[6] J. H. Bertera and K. Rayner. Eye Movements and The Span of Effective Stimulusin Visual Search. Perception & Psychophysics, 62(3):576 – 585, 2000.
[7] X. Bi, B. A. Smith, and S. Zhai. Quasi-qwerty Soft Keyboard Optimization. InProceedings of Human Factors in Computing Systems, pages 283 – 286, Atlanta,USA, 2010.
[8] A. B. Bond. The Bead Game: Response Strategies in Free Assortment. HumanFactors: The Journal of the Human Factors and Ergonomics Society, 24(1):101–110, 1982.
[9] C. J. C. Burges. A Tutorial on Support Vector Machines for Pattern Recognition.Data Mining and Knowledge Discovery, 2:121–167, 1998.
87
References
[10] M. D. Byrne, J. R. Anderson, S. Douglass, and M. Matessa. Eye Tracking theVisual Search of Click-Down Menus. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, CHI ’99, pages 402–409, Pittsburgh,United States, May 15–20, 1999.
[11] L. J. Cao and F. E. H. Tay. Support Vector Machine with Adaptive Parametersin Financial Time Series Forecasting. IEEE Transactions on Neural Networks,14(6):1506–1518, 2003.
[12] S. K. Card, T. P. Moran, and A. Newell. The Keystroke-Level Model for UserPerformance Time with Interactive Systems. Communication of ACM, 23(7):396–410, 1980.
[13] S. K. Card, T. P. Moran, and A. Newell. The Psychology of Human-ComputerInteraction. Lawrence Erlbaum Associates, Hillsdale, USA, 1983.
[14] A. Clark. Some Logical Features of Feature Integration. Neuronal Coding ofPerceptual Systems, 9(2):3–20, 2001.
[15] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A Fast and ElitistMultiobjective Genetic Algorithm : NSGA-II. IEEE Transactions on EvolutionaryComputation, 6(2):182–197, 2002.
[16] S. Deshwal and K. Deb. Design of an Optimal Hindi Keyboard for Convenient andEfficient Use. Technical Report, Indian Institute of Technology, Kanpur, 2003.
[17] M. Dorigo and L. M. Gambardella. Ant System : Optimization by a Colony ofCooperating Agents. IEEE Transactions on Systems Man Cybernetics, Part B,26(2):29 – 41, 1996.
[18] B. A. Dosher. Models of Visual Search: Finding a Face in a Crowd. An Invitationto Cognitive Science: Methods, Models and Conceptual Issues, 4(1):455–521, 1998.
[19] J. Duncan and G. W. Humphreys. Visual Search and Stimulus Similarity.Psychological Review, 96(3):433, 1989.
[20] M. D. Dunlop and J. Levine. Multidimensional Pareto Optimization ofTouchscreen Keyboards for Speed, Familiarity and Improved Spell Checking. InProceedings of Human Factors in Computing Systems, pages 2669 – 2678, Austin,USA, 2012.
88
References
[21] J. Eggers, D. Feillet, S. Kehl, M. O. Wagner, and B. Yannou. Optimization ofthe Keyboard Arrangement Problem using an Ant Colony Algorithm. EuropeanJournal of Operational Research, 148(3):672 – 686, 2003.
[22] S. P. Everett and M. D. Byrne. Unintended Effects: Varying Icon Spacing ChangesUsers’ Visual Search Strategy. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’04, pages 695–702, Vienna, Austria, Apr 24–29, 2004.
[23] J. M. Findlay and I. D. Gilchrist. Eye Guidance in Reading and Scene Perception,chapter Eye Guidance and Visual Search, pages 295 – 312. Elsevier, Amsterdam,Netherlands, 1998.
[24] J. M. Findlay and I. D. Gilchrist. Active Vision: The Psychology of Looking andSeeing. Oxford University Press, Oxford, UK, 2003.
[25] The first GUIs. Xerox 8010 Star Information System. [Online] Available:http://www.catb.org/~esr/writings/taouu/html/ch02s05.html. accessed onDecember 2011.
[26] P. M. Fitts. The Information Capacity of the Human Motor System in Controllingthe Amplitude of Movement. Journal of Experimental Psychology, 47(6):381 – 391,1954.
[27] M. D. Fleetwood and M. D. Byrne. Modeling the Visual Search of Displays:A Revised ACT-R Model of Icon Search Based on Eye-Tracking Data. Human-Computer Interaction, 21(2):153–197, 2006.
[28] C. M. Fonseca and P. J. Fleming. Genetic Algorithms for MultiobjectiveOptimization: Formulation, Discussion and Generalization. In Proceedings of the5th International Conference on Genetic Algorithms, pages 416 – 423, California,USA, 1993.
[29] W. Fu and P. Pirolli. SNIF-ACT: A Cognitive Model of User Navigation on theWorld Wide Web. Human Computer Interaction, 22(4):355 – 412, 2007.
[30] Gate2Home. [Online] Available: http://gate2home.com/Bengali-Keyboard#INBengaliInscript. accessed on August 2012.
[31] C. O. Getschow, M. J. Rosen, and Goodenough-Trepagnier. A SystematicApproach to Design a Minimum Distance Alphabetical Keyboard. In Proceedings
89
References
of 9th Annual Conference of Rehabilitation Engineering Society of North America,Minneapolis, USA, 1986.
[32] D. E. Goldberg and J. H. Holland. Genetic Algorithms and Machine Learning.Machine Learning, 3(2):95 – 99, 2005.
[33] Guruji. Indian Internet Search Engine. [Online] Available: http://www.guruji.com/hi/index.html. accessed on February 2012.
[34] P. Hajela and C. Y. Lin. Genetic Search Strategies in Multicriterion OptimalDesign. Structural and Multidisciplinary Optimization, 4(2):99 – 107, 1992.
[35] T. Halverson and A. J. Hornof. A Minimal Model of For Predicting Visual Searchin Human-Computer Interaction. In Conference on Human Factors in ComputingSystems, pages 431 – 434, CA, USA, 2007.
[36] S. A. Herd and R. C. O’Reilly. Serial Visual Search from a Parallel Model. VisionResearch, 45(24):2987–2992, 2005.
[37] W. E. Hick. On the Rate of Gain of Information. The Quarterly Journal ofExperimental Psychology, 4(1):11–26, 1952.
[38] T. P. Hong, H. S. Wang, W. Y. Lin, and W. Y. Lee. Evolution of AppropriateCrossover and Mutation Operators in a Genetic Process. Applied Intelligence,16(1):7 – 17, 2001.
[39] W. C. Hong. A Hybrid Support Vector Machine Regression for Exchange RatePrediction. International Journal of Information and Management Sciences,17(2):19–32, 2006.
[40] J. Horn, N. Nafploitis, and D. E. Goldberg. A Niched Pareto Genetic Algorithmfor Multiobjective Optimization. In Proceedings of the 1st IEEE Conference onEvolutionary Computation, pages 82 – 87, New Jersey, USA, 1994.
[41] A. J. Hornof. Visual Search and Mouse-Pointing in Labeled Versus UnlabeledTwo-Dimensional Visual Hierarchies. ACM Transactions on Computer-HumanInteraction (TOCHI), 8(3):171–197, 2001.
[42] R. Hyman. Stimulus Information as a Determinant of Reaction Time. Journal ofExperimental Psychology: General, 45(3):188–196, 1953.
90
References
[43] IBM. SPSS Statistics. [Online], Available: http://www-01.ibm.com/software/analytics/spss/products/statistics/. accessed on June 2012.
[44] B. E. John and D. E. Kieras. The GOMS Family of User Interface AnalysisTechniques: Comparison and Contrast. ACM Transaction on Computer HumanInteraction, 3(4):320–351, 1996.
[45] B. E. John and D. E. Kieras. Using GOMS for User Interface Design andEvaluation: Which Technique? ACM Transaction on Computer HumanInteraction, 3(4):287–319, 1996.
[46] B. E. John and D. D. Salvucci. Multipurpose Prototypes for Assessing UserInterfaces in Pervasive Computing Systems. IEEE Pervasive Computing, 4(4):27–34, Dec 2005.
[47] B. E. John, D. D. Salvucci, P. Centgraf, and K. Prevas. Predictive HumanPerformance Modeling Made Easy. In Proceedings of the International Conferenceon Cognitive Modeling, pages 455 – 462, Pittsburgh, Pennsylvania, 2004.
[48] V. Khare, X. Yao, and K. Deb. Performance Scaling of Multiobjective EvolutionaryAlgorithms. In Proceedings of the 2nd International Conference on EvolutionaryMulti-Criterion Optimization, pages 376 – 390, Faro, Portugal, 2003.
[49] D. E. Kieras and D. E. Meyer. An Overview of the EPIC Architecture forCognition and Performance with Application to Human-Computer Interaction.Human Computer Interaction, 12(4):391–438, Dec 1997.
[50] R. M. Klein and W. J. MacInnes. Inhibition of Return is a Foraging Facilitator inVisual Search. Psychological Science, 10(4):346 – 352, 1999.
[51] J. D. Knowles and D. W. Corne. Approximating the Nondominated Front usingthe Pareto Archived Evolution Strategy. Evolutionary Computation, 8(2):149 – 72,2000.
[52] H. H. Koester and S. P. Levine. Modeling the Speed of Text Entry with a WordPrediction Interface. IEEE Transactions on Rehabilitation Engnieering, 2(3):177–187, 1994.
[53] H. H. Koester and S. P. Levine. Keystroke Level Models for User Performance withWord Prediction. Augmentative and Alternative Communication, 13(4):239–257,1997.
91
References
[54] H. H. Koester and S. P. Levine. Model Simulation of User Performance with WordPrediction. Augmentative and Alternative Communication, 14(1):25–36, 1998.
[55] A. F. Kramer, W. R. Boot, J. S. McCarley, M. S. Peterson, A. Colcombe, andC. T. Scialfa. Aging, Memory and Visual Search. Acta Psychologica, 122(3):288–304, 2006.
[56] J. R. Lewis, P. J. Kennedy, and M. J. LaLomia. Development of a Digram-basedTyping Key Layout for Single-Finger/Stylus Input. In Proceedings of 43rd AnnualMeeting of The Human Factors and Ergonomics Society, pages 415–419, 1999.
[57] Lipik. A Predictive Text Input System. [Online] Available: http://www.lipik.in.accessed on August 2012.
[58] G. D. Logan. Attention in Character Classification Tasks: Evidence for TheAutomaticity of Component Stages. Journal of Experimental Psychology: General,107(1):32 – 63, 1978.
[59] G. D. Logan. On The Use of Concurrent Memory Load to Measure Attentionand Automaticity. Journal of Experimental Psychology: Human Perception &Performance, 5(2):189 – 207, 1979.
[60] R. H. Logie. Visuo-spatial Working Memory. Lawrence Erlbaum, Hove, UK, 1995.
[61] G. L. Lohse. A Cognitive Model for Understanding Graphical Perception. HumanComputer Interaction, 8(4):353–388, Dec 1993.
[62] S. J. Luck and E. K. Vogel. The Capacity of Visual Working Memory for Featuresand Conjunctions. Nature, 390(6657):279 – 281, 1997.
[63] I. S. MacKenzie. Fitts Law as a Research and Design Tool in Human ComputerInteraction. Human Computer Interaction, 7(1):91–139, 1992.
[64] I. S. MacKenzie and R. W. Soukoreff. Text Entry for Mobile Computing: Modelsand Methods, Theory and Practice. Human Computer Interaction, 17(2-3):147–198, 2002.
[65] I. S. MacKenzie and K. Tanaka-Ishii. Text Entry Systems: Mobility, Accessibility,Universality. Morgan Kaufmann Inc., San Francisco, CA, USA, 2007.
[66] I. S. MacKenzie and S. X. Zhang. The Design and Evaluation of a High-performance Soft Keyboard. In Proceedings of the SIGCHI Conference on Human
92
References
Factors in Computing Systems, CHI ’99, pages 25–31, Pittsburgh, United States,May 15–20, 1999.
[67] I. S. MacKenzie, S. X. Zhang, and R. W. Soukoreff. Text Entry using SoftKeyboards. Behavior & Information Technology, 18(4):235–244, 1999.
[68] E. McSorley and J. M. Findlay. Visual Search in Depth. Vision Research,41(26):3487–3496, 2001.
[69] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller.Equation of State Calculations by Fast Computing Machines. The Journal ofChemical Physics, 21(6):1087–1092, 1953.
[70] D. C. Montgomery and G. C. Runger. Applied Statistics and Probability forEngineers. John Wiley & Sons, 5th edition, 2010.
[71] A. L. Nagy, K. E. Neriani, and T. L. Young. Effects of Target and DistractorHeterogeneity on Search for a Color Target. Vision Research, 45(14):1885–1899,2005.
[72] R. Näsänen and H. Ojanpää. Effect of Image Contrast and Sharpness on VisualSearch for Computer Icons. Displays, 24(3):137–144, 2003.
[73] M. B. Neider and G. J. Zelinsky. Searching for Camouflaged Targets Effects ofTarget-Background Similarity on Visual Search. Vision Research, 46(14):2217–2235, 2006.
[74] S. Oh and M. Kim. The Role of Spatial Working Memory in Visual SearchEfficiency. Psychonomic Bulletin & Review, 11(2):275 – 281, 2004.
[75] OmicronLab. Avro Keyboard. [Online], Available: http://www.omicronlab.com/avro-keyboard.html, 2003. accessed on December 2012.
[76] J. Palmer, P. Verghese, and M. Pavel. The Psychophysics of Visual Search. VisionResearch, 40(10):1227–1268, 2000.
[77] M. Pomplun, E. M. Reingold, and J. Shen. Area Activation: A ComputationalModel of Saccadic Selectivity in Visual Search. Cognitive Science, 27(2):299–312,2003.
[78] GNU Project. GNU Octave. [Online], Available: http://www.gnu.org/software/octave/. accessed on June 2012.
93
References
[79] M. Raynal and N. Vigouroux. Genetic Algorithm to Generate Optimized SoftKeyboard. In Proceedings of Human Factors in Computing Systems - ExtendedAbstracts, pages 1729 – 1732, Portland, USA, 2005.
[80] D. D. Salvucci. An Integrated Model of Eye Movements and Visual Encoding.Cognitive Systems Research, 1(4):201 – 220, 2001.
[81] D. D. Salvucci. Predicting the Effects of In-Car Interface Use on DriverPerformance: An Integrated Model Approach. International Journal of Human-Computer Studies, 55(1):82 – 107, 2001.
[82] D. Samanta, S. Sarcar, and S. Ghosh. An Approach to DesignVirtual Keyboards for Text Composition in Indian Languages.International Journal of Human-Computer Interaction, [Online] Available:http://dx.doi.org/10.1080/10447318.2012.728483, 2012.
[83] J. D. Schaffer. Multiple Objective Optimization with Vector Evaluated GeneticAlgorithms. In Proceedings of the 1st International Conference on GeneticAlgorithms, pages 93 – 100, New Jersey, USA, 1985.
[84] R. Sivaraj and T. Ravichandran. A Review of Selection Methods in GeneticAlgorithm. International Journal of Engineering Science and Technology,3(5):3792 – 3797, 2011.
[85] A. J. Smola and B. Schölkopf. A Tutorial on Support Vector Regression. Statisticsand Computing, 14(3):199–222, 2004.
[86] W. Soukoreff and I. S. MacKenzie. Theorectical Upper and Lower Bounds onTyping Speed using a Stylus and Soft Keyboard. Behaviour & InformationTechnology, 14(6):370–379, 1995.
[87] N. Srinivas and K. Deb. Muiltiobjective Optimization using Nondominated Sortingin Genetic Algorithms. Evolutionary Computation, 2(3):221 – 248, 1994.
[88] N. A. Taatgen, H. V. Rijn, and J. R. Anderson. An Integrated Theory ofProspective Time Interval Estimation: The Role of Cognition, Attention andLearning. Psychological Review, 114(3):577 – 598, 2007.
[89] TDIL. iLeap - Multilingual Word Processor. [Online] Available: http://www.cdac.in/html/gist/products/ileap.asp. accessed on June 2011.
94
References
[90] L. Teo and B. E. John. Towards a Tool for Predicting Goal-directed ExploratoryBehavior. Human Factors and Ergonomics Society Annual Meeting Proceedings,52(5):950–954, 2008.
[91] I. Tollinger, R. L. Lewis, M. McCurdy, P. Tollinger, A. Vera, A. Howes, andL. Pelton. Supporting Efficient Development of Cognitive Models at Multiple SkillLevels: Exploring Recent Advances in Constraint-based Modeling. In Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems, CHI ’05,pages 411–420, Portland, USA, Apr 2–7 2005.
[92] T. B. Trafalis. Support Vector Machine for Regression and Applications toFinancial Forecasting. In Proceedings of the International Joint Conference onNeural Networks, pages 348–353, Norman, USA, 2000.
[93] A. M. Treisman and G. Gelade. A Feature-Integration Theory of Attention.Cognitive Psychology, 12(1):97–136, 1980.
[94] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 2nd edition,2000.
[95] J. M. Wolfe. Guided Search 2.0: A Revised Model of Visual Search. PsychonomicBulletin & Review, 1(2):202–238, 1994.
[96] J M. Wolfe. What can 1 Million Trials Tell Us About Visual Search? PsychologicalScience, 9(1):33–39, 1998.
[97] J. M. Wolfe. Guided Search 4.0: Current Progress with a Model of Visual Search.Integrated Models of Cognitive Systems, 25(3):99–119, 2007.
[98] J. M. Wolfe, K. R. Cave, and S. L. Franzel. Guided Search: An Alternative to theFeature Integration Model for Visual Search. Journal of Experimental Psychology:Human Perception and Performance, 15(3):419–433, 1989.
[99] G. F. Woodman, E. K. Vogel, and S. J. Luck. Visual Search Remains Efficientwhen Visual Working Memory is Full. Psychological Science, 12(3):219 – 224, 2001.
[100] A. H. Wright. Genetic Algorithms for Real Parameter Optimization. Foundationsof Genetic Algorithms, 1:205 – 218, 1991.
[101] C. H. Wu, J. M. Ho, and D. T. Lee. Travel Time Prediction with Support VectorRegression. IEEE Transactions on Intelligent Transportation Systems, 5(4):276–281, 2004.
95
References
[102] H. Yang, L. Chan, and I. King. Intelligent Data Engineering and AutomatedLearning, chapter Support Vector Machine Regression for Volatile Stock MarketPrediction, pages 391–396. Springer Berlin Heidelberg, 2002.
[103] P. Y. Yin, F. Glover, M. Laguna, and J. X. Zhu. Cyber Swarm Algorithms- Improving Particle Swarm Optimization using Adaptive Memory Strategies.European Journal of Operational Research, 201(2):377 – 389, 2010.
[104] P. Y. Yin and E. P. Su. Cyber Swarm Optimization for General KeyboardArrangement Problem. International Journal of Industrial Ergonomics, 41(1):43– 52, 2011.
[105] S. Zhai, M. Hunter, and B. A. Smith. Performance Optimization of VirtualKeyboards. Human Computer Interaction, 17(2):229–269, 2002.
[106] S. Zhai, P. Kristensson, and B. A. Smith. In Search of Effective Text InputInterfaces for off the Desktop Computing. Interacting with Computers, 17(3):229–250, 2005.
[107] E. Zitzler and L. Thiele. Multiobjective Evolutionary Algorithms: A ComparativeCase Study and the Strength Pareto Approach. IEEE Transactions onEvolutionary Computation, 3(4):257 – 271, 1999.
96