analysis of modified delphi card sorting - uni-paderborn.de · ed delphi card sorting experiment...

Analysis of Modified Delphi Card Sorting

by

Luqman Ahmad

Fakultat fur Elektrotechnik, Informatik und MathematikHeinz Nixdorf Institut und Institut fur InformatikFachgebiet SoftwaretechnikWarburger Straße 10033098 Paderborn

Analysis of Modified Delphi CardSorting

Master’s ThesisSubmitted to the Software Engineering Research Group

in Partial Fulfillment of the Requirements for theDegree of

Master of Science

byLuqman AhmadGuldenpfennigweg 25

33100 Paderborn

Thesis Supervisor:Prof. Dr. Gerd Szwillus

andProf. Dr.-Ing. Reinhard Keil

Paderborn, July 13, 2017

Statutory Declaration

I hereby declare that I prepared this thesis entirely on my own and have notused outside sources without declaration in the text. Any concepts or quotationsapplicable to these sources are clearly attributed to them. This thesis has notbeen submitted in the same or substantially similar version to any other authorityfor grading and has not been published elsewhere.

City, Date Signature

v

Abstract

Experts have emphasized to employ Card Sorting technique to find the qualityinformation structure according to the user’s perceptive. Modified Delphi CardSorting, not only, provides better quality results as compared to classical cardsorting methods but also reduces the time, cost and human efforts. It differs fromclassical card sorting methods in a way that, the seed participant has to create amodel from the scratch and other participants have to make changes to the modelgenerated by its predecessor, only if necessary.

All the existing analysis approaches on the results of a Card Sorting experi-ment uses either Qualitative or Quantitative approach. The aim of this work isto combine the characteristics of both approaches to analyze the results of Modi-fied Delphi Card Sorting experiment with the help of visualization. A prototypicalimplementation of this work is to be embedded into an existing web-based applica-tion MoDeCaSo which is used to conduct Card Sorting experiment using ModifiedDelphi approach.

Keywords: Card Sorting, Modified Delphi Card Sorting, MoDeCaSo, Analysis,Qualitative Analysis Approach, Quantitative Analysis Approach, Category BasedAnalysis Approach, Visualization

vii

Acknowledgment

First of all, I would like to express my deepest gratitude to Prof. Dr. Gerd Szwillusfor providing me such an interesting Master’s thesis topic. His continuous support,constructive criticism and guidance were of great help to me. His encouragementthroughout my thesis work drove me toward my goal.

I would also like to thank Peter Folta for his work that provides the basis tomy thesis.

Above all, I would like to thank my parents for their prayers and endless love.I am also thankful to my wife and my brother for supporting and encouraging meduring the whole time of my work.

ix

Contents

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Solution Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Card Sorting 52.1 Introduction to Card Sorting . . . . . . . . . . . . . . . . . . . . . 52.2 How to Conduct Card Sorting Experiment . . . . . . . . . . . . . 6

2.2.1 Select Content for Cards . . . . . . . . . . . . . . . . . . . 72.2.2 Select Type of Card Sorting . . . . . . . . . . . . . . . . . 72.2.3 Select Participants . . . . . . . . . . . . . . . . . . . . . . 72.2.4 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.5 Analyze Results . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Types of Card Sorting . . . . . . . . . . . . . . . . . . . . . . . . 92.3.1 Open Card Sorting . . . . . . . . . . . . . . . . . . . . . . 92.3.2 Closed Card Sorting . . . . . . . . . . . . . . . . . . . . . 9

2.4 Advantages and Disadvantages of Card Sorting . . . . . . . . . . 102.4.1 Advantages of Card Sorting . . . . . . . . . . . . . . . . . 102.4.2 Disadvantages of Card Sorting . . . . . . . . . . . . . . . . 11

2.5 Need of another Card Sorting Technique? . . . . . . . . . . . . . . 112.6 Modified Delphi Card Sorting . . . . . . . . . . . . . . . . . . . . 11

2.6.1 Steps of Modified Delphi Card Sorting . . . . . . . . . . . 122.6.2 Moderator . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.6.3 Seed Participant . . . . . . . . . . . . . . . . . . . . . . . 132.6.4 Benefits of Modified Delphi Card Sorting . . . . . . . . . . 13

2.7 Online (Remote) Card Sorting Tools . . . . . . . . . . . . . . . . 132.7.1 OptimalSort . . . . . . . . . . . . . . . . . . . . . . . . . . 142.7.2 WeCaSo . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.7.3 SimpleCardSort . . . . . . . . . . . . . . . . . . . . . . . . 15

2.8 Need of another Online Card Sorting Tool? . . . . . . . . . . . . . 16

3 Previous Work 173.1 MoDeCaSo (Modified Delphi Card Sorting Experiments) . . . . . 173.2 MoDeCaSo’s Architecture . . . . . . . . . . . . . . . . . . . . . . 17

3.2.1 Server Side . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2.2 Client Side . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

xi

Contents

3.3 Walkthrough of MoDeCaSo . . . . . . . . . . . . . . . . . . . . . 203.3.1 Moderators and Administrators . . . . . . . . . . . . . . . 203.3.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4 The “BBC Online” Experiment and Results . . . . . . . . . . . . 223.4.1 BBC Online . . . . . . . . . . . . . . . . . . . . . . . . . . 223.4.2 Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.4.3 Results of Experiment . . . . . . . . . . . . . . . . . . . . 23

4 Analysis 274.1 Types of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.1 Heatmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.1.2 Proximity Matrix . . . . . . . . . . . . . . . . . . . . . . . 284.1.3 Section Label Analysis . . . . . . . . . . . . . . . . . . . . 294.1.4 DBSCAN Clustering . . . . . . . . . . . . . . . . . . . . . 314.1.5 Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . 31

4.2 Analysis Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 344.3 Category Based Analysis Approach . . . . . . . . . . . . . . . . . 34

4.3.1 Overview of the Category Based Analysis Approach . . . . 354.3.2 Detailed Procedure of Category Based Analysis Approach 35

5 Prototype Implementation 495.1 Technologies Used . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.1.1 PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.1.2 Slim Framework . . . . . . . . . . . . . . . . . . . . . . . . 495.1.3 Slim-jsonAPI . . . . . . . . . . . . . . . . . . . . . . . . . 505.1.4 Basic technologies used on front end . . . . . . . . . . . . 505.1.5 Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.1.6 AJAX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.1.7 jQuery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.1.8 AngularJS . . . . . . . . . . . . . . . . . . . . . . . . . . . 515.1.9 Angular-nvd3 . . . . . . . . . . . . . . . . . . . . . . . . . 515.1.10 MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.2 Prototype’s Architecture . . . . . . . . . . . . . . . . . . . . . . . 525.2.1 Model-View-Controller . . . . . . . . . . . . . . . . . . . . 525.2.2 Front End and Back End Communication . . . . . . . . . 535.2.3 Folder Structure . . . . . . . . . . . . . . . . . . . . . . . . 535.2.4 Navigation Flow . . . . . . . . . . . . . . . . . . . . . . . 54

5.3 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.3.1 Database Structure . . . . . . . . . . . . . . . . . . . . . . 555.3.2 Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.4 Installation and Configuration . . . . . . . . . . . . . . . . . . . . 565.5 Application Walkthrough . . . . . . . . . . . . . . . . . . . . . . . 56

5.5.1 User Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.5.2 Project Configuration . . . . . . . . . . . . . . . . . . . . . 57

xii

Contents

5.5.3 View Results . . . . . . . . . . . . . . . . . . . . . . . . . 585.5.4 View/Edit Messages . . . . . . . . . . . . . . . . . . . . . 605.5.5 Analysis Process . . . . . . . . . . . . . . . . . . . . . . . 615.5.6 View Analysis Results . . . . . . . . . . . . . . . . . . . . 695.5.7 Export Data (JSON or CSV Form) . . . . . . . . . . . . . 70

6 Evaluation 756.1 Choosing Project - “BBC Online” . . . . . . . . . . . . . . . . . . 756.2 Role of Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . 756.3 Results of the Analysis . . . . . . . . . . . . . . . . . . . . . . . . 756.4 Analysis Strategy to get the Resulting Model . . . . . . . . . . . . 77

7 Summary and Future Work 797.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.2.1 Improvement in Analysis Approach . . . . . . . . . . . . . 807.2.2 Importing data from other tools for analysis . . . . . . . . 807.2.3 Standalone application . . . . . . . . . . . . . . . . . . . . 807.2.4 Refine Categories before Analysis . . . . . . . . . . . . . . 80

Bibliography 81

xiii

List of Acronyms

AJAX Asynchronous JavaScript and XML

API Application Programming Interface

BBC British Broadcasting Corporation

CS Card Sorting

CSS Cascading Style Sheets

CSV Comma Separated Values

DBSCAN Density-based spatial clustering of applications with noise

DOM Document Object Model

HTML Hypertext Markup Language

JSON JavaScript Object Notation

MoDeCaSo Modified Delphi Card Sorting Experiments

MVC Model-View-Controller

PHP Hypertext Preprocessor

SLA Section Label Analysis

SPA Single-page Application

URL Universal Resource Locator

XML Extensible Markup Language

xv

List of Figures

2.1 Example of Card Sorting, an organized bookshelf . . . . . . . . . 52.2 Snapshot of BBC website . . . . . . . . . . . . . . . . . . . . . . . 62.3 An example of card . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4 Execution of Card Sorting experiment . . . . . . . . . . . . . . . 82.5 Example of possible outcome for a participant . . . . . . . . . . . 82.6 Example of Open Card Sorting . . . . . . . . . . . . . . . . . . . 92.7 Example of Closed Card Sorting . . . . . . . . . . . . . . . . . . . 102.8 Example workflow of Modified Delphi Card Sorting . . . . . . . . 122.9 Demo of Card Sorting using OptimalSort . . . . . . . . . . . . . . 142.10 Demo of Card Sorting using WeCaSo . . . . . . . . . . . . . . . . 152.11 Demo of Card Sorting using SimpleCardSort . . . . . . . . . . . . 15

3.1 MoDeCaSo’s Logo . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2 Project Configuration Page . . . . . . . . . . . . . . . . . . . . . 213.3 MoDeCaSo’s workspace . . . . . . . . . . . . . . . . . . . . . . . . 213.4 BBC Online experiment’s Cards . . . . . . . . . . . . . . . . . . . 23

4.1 Example of Heatmap . . . . . . . . . . . . . . . . . . . . . . . . . 274.2 Example of Proximity matrix . . . . . . . . . . . . . . . . . . . . 284.3 Example of Section Label Analysis . . . . . . . . . . . . . . . . . 294.4 Example of DBSCAN Analysis . . . . . . . . . . . . . . . . . . . 304.5 Types of distances between clusters . . . . . . . . . . . . . . . . . 324.6 Clustering in Single, Complete and Group Average Linkage . . . . 324.7 Example of Dendogram . . . . . . . . . . . . . . . . . . . . . . . . 33

5.1 MVC Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.2 Folder Structure of Application . . . . . . . . . . . . . . . . . . . 535.3 Navigation Flow of the work . . . . . . . . . . . . . . . . . . . . . 545.4 Database Structure . . . . . . . . . . . . . . . . . . . . . . . . . . 555.5 MoDeCaSo’s Login Page . . . . . . . . . . . . . . . . . . . . . . . 575.6 Project Configuration Page . . . . . . . . . . . . . . . . . . . . . 585.7 View Results Page . . . . . . . . . . . . . . . . . . . . . . . . . . 595.8 View of a Participant’s Model Page . . . . . . . . . . . . . . . . . 595.9 Information Messages . . . . . . . . . . . . . . . . . . . . . . . . . 605.10 Edit Messages dialogue . . . . . . . . . . . . . . . . . . . . . . . . 615.11 View of Information Dialogue on Category Analysis Page . . . . . 625.12 Category Analysis Page . . . . . . . . . . . . . . . . . . . . . . . 635.13 Selection of Categories for final model . . . . . . . . . . . . . . . . 64

xvii

List of Figures

5.14 View of Information Dialogue on Suggested Solution Page . . . . 655.15 Warning for unnecessary selected categories . . . . . . . . . . . . 655.16 View Suggested Solution Page . . . . . . . . . . . . . . . . . . . . 665.17 Drag-and-drop of unsorted card . . . . . . . . . . . . . . . . . . . 675.18 Add comments dialogue . . . . . . . . . . . . . . . . . . . . . . . 685.19 Thank you dialogue . . . . . . . . . . . . . . . . . . . . . . . . . . 685.20 View Analysis Results Page . . . . . . . . . . . . . . . . . . . . . 695.21 View of an analyzed Model . . . . . . . . . . . . . . . . . . . . . . 70

xviii

List of Tables

3.1 Core technologies used by MoDeCaSo on the server side . . . . . . 183.2 PHP Frameworks used by MoDeCaSo on server side . . . . . . . . 193.3 Frameworks used by MoDeCaSo on client side . . . . . . . . . . . 193.4 Resulting models of participant 1,2 and 3 . . . . . . . . . . . . . . 243.5 Resulting models of participant 4 and 5 . . . . . . . . . . . . . . . 25

5.1 Database Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.1 Resulting model after applying Category Based Analysis Approach 76

xix

Listings

4.1 Example of a participant’s model . . . . . . . . . . . . . . . . . . 364.2 Example of re-arranged model of a participant . . . . . . . . . . . 374.3 Part of re-arranged model of a participant . . . . . . . . . . . . . 404.4 Example model of participants related to a specific category . . . 404.5 Example model of participants with the change in category . . . . 424.6 Example of graph data model . . . . . . . . . . . . . . . . . . . . 43

5.1 Example of exported data in CSV . . . . . . . . . . . . . . . . . . 705.2 Example of exported data in JSON . . . . . . . . . . . . . . . . . 71

xxi

1 Introduction

Finding an accurate logical structure for organizing the content of a product isalways a daunting task for the developers. An ill-structured information will notonly give a negative impression of the product to the customer but also prohibitthem from using it in future. One approach to overcome this problem is to useCard Sorting (CS) which is also recommended by the experts [Spe04].

CS is an approach to structure the information into logical groups. If CS hasbeen used, it can provide a positive feeling to the customer because the informa-tion will be well structured and easy to find without taking so much valuable timeof the customer.

CS has two main variants, namely “Open Card Sorting” and “Closed Card Sort-ing”. Closed CS is used to validate an existing structure and Open CS is used toget the initial structure of an information architecture. Open CS is the preferredway as it gives more freedom to participants in terms of creating new categorieson their own and choosing cards for these categories just as they think which, inturn, bringing out the participant’s own perspective.

Open CS is often criticized because this method is very time consuming bothin terms of conducting an experiment and later analyzing the results of the ex-periment. Small scale companies, for conducting this type of experiment, haveto bear the cost of many participants, moderator and lab which can easily crosstheir budget. To overcome these flaws, Modified Delphi Card Sorting can be usedbecause it does not require so many participants for an experiment and the out-put generated by one participant is handed over to next participant. Hence, nextparticipant does not need to model everything from the scratch that saves theconsiderable amount of time as compared to Open CS.

1.1 Motivation

Every CS experiment results in the mental model of participants which can bedifferent from each other. Each obtained model reflects different view of the dif-ferent participants but only one model has to be used for the final structure of theproduct. This raises a question that out of these several models which one shouldbe used as the final model? A simplest and most obvious answer for this questionis to use any model at random. Using this approach does not guarantee to pro-

1

1. Introduction

vide the best model to be used for the product. Another approach is to analyzeall the resulting model of participants and take the best combination out of it.This approach is superior to the former approach in terms of that it promises tofind the best model, which is the combination of all the views which are gatheredduring the experiment.

Analysis of any CS experiment can be either “Qualitative” or “Quantitative”.Qualitative analysis is to “... looking for patterns in the sorts”[Naw12] and quan-titative analysis is to “use different tools to interpret the users’ sorting. Thesetools use algorithms such as cluster analysis and similarity matrixes to arrive atan interpretation”[Naw12].

There are many tools available, both online and offline, for analyzing the resultsgenerated by the CS experiment and each one of them uses either one of the above-mentioned approaches. These two approaches are matured enough to deal withtheir own set of data but the use of only one approach restricts to be benefited fromthe other approach. This restriction raises a need for having a hybrid approachwhich combines both approaches into one approach. This hybrid approach willhelp to analyze models of participants more effectively and that will increase thereliability of the final model after analysis.

1.2 Solution Idea

The objective of this thesis is to provide a hybrid analysis approach for ModifiedDelphi Card Sorting which tracks the changes of categories with respect to partic-ipants. This approach plots these category changes using a line graph to provide abetter visualization of the results. Unlike other approaches, this approach involvesanalyzer to pick the categories for the final output model by looking at the trendof the categories. To complete the process of analysis, this approach must be ableto suggest a final model to the analyzer based on the categories selected by theanalyzer. The analyzer must have the freedom to make changes in the suggestedmodel, if necessary, before finalizing the model.

1.3 Overview

This thesis is organized into a number of chapters. The chapters are organized ina systematic order to get a meaningful concept.

• Card Sorting

This chapter explains CS in detail to form the basis of this work. It discussesdifferent types of CS and different available tools for CS to justify the use

2

1.3 Overview

of Modified Delphi Card Sorting approach and MoDeCaSo, an online webapplication that is used to conduct the CS experiment using Modified Delphiapproach.

• Previous Work

This thesis is an extension of the work done by Peter Folta in his thesis.He has created a web application named as MoDeCaSo that can be usedto conduct the CS experiment using Modified Delphi approach. This chap-ter briefly describes the architecture and walkthrough of the application.This also presents the results of a real-time experiment conducted using thisapplication.

• Analysis

This chapter illustrates different types of analysis tools and analysis ap-proaches to justify a new approach which is the actual work of this thesis.It also states and explains the newly proposed approach in detail.

• Prototype Implementation

This chapter documents the technologies used to implement the prototypefor the proposed analysis. It also talks about the architecture of the proto-type and in the end a complete walkthrough of the implemented prototypeis presented.

• Evaluation

This chapter evaluates the implemented prototype and reports the results.

• Summary and Future Work

This chapter presents the summary of the whole work and suggests exten-sions that can be made in the future.

3

2 Card Sorting

This thesis is all about to analyze the results of CS experiment so understandingof CS is imperative. This chapter explains CS and its variants in details.

2.1 Introduction to Card Sorting

Card Sorting (CS) is a technique to organize information into groups. Historyof CS is very old as categorizing the knowledge is something humans have beendoing for centuries.

CS was originally used by psychologists as a formal technique to study that howpeople categorize their knowledge [WW08][Hud14].

Figure 2.1: Example of Card Sorting, an organized bookshelf [pin]

One can think a bookshelf as an example of CS. Figure 2.1 shows an organizedbookshelf. It might be organized by genre, author, size or number of pages butwhatever scheme is used to organize that bookshelf, when a person needs a book,it is easy to find that because of sorting that is applied on the bookshelf.

Another sophisticated example of CS, for better understanding, is to consider

5

2. Card Sorting

the website of British Broadcasting Corporation (BBC)1. According to [BBC],this website has several million pages. Figure 2.2 shows the snapshot of the BBCwebsite in which data items are organized into appropriate categories. Due to thisorganization of data into categories, the user will directly visit the Weather Pageto see the “Weather”.

Figure 2.2: Snapshot of BBC website

According to [Spe04], “Card sorting is a great, reliable, inexpensive method forfinding patterns in how users would expect to find content or functionality.”

In this digital age, CS is mostly used by information architects or professionalsto either get a structure of the computer application or to improve the existingstructure.

[Spe04] describes that CS is a simple, quick and reliable approach that helpsto generate the overall structure of information. It also suggests for navigation,menus, and possible taxonomies. As CS is based on real user input so it helpsto find the patterns that how a user wants to see the structure or functionality.These patterns are referred as “Mental Model” of users. Using these mental modelof users, the structure of a website or computer application can be achieved orcan be improved.

2.2 How to Conduct Card Sorting Experiment

Performing a CS experiment is fairly straight forward. A set of cards are given tothe participants and they pile cards into categories. According to [Gha15], core

1BBC: http://www.bbc.com/

6

http://www.bbc.com/

2.2 How to Conduct Card Sorting Experiment

steps of CS are as follows.

2.2.1 Select Content for Cards

The first step is to determine the contents. A list of contents should be made andeach item on this list should be placed on a card. Contents for the cards can befunctionality or individual pages.

Figure 2.3: An example of card - own image

Figure 2.3 illustrates a sample card. A card should have a unique ID to dis-tinguish from other cards. Card title is very important. It should be short butself-explanatory so that participants can understand the content properly but ifneeded, a short description can be added for explanation. In manual CS experi-ment, small pieces of paper are used as cards.

There are many different point of views that how many cards should be used.[Spe04] describes that number of cards should be in between 30 to 100. As below30 does not allow enough merging and over 100 will be a lot of waste of time.

2.2.2 Select Type of Card Sorting

According to the need, type of CS can be selected. Types of CS are defined innext section 2.3.

2.2.3 Select Participants

Participants should come from the user group. They actually are or will be theusers of the product for which the mental model of users is required. The numberof participants can vary according to need. [Nie04a] suggests that 15 participantsare good whereas many other researchers including [TW04] suggest between 25 to30 participants.

7

2. Card Sorting

2.2.4 Execution

After having cards and participants, the CS experiment can be executed. Par-ticipants should be explained in plain language that how the experiment works.They should be in a comfortable environment.

Figure 2.4: Execution of Card Sorting experiment [Yan].

Figure 2.5: Example of possible outcome for a participant [Gha15].

Figure 2.4 shows the execution of a manual CS experiment and Figure 2.5 showsthe possible outcome of an experiment in the form of cards and categories for aparticipant.

8

2.3 Types of Card Sorting

2.2.5 Analyze Results

After the completion of a CS experiment, the bulk of data will be received andcan be analyzed with different existing approaches to get the final model of theproduct.

2.3 Types of Card Sorting

There are different types of CS to perform that yields different results. Choosingthe right type of CS is crucial as it would effect the outcome of the project. Thereare two primary variants of CS, Open Card Sorting and Closed Card Sorting.These two variants are discussed in the following sections.

2.3.1 Open Card Sorting

In Open Card Sorting, participants of the CS experiment are given a set of cardsand asked to group them according to their own understanding. They make thelabel for the group which they have created. The Label is commonly known as“Category” in CS experiments.

Figure 2.6: Example of Open Card Sorting [Sin13].

Figure 2.6 describes an Open Card Sorting method in which the whole process isdescribed for open card sorting. Open Card Sorting gives freedom to participantsto make group of cards and name that group.

2.3.2 Closed Card Sorting

In Closed Card Sorting, participants are provided a set of cards and predeterminedgroup names (Label, Category). Then they are asked to put cards into predefinedcategories.

Figure 2.7 describes the whole process of Closed Card Sorting method. ClosedCard Sorting also gives freedom to participants to make group of cards but it

9

2. Card Sorting

Figure 2.7: Example of Closed Card Sorting [Sin13].

does not allow them to name these groups as categories are predefined by theexperiment creator.

2.4 Advantages and Disadvantages of Card Sorting

There are advantages and disadvantages of CS so these pros and cons should bekept in mind before using CS for an experiment. According to [Adv], [Pro16] and[Spe04], some of the advantages and disadvantages are described in the followingsections.

2.4.1 Advantages of Card Sorting

• Simple CS is very simple and easy for participants and organizers.

• Cheap It is cheap as only few things are required for whole experiment e.g,pen, paper etc.

• Established It is used by professionals from more than 10 years.

• User focused Experiment gives the insight of real user mental model.

• Provides a good foundation The results might not be the final one butit provides good foundation about the structure of the product.

10

2.5 Need of another Card Sorting Technique?

2.4.2 Disadvantages of Card Sorting

• Varied results Results of the experiment may vary widely between partic-ipants.

• Analysis can be time consuming Analysis of results is very time con-suming.

• May not deep enough If the user can not place data into context then itgives only top level analysis.

2.5 Need of another Card Sorting Technique?

In general, Closed CS is used to validate an existing structure and Open CS is usedto get the initial structure of an information architecture. Open CS gives freedomto participants that they can create their own categories whereas, in Closed CS,participants do not have much freedom because of predefined categories. Thisfreedom makes Open CS one of the strongest methods to draw out the mentalmodels of participants [Pau08].

As describes in an article [Pau08], a major drawback of Open CS is that some-times it is very time consuming. How many participants should be included ina CS experiment is under debate as some researchers suggest 10 to 15 and somefrom 20 to 30 [Nie04a] [TW04]. [TW04] states that 20 to 30 participants shouldbe included to get a meaningful result in an Open CS experiment.

With so many participants involved in an Open CS experiment with a set of50 to 100 cards, it takes too much time to analyze the resulting data. There aresome more factors like renting a lab, providing moderator and participants feesmake Open CS experiment too much expensive for small companies as well as foran individual [Pau08].

An informal poll result shows that many experts are using 6 to 12 participantsfor CS experiment for avoiding the above given drawbacks and still getting goodresults [Pau08].

Therefore a need for another CS technique is there as the replacement of anOpen CS technique with 6 to 12 participants that works well with results andmore efficient in time and cost.

2.6 Modified Delphi Card Sorting

“Celeste Lyn Paul” [Pau08] invented a new technique for CS that is named as“Modified Delphi Card Sorting” to overcome the flaws of Open CS. Modified

11

2. Card Sorting

Delphi Card Sorting is similar to Open CS as participants have same the freedomas they have in Open CS. They can change category names and can put cardsto any categories according to their own mental model. The difference is thatin Modified Delphi Card Sorting, participants are given the combined results ofprevious participants and they can modify it after reviewing it which makes thistechnique time efficient.

2.6.1 Steps of Modified Delphi Card Sorting

Modified Delphi Card Sorting has four steps [Pau08] which are also shown in theFigure 2.8.

1. The seed participant (first participant) creates the initial model from givencards.

2. Following participants are given the previous participant’s model and theycan modify it after reviewing.

3. The card structure will be changed during the process and at the end amodel will be achieved having inputs from all the participants.

4. A consensus is reached when the model is stabilized and no big changesoccur.

Figure 2.8: Example workflow of Modified Delphi Card Sorting [Pau08].

12

2.7 Online (Remote) Card Sorting Tools

2.6.2 Moderator

The whole process is managed and monitored by a Moderator. Moderator col-lects the previous model and represents it to the new participant. Moderator isvery important because of direct connection with all the participants and havingcontrol over models and object view of information. Moderator should remainneutral after having object view of information even if it conflicts moderator’sown personal interest.

2.6.3 Seed Participant

Seed participant has a significant impact on Modified Delphi Card Sorting as seedparticipant has to generate an initial model from scratch that takes a considerableamount of time as compared to other participants. Since this initial model issupplied to other participants, initial model can have a certain impact on modelsgenerated by following participants. Seed participant can be a single participant,can be two participants or can be a group of participants [SC13].

2.6.4 Benefits of Modified Delphi Card Sorting

According to the inventor of Modified Delphi Card Sorting, some of the benefitsare as follows [Pau08].

• Result It improves the quality of results of all participants as comparedto Open CS because Modified Delphi Card Sorting works only on a singlemodel.

• Time By using this practice, a huge amount of time can be saved duringthe experiment and even during the analysis.

• Cost By reducing the huge amount of time, cost of the experiment is alsoreduced.


Online CS tools are very popular these days. By using these tools, cost of arrang-ing a laboratory can be saved and it is easy to use instead of working with theclassical pen and paper based CS.

[Sin13] states that online CS tools should be used instead of manual CS becausenowadays a website or a web service is used by all over the world unless a websiteor a web service is geological focused. For a CS experiment on an application thatis used by all over the world, participants should be from all over the world. Soonly with the help of online CS tools, these experiments can be done to get the

13

2. Card Sorting

real user input.

Here are some of the online CS tools which are being used for conducting CSexperiments.

2.7.1 OptimalSort

OptimalSort2 [Opt] is a high quality CS online application that supports Open andClosed CS. This also provides real-time results and graphs for analysis. Figure 2.9shows a demo of CS using OptimalSort.

Figure 2.9: Demo of Card Sorting using OptimalSort [Opt].

2.7.2 WeCaSo

WeCaSo3 [WeC] is an online CS web application that supports Open and ClosedCS. Figure 2.10 shows a demo of CS using WeCaSo. WeCaSo was developedin Paderborn University4 as a bachelor thesis in which CS experiment can beperformed in different languages.

2OptimalSort: https://www.optimalworkshop.com/optimalsort3WeCaSo: http://wecaso.de/4Paderborn University: https://www.uni-paderborn.de/

14

https://www.optimalworkshop.com/optimalsort

http://wecaso.de/

https://www.uni-paderborn.de/


Figure 2.10: Demo of Card Sorting using WeCaSo [WeC].

2.7.3 SimpleCardSort

SimpleCardSort5 [Sim] supports Open and Closed CS with analysis. Figure 2.11shows a demo of CS using SimpleCardSort

Figure 2.11: Demo of Card Sorting using SimpleCardSort [Sim].

5SimpleCardSort: https://www.simplecardsort.com/

15

https://www.simplecardsort.com/

2. Card Sorting

2.8 Need of another Online Card Sorting Tool?

A lot of web-based applications exist in which some of them have been mentionedin the previous section and they all support Open and Closed CS with manyfeatures but an online application that supports Modified Delphi Card Sorting isvery rare.

Tim Kuhl [Kuh14] has to do manual experiment on Modified Delphi Card Sort-ing during his bachelor thesis as he was doing an experiment to compare OpenCS with Modified Delphi Card Sorting.

Due to lack of available CS application based on Modified Delphi technique,Peter Folta has developed, during his Bachelor Thesis [Fol15], a web applicationthat can be used to conduct a CS experiment using Modified Delphi approachwhich is explained in the next Chapter 3.

16

3 Previous Work

This thesis is an extension of Peter Folta’s Bachelor Thesis [Fol15] in which he hasdeveloped a web application that can be used to conduct a CS experiment usingModified Delphi approach. This chapter will explain briefly about Peter Folta’swork and results which are the initial point of this thesis that is to implement andanalyze the generated results of this web application with our approach.

3.1 MoDeCaSo (Modified Delphi Card SortingExperiments)

Peter Folta has developed a web application named as Modified Delphi CardSorting Experiments (MoDeCaSo). Using this application, a CS experiment usingModified Delphi approach can be conducted.

Figure 3.1: MoDeCaSo’s Logo

3.2 MoDeCaSo’s Architecture

MoDeCaSo’ architecture is divided into two parts:

• Back end (Server Side)

• Font end (Client Side)

3.2.1 Server Side

According to [Fol15], server side of MoDeCaSo is divided into three components:

1. Web application API

17

3. Previous Work

2. Central database

3. Maintenance service, which runs regardless of any user or client interaction.

The purpose of database is to store data used by MoDeCaSo permanently andweb application APIs are to handle all client’s requests. Maintenance service,which runs always in the background regardless of any client interaction, is tosend reminder through emails to participants for time limit or for inviting the nextparticipant to participate when previous participant’s time is over. Maintenanceservice communicates with web application APIs to perform these tasks.

Technology Description

Debian Debian1 is free Linux-based operating systemand among the most popular operating sys-tem used on web servers worldwide. It islicensed under the GNU General Public Li-cense [Deb].

Apache HTTP Server Apache HTTP Server2 is the most used webserver software. It can be used for many op-erating systems but mostly used for Unix-likeoperating system [Apa].

PHP PHP3 is a server scripting language that iswidely used for web development and can beembedded into HTML [PHP].

MySQL Server MySQL4 is open-source relational databasemanagement system. It is widely used withPHP for web applications [MyS].

Table 3.1: Core technologies used by MoDeCaSo on the server side

Table 3.1 shows the core technologies used by MoDeCaSo on the server side. Ituses Debian operating system and Apache HTTP Server as a web server. It usesUnix cron job for maintenance services. From version 5.0, Hypertext Preproces-sor (PHP) supports object-oriented programming and widely used with MySQLdatabase. Using these technologies make MoDeCaSo attractive and powerful.

1Debian: https://www.debian.org/2Apache: https://httpd.apache.org/3PHP: http://php.net/4MySQL: https://www.mysql.com/

18

https://www.debian.org/

https://httpd.apache.org/

http://php.net/

https://www.mysql.com/

3.2 MoDeCaSo’s Architecture

Framework Description

Slim Framework Slim Framework5 is a PHP micro frameworkused to create powerful web applications andAPIs [Slib].

Slim-JSON API To implement JSON APIs using Slim.

Table 3.2: PHP Frameworks used by MoDeCaSo on server side

Table 3.2 lists frameworks used in MoDeCaSo. As MoDeCaSo uses PHP envi-ronment on the server side so using Slim as a PHP framework make MoDeCaSomore reliable, efficient and secure as in frameworks many programming tasks havealready been solved and are given to utilize with minimum time efforts. The SlimFramework simplifies providing an HTTP API over GET and POST requests.Slim-JSON API is an efficient extension of the Slim Framework used for conver-sion of JavaScript Object Notation (JSON) objects into PHP objects and viceversa as MoDeCaSo uses JSON objects to communicate between the front endand back end.

3.2.2 Client Side

As illustrated in [Fol15], MoDeCaSo’s client side is a modern Hypertext MarkupLanguage (HTML) web application that uses Javascript and Asynchronous JavaScriptand XML (AJAX) for providing an efficient and desktop-like user experience.

MoDeCaSo is a Single-page Application (SPA) that runs in a single HTML pagewhich is loaded initially on request and all other contents are loaded on this pagewithout loading a full page. This provides a fresh and desktop-like look and feel.

Framework Description

Bootstrap Bootstrap6 is a open source free framework. Itcontains HTML, CSS and Javascript templates fordesigning web applications [Boob].

AngularJS AngularJS7 is an open source Model-View-Controller Javascript framework used for dynamicviews in web applications [Angb].

jQuery jQuery8 is Javascript library enables to doJavascript tasks in less time and in efficientway [jQu].

Table 3.3: Frameworks used by MoDeCaSo on client side

5Slim: https://www.slimframework.com/

19

https://www.slimframework.com/

3. Previous Work

Table 3.3 shows frameworks used by MoDeCaSo on client side. Bootstrap isused to give MoDeCaSo a modern web application look, AngularJS is a Model-View-Controller (MVC) framework that is used to build SPA and jQuery providesdesktop-like user interaction with web application. MoDeCaSo uses jQuery toprovide drag-and-drop functionality during CS experiment.

3.3 Walkthrough of MoDeCaSo

This section demonstrates the walkthrough of MoDeCaSo. This is necessary forthis thesis as one should know that how CS experiment is done with ModifiedDelphi approach and at the end how results are gathered from the CS experi-ment using MoDeCaSo. This section will provide only a short introduction ofMoDeCaSo usage and describe concisely that how a CS experiment can be doneusing MoDeCaSo.

As three types of users can access MoDeCaSo so walkthrough of MoDeCaSowill be explained according to the user roles.

3.3.1 Moderators and Administrators

Moderators are the users who have the rights to log into MoDeCaSo and can starta new CS experiment. They have the full control on their own created projects.

Administrators have all the rights of moderators but, in addition, they have fullcontrol of all the projects in MoDeCaSo. They can also have the rights to see allusers of MoDeCaSo and can add new moderators and administrators.

3.3.1.1 Project Configuration

After logging into MoDeCaSo, moderators or administrators can create a newproject for CS experiment. After creating a new project, they can add, edit,import or export cards for the experiment in Cards Tab. In Participants Tab,moderators or administrators could add, edit or delete participants for the exper-iment. First added participant will be seed participant of the experiment. More-over, in Messages Tab and Project Setting Tab, messages can be edited accordingto project and time to complete participation in the experiment can be set. Aftersetting all the requirements, moderators or administrators could start the projectby clicking on Start Project button. When a project is in running state, seedparticipant will be invited through an email to participate in the project and toperform CS experiment. Figure 3.2 shows the view of project configuration page.

6Bootstrap: http://getbootstrap.com/7AngularJS: https://angularjs.org/8jQuery: https://jquery.com/

20

http://getbootstrap.com/

https://angularjs.org/

https://jquery.com/

3.3 Walkthrough of MoDeCaSo

Figure 3.2: Project Configuration Page

3.3.1.2 View Results

Moderators or administrators can see submitted models of all participants byclicking on View Results button.

3.3.2 Participants

Figure 3.3: MoDeCaSo’s workspace

Participants can not log into MoDeCaSo. They are invited to participate inModified Delphi Card Sorting experiment through email. After completing their

21

3. Previous Work

submission, they can not access MoDeCaSo until they are invited for another ex-periment.

Participants can access into MoDeCaSo by clicking on an invitation link whichis sent to them by email. After accessing to MoDeCaSo, they can add categoriesand can drag and drop unsorted cards to categories in order to create a sortedmodel. After creating a sorted model, they can click on Save and Submit buttonto complete their submission. Figure 3.3 shows the MoDeCaSo’s workspace page.

3.4 The “BBC Online” Experiment and Results

As this thesis is to analyze results of Modified Delphi Card Sorting experiment,therefore, the need of results of this experiment is compulsory. After getting theseresults, the analysis can be applied to these results.

3.4.1 BBC Online

Peter Folta has evaluated MoDeCaSo in real-time by doing a project named as“BBC Online” which is based on the data of BBC9 website. This website hasseveral million pages [BBC].

Participants of this experiment were given 4 days to complete their participa-tion.

3.4.2 Cards

32 cards were selected for this experiment. These cards are pages which are di-rectly available on BBC’s website.

Figure 3.4 on Page 23 shows all cards selected for the experiment. These cardsonly show a small selection of pages available on BBC’s website.

9BBC: http://www.bbc.co.uk/

22

http://www.bbc.co.uk/


Figure 3.4: BBC Online experiment’s Cards

3.4.3 Results of Experiment

7 participants were invited for the participation of this experiment and 5 of themhave successfully submitted their personal hierarchical model using MoDeCaSo.Table 3.4 and 3.5 on Page 24 and Page 25 shows the resulting models of par-ticipants which have been submitted successfully. Text in Red color shows thecategory and text in black color shows the card.

23

3. Previous Work

Participant 1 Participant 2 Participant 3

NewsBusinessScienceTechUKWorldSportsAthleticsCricketCyclingFootballFormula 1GolfRugby UTennisNatureBritish WildlifeEarth on TV & RadioLife through the LensMediaEntertainment & ArtsFilmGamesMusicCookingChefsHealthIngredientsRecipesTechniquesLifestyleMagazineStyleTravelDestinationsIn PicturesInterestsKidsMake & ColourStory Time

NewsBusinessScienceUKWorldSportsAthleticsCricketCyclingFootballFormula 1GolfRugby UTennisNatureBritish WildlifeEarth on TV & RadioLife through the LensMediaEntertainment & ArtsFilmMusicFoodChefsIngredientsRecipesTechniquesLifestyleHealthInterestsMagazineStyleTechTravelDestinationsIn PicturesKidsGamesMake & ColourStory Time

NewsBusinessUKWorldSportsAthleticsCricketCyclingFootballFormula 1GolfRugby UTennisNatureBritish WildlifeLife through the LensScienceMediaEntertainment & ArtsFilmMusicFoodChefsIngredientsRecipesTechniquesLifestyleHealthInterestsMagazineStyleTechTravelDestinationsEarth on TV & RadioIn PicturesKidsGamesMake & ColourStory Time

Table 3.4: Resulting models of participant 1,2 and 3

24


Participant 4 Participant 5

WorkBusinessChefsScienceTechSportsAthleticsCricketCyclingFootballFormula 1GolfRugby UTennisNatureBritish WildlifeLife through the LensMediaEntertainment & ArtsFilmMusicFoodIngredientsRecipesTechniquesLifestyleHealthInterestsMagazineStyleTravelDestinationsEarth on TV & RadioIn PicturesUKWorldKidsGamesMake & ColourStory Time

EconomyBusinessChefsSportsAthleticsCricketCyclingFootballFormula 1GolfRugby UTennisCulture, Media & DigitalBritish WildlifeEntertainment & ArtsFilmGamesLife through the LensMusicTechKnowledge & HealthHealthIngredientsRecipesScienceTechniquesLifestyleInterestsMagazineStyleTravelDestinationsEarth on TV & RadioIn PicturesUKWorldKidsMake & ColourStory Time

Table 3.5: Resulting models of participant 4 and 5

25

4 Analysis

In Chapter 2, CS has been discussed in detail. Different types of CS and how toperform a CS experiment with participants have been explained there. After theexperiment, it is the work of data experts to analyze this data that came throughthe CS experiment to get the final model for the application. In this chapter, sometechniques to analyze the data will be presented and at the end a new approachwill be discussed that is the work of this thesis.

4.1 Types of Analysis

There are many techniques to analyze data after CS experiment is completed.Some of these techniques are discussed below.

4.1.1 Heatmap

Figure 4.1: Example of Heatmap analysis of BBC Experiment created in[SHMW15]

27

4. Analysis

Heatmap is a very basic analysis technique in which one can see that how manytimes a card is placed in the same category by different participants. It is two-dimensional representation of data in which normally cards are shown on x-axisand categories are shown on y-axis.

Figure 4.1 shows an example of Heatmap. Values of table are coded in colorbased scheme that is, highest frequency is shown in “Red” color, lowest frequencyis in “Blue” and intermediate is from “Red” to “Blue”. A lot of “Red” color in thetable represents a visible trend in the sortings whereas a lot of “Blue” representsno visible trend in the sortings.

4.1.2 Proximity Matrix

Proximity matrix is another way to analyze the results of a CS experiment. Thismethod finds the semantic indifference between a pair of card.

In this method, if two cards are placed together most of the time then they areconsidered semantically similar and if they are placed rarely together, they areconsidered semantically different [Szw14].

Figure 4.2: Example of Proximity matrix [Szw14].

Figure 4.2 shows an example of proximity matrix as a two-dimensional matrixwith cards on both axes. As one can see that distance between “Student Support”and “Money Matters” is calculated as 0.76 and marked in yellow circle. Thedistance which is in percentage is calculated by the formula:

28


distance = d/(d + e) (4.1)

where e = how often these pair of cards placed together and d = how oftenthese pair of cards placed separately.

Distance is measured in percentage from 0 to 100. Red color represents thatcards are semantically similar and blue color shows that cards are semanticallydifferent.

4.1.3 Section Label Analysis

Section Label Analysis (SLA) is a very simple method of analysis. A threshold(p%) is set manually in percentage and SLA places a card into category if atleast p% of participants have put that card into same category. For some cardsthreshold can not be reached and in this case those cards left as Uncategorized inSLA [SHMW15]. Figure 4.3 shows an example of SLA.

Figure 4.3: Example of Section Label Analysis of BBC Experiment created in[SHMW15] (adjusted for scaling)

29

4. Analysis

Figure 4.4: Example of DBSCAN Analysis of BBC Experiment with differentthresholds created in [SHMW15]

30


4.1.4 DBSCAN Clustering

Density-based spatial clustering of applications with noise (DBSCAN) is a dataclustering algorithm. It groups items which are packed together [DBS]. It is oneof the most common clustering algorithms.

As stated in [Szw14], it calculates distance between a card and category andalso calculates distance between two cards. It requires proximity matrix and r =radius which can be 1 to n as n is the number of test subjects as an initial input.After running the algorithm, it results in clusters with items in it. Figure 4.4 onPage 30 shows an example result of DBSCAN with threshold.

4.1.5 Hierarchical Clustering

As described in [Szw14], Hierarchical Clustering is used to identify clusters (cat-egories) in a given set of items (cards). It requires Proximity matrix (Distancematrix or Similarity matrix) in the initial point and after the clustering the outputcan be seen as a graphical tree-like structure known as “Dendogram” [Dena][Denb].There are two types of Hierarchical Clustering which are explained below.

4.1.5.1 Agglomerative Hierarchical Clustering

Procedure of Hierarchical Clustering is as follows[Rie][DJ88].

1. Calculate Proximity matrix.

2. Each item is considered as a cluster initially.

3. Merge the two nearest clusters into one cluster.

4. Update Proximity matrix.

5. Repeat that process until one cluster is left.

Two clusters are merged according to distance between them. There are thevarious methods to calculate distance between clusters.

31

4. Analysis

Figure 4.5: Types of distances between clusters [Rie].

• Single Linkage Clustering - In this type of clustering, clusters are mergedon the basis of shortest distance between two clusters and this process con-tinues until only one cluster is remained [Sin]. In the figure 4.5, single linkagecan be seen in blue color.

• Complete Linkage Clustering - Farthest distance between two clustersis taken into account in this type of clustering [Com]. Complete linkage canbe seen in green color in the figure 4.5.

• Average Linkage Clustering - In this type of clustering, the average ofall distances between pairs of items are taken into account. Figure 4.5 showsaverage linkage clustering in yellow color.

Figure 4.6: Clustering in Single, Complete and Group Average Linkage [Rie].

32


Figure 4.6 on Page 32 explains how clustering of items take place. MIN refersto single linkage clustering, MAX refers to complete linkage clustering and GroupAverage refers to group average linkage clustering.

The results of hierarchical cluster analysis is represented with a “Dendogram”.A dendogram is a tree-like structure which shows how the clusters are arranged.In dendogram, clusters are represented as nodes and length of these clusters can beseen on horizontal scale [Denb][Dena]. Figure 4.7 shows an example of dendogram.

Figure 4.7: Example of Dendogram created in [Opt].

4.1.5.2 Divisive Hierarchical Clustering

Procedure of Divisive Hierarchical Clustering is opposite to Agglomerative Hierar-chical Clustering. In Divisive Hierarchical Clustering set of items are considered asa single cluster initially and splitted into sub-clusters until one elementary clusteris left.

33

4. Analysis

4.2 Analysis Approaches

Analysis approaches can be divided into two different approaches [San16] [Nie04b]:

1. Quantitative Approach

2. Qualitative Approach

As stated in [Naw12], qualitative analysis is to “... looking for patterns in thesorts” [Naw12] and quantitative analysis is to “use different tools to interpret theusers’ sorting. These tools use algorithms such as cluster analysis and similaritymatrixes to arrive at an interpretation” [Naw12].

Researchers have shown the benefits of qualitative methods over quantitativemethods as quantitative methods are too narrow to be useful and can be some-times misleading [Nie04b][Roh14]. They have shown more interest in qualitativemethods to be used for analysis [Bla14] but to analyze data with validity and reli-ability, both methods should be used in combination. They are combined becausea single method can not analyze effectively [Bla14][Pau14][Han14].

Section 4.1 describes many types of analysis. They are very useful in manyconditions but they all do either quantitative or qualitative analysis. So, theneed of a new analysis approach is there and this thesis proposes a new analysisapproach to overcome these drawbacks.

4.3 Category Based Analysis Approach

This thesis proposed a new hybrid approach using both qualitative and quantita-tive methods to analyze results of Modified Delphi Card Sorting experiment withthe help of graph visualization. This approach will be referred in this documentas “Category Based Analysis Approach”. Now we take a look at what aresome important factors, which are used in this approach, that make this approachstandout from other analysis approaches.

• As qualitative analysis methods provide rich and in-depth analysis but arenot suitable for large CS experiments [Pau14]. As Modified Delphi CardSorting is not a large CS experiment (it uses 6 to 12 participants [Pau07]),so the use of qualitative analysis method is suitable.

• Quantitative approaches provide automated and repeatable results. In thestarting point of this proposed approach, data is in the quantitative form sothe quantitative approach is imperative to use [Pau14].

34


• Visualization is used for both approaches and it helps to see the results inmuch better form as compared to other non-visual analysis methods. Hence,this proposed analysis uses graph visualization (Line Graph) for better andeasy understanding of the results for analysis.

• Many other analysis approaches do not allow analyzer to involve in theanalysis process as they only have to give input to get output but thisproposed approach gives full control to analyzer over analysis process. Itactually helps analyzer to select a final model out of many models whereanalyzer has to make decisions on his own to get the better result.

4.3.1 Overview of the Category Based Analysis Approach

In this approach, results of a CS experiment using Modified Delphi approach willbe converted to a graph. Graph will show the category changes, if it changes orremains same, from one participant to another on the basis of cards in a category.It shows categories of participants in a pattern that how one category changesfrom one participant to next the participant.

This approach helps analyzer to select the final model by showing the changein categories according to participants. Final model is still based on the selectionof categories by the analyzer. Now analyzer of the experiment can see the pat-tern of categories that can be changed from one participant to next participant orremained same. Analyzer selects the categories for the final model by looking atthe pattern of categories shown on line graph.

After selecting categories for the final model, this approach suggests a finalmodel by taking selected categories by the analyzer and placing cards in thesecategories according to most number of occurrences of cards in these categoriessorted by participants during the experiment. Now it is up to the analyzer tomark this suggested model as the final model or to make further changes in it.

4.3.2 Detailed Procedure of Category Based AnalysisApproach

Whole procedure of the proposed approach can be divided into two parts whichare given in next sections in detail.

Note: For better understanding, procedure of the proposed anal-ysis approach will be explained by applying “Category BasedAnalysis Approach” on “BBC Online” experiment’s results. Ex-ample of JSON data will be shown in each step for better under-standing.

35

4. Analysis

4.3.2.1 Visualization of Category Selection Analysis

This part of analysis will be explained step by step in next sections.

4.3.2.1.1 Get model of participants

As this thesis is to analyze the results of a CS experiment using Modified Delphiapproach so the starting point the data is in the form of models. This data wascreated by the participants during the experiment. Listing 4.1 shows an exampleof a model created by a participant during the experiment in JSON form. Theparticipant is shown on Line 2, a category on Line 3 and cards in this categoryfrom Lines 4–8.

1 {2 "Peter Folta": {

3 "News": [

4 "Business",

5 "Science",

6 "Tech",

7 "UK",

8 "World"

9 ],

10 "Sports": [

11 "Athletics",

12 "Cricket",

13 "Cycling",

14 "Football",

15 "Formula 1",

16 "Golf",

17 "Rugby U",

18 "Tennis"

19 ],

20 "Nature": [

21 "British Wildlife",

22 "Earth on TV & Radio",

23 "Life through the Lens"

24 ],

25 "Media": [

26 "Entertainment & Arts",

27 "Film",

28 "Games",

29 "Music"

30 ],

36


31 "Cooking": [

32 "Chefs",

33 "Health",

34 "Ingredients",

35 "Recipes",

36 "Techniques"

37 ],

38 "Lifestyle": [

39 "Magazine",

40 "Style"

41 ],

42 "Travel": [

43 "Destinations",

44 "In Pictures",

45 "Interests"

46 ],

47 "Kids": [

48 "Make & Colour",

49 "Story Time"

50 ]

51 }52 }

Listing 4.1: Example of a participant’s model

4.3.2.1.2 Re-arranging model of participants

In this step, model of participants will be re-arranged through a process. Everycard from a participant’s model is compared with the previous participant’s modelto know that in which category this card is arranged by the previous participant.Both categories (current participant’s category and previous participant’s cate-gory) of card are stored with the card in re-arranged model. This step shows therelation of every card to the current participant’s model as well as to the previousparticipant’s model. Listing 4.2 illustrates the relation of cards to current partici-pant and previous participant. One can see that cards from Lines 39–42 belong toFood category (Line 37) in the current participant’s model and Cooking category(Line 38) in the previous participant’s model.

1 {2 "Chloe Allison": {

3 "News": {

4 "News": [

5 "Business",

37

4. Analysis

6 "Science",

7 "UK",

8 "World"

9 ]

10 },11 "Sports": {

12 "Sports": [

13 "Athletics",

14 "Cricket",

15 "Cycling",

16 "Football",

17 "Formula 1",

18 "Golf",

19 "Rugby U",

20 "Tennis"

21 ]

22 },23 "Nature": {

24 "Nature": [

25 "British Wildlife",

26 "Earth on TV & Radio",

27 "Life through the Lens"

28 ]

29 },30 "Media": {

31 "Media": [

32 "Entertainment & Arts",

33 "Film",

34 "Music"

35 ]

36 },37 "Food": {

38 "Cooking": [

39 "Chefs",

40 "Ingredients",

41 "Recipes",

42 "Techniques"

43 ]

44 },45 "Lifestyle": {

46 "News": [

47 "Tech"

48 ],

49 "Cooking": [

38


50 "Health"

51 ],

52 "Lifestyle": [

53 "Magazine",

54 "Style"

55 ],

56 "Travel": [

57 "Interests"

58 ]

59 },60 "Travel": {

61 "Travel": [

62 "Destinations",

63 "In Pictures"

64 ]

65 },66 "Kids": {

67 "Media": [

68 "Games"

69 ],

70 "Kids": [

71 "Make & Colour",

72 "Story Time"

73 ]

74 }75 }76 }

Listing 4.2: Example of re-arranged model of a participant

4.3.2.1.3 Make intermediate Models for further processing

By using the re-arranged model of participants, a list is generated of all cate-gories made by participants. Each category will have all participants those haveused this category in their resulting model or they have changed this categoryto a new category in their model as compared to previous participant. Now thequestion is that how a change in the category can be determined?

This approach compares all models of participants in a way that every par-ticipant’s model compared with the previous participant’s model to find out thechange in the category. If current participant has used the same categories asused by previous participant then no changes in the categories will be stored. Ifa category which is used by current participant is not in the previous partici-pant’s model then a change is there. From the re-arranged model of participants,

39

4. Analysis

a change in the category can be determined by comparing a category with innercategories which belong to previous participant. Now the next question is thatwhich of the previous participant’s category is changed?

1 "Knowledge & Health": {

2 "Work": [

3 "Science"

4 ],

5 "Food": [

6 "Ingredients",

7 "Recipes",

8 "Techniques"

9 ],

10 "Lifestyle": [

11 "Health"

12 ]

13 }

Listing 4.3: Part of re-arranged model of a participant

Let us understand it with the help of Listing 4.3 as an example which is a part ofre-arranged model. Let us assume that “Knowledge & Health” is a new categorycreated by current participant. By looking at the cards in this category, one cansee that three out of five cards of “Knowledge & Health” category previously be-longed to “Food” category, one card previously belonged to “Work” category andone card previously belonged to “Lifestyle” category of the previous participant’smodel. Now it can be determined on the basis of the most number of cards that“Knowledge & Health” category was “Food” in the previous participant’s model.So in this way change of category is measured in this approach.

By using above method, two models can be generated for further processing.

1. A model of all categories those are used in the experiment and participantsbelonging to these categories. A participant, who has created a category orchanged it, is considered that he belongs to that category. Listing 4.4 showsthe example model after the process in JSON form.

2. A model of participants those have changed the categories. Both the newcategory and the changed category will be saved under the participant whohas been responsible for this change. Listing 4.5 on Page 42 shows theexample model after the process in JSON form.

1 {2 "News": [

3 "Peter Folta",

40


4 "Chloe Allison",

5 "Gerd Szwillus",

6 "Sonja Folta"

7 ],

8 "Sports": [

9 "Peter Folta",

10 "Chloe Allison",

11 "Gerd Szwillus",

12 "Sonja Folta",

13 "Fabian Mense"

14 ],

15 "Nature": [

16 "Peter Folta",

17 "Chloe Allison",

18 "Gerd Szwillus",

19 "Sonja Folta"

20 ],

21 "Media": [

22 "Peter Folta",

23 "Chloe Allison",

24 "Gerd Szwillus",

25 "Sonja Folta",

26 "Fabian Mense"

27 ],

28 "Cooking": [

29 "Peter Folta",

30 "Chloe Allison"

31 ],

32 "Lifestyle": [

33 "Peter Folta",

34 "Chloe Allison",

35 "Gerd Szwillus",

36 "Sonja Folta",

37 "Fabian Mense"

38 ],

39 "Travel": [

40 "Peter Folta",

41 "Chloe Allison",

42 "Gerd Szwillus",

43 "Sonja Folta",

44 "Fabian Mense"

45 ],

46 "Kids": [

47 "Peter Folta",

41

4. Analysis

48 "Chloe Allison",

49 "Gerd Szwillus",

50 "Sonja Folta",

51 "Fabian Mense"

52 ],

53 "Food": [

54 "Chloe Allison",

55 "Gerd Szwillus",

56 "Sonja Folta",

57 "Fabian Mense"

58 ],

59 "Work": [

60 "Sonja Folta",

61 "Fabian Mense"

62 ],

63 "Economy": [

64 "Fabian Mense"

65 ],

66 "Culture , Media & Digital": [

67 "Fabian Mense"

68 ],

69 "Knowledge & Health": [

70 "Fabian Mense"

71 ]

72 }

Listing 4.4: Example model of participants related to a specific category

1 {2 "Chloe Allison": {

3 "Cooking": "Food"

4 },5 "Sonja Folta": {

6 "News": "Work"

7 },8 "Fabian Mense": {

9 "Work": "Economy",

10 "Media": "Culture , Media & Digital",

11 "Food": "Knowledge & Health"

12 }13 }

Listing 4.5: Example model of participants with the change in category

42


4.3.2.1.4 Make model for graph data

This approach uses “Line Graph or Line Chart” for visualization after process-ing. Line graph is a graph that shows the change in one quantity according toanother quantity. It uses points those are connected by line to show the change.It is similar as “scatter plot” but points are joined with straight lines [Lina][Linb].

Line graph is perfect for “Category Based Analysis Approach” as this approachshows change in the categories with respect to participants. Participants will beused on the x-axis and categories will be used on the y-axis.

Now, the next task is to compute the points for line graph to show the changesin the categories. Because participants are ordered in a sequence in the ModifiedDelphi approach so, they can be labeled with some constant numeric values toshow in the line graph. The total number of categories used in the experiment arealso known so these categories can also be labeled as constant numeric values tobe used in the line graph.

Every category will have its own line in the graph. From the intermediate cal-culated models, a new model named as “Graph data model” is created which willbe used as input to line graph.

1 {2 "News": [

3 {4 "label": "Peter Folta", "value": 0

5 },6 {7 "label": "Chloe Allison", "value": 0

8 },9 {

10 "label": "Gerd Szwillus", "value": 0

11 },12 {13 "label": "Sonja Folta", "value": 9

14 }15 ],

16 "Sports": [



43

4. Analysis

22 },23 {24 "label": "Gerd Szwillus", "value": 1


28 },29 {30 "label": "Fabian Mense", "value": 1

31 }32 ],

33 "Nature": [


36 },37 {38 "label": "Chloe Allison", "value ": 2



45 }46 ],

47 "Media": [


50 },51 {52 "label": "Chloe Allison", "value ": 3




62 }63 ],

64 "Cooking": [

65 {

44


66 "label": "Peter Folta", "value": 4


70 }71 ],

72 "Lifestyle": [






87 }88 ],

89 "Travel": [




98 },99 {

100 "label": "Sonja Folta", "value": 6


104 }105 ],

106 "Kids": [

107 {108 "label": "Peter Folta", value": 7

109 },

45

4. Analysis

110 {111 "label": "Chloe Allison", "value": 7




121 }122 ],

123 "Food": [

124 {125 "label": "Chloe Allison", "value": 8




135 }136 ],

137 "Work": [

138 {139 "label": "Sonja Folta", "value": 9


143 }144 ],

145 "Economy": [

146 {147 "label": "Fabian Mense","value ": 10

148 }149 ],

150 "Culture , Media & Digital": [

151 {152 "label": "Fabian Mense", "value": 11

153 }

46


154 ],

155 "Knowledge & Health": [

156 {157 "label": "Fabian Mense", "value": 12

158 }159 ]

160 }

Listing 4.6: Example of graph data model

Listing 4.6 shows the Graph data model as an example. One can see the datathat will be used in line graph for “Cooking” category (Line 64) which has constantnumeric value 4. This category is related to two participants, one who has cre-ated this category (Peter Folta, Line 66) and second (Chloe Allison, Line 69) whohas changed this category to “Food” category which has constant numeric value 8.

Implementation of line graph using Graph data model can be seen in figure 5.12on Page 63

4.3.2.2 Suggested Solution on the basis of Selected Categories

After line graph is plotted, the analyzer can see the changes in the categories withrespect to participants. Now, the analyzer will select the categories for the finalmodel by analyzing the graph carefully.

After selecting categories for the final model, “Category Based Analysis Ap-proach” uses the quantitative analysis approach to suggest a final model on thebasis of selected categories. It calculates the “Heatmap” model from the modelsof all participants to place the cards into selected categories on the basis of mostnumber of occurrence of a card in a category sorted by participants of the experi-ment. Figure 4.1 shows the “Heatmap” of BBC experiment in a visual form. TheAnalyzer still has control to make changes in the final model. Further changes canbe made to the suggested model by analyzer if required. Suggested model can beselected as a final model that can be used for the final structure of the product orthe application.

47

5 Prototype Implementation

This thesis is an extension of Peter Folta’s Bachelor Thesis [Fol15] in which hehas developed a web application MoDeCaSo. This web application can be used toconduct a CS experiment using Modified Delphi approach. In this thesis, resultsof a CS experiment using Modified Delphi approach has taken as input and im-plement the “Category Based Analysis Approach” prototypically to find out thefinal model. This chapter will explain the technical aspects of the prototype im-plementation. We will focus only on the prototype implementation of the analysispart. For the prior work’s implementation part, one can see the chapter 3 for briefinformation and Peter Folta’s Bachelor Thesis [Fol15] for detailed information.

5.1 Technologies Used

This section will explain the technologies and frameworks used to implement theprototype of the analysis.

5.1.1 PHP

PHP1 is a server scripting language that is widely used for web development andcan be embedded into HTML [PHP].

PHP can be used on all major operating systems and with nearly all databases.It also supports object-oriented programming that makes it more powerful.

5.1.2 Slim Framework

Slim Framework2 is a PHP micro framework used to create powerful web appli-cations and APIs [Slib].

Slim Framework fulfills nearly all requirements on the server side. Here aresome of the key features of this framework [Slia].

1. HTTP Router

1PHP: http://php.net/2Slim: https://www.slimframework.com/

49

http://php.net/


5. Prototype Implementation

2. Middleware

3. PSR-7 Support

4. Dependency Injection

5. Template rendering

6. HTTP Caching

7. Flash messages

8. Encrypted cookies

9. Logging

10. Error handling and debugging

11. Slack Integration

12. Create REST applications

13. Support all the HTTP methods (GET, POST, PUT and DELETE)

5.1.3 Slim-jsonAPI

Slim-jsonAPI is a framework to implement JSON APIs using Slim.

5.1.4 Basic technologies used on front end

As our analysis prototype implementation is a web application and a web applica-tion can not be built without HTML, Cascading Style Sheets (CSS) and Javascriptso these technologies are also used in the implementation of this prototype.

This prototype is a modern HTML web application that uses Javascript andCSS for providing an efficient and desktop-like user experience. This is also a SPAthat runs in a single HTML page which is loaded initially on request and all othercontents are loaded on this page without loading a full page that provides a freshand desktop-like look and feel.

5.1.5 Bootstrap

Bootstrap3 is an open-source framework developed by Twitter. It consists ofHTML, CSS and Javascript code that helps developers to create cross-browserweb applications [Booa]. Accrding to [Booa], some of the main reasons to use thisframework are:

3Bootstrap: http://getbootstrap.com/

50

http://getbootstrap.com/

5.1 Technologies Used

• Easy to get started

• Great grid system

• Base styling for most HTML elements (Typography, Code, Tables, Forms,Buttons, Images, Icons)

• Extensive list of components

• Bundled Javascript plugins

5.1.6 AJAX

AJAX is a client-side script that is used to communicate to and from server ordatabase without page refresh [Tec13]. Extensible Markup Language (XML) isused as a standard for AJAX but in this application JSON is used for passingdata as JSON is fast and takes less space [JSO].

5.1.7 jQuery

jQuery4 is Javascript library enables to do Javascript tasks in less time and inefficient way. It is mostly used for Document Object Model (DOM) manipulation[jQu].

5.1.8 AngularJS

AngularJS5 is an open source MVC Javascript framework used for dynamic viewsin web applications [Angb]. This application uses AngularJS to create modernSPA.

5.1.9 Angular-nvd3

D3.js6 is a Javascript library that brings data into life with powerful visualization.With this, any kind of visualization can be shown as it supports a huge numberof graph types. D3.js is extremely fast and supports large dataset with any typeof animation [D3].

NVD3.js7 is also a javascript library built on top of D3.js to build re-usablecharts keeping that every component remained customizable on D3.js level [NVD].

4jQuery: https://jquery.com/5AngularJS: https://angularjs.org/6D3.js: https://d3js.org/7NVD3.js: http://nvd3.org/

51

https://jquery.com/


https://d3js.org/

http://nvd3.org/


Angular-nvd38, a javascript library, makes it easier to work with NVD3.js viaJSON API. It completely preserves the structure of NVD3.js in the directiveJSON structure that makes it very simple to customize while using JSON ap-proach [Anga].

This application uses the line graph to visualize the data. With using AngularJSas MVC framework and JSON for passing data, Angular-nvd3 becomes the perfectchoice to use it for visualization.

5.1.10 MySQL

MySQL9 is open-source relational database management system. It is widely usedwith PHP for web applications [MyS]. Having properties like security, reliability,scalability and easy to use, MySQL is used as a database system.

5.2 Prototype’s Architecture

This section will explain that how the architecture of the prototype has beendesigned.

5.2.1 Model-View-Controller

MVC is a software design pattern used to separate different concerns of the appli-cation into three components. Model component has the data and the logic, Viewcontains the visual layout and Controller connects the Model and View together.An MVC project can have more components as in the modern web applications,a router component is used to handle URL routing [Raj16].

Figure 5.1: MVC Flow [Raj16]

8Angular-nvd3.js: http://krispo.github.io/angular-nvd3/9MySQL: https://www.mysql.com/

52

http://krispo.github.io/angular-nvd3/

https://www.mysql.com/

5.2 Prototype’s Architecture

Figure 5.1 shows the basic MVC flow. This prototype is also using MVC toseparate the concerns of the application. In this prototype, AngularJS is respon-sible for URL routing. Views have been built in HTML, CSS, Javascript andmany frameworks for providing an efficient and desktop-like user experience. Italso uses AngularJS in the View component so that developer does not need toupdate the view. Model and Controller components have been built using SlimFramework.

5.2.2 Front End and Back End Communication

MoDeCaSo’s implementation is divided into two parts: Server Side (Back end)and Client Side (Front end). The Front End and the Back End communi-cates with JSON objects. Both requests and responses are done with JSON ob-jects. One of the many benefits of using JSON is that it can easily converted intoJavascript.

5.2.3 Folder Structure

Figure 5.2: Folder Structure of Application [Sub]

53


Figure 5.2 represents the folder structure of the implementation where two partsFront end and Back end can be seen separately. In the front end, css folder(including CSS), js folder (including Javascript), tpl folder (including HTML)and some other folders together forms the View component. On the Server sideModel and Controller components can be seen.

5.2.4 Navigation Flow

Figure 5.3 shows the flow of the work. It is representing only the Analysis workthat starts from Project Configuration page. As shown in the figure 5.3 the ana-lyzer can see the resulting models of participants, can perform the analysis or canview the analyzed models.

Figure 5.3: Navigation Flow of the work [Flo]

5.3 Database

This prototype uses a relational database in order to store data effectively, safelyand scalably.

54

5.3 Database

5.3.1 Database Structure

Complete structure of the database can be seen in Figure 5.4 where all relationsbetween tables with primary and foreign keys are shown.

Figure 5.4: Database Structure made in [Ver]

5.3.2 Tables

Table 5.1 shows all the tables used for this application. It also illustrates thepurpose of each table in the database.

55


Table Description

users Stores information about users of the appli-cation.

user tokens To store the Application Programming In-terface (API) keys that is generated by backend for users.

projects This table store data about all projectsproject participants Information about the participants related to

project are stored in this table.project messages This table holds all messages of projects.project cards Every information about cards added for the

experiment under a specific project is storedhere.

experiment categories Categories that are made by participantsduring the experiment are saved in this ta-ble. Relation of associated project and par-ticipant with the category can be identified.

experiment models All resulting model of participants are storedin this table.

experiment analysis This table stores the analyzer of the projectwith the comments.

experiment final models Models after analysis are saved in this table.

Table 5.1: Database Tables

5.4 Installation and Configuration

As described several times that the prototype implementation is an extensionto MoDeCaSo. The installation and configuration process is the same as used byMoDeCaSo. This process can be seen in the installation and configuration processof MoDeCaSo from Peter Folta’s thesis which is given in Chapter 5 “Installing,Configuring and Using MoDeCaSo” [Fol15].

5.5 Application Walkthrough

This section provides a complete walkthrough of the extended work that has beendone during the duration of this thesis. It not only provides the information aboutthe workflow of the work but also can be used as a user manual. This section willexplain every view of the work step by step. As it is described earlier that thiswork is the extension of work done by Peter Folta so only the workflow of theextended work will be explained. All the images used for the workflow are takenfrom the application itself and “BBC Online” experiment is taken as an example

56


experiment which is done in real-time by Peter Folta in his evaluation part of thethesis.

5.5.1 User Roles

As mentioned before that MoDeCaSo distinguishes the roles of different users. Amoderator or administrator can log into MoDeCaSo to control the things aboutthe projects (see figure 5.5). Moderator has the control over his own projectswhere as an administrator has full rights to work with any project. To do theanalysis of the project a moderator or administrator must log into MoDeCaSowhere as the participants have no access to log into MoDeCaSo. They can onlyparticipate in the project to submit their model for an experiment when notifiedvia email.

Figure 5.5: MoDeCaSo’s Login Page

5.5.2 Project Configuration

After login, moderator or administrator will select the project (experiment usingModified Delphi approach will be referred as Project). Then will to go to ProjectConfiguration page where several configurations about the project can be done.But for the scope of this work, the project is already finished and only analysis

57


has to be done so no further need to work is needed on any further configurations.Views those are related to this works from the Project Configuration page (seefigure 5.6) will be explained in next sections.

Figure 5.6: Project Configuration Page

5.5.3 View Results

Clicking on View Results button from the Project Configuration page will lead tothe page where all the results of a completed project can be seen.

Figure 5.7 on Page 59 shows the results of “BBC Online” project. It showsthe information of all participants including seed participant. Personal data ofparticipants, order of participants in the experiment and the status that eitherthey have completed their task or not can be seen on this page.

All the resulting model of participants of a project can be exported in a singlefile, either in JSON form or in CSV form, by clicking on Export Project Modelbutton. The exported project model can be used for any other processing. Eachmodel of a participant can also be exported separately, either in JSON form orin CSV form, by clicking on Export Model button which can be seen next to eachparticipant.

58


Figure 5.7: View Results Page

Figure 5.8: View of a Participant’s Model Page

59


Model of every participant can be viewed by clicking View Model button whichis located next to each participant. In figure 5.8, model of a participant canbe seen. All categories and cards related to this model are shown. Model of aparticipant can also be exported, either in JSON form or in CSV form, from thispage by clicking on Export Model button.

5.5.4 View/Edit Messages

In the Project Configuration page, default messages that will be shown to analyzer(moderator or administrator) during the analysis can be seen under the Messagestab. These messages contain the information and the instructions that should befollowed by the analyzer during the process. Figure 5.9 shows the messages underMessage tab which can be edited by hovering on the message and then click onEdit Text button. Clicking on Edit Text button will open a dialogue box wheredefault message can be edited according to the need (see Figure 5.10 on Page 61).

Figure 5.9: Information Messages

60


Figure 5.10: Edit Messages dialogue

5.5.5 Analysis Process

Walkthrough of the Analysis Process can be divided into two parts:

1. Category Analysis

2. Suggested Solution

5.5.5.1 Category Analysis

Analysis process can be started by clicking on Start Analysis button on the ProjectConfiguration page that will lead to the Category Analysis page.

61


Before showing the Category Analysis page, an information dialogue (see figure5.11) will appear that shows the information about the Category Analysis pageand the instructions for the analyzer (moderator or administrator) to do the anal-ysis. After reading the text on the dialogue, the analyzer will click on the Startbutton, which is located on the bottom of information dialogue, to see the Cat-egory Analysis page. This dialogue can be seen in the future by clicking on InfoMessage button which is located on the top right of the Category Analysis page.

Figure 5.11: View of Information Dialogue on Category Analysis Page

Figure 5.12 on Page 63 shows the Category Analysis page in which a line graphbetween all categories, created during the CS experiment, and participants of theexperiment can be seen. Participants are shown on the x-axis and categories areshown on the y-axis. For the analyzer, understanding of the line graph is veryimportant. Without the understanding of the line graph, results of the analysismight not be useful.

Line graph represents the change of categories according to the participants.Each color in the line graph represents the category progression with respect tothese participants. This color will remain same and the line will be horizontal overthe next participants if no changes occurs in the categories. In figure 5.12, Sportcategory that is represented by the red line can be seen. It remains horizontalbecause there is no change of this category with respect to all participants. Achange in the category between two participants can be seen if it will not remainhorizontal between two participants. Figure 5.13 on Page 64 shows the example of

62


Figure 5.12: Category Analysis Page

the change in category. We can see that the Cooking category, created by the firstparticipant, is changed to the Food category in the second participant’s model.The tooltip on the Food category of the second participant shows Cooking(4) thatmeans Cooking category is changed to Food category on the basis of 4 cards. 4cards are the most number of cards in the Food category which were also in theCooking category in the previous participant’s model.

After understanding the line graph, the analyzer will select the categories forthe final model by deeply analyzing the change in the categories over the partic-ipants. Categories for the final model can be selected by clicking the categories,which are given on top of line graph (in the Legend of line graph). Figure 5.13on Page 64 displays the selection of categories from the legend of the line graph.Selected categories will be highlighted in green on the right column, which is titledas Selected Categories.

After selecting the categories for the final model, Show Suggested Solution but-

63


Figure 5.13: Selection of Categories for final model

ton, which is located on the top left, will be enabled. The Suggested Solution pageon the basis of selected categories can be seen by clicking this button.

5.5.5.2 Suggested Solution

Before showing the Suggested Solution page, an information dialogue (see figure5.14 on Page 65) will appear. This information dialogue shows the informationabout the Suggested Solution page and the instructions for the analyzer (mod-erator or administrator) to perform the analysis. After reading the text on thedialogue, the analyzer will click on the Start button, which is located on the bot-tom of information dialogue, to see the Suggested Solution page. This dialoguecan be seen in the future by clicking on Info Message button which is located onthe top right of the Suggested Solution page.

64


Figure 5.14: View of Information Dialogue on Suggested Solution Page

Figure 5.15: Warning for unnecessary selected categories

If several categories are selected and the cards are not placed in all of the se-lected categories, a warning message will be shown as seen in Figure 5.15. To

65


remove this warning message analyzer should go back to the Category Analysispage and unselected the unnecessary categories. If the analyzer saves that modelas final model, the unnecessary categories will not be saved in the final model.

Figure 5.16 displays the Suggested Solution page in which the suggested model,on the basis of selected categories, can be seen on the right column. Cards areplaced into the selected categories by using the “Heatmap” method which is ex-plained previously in Chapter 4 Section 4.1.1.

Figure 5.16: View of Suggested Solution Page

On the left column, unsorted cards are shown. All cards must be sorted, in orderto be select as the final model. For this, analyzer can go back to Category Analysispage by clicking on Back to Category Analysis button to go back and select morecategories. By doing so these unsorted cards can be sorted in selected categoriesautomatically by the system. For help, analyzer can click on the ( ? ) sign ontop right corner of each unsorted cards. On mouse hover the tooltip shows thecategories in which this card has been sorted by the participants (see Figure 5.16).

66


The analyzer can also drag-and-drop the unsorted cards into suggested model, ifmore categories are not required (see Figure 5.17).

Figure 5.17: Drag-and-drop of unsorted card

When there is no unsorted card left, the model is then ready to be submittedas the final model. Analyzer can also add comments to this model by clicking onAdd Comments button. After clicking on Add Comments button a dialogue willappear where the analyzer’s comments can be added (see Figure 5.18).

67


Figure 5.18: Add comments dialogue

Clicking on Select as Final Model button will save the model in the databaseas a final model of the experiment. A Thank You dialogue (see Figure 5.19) willappear to show the confirmation of submission. By clicking on Close button onthe Thank You dialogue, it leads to the View Analysis Result page where analyzercan see the final submitted model.

Figure 5.19: Thank you dialogue

68


5.5.6 View Analysis Results

Clicking on View Analysis Results button from the Project Configuration pagewill lead to the page where all the analyzed results of a project can be seen.

Figure 5.20: View Analysis Results Page

Figure 5.7 shows the analyzed results of “BBC Online” project. It lists all theanalyzed models with the information of analyzers. Comments that were addedwith the analyzed model can also be seen on this page. Each analyzed model canbe exported from this page, either in JSON form or in CSV form, by clicking onExport Model button which can be seen next to each analyzer.

Model of every analyzed model can be viewed by clicking on the View Modelbutton. The button is located next to each analyzer. Figure 5.21 shows theAnalysis Result page where an analyzed model can be seen. All categories andcards related to this Model are shown here. An analyzed model can also beexported from this page, either in JSON form or in CSV form, by clicking on theExport Model button.

69


Figure 5.21: View of an analyzed Model

5.5.7 Export Data (JSON or CSV Form)

All the models which are created by the participants separately/collectivity or themodels which are analyzed by the analyzers during the analysis can be exportedfor any further use either in JSON form or in CSV form. Listing 5.1 and 5.2 showsthe example data that is exported in JSON and CSV form.

1 "user_id","card_label","card_id","category_label","

category_id"

2 "1", "Chefs", "414", "Food", "9"

3 "1", "Ingredients", "415", "Food", "9"

4 "1", "Recipes", "413", "Food", "9"

5 "1", "Techniques", "416", "Food", "9"

70


6 "1", "Games", "423", "Kids", "8"

7 "1", "Make & Colour", "422", "Kids", "8"

8 "1", "Story Time", "424", "Kids", "8"

9 "1", "Health", "398", "Lifestyle", "6"

10 "1", "Interests", "418", "Lifestyle", "6"

11 "1", "Magazine", "396", "Lifestyle", "6"

12 "1", "Style", "421", "Lifestyle", "6"

13 "1", "Tech", "394", "Lifestyle", "6"

14 "1", "Entertainment & Arts", "397", "Media", "4"

15 "1", "Film", "419", "Media", "4"

16 "1", "Music", "420", "Media", "4"

17 "1", "British Wildlife", "410", "Nature", "3"

18 "1", "Life through the Lens", "411", "Nature", "3"

19 "1", "Business", "393", "News", "1"

20 "1", "Science", "395", "News", "1"

21 "1", "UK", "392", "News", "1"

22 "1", "World", "391", "News", "1"

23 "1", "Athletics", "406", "Sports", "2"

24 "1", "Cricket", "402", "Sports", "2"

25 "1", "Cycling", "407", "Sports", "2"

26 "1", "Football", "400", "Sports", "2"

27 "1", "Formula 1", "401", "Sports", "2"

28 "1", "Golf", "405", "Sports", "2"

29 "1", "Rugby U", "403", "Sports", "2"

30 "1", "Tennis", "404", "Sports", "2"

31 "1", "Destinations", "417", "Travel", "7"

32 "1", "Earth on TV & Radio", "412", "Travel", "7"

33 "1", "In Pictures", "399", "Travel", "7"

Listing 5.1: Example of exported data in CSV

1 [

2 {3 "user": {

4 "user_id": 1

5 },6 "category": [

7

8 [....]

9

10 {11 "category_id": "709",

12 "category_label": "Lifestyle",

13 "cards": [

71


14 {15 "card_id": "398",

16 "card_label": "Health",

17 "tooltip": ""

18 },19 {20 "card_id": "418",

21 "card_label": "Interests",

22 "tooltip": ""

23 },24 {25 "card_id": "396",

26 "card_label": "Magazine",

27 "tooltip": ""

28 },29 {30 "card_id": "421",

31 "card_label": "Style",

32 "tooltip": "Fashion"

33 },34 {35 "card_id": "394",

36 "card_label": "Tech",

37 "tooltip": ""

38 }39 ]

40 },41 {42 "category_id": "714",

43 "category_label": "Media",

44 "cards": [

45 {46 "card_id": "397",

47 "card_label": "Entertainment &

Arts",

48 "tooltip": ""

49 },50 {51 "card_id": "419",

52 "card_label": "Film",

53 "tooltip": ""

54 },55 {56 "card_id": "420",

72


57 "card_label": "Music",

58 "tooltip": ""

59 }60 ]

61 },62 {63 "category_id": "717",

64 "category_label": "Nature",

65 "cards": [

66 {67 "card_id": "410",

68 "card_label": "British Wildlife

",

69 "tooltip": ""

70 },71 {72 "card_id": "411",

73 "card_label": "Life through the

Lens",

74 "tooltip": "Photographs of

nature"

75 }76 ]

77 },78

79 [....]

80 ]

81 }82 ]

Listing 5.2: Example of exported data in JSON

73

6 Evaluation

This thesis implements an extension that is made to MoDeCaSo to analyze theresults of CS experiment using Modified Delphi approach. This extension shouldprovide a model as a result after applying the “Category Based Analysis Ap-proach”. In this chapter, the application will be evaluated to verify that it cananalyze the results of CS experiment using Modified Delphi approach and to pro-vide a resulting model after the analysis.

6.1 Choosing Project - “BBC Online”

As stated before that Peter Folta has evaluated his part by selecting BBC web-site. He has conducted a CS experiment using Modified Delphi approach on BBCwebsite with the help of web application (MoDeCaSo) which he has made dur-ing his work. He has created a project named as “BBC Online” to conduct thisexperiment in the MoDeCaSo. A short description about his evaluation usingMoDeCaSo on BBC website is given in Chapter 3 Section 3.4. Resulting Modelsof all participants of “BBC Online” project can be seen in Table 3.4 and 3.5.

6.2 Role of Analyzer

This proposed approach to analyze the results of CS experiment using ModifiedDelphi approach gives analyzer freedom to fully involved in the analysis process.The resulting model is dependable on the analyzer that how presented line graph,which shows the changes in categories over participants, is understood by theanalyzer and how categories are selected by the analyzer.

6.3 Results of the Analysis

The evaluation process is completed and it shows that this extension of MoDeCaSois fully able to analyze the results of CS experiment using Modified Delphi ap-proach and generates the model which can be used as the structure of the appli-cation.

Table 6.1 shows the resulting model obtained after the evaluation of this pro-totype. This model shows that newly proposed approach and its implementationcan input the results of any CS experiment using Modified Delphi approach and

75

6. Evaluation

results in a model that can be used as the structure of the application for whichthe experiment was held.

Category Cards

Food

ChefsIngredientsRecipesTechniques

KidsGamesMake & ColourStory Time

Lifestyle

HealthInterestsMagazineStyleTech

MediaEntertainment & ArtsFilmMusic

NatureBritish WildlifeLife through the Lens

News

BusinessScienceUKWorld

Sports

AthleticsCricketCyclingFootballFormula 1GolfRugby UTennis

TravelDestinations & Earth on TVRadioIn Pictures

Table 6.1: Resulting model after applying Category Based Analysis Approach

76

6.4 Analysis Strategy to get the Resulting Model

6.4 Analysis Strategy to get the Resulting Model

How the resulting model is achieved, let us have a look on the applied strategy.Kids, Travel, Lifestyle and Sports categories used by all participants so first thesecategories were selected. Four out of five participants have used Media and Naturein their Model. So these categories were also selected. Now Food and Newshave been used by three participants. If News is selected then four cards areremained unsorted and if Food is selected then only one card is left unsorted. Ifseven categories are required for the final model then Food can be selected as theseventh category and one card that is left unsorted can be sorted using drag-and-drop method by the analyzer. Currently, both cards are selected and all cards aresorted in the selected categories and this model is achieved as a resulting modelof the experiment which can be seen in Table 6.1.

77

7 Summary and Future Work

This chapter provides a summary of this thesis and proposes some of the futurework that can be taken as the reference for the next possible extension of thiswork.

7.1 Summary

The aim of this thesis was to provide an analysis approach which combines Qual-itative and Quantitative analysis approaches and prototypically implement thisapproach into an application with visualization. To realize this work, CS, itsvariants and different available tools have been explained in detail to justify thatModified Delphi Card Sorting generates quality results with less time and minimalcost. This thesis is the extension of the work done by Peter Folta who has createda web-based application (MoDeCaSo) to conduct Card Sorting experiment usingModified Delphi approach that is discussed briefly in this work.

A new hybrid analysis approach has been designed that uses Qualitative analy-sis, Quantitative analysis and visualization to provide a better insight of the resultsof Modified-Delphi Card Sorting experiment. This approach gives analyzer a fullcontrol to select the categories for the final model by looking at the changes incategories with respect to participants of the experiment and make changes in thesuggested final model based on the selected categories. Later this approach hasbeen successfully implemented and integrated into MoDeCaSo. The implementedapproach uses latest technologies to visualize the changes in categories with re-spect to participants of the experiment.

The result of this new hybrid approach is evaluated and analyzed for “BBCOnline” experiment. The final resulting model after applying the “Category BasedAnalysis Approach” is given in Table 6.1 which shows that this approach is fullyable to analyze the result of CS experiment using Modified Delphi approach. Thisapproach is very simple and easy to use (thanks to visualization of data duringthe analysis) and also provides full control to analyzer during the analysis process,makes it a better candidate among others.

79

7. Summary and Future Work

7.2 Future Work

This thesis proposed a new analysis approach and implemented this approach asan extension to MoDeCaSo. The proposed analysis approach and implementationcan be extended in several ways, some of them are discussed below.

7.2.1 Improvement in Analysis Approach

Currently, this approach identifies the change in a category on the basis of mostnumber of cards in the current participant’s model belonging to the category inthe previous participant’s model. If every card in the current participant’s modelbelongs to the different categories in the previous participant’s model then we cannot predict the category that is changed. In this scenario, this approach picksone category at random out of all candidate categories. This can be improved byfurther research to make it more efficient algorithm.

7.2.2 Importing data from other tools for analysis

Currently, this approach is integrated within MoDeCaSo which restricts this ap-proach to be used for the results generated by the other applications which arealso conducting CS experiment using Modified Delphi approach. One suggestedextension can be that to provide an import functionality which imports the resultsof an external CS tool and analyze that imported data to generate the final model.Compatibility can only be achieved when the imported results were generated bya tool which also uses Modified Delphi approach. The imported data format canbe chosen from various available data formats like JSON, CSV, XML etc.

7.2.3 Standalone application

This thesis is currently working in the MoDeCaSo as an extension to it. In thefuture, this extension can be made in a separate application so that any one who isusing some other tool to conduct CS experiment using Modified Delphi approachcan use it for analysis like Casolysis2.0 [SHMW15] in which analysis can be doneindependently of results generated by any CS tool.

7.2.4 Refine Categories before Analysis

Before the analysis, the categories which have same semantic meanings can becombined together for getting the better final structure. Currently, “Sport” and“Sports” are considered different categories that can effect on the resulting model.So, before analysis, an algorithm can be applied that deals with this situation.Latent Semantic Analysis [LD97] can be used as the starting point to solve thissituation.

80

Bibliography

[Adv] Advantages and disadvantages - card sorting. on-line at: https://dirtarchitecture.wordpress.com/

advantages-and-disadvantages/ accessed on: 12-07-2017.

[Anga] Angular-nvd3. online at: http://krispo.github.io/

angular-nvd3/ accessed on: 12-07-2017.

[Angb] Angularjs - superheroic javascript mvw framework. online at: https://angularjs.org/ accessed on: 12-07-2017.

[Apa] Apache http server. online at: https://en.wikipedia.org/wiki/

Apache_HTTP_Server accessed on: 12-07-2017.

[BBC] Bbc online. online at: https://en.wikipedia.org/wiki/BBC_

Online accessed on: 12-07-2017.

[Bla14] Ann Blandford. The Encyclopedia of Human-Computer Interaction- Semi-structured qualitative studies. Interaction design foundation,2nd edition, 2014.

[Booa] About bootstrap framework. online at: http://tech.queryhome.

com/51419/about-bootstrap-framework accessed on: 12-07-2017.

[Boob] Bootstrap (front-end framework). online at: https:

//en.wikipedia.org/wiki/Bootstrap_(front-end_framework)

accessed on: 12-07-2017.

[Com] Complete-linkage clustering. online at: https://en.wikipedia.org/wiki/Complete-linkage_clustering accessed on: 12-07-2017.

[D3] D3.js - data-driven documents. online at: https://d3js.org/ ac-cessed on: 12-07-2017.

[DBS] Dbscan. online at: https://en.wikipedia.org/wiki/DBSCAN ac-cessed on: 12-07-2017.

[Deb] Debian. online at: https://en.wikipedia.org/wiki/Debian ac-cessed on: 12-07-2017.

[Dena] Dendrogram. online at: https://en.wikipedia.org/wiki/

Dendrogram accessed on: 12-07-2017.

81

https://dirtarchitecture.wordpress.com/advantages-and-disadvantages/

https://dirtarchitecture.wordpress.com/advantages-and-disadvantages/





https://en.wikipedia.org/wiki/Apache_HTTP_Server

https://en.wikipedia.org/wiki/Apache_HTTP_Server

https://en.wikipedia.org/wiki/BBC_Online

https://en.wikipedia.org/wiki/BBC_Online

http://tech.queryhome.com/51419/about-bootstrap-framework

http://tech.queryhome.com/51419/about-bootstrap-framework

https://en.wikipedia.org/wiki/Bootstrap_(front-end_framework)

https://en.wikipedia.org/wiki/Bootstrap_(front-end_framework)

https://en.wikipedia.org/wiki/Complete-linkage_clustering

https://en.wikipedia.org/wiki/Complete-linkage_clustering

https://d3js.org/

https://en.wikipedia.org/wiki/DBSCAN

https://en.wikipedia.org/wiki/Debian

https://en.wikipedia.org/wiki/Dendrogram

https://en.wikipedia.org/wiki/Dendrogram

Bibliography

[Denb] Dendrogram. online at: http://www.statistics.com/glossary&

term_id=742 accessed on: 12-07-2017.

[DJ88] Richard C Dubes and Anil K Jain. Algorithms for clustering data.1988.

[Flo] Flowchart maker and online diagram software. online at: https:

//www.draw.io/ accessed on: 12-07-2017.

[Fol15] Peter Folta. Design and implementation of a web application for amodified delphi approach to card sorting. May 2015.

[Gha15] Armen Ghazarian. How to conduct a card sorting study?, December2015. online at: https://designmodo.com/card-sorting-study/


[Han14] Steven Hannah. Sorting out card sorting: Comparing methods forinformation architects, usability specialists, and other practitioners.Master’s thesis, University of Oregon, June 2014.

[Hud14] William Hudson. The Encyclopedia of Human-Computer Interaction- Card Sorting. Interaction design foundation, 2nd edition, 2014.

[jQu] jquery. online at: https://jquery.com/ accessed on: 12-07-2017.

[JSO] Why json is better than xml. online at: https://blog.

cloud-elements.com/json-better-xml accessed on: 12-07-2017.

[Kuh14] Timo Kuhl. Modified-delphi card sorting - eine sinnvolle alternativezu bereits bestehenden methoden? 2014. Bachelor Thesis.

[LD97] Thomas K Landauer and Susan T. Dumais. A solution to plato’sproblem: The latent semantic analysis theory of acquisition, induc-tion, and representation of knowledge. 1997.

[Lina] Line chart. online at: https://en.wikipedia.org/wiki/Line_

chart accessed on: 12-07-2017.

[Linb] Line graph. online at: https://www.mathsisfun.com/definitions/line-graph.html accessed on: 12-07-2017.

[MyS] About mysql. online at: https://www.mysql.com/about/ accessedon: 12-07-2017.

[Naw12] Ather Nawaz. A comparison of card-sorting analysis methods, 2012.

[Nie04a] Jakob Nielsen. Card sorting: How many users to test.July 2004. online at: https://www.nngroup.com/articles/

card-sorting-how-many-users-to-test/ accessed on: 12-07-2017.

82

http://www.statistics.com/glossary&term_id=742

http://www.statistics.com/glossary&term_id=742

https://www.draw.io/

https://www.draw.io/

https://designmodo.com/card-sorting-study/

https://jquery.com/

https://blog.cloud-elements.com/json-better-xml

https://blog.cloud-elements.com/json-better-xml

https://en.wikipedia.org/wiki/Line_chart

https://en.wikipedia.org/wiki/Line_chart

https://www.mathsisfun.com/definitions/line-graph.html

https://www.mathsisfun.com/definitions/line-graph.html

https://www.mysql.com/about/

https://www.nngroup.com/articles/card-sorting-how-many-users-to-test/

https://www.nngroup.com/articles/card-sorting-how-many-users-to-test/

Bibliography

[Nie04b] Jakob Nielsen. Risks of quantitative studies. March2004. online at: https://www.nngroup.com/articles/

risks-of-quantitative-studies/ accessed on: 12-07-2017.

[NVD] Nvd3 re-usable charts for d3.js. online at: http://nvd3.org/ ac-cessed on: 12-07-2017.

[Opt] Optimal sort - online card sorting software. online at: https://www.optimalworkshop.com/optimalsort accessed on: 12-07-2017.

[Pau07] Celeste Lyn Paul. Investigation of applying the delphi method to anew card sorting technique. June 2007.

[Pau08] Celeste Lyn Paul. A modified delphi approach to a new card sortingmethodology. 4:7–30, May 2008.

[Pau14] Celeste Lyn Paul. Analyzing card-sorting data using graph visualiza-tion. 9:87–104, May 2014.

[PHP] What is php? online at: http://php.net/manual/en/

intro-whatis.php accessed on: 12-07-2017.

[pin] pinterest. online at: https://www.pinterest.com/ accessed on: 12-07-2017.

[Pro16] The pros and cons of card sorting in ux research, 2016. onlineat: https://www.interaction-design.org/literature/article/

the-pros-and-cons-of-card-sorting-in-ux-research accessedon: 12-07-2017.

[Raj16] Juma Rajogi. An introduction to mvc in php usingthe slim framework - web development. April 2016.online at: https://www.madebymagnitude.com/blog/

an-introduction-to-mvc-in-php-using-the-slim-framework/


[Rie] Mirek Riedewald. Data mining techniques:cluster analysis. online at:http://www.ccs.neu.edu/home/mirek/classes/2012-S-CS6220/

Slides/Lecture4-Clustering.pdf accessed on: 12-07-2017.

[Roh14] Christian Rohrer. When to use which user-experience research meth-ods. October 2014. online at: https://www.nngroup.com/articles/which-ux-research-methods/ accessed on: 12-07-2017.

[San16] Gilmar J. Santos. Card sort technique as a qualitative substitute forquantitative exploratory factor analysis. Corporate Communications:An International Journal, 11:288 – 302, 2016.

83

https://www.nngroup.com/articles/risks-of-quantitative-studies/

https://www.nngroup.com/articles/risks-of-quantitative-studies/

http://nvd3.org/



http://php.net/manual/en/intro-whatis.php

http://php.net/manual/en/intro-whatis.php

https://www.pinterest.com/

https://www.interaction-design.org/literature/article/the-pros-and-cons-of-card-sorting-in-ux-research

https://www.interaction-design.org/literature/article/the-pros-and-cons-of-card-sorting-in-ux-research

https://www.madebymagnitude.com/blog/an-introduction-to-mvc-in-php-using-the-slim-framework/

https://www.madebymagnitude.com/blog/an-introduction-to-mvc-in-php-using-the-slim-framework/

http://www.ccs.neu.edu/home/mirek/classes/2012-S-CS6220/Slides/Lecture4-Clustering.pdf

http://www.ccs.neu.edu/home/mirek/classes/2012-S-CS6220/Slides/Lecture4-Clustering.pdf

https://www.nngroup.com/articles/which-ux-research-methods/

https://www.nngroup.com/articles/which-ux-research-methods/

Bibliography

[SC13] Jiann-Cherng Shieh and Yi-Shiou Chiou. The impact study of seedparticipant to modified delphi card sorting. Journal of Libray andInformation Science Research, December 2013.

[SHMW15] Gerd Szwillus, Adrian Hulsmann, Yevgen Mexin, and Anastasia Waw-ilow. Casolysis 2.0 - Flexible Auswertung von Card Sorting Experi-menten. De Gruyter Oldenbourg, Berlin, 2015.

[Sim] Simple card sort. online at: https://www.simplecardsort.com/ ac-cessed on: 12-07-2017.

[Sin] Single-linkage clustering. online at: https://en.wikipedia.org/

wiki/Single-linkage_clustering accessed on: 12-07-2017.

[Sin13] Sergey Sinyakov. Can card sorting improve the usability of yourdesigns?, May 2013. online at: http://designinstruct.com/

web-design/card-sorting-usability-designs/ accessed on: 12-07-2017.

[Slia] New amazing features of slim framework. on-line at: https://www.agriya.com/blog/2016/10/25/

new-amazing-features-and-advantages-slim-framework/ ac-cessed on: 12-07-2017.

[Slib] Slim framework. online at: https://www.slimframework.com/ ac-cessed on: 12-07-2017.

[Spe04] Donna Spencer. Card sorting: a definitive guide,2004. online at: http://boxesandarrows.com/

card-sorting-a-definitive-guide/ accessed on: 12-07-2017.

[Sub] Sublime text. online at: https://www.sublimetext.com/ accessedon: 12-07-2017.

[Szw14] Prof. Dr. Gerd Szwillus. Card sorting. usability engineering practice,WS 2013/2014.

[Tec13] Segue Technologies. What is ajax and where is it used in technology?,2013. online at: http://www.seguetech.com/ajax-technology/ ac-cessed on: 12-07-2017.

[TW04] Tom Tullis and Larry E. Wood. How many users are enough for a card-sorting study?, 2004. online at: http://citeseerx.ist.psu.edu/

viewdoc/download?doi=10.1.1.514.3907&rep=rep1&type=pdf ac-cessed on: 12-07-2017.

[Ver] Vertabelo - design your database online. online at: http://www.

vertabelo.com/ accessed on: 12-07-2017.

84

https://www.simplecardsort.com/

https://en.wikipedia.org/wiki/Single-linkage_clustering

https://en.wikipedia.org/wiki/Single-linkage_clustering

http://designinstruct.com/web-design/card-sorting-usability-designs/

http://designinstruct.com/web-design/card-sorting-usability-designs/

https://www.agriya.com/blog/2016/10/25/new-amazing-features-and-advantages-slim-framework/

https://www.agriya.com/blog/2016/10/25/new-amazing-features-and-advantages-slim-framework/


http://boxesandarrows.com/card-sorting-a-definitive-guide/

http://boxesandarrows.com/card-sorting-a-definitive-guide/

https://www.sublimetext.com/

http://www.seguetech.com/ajax-technology/

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.514.3907&rep=rep1&type=pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.514.3907&rep=rep1&type=pdf

http://www.vertabelo.com/

http://www.vertabelo.com/

Bibliography

[WeC] Wecaso. online at: http://wecaso.de/ accessed on: 12-07-2017.

[WW08] Jed R. Wood and Larry E. Wood. Card sorting: Current practicesand beyond. Journal of Usability Studies, 4, November 2008.

[Yan] Yandle. online at: https://www.flickr.com/photos/yandle/ ac-cessed on: 12-07-2017.

85

http://wecaso.de/

https://www.flickr.com/photos/yandle/

analysis of modified delphi card sorting - uni-paderborn.de · ed delphi card sorting experiment...

Documents