SUNIL GAHLAWATLU GAN

MIYA SYLVESTERYIRAN WANG

Usability and Utility of

TopicLensa Visualization System for the Exploration of Topic Models

Topic Models

Topic models provide a simple way to analyze large volumes of unlabeled text.

Topic Models

A "topic" consists of a cluster of words that frequently occur together.

Topic Models

An example of Topic Modeling : Twitter tweets search for "Hollywood"

Why develop these kinds of Models?

Organizing and finding patterns in data has become important in the sciences, humanities, industry, and culture.

Topic models can be built for images, social networks, music, purchase histories, computer code, genetic data, click-through-data...

TopicLens Interface

TopicLens Interface

TopicLens Interface

Goals

Get a good understanding whether the user editing function in TopicLens is useful or necessary.

Identify and classify any usability defects.

Study whether the design elements and aesthetics enhance the Usablity of the program.  

Evaluate User satisfaction/User experience in terms of ease of learning, retention, efficiency, and reliability.

A Detailed Summary of Usability Study

Primary focus on User Testing Initial Heuristic Analysis Short Survey and Interview with Participants

6~10 Participants Two sub-groups to evaluate different usage behavior

3~5 Students, 3~5 Non-students Recruited from grad student housing, university center, etc.

Recording: Audio and Screen Video CaptureTime: 30~40 minutesLocation: Donald Bren Hall

Part1: First Impression on Visual Presentation We will briefly explain about TopicLens to the participant  Participants will be asked to think aloud as they experiment

with the interface. Editing Mode: OFF 

A Detailed Summary of Usability Study

Part2: Details on User Editing Functionality We will give participants a few scenarios to explore all the functions

available on TopicLens (merging/filtering/hiding) and answer any questions the participant may have about these functions. Bill is Professor at a research university who interested in

software/hardware and security. Bill will be using the merging function of TopicLens to visually organize his topics.

Hazel is a movie buff and she wants to see topics related to all movies. She’ll be filtering function to find the topics related to movies. Hazel has filtered her movie topics but there are still too many for her

to sort through. She will use the hiding function to better suit her needs.

Participants will be asked to think aloud as they complete the tasks. Editing Mode: ON

A Detailed Summary of Usability Study

Part3: Survey and Interview Survey:

Participants will be asked to complete a short survey (adapted from System Usability Scale) on the functionality of TopicLens. The following are examples participants would rate on a scale of 1(strongly disagree) to 5(strongly agree):

I thought the system was easy to use.

I found the various functions in this system were well integrated.

I found the system unnecessarily complex. 

A Detailed Summary of Usability Study

Interview: We will also ask Participants open-ended questions that focus on editing:

Was it easy to understand the purpose of merging/hiding/filtering the topics or not?

After the editing, does the visual layout of the topics make more or less sense to you?

Right now, once you merge two topics, the nodes associated are either related to one or both topics. Do you prefer this, or do you prefer to see topics that are only related to both topics you merged? (For Participants who like the editing function, we will ask detailed questions about the AND/OR Boolean for two or more topics.)

Would you use the editing function or not? Would you recommend the editing function to a friend or not?

A Detailed Summary of Usability Study

Plans to analyze usability test data

Qualitative "think aloud" data We will play back the audio recording of each participant during their

period of experimenting the system and transcribe those that we find most useful.

We will discuss any emergent patterns of user intentions, frustrations, and opinions that were expressed by Participants as a group.

Survey results We will use Excel to perform basic analysis of the rating scales We will also compare and contrast the ratings and their open-ended

comments.

Open-ended interview question We will transcribe the audio-taped open comments of each participant. We will use content analysis on the open comments.

Challenges

Environment configurationFinding participantsStriving for consistency Instructions Interview questions Testing situation“Hard” participantsTransforming data into findings and practical

recommendations

Timeline

WEEK PlanPeopl

e

Week 3~4 Identify topics for questions and user sub-population profiles.   Group

Week 4 Create an activity proposal Group

Week 4~5

Word the questions appropriately  Group

Distribute the questions to customer for review Yiran

Find the SUS for the questionnaire Yiran

Week 5

 Identify and recruit participants for our study Group

 Determine the roles for our Usability Test Group

Prepare interview materials including documentation for confidentiality, consent forms

Miya

Acquire location in Donald Bren Hall Mark

Acquire incentives (starbucks, coupons, etc.) Sunil

Week 6Conduct pilot test, make necessary changes to questions and procedure

based on pilotGroup

Week 7~8 Set up location with all materials necessary for activity Mark

Call and confirm with participants and begin Usability Studies Miya

Week 8~10 Analyze the data Group

Week 11 Submit the final report Group

 Drawbacks of TopicLens Observed

Font size is not adjustableThe background colors of program boxes

sometimes make the label unreadableRange slider design could be betterTitles of topics overlap when dragged

togetherReset/return button missing