the science of great site navigation: online card sorting + tree testing
DESCRIPTION
TRANSCRIPT
The Science of Great Site Navigation: Online Card Sorting + Tree Testing
Jeff Sauro | Measuring Usability LLC
@userzoom #uzwebinar
CARDS AS PSYCHOLOGICAL APPARATUS
Traditional Card Sort
Online Card Sorting
Card Sorting is as common as Lab based Usability Testing
Source: 2011 UxPA Salary Survey
Terms & Concepts
• Open Sort: Users sort items into groups and give the groups a name.
Closed Sort: Users sort items into previously defined category names.
• Reverse Card Sort (Tree Test) : Users are asked to locate items in a hierarchy (no design)
• Most Users Start Browsing vs Searching: Across 9 websites and 25 tasks we found on average 86% start browsing
http://www.measuringusability.com/blog/card-sorting.phphttp://www.measuringusability.com/blog/search-browse.php
Target.com Case Study
Method & FAQs
• We want to improve the Target.com navigation. – Obtain a baseline set of metrics and problem spots– Make improvements– Compare against baseline
• Pick a variety of items to test the taxonomy• Conduct an Open Sort then Tree-Test• Use Product Names with some level of context
40 Items Across Departments1. Presto GranPappy Electric Deep Fryer2. Black CleanAirBall Air Purifier3. Nano Tech Shaver4. Hamilton Beach FashionFirst Medium Iron5. Brother International Sewing Machine-
XR9500PRW6. T3 White Micro Featherweight Hair Dryer7. Signature Sleep Silhouette 10” Queen8. Thomas O’Brien Canterbury Blue Bath Rug9. Fieldcrest Luxury Towel Collection10. Privet House Sugar Sack Pillow11. P Dog ZIP Lead Retractable Blue Leash12. Kitchen Aid 3pc BBQ Set13. Women’s Mossimo Kasey Boot14. Women’s Mossimo Pearce Black Leather Pumps15. Men’s Brown Swiss Gear Jackson Hiker16. Totes Clear Bubble Umbrella17. Merona Yellow Scarf Tote18. Wildkin Kaleidoscope Backpack19. NBA Golden State Warriors Blue Water Bottle20. Grey Stripe Merona Men’s Short-Sleeve Polo
21. Corellian Freighter Men’s Graphic Tee22. Inset Mother of Pearl Stainless Steel Cuff Links23. Xhilaration Black Large Sunglasses24. Wrangler Mens Reversible Belt25. Cherokee Boys Drawstring Cargo Short26. Girls Danshuz Pink Short-Sleeve Sparkle Straight
Dress27. Tiddliwinks Blue Dot Plush Monkey28. Black Griffin Case for Apple iPod touch29. Marvel POWER ATTACK AVENGER: IRONMAN30. Titanic (2 Discs) (Blu-ray/DVD)31. PC Diablo 3 Computer Game32. PC Treasures FlipIt 10” Tablet Sleeve33. Commodore Big And Tall Executive Oversize Chair34. Classic Borders Wedding Invitation and Response
Card35. Hello Kitty Party Favors 32-pc.36. Women’s Obsession by Calvin Klein Eau de
Toilette37. C9 by Champion Women’s Molded Cup Racer
Back Sports Bra38. Coffee Shop Outerwear Women’s Plus-Size Spring
Twill Coat39. Women’s One Shoulder Shantung Dress40. $5-$1,000 Iconic Puppy GiftCard
Overview
• 50 Participants– 65% Female– 2 Excluded for Quality Control (Haphazard Sorts)
• Sort 40 Items into 10 Fixed Categories• Median Study Time: 11 Minutes• Output– Dendrogram of Clusters– Open Category Names– Most Difficult to Sort
12 Clusters
1. Men’s / Boys2. Women’s / Girls3. Accessories 4. Runt: NBA5. Kitchenware6. Bed + Bath7. Runt : Furniture8. Toys9. Electronics10. Runt: Shaver11. Misc/Pets: Gift Card & Leash12. Misc: Party Favors & Invites
Dendrogram
Most Popular Category Labels
Similar Category Names CombinedPercentages are % of 48 participants who labeled a category, so % adds up to more than 100%Error Bars are 90% Confidence Intervals
Most Difficult Items to Categorize
Percentages are % of 48 participants who selected item, so % add up to more than 100. Error Bars are 90% Confidence Intervals
Dendrogram Close Up
Item 1st % 2nd % 3rd % 4th % 5th %Black Griffin Case for Apple iPod touch Electronics 65 Entertainment 15 Accessories 10 Misc 4 Gifts 2
PC Diablo 3 Computer Game Electronics 63 Entertainment 29 Misc 4 Other 2 Gifts 2PC Treasures FlipIt 10”Tablet Sleeve Electronics 58 Entertainment 13 Misc 10 Accessories 10 Gifts 2Grey Stripe Merona Men’s Short-Sleeve Polo Men 56 Clothing 40 Boys 2 Accessories 2Corellian Freighter Men’s Graphic Tee Men 56 Clothing 38 Sporting Goods 2 Accessories 2 Boys 2C9 by Champion Women’s Molded Cup Racer Back Sports Bra Womens 54 Clothing 38 Girls 4 Sporting Goods 2 Accessories 2Coffee Shop Outerwear Womens Plus-Size Spring Twill Coat Womens 54 Clothing 38 Girls 4 Homeware 2 Accessories 2
Black CleanAirBall Air Purifier Homeware 52 Electronics 13 Misc 8 Appliances 8 Bed & bath 2Women’s Mossimo Kasey Boot Womens 52 Clothing 27 Shoes 15 Girls 4 Accessories 2Women’s Mossimo Pearce Black Leather Pumps Womens 52 Clothing 27 Shoes 13 Accessories 4 Girls 4Women’s One Shoulder Shantung Dress Womens 52 Clothing 40 Girls 4 retail 2 Accessories 2P Dog ZIP Lead Retractable Blue Leash Pets 50 Misc 15 Accessories 10 Outdoor 8 Homeware 6Titanic (2 Discs) (Blu-ray/DVD) Electronics 50 Entertainment
40 Misc 4 Womens 2 Gifts 2
Men’s Brown Swiss Gear Jackson Hiker Men 48 Clothing 35 Accessories 6 Shoes 4 Outdoor 4Hamilton Beach FashionFirst Medium Iron Homeware 46 Electronics 15 Appliances 10 Misc 8 Womens 6Brother International Sewing Machine-XR9500PRW Homeware 46 Electronics 19 Appliances 10 Misc 10 Womens 4Xhilaration Black Large Sunglasses Accessories 46 Womens 25 Clothing 10 Men 6 Misc 6Wrangler Mens Reversible Belt Men 46 Accessories 29 Clothing 19 Womens 2 Boys 2
Totes Clear Bubble Umbrella Accessories 42 Homeware 15 Womens 13 Misc 10 Outdoor 4
Items with > 40% of Votes as Primary Category
Item 1st % 2nd % 3rd % 4th % 5th %Signature Sleep Silhouette Queen Homeware 40 Bedroom 17 Bed & Bath 10 Furniture 10 Accessories 4
Privet House Sugar Sack Pillow Homeware 40 Bedroom 17 Bed & Bath 13 Furniture 6 Misc 4Cherokee Boys Drawstring Cargo Short Clothing 40 Men 31 Kids 17 Boys 8 Homeware 2Girls Danshuz Pink Short-Sleeve Sparkle Straight Dress Clothing 40 Womens 35 Kids 15 Girls 8 Accessories 2
Nano Tech Shaver Electronics 38 Bath 21 Homeware 13 Men 10 Misc 4
Thomas O’Brien Canterbury Blue Bath Rug Homeware 38 Bath 25 Bed & Bath 13 Misc 6 Bed bath 4
Kitchen Aid 3pc BBQ Set Homeware 38 Kitchen 33 Appliances 10 Outdoor 4 Misc 4Women’s Obsession by Calvin Klein Eau de Toilette Womens 38 Bath 17 fragrance 10 Accessories 10 Clothing 6$5-$1,000 Iconic Puppy GiftCard Pets 38 Misc 15 Gifts 13 Cards/Party 8 Accessories 4Presto Gran Pappy Electric Deep Fryer Homeware 35 Kitchen 31 Electronics 13 Appliances 10 Misc 6
Merona Yellow Scarf Tote Womens 35 Accessories 29 Clothing 21 Homeware 6 Luggage 4Commodore Big And Tall Executive Oversize Chair Homeware 35 Furniture 31 Misc 10 Office 6 Other 4T3 White Micro Featherweight Hair Dryer Homeware 33 Bath 17 Electronics 15 Misc 10 Appliances 6Fieldcrest Luxury Towel Collection Homeware 33 Bath 31 Bed & Bath 13 Bedroom 4 Misc 4Inset Mother of Pearl Stainless Steel Cuff Links Accessories 33 Men 31 Womens 15 Clothing 13 Jewelry 6Classic Borders Wedding Invitation and Response Card Cards/Party 33 Misc 17 Accessories 13 Gifts 6 Womens 6Wildkin Kaleidoscope Backpack Accessories 31 Kids 19 Misc 10 Clothing 10 Outdoor 8
Hello Kitty Party Favors 32-pc. Cards/Party 29 Misc 17 Kids 15 Gifts 6 Accessories 6Marvel POWER ATTACK AVENGER: IRONMAN Entertainment 27 Toys 19 Electronics 19 Kids 13 Misc 10Tiddliwinks Blue Dot Plush Monkey Toys 23 Kids 23 Misc 19 Homeware 4 Accessories 4NBA Golden State Warriors Blue Water Bottle Misc 21 Accessories 21 Sporting Goods 13 Kitchen 10 Men 10
Items with < 40% of Votes as Primary Category
Tree Test
Selected 14 Items From Open Sort
1. Wrangler Mens Reversible Belt2. Marvel POWER ATTACK AVENGER: IRONMAN3. Women’s One Shoulder Shantung Dress4. Grey Stripe Merona Men’s Short-Sleeve Polo5. Inset Mother of Pearl Stainless Steel Cuff Links6. Women’s Obsession by Calvin Klein Eau de Toilette7. Tiddliwinks Blue Dot Plush Monkey8. Hello Kitty Party Favors 32-pc.9. Nano Tech Shaver10. Brother International Sewing Machine-XR9500PRW11. Classic Borders Wedding Invitation and Response Card12. Kitchen Aid 3pc BBQ Set13. Men’s Brown Swiss Gear Jackson Hiker14. Wildkin Kaleidoscope Backpack
Overview
• 50 Participants• 68% Female
• Median Study Time 17 Minutes• Task Based Metrics• Confidence• Difficulty (SEQ)• Success• Time
Post Task Questions in Userzoom
Single Ease Question: SEQ
Tree Testing in Userzoom (User’s View)
Successfully Located Item
Hardest Items to Locate
Brother Sewing Machine 64% -The sewing machine didn’t have a separate section that I could find- it could fit into a lot of categories and it made it hard to nail down
Wildkin Kalidescope Backpack 38% - Not sure if it’s a child’s toy or real camping gear- Expected to find it in the children’s section
Hello Kitty Party Favors 36% - I didn’t find a section for things like Cards or Party Favors Could be for both men and women- not apparent under what main heading it would be under.
Men’s Brown Swiss Gear Jackson Hiker 34% - Did not know what this was, a boot? A Jacket?
Classis Border Wedding Invitations 39% - Was expecting to find a stationary or cards section
Twiddledinks Blue Dot Plush Monkey 12% - Did not know if this was a toy- No Stuffed animal section
Kitchenaid 3-pc BBQ set 10% -Did not know if it was Kitchen or a small appliance- Could be categorized under kitchen or outdoors
Marvel Attack Avengers: Ironman 8% - Did not know what kind of Toy it was- List of Toys was too long, it was hard to eliminate
Nano Tech Shaver 6% - Is it for a man or a women?- Did not know if it was electronics or another category
Women’s Obsession by Calvin Klein 6% - Did not know which category to put it under
Mother of Pearl Stainless Steele Cuff links 2% - Could have fit into more than one category
Item % that found it difficult Reason it was difficult
Verbatim Reasons For Difficulty
How difficult or easy was it to complete the task
Very EasyVery Difficult
Percentages are normalized percentile ranks from across 200 tasks. 50% = Average.
Confidence
Not at all Confident
Extremely Confident
Confidence & Success
DisastersDifficult
Deceptive
Hardest Items to Categorize – Tree Test Vs Open Sort
Difficulty sorting explains about 16% of difficulty
finding.
Sample Size ConsiderationsMargin of Error
90% Confidence (+/-) Sample Size24% 1020% 1515% 2814% 3213% 3812% 4511% 5410% 659% 818% 1037% 1366% 1865% 2684% 4213% 749
See Chapter 6 in Quantifying the User Experience
Conclusions
• Determine if new categories should be created or consolidated
• Look to confirm or deny hypotheses about categories and findability– Gender Searching– Sub Categories
• Watch for “false” sorts• If “key” items are hard to find, move them,
test them• It’s hard to do a Card Sort without a Tree Test
Find out More