Strategies LLCTaxonomy
October 7, 2009 Copyright 2009 Taxonomy Strategies LLC. All rights reserved.
How to Validate a Taxonomy
ASIST Workshop Preview
2Taxonomy Strategies LLC The business of organized information
What is a Taxonomy?
v A categorization framework agreed upon by business and content owners (with the help of subject matter experts) that will be used to tag content.
6 broad, discrete divisions (called facets) 2-3 levels deep. Up to 15 terms at each level. 1200 terms total. With some logic—hierarchical, equivalent and associative relationships
between terms.
3Taxonomy Strategies LLC The business of organized information
Why taxonomy is important
4Taxonomy Strategies LLC The business of organized information
Why is taxonomy important?
v Easier information management.v Flexibility to respond to changing needs.v Foundation for findability and usability.
5Taxonomy Strategies LLC The business of organized information
How can you know if a taxonomy will work?
v Tagging content.v Publishing content.v Finding and using content in user-facing applications.
6Taxonomy Strategies LLC The business of organized information
What types of taxonomy validation can you do?
v Validating taxonomy design.v Validating taxonomy usability.v Validating taxonomy use in a collection.
7Taxonomy Strategies LLC The business of organized information
Taxonomy design validation methodsMethod Process Who Requires Validation
Web tool Open Card sort
Representative users
Rough taxonomy Card sort analysis (emergent patterns)
Cards & markup
Delphi card sort
Representative users
Rough taxonomy End result (consensus)
Walk-thru Show & explain
Stakeholders Rough taxonomy Approach Appropriateness to task
Walk-thru Check conformance to editorial rules
Taxonomist Draft taxonomy Editorial rules
Consistent look & feel
Matrix or cards
Closed card sort
End users (or surrogates)
Rough taxonomy Top queries, etc.
Consistent results
User satisfaction
Survey End users (or surrogates)
Draft taxonomy Reaction to taxonomy
Tagging samples
Tag sample content with taxonomy
Taxonomist Indexers
Sample content Draft taxonomy
(or better)
Content “fit” Fills out content inventory Training materials for people &
algorithms Basis for quantitative methods
8Taxonomy Strategies LLC The business of organized information
Usability validation methods
Method Process Who Requires ValidationUsability Testing
Task-based scenario testing (Find it, Tag it, etc.)
End users (or surrogates)
Draft taxonomy Tasks & answers
Tasks are completed successfully Time to complete task is reduced
User Satisfaction
Survey End users (or surrogates)
Draft taxonomy UI mockup,
Search prototype
Reaction to taxonomy Reaction to new interface,
Reaction to search results
9Taxonomy Strategies LLC The business of organized information
Collection analysis validation methods
Method Process Who Requires ValidationDistribution Statistical
analysis Taxonomist Analyst
Tagged collection
Do categories follow Zipf distribution?
Do % of categories equal % of documents?
Identify candidates for merging & splitting.
Query log & click trail examination
Clustering & statistical analysis
Taxonomist Analyst
Search logs Web analytics
Do (clustered) queries follow Zipf distribution?
Are top (clustered) queries in taxonomy?
10Taxonomy Strategies LLC The business of organized information
Full-day taxonomy validation workshop agenda
9:00-9:30 30 min Introduction9:30-10:00 30 min Taxonomy & validation overview
10:00-10:30 30 min Taxonomy design validation10:30-10:45 15 min Coffee Break10:45-11:30 45 min Delphi card sorting exercise11:30-12:00 30 min Closed card sorting
12:00-1:00 60 min Lunch1:00-1:45 45 min Closed card sorting exercise1:45-2:15 30 min Online tools2:15-2:45 30 Min Task-based validation methods2:45-3:00 15 min Coffee break3:00-3:45 45 min Task-based validation exercise3:45-4:15 30 min Collection analysis overview4:15-5:00 45 min Q&A, Closing
Hands-on Exercises BreaksLectures
11Taxonomy Strategies LLC The business of organized information
Closed card sorting case study
v Project: Develop Substance Abuse and Mental Health Services Administration (SAMHSA) taxonomy to be the backbone of a unified agency information service that collects, integrates, and reports on customer interaction data in ways that help the agency plan health communications products, messages and outreach.
v Task: Sort popular queries (words and phrases) from search logs into the most likely Taxonomy facet.
12Taxonomy Strategies LLC The business of organized information
Term sorting data collection form
Term Content
Type AudiencePopulation
Groups SubstancesConditions &
Disorders
Intervention & Treatment
TopicsProfessional &
Research TopicsGeographic Locations
Alcohol Anger Anxiety Autism Binge Eating Bipolar Disorder Change Management Child Co-occurring Disorders Counseling Depression Fulton County Health Literacy Immigrant Adolescents Inhalants Low Income Methadone Native American Culture Ohio Opium Posters PTSD Residential Treatment Rohypnol Smoking Stress Substance Abuse Suicide University Students Vicodin Webcast
13Taxonomy Strategies LLC The business of organized information
Summary of term sorting results
Term Content
Type AudiencePopulation
Groups SubstancesConditions &
DisordersIntervention &
Treatment TopicsProfessional &
Research TopicsGeographic Locations None
Alcohol 1 10 1 Anger 9 1 2 Anxiety 12 Autism 12 Binge Eating 10 1 1 Bipolar Disorder 12 Change Management 2 9 1Child 12 Co-occurring Disorders 11 1 Counseling 9 3 Depression 12 Fulton County 12 Health Literacy 12 Immigrant Adolescents 12 Inhalants 10 1 1Low Income 1 9 2 Methadone 10 2 Native American Culture 1 7 4 Ohio 12 Opium 12 Posters 9 1 1 1PTSD 12 Residential Treatment 1 11 Rohypnol 10 1 1Smoking 1 5 3 1 2 Stress 10 1 1 Substance Abuse 2 1 1 4 1 3 Suicide 1 7 1 3 University Students 12 Vicodin 11 1 Webcast 9 1 2
Frequently chosen related category Frequently chosen incorrect categoryCorrect category
14Taxonomy Strategies LLC The business of organized information
Percentage of popular search terms sorted correctly
Anxiet
yAuti
sm
Bipola
r Diso
rder
Child
Depre
ssion
Fulto
n Cou
nty
Health
Lite
racy
Imm
igran
t Ado
lesce
nts
OhioOpiu
mPT
SD
Univer
sity S
tude
nts
Co-o
ccur
ring D
isord
ers
Resid
ential
Trea
tmen
tVico
dinIn
halan
tsRo
hypn
olAlco
hol
Binge
Eating
Stre
ss
Chan
ge M
anag
emen
tPo
sters
Anger
Coun
selin
gLo
w Inco
me
Web
cast
Native
Am
erica
n Cult
ure
Suici
deSm
oking
Subs
tanc
e Abu
seM
etha
done
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
15Taxonomy Strategies LLC The business of organized information
Blind sorting of popular search terms (n=12)
50-60%(7%)
25-50%(6%)
< 25%(3%)
84% of terms were correctly sorted 60-100% of the time.
Results: Excellent
Difficultiesv For Methadone, confusion when, in this case, a substance is a treatment.v For general terms such as Smoking, Substance Abuse and Suicide, confusion about
whether these are Conditions or Research topics.
16Taxonomy Strategies LLC The business of organized information
Search terms sorting task user rating (n=12)
Easy42%
Easy/Medium25%
Medium33%
No one rated the task difficult!
17Taxonomy Strategies LLC The business of organized information
Task-based case study
v Project: Develop American Society of Civil Engineers (ASCE) taxonomy to draw from existing databases and content sources, and provision the next generation websites being built to support this non-profit, membership association’s activities.
v Task: Find specific content (web pages)
18Taxonomy Strategies LLC The business of organized information
Find web pages
A Audiences
C Content Types
E Event Types
L Locations
O Organizations
T Topics
T TopicsT.1 Architectural EngineeringT.2 Coasts & waterwaysT.3 ConstructionT.4 Cross-Cutting TopicsT.5 Disaster & Hazard
ManagementT.6 Education & Career
DevelopmentT.7 Engineering MechanicsT.8 EnergyT.9 EnvironmentT.10 Geotechnical EngineeringT.11 People, Projects & HeritageT.12 Planning & DevelopmentT.13 Professional IssuesT.14 Project ManagementT.15 Structural EngineeringT.16 TransportationT.17 Water & Wastewater
T.6 Education & Career Development
T.6.1 Continuing EducationT.6.2 Engineering EducationT.6.3 Management &
Professional DevelopmentT.6.4 Scholarships, Internships
& Competitions
ASCE Continuing Education http://www.asce.org/conted/
19Taxonomy Strategies LLC The business of organized information
Summary of navigation results trial
JF KSS MJ KS IM VM KSM GSB KCH
Trial 1 2 3 4 5 6 7 8 9 Key
Navigation
ASCE Continuing Education A > C > T A > O > C C > A > T C > E
Audience A.27
Content Type C.12
Event Type E.4 E.4 E.4 E.4
Location
Organization
Topic T.6.1 T.6.1 T.6.1
Difficulty Difficult Easy Difficult Easy Easy Difficult Medium Easy Medium Difficulty
Frequently chosen related category Frequently chosen incorrect categoryCorrect category
Gave up
20Taxonomy Strategies LLC The business of organized information
Overall navigation task performance (n=54)
Correct41%
Alternative46%
In-correct9%
Gave Up4%
v 87% navigated as predicted or used a reasonable alternative.v In only 4% of the trials, did the subject give up.
21Taxonomy Strategies LLC The business of organized information
Overall user rating of navigation task (n=9)
No one rated the overall task Difficult!
Easy33%
Medium67%
Strategies LLCTaxonomy
October 7, 2009 Copyright 2009 Taxonomy Strategies LLC. All rights reserved.
QuestionsASIST Annual Meeting - Full-Day Workshop
November 7, Vancouver, BC http://www.asist.org/Conferences/AM09/taxonomy.html
Gilbane Conference - Half-Day WorkshopDecember 1, Boston, MA
http://gilbaneboston.com/workshops.html#workshop
Joseph A. [email protected]
http://ww.taxonomystrategies.com