©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
PhillyCHI Site Search Analytics: Conversations with your customers
Louis Rosenfeld [email protected] Philadelphia, PA • April 2, 2008
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
About us
Me
You?
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Anatomy of a search log (from Google Search Appliance) Critical elements in bold: IP address, time/date stamp, query, and # of results:
XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=regional+transportation+governance+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Querying the queries: Generic questions to help get started
1. What are the most frequent unique queries? 2. Are frequent queries retrieving quality results? 3. Click-through rates per frequent query? 4. Most frequently clicked result per query? 5. Which frequent queries retrieve zero results? 6. What are the referrer pages for frequent queries? 7. Which queries retrieve popular documents? 8. What interesting patterns emerge in general?
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Improving metadata and navigation: How SSA can help
1. Analyze for metadata value tone, granularity, and trends
2. Determine (and invest in) important metadata attributes
3. Study queries that follow navigation failures/changes to create better contextual navigation
4. Look to improve “top-down” navigation (e.g., indices, taxonomy, main page)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Determining metadata values through clustering
One week’s top 50 queries, clustered by topic
Represents 20% of all search activity
< 1 hour of work Can help suggest
preferred terms
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Determining metadata values through similar queries
BBC explores similar queries (i.e., collaborative filtering)
Helps with misspellings, synonyms, best bets
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Determining metadata attributes (AKA types, fields)
Top queries clustered and prioritized by attributes: Place Dept./
Program Service Task
1-2 hours work
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Navigation failures/goal changes: (from Google Analytics)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Supporting contextual navigation through embedded queries
SLI Systems can display other queries related to a particular query…
…and queries related to a particular document
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Hybrid A-Z index example: Michigan State University
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Improving content: How SSA can help
1. Verify content spidering 2. Plug content gaps
(placeholders, new content, appropriate types, best bets)
3. Improve content positioning (highlighting, demotion, deletion)
4. Prioritize content development (invest in appropriate topics and types, more efficient migration and inventory)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Developing new content: Addressing emergent content needs
Using SSA to plug gaps Failure analysis suggests
unmet content needs by topic (e.g., frequent queries for “iPod wristband” and “iPod holder for jogging”)
Content typing (covered later) suggests what kind of content to create (e.g., “product comparison” and “specifications”)
0 results report (from behaviortracking.com)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Developing new content: Addressing emergent content needs
Use SSA to develop editorial agenda; determine relative “demand” for content topics and document types
Available content (supply) Content by query (demand)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Best bets: HP example (exposes content model)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Monitoring trends: Tuning content to meet demand
How might such trends impact your content? Search engine? Navigation?
From behaviortracking.com
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Monitoring trends
Short head and long tail generally predictable
“Middle torso” is interesting area to monitor
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Comparing short head and long tail queries (from Vanguard)
Short head: common queries Long tail: common queries
Beneficiary form 401(k) beneficiary career forms amt money market location loans calculator
403(b)(7) account asset transfer authorization automatic investing Wire transfer instructions adoption agreement international wire transfers socially responsible investing Vanguard tax identification number IRA Asset Transfer form fdic insured account early withdrawal penalties
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
The shape of your content: Determining content types
Top queries clustered and prioritized by content types: Application News/
Announce-ments
Main page Contact info Instructions
1-2 hours work
“What kind of page would users want when they searched this term?”
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Content types help expose content models, improve navigation
23
artist descriptions
album reviews
album pages
artist bios discography
TV listings
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Engaging content owners: Showing how users find their content
Connecting pages that are found through search…
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Engaging content owners: Showing how users find their content
…with how those pages were found.
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Engaging content owners: Comparing time periods
Showing trends
(from behaviortracking.com)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Improving search: How SSA can help
1. Understand how search works (and what can be changed/fixed)
2. Validate “the box” UI design 3. Look for opportunities to “soup up” queries
(e.g., syntax, spell-checking) 4. Design better search results (individual and
groups) 5. Blow up “advanced search” and
contextualize functions 6. Buy a better engine—or improve your own
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Improving “the box”: Initial query entry UI
Validate width of query entry field Use query data to calculate “high
number” (e.g., 90th percentile) for query length in characters
Does your entry field accommodate that length?
Amazon: 65 characters
Microsoft: 25 characters
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Improving “the box”: Search suggestions
Populate query entry with common queries (Ask)
Alternative: use controlled vocabulary (Netflix movie titles)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Query enhancement: Handling unique query types
Little fixes can add up Spell-checking Proper names (people, places) Unique identifiers (e.g., SKUs, ISBNs)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Query enhancement: Handling unique query types (cont.)
Little fixes can add up Acronyms (glossary lookup)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Search results design: What to show per individual result
…more descriptive information for open-ended searchers
…less information for known-item searchers
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Search results design: SSA helps determine sorting
Financial Times found a high level of dates within queries
Solution: support chronological sorting and filtering
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Search results design: SSA helps determine typing
Content typing can enable: Filtering by exposing types
(HP) Faceted classification/
navigation (WebMD)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Search results design: Which results where?
#10 result is clicked through more often than #s 6, 7, 8, and 9 (ten results per page)
From SLI Systems (www.sli-systems.com)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Rethinking advanced search: Which functions go where (or away)?
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Rethinking advanced search: “Revise your Search” in action
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
How we determine what users want
Qualitative research (observation) Field studies Focus groups Card sorting Usability testing and task analysis
Quantitative research (data-driven) Help/reference desk/switchboard Web analytics (SEO/SEM, clickstream, SSA)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
SSA as a user research method: How is it different?
Data that’s: Real High volume Readily available
Analysis that’s: Quantitative Inexpensive Scalable Complementary to other user research methods
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Comparing SSA with other quantitative user research methods
Data-driven methods Clickstream/server log analysis: tells you where
more than what users wants (complementary) Event logging: comparable but often less data
and usually at a different point in customer/user lifecycle
SEO/SEM: Different query types (finding where the answer is versus finding the answer); often focused on commerce conversions
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Comparing referral queries with local queries
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
SSA and qualitative user research methods
What questions lead to Why questions Examples
Personas: SSA supplements fictitious construct with real data (i.e., likely queries a persona might have)
Task analysis: SSA helps identify common tasks to evaluate
Field studies: SSA suggests issues to monitor in the field
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Augmenting personas and audience profiles with frequent queries
Persona example (from Adaptive Path)
Frequent queries added (in green)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Using common queries to enhance/fuel task analyses
Use for tasks that are more what than how (i.e., testing information needs rather than functions)
Examples • Can you find a map of the campus? • What study abroad options are available to MSU students? • When is the last home football game of the season?
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Identifying trends to monitor during field studies
Example: catalog clothing retailer SSA showed:
Preponderance of SKUs, even though they were not displayed on the site
What were they doing there? Subsequent field study showed:
Customers preferred print catalog to web catalog to identify products
Customers preferred web catalog for ordering Action: add SKUs to web catalog
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Combining quantitative and qualitative
Exploring the needs of a particular audience: users of BBC children’s content
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
The power of data: Why UX needs SSA
1. Balance: UX methodologies/practitioners are often quantitatively weak
2. Legitimacy: Data-driven analyses make an impression on people hard to impress
3. Cost: The data is available; the analysis scales well with available time
4. Fidelity: The data is behavioral, real, and voluminous
5. Comprehensiveness: Quantitative methods complement and help improve qualitative methods (and vice versa)
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Please share your SSA knowledge: Visit our book in progress site Search Analytics for Your Site:
Conversations with your Customers by Louis Rosenfeld and Richard Wiggins (Rosenfeld Media, 2008)
Site URL: www.rosenfeldmedia.com/books/searchanalytics/ Feed URL: feeds.rosenfeldmedia.com/searchanalytics/
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Slides and discounts
Slides available here: http://www.slideshare.net/lrosenfeld
For a discount, use code…
PHILLYCHI
…at the Rosenfeld Media site for 10% off all of our titles
©2008 Louis Rosenfeld, LLC (www.louisrosenfeld.com). All rights reserved.
Contact Information
Louis Rosenfeld Rosenfeld Media, LLC 705 Carroll Street, #2L Brooklyn, NY 11215 USA
www.louisrosenfeld.com www.rosenfeldmedia.com