the searchmaster's toolbox
DESCRIPTION
The SearchMaster's Toolbox. David Hawking. ECIR Industry Day 01 Apr 2010. UK Customers. From 2004/5 : Staffordshire University, Scottish Care Commission From 2009 :The Electoral Commission, Digital UK, Hargreaves Lansdown - PowerPoint PPT PresentationTRANSCRIPT
The SearchMaster's Toolbox
ECIR Industry Day 01 Apr 2010
David Hawking
UK Customers• From 2004/5: Staffordshire University,
Scottish Care Commission
• From 2009:The Electoral Commission, Digital UK, Hargreaves Lansdown
• From 2010: London School of Economics and Political Science, Incisive Media, British Medical Journal, East Ayrshire Council, ...
“Search is life”
Costs of poor search• Butler Group: Up to 10% of salary costs
wasted through ineffective search• IDC: A company with 1000 information
workers can expect to waste more than $5M p.a. due to poor search
• Accenture: A survey of 1000 middle managers spend as long as 2 hrs/day searching for information.
Who's the SearchMaster in your organisation?
Stakeholders expect every SearchMaster to do her duty!
• To make external website search work– Sales conversions– Information dissemination– Reduced inquiry handling load
• To provide effective search of corporate information– Happy, productive employees (plus students
and other stakeholders)
Give them the tools and they will do the job!
• Searchmaster• End-user
• Simple• Powerful
1. The basic search tool• Should:
– Have good performance out of the box, without weeks of implementation.
– Be simple to configure– Avoid features which are too complex to use or
set up.– Be able to cover your content and scale to the
necessary level
2. FineTuner• Every search deployment is different
– Web, database, fileshare, Lotus
• The weighting of ranking features must accommodate to the differences
• Manual tweaking is fraught with danger– Fix one query, break a dozen
• Make a test file and use a tuning tool to learn feature weightings
Testfile Desiderata• Representative of real workload
– Need an unbiased sample
• Many queries (typically >> 100)• Multiple weighted answers (where
applicable)• Redirects• Equivalent answers• See es.csiro.au/C-TEST/
Academic Research on Evaluation
• Masses of academic research• How does it translate to tuning an
enterprise search system?– Setting good defaults– Tuning to specific characteristics in hundreds
of customer deployments
• Note: the system starts with no user interaction data.
• Creation of testfiles must be affordable.
Spreadsheet testfileemployment health.gov.au/health-career-vacant.htm
jobs health.gov.au/health-career-vacant.htm
vacancies health.gov.au/health-career-vacant.htm
recruitment health.gov.au/health-career-vacant.htm
tenders health.gov.au/list-of-tenders-and-grants.htm
grants health.gov.au/list-of-tenders-and-grants.htm
tenders health.gov.au/list-of-tenders-and-grants.htm
mental health health.gov.au/mental-health-and-wellbeing
mental health strategyhealth.gov.au/mental-strategy
aged care health.gov.au/aged-care.htm
LSE Case Study
Sources of testfiles at LSE• A-Z Sitemap (>500 entries)
– Biased toward anchortext
• Keymatches file (>500 entries)– Pessimistic
• Click data (>250 queries with > t clicks)– Biased toward clicks – 100% success!
• Pop/crit queries (134 manually judged)
All biased – Use a sampling tool!
1 2
3
dim2
dim1
Dimension-at-a-time tuning
Out of boxAs configured
-daat (tuned)-daat20000 (tuned)
-daat0/TAAT (tuned)
0
5
10
15
20
25
30
Popular/Critical Set
Fine Tuning Summary• Tuning a large number of dimensions
(Funnelback FineTune covers 38)• Millions of query executions• Achieves substantial gains
But why do queries still fail?
• Misspelled– Europian Conferense oninformation retreival
• Query words don't match document– “door” or “MOPEM” v. “manually operated
personnel egress mechanism”
• There is no answer to that question.– Maybe there should be– Scope issues.
Need more tools!
3. Spelling suggestion tools• Suggestions may be useful even if words
are correctly spelled:– Carlton furball club → Carlton football club
• Suggestions based on whole query, not word-by-word
• Don't suggest queries which make no sense in the collection being searched
• Autocompletion: Guide users to the best query
• Context is king
4. Query expansion tools• Manual rules:
– Rego → [registration rego]– MOPEM →[“manually operated personnel
egress mechanism door”]
• Related queries (automatic)– Based on co-clicking
• Contextual navigation (on-the-fly)– Finding superphrases in a deep result set
• Faceting (semi-automatic)
5. Reporting and alerting tools• Reporting on Queries which:
– Produced no results– Logged behaviour suggestive of unfulfilment
• Alerting when:– Submissions of a query (or group of related
queries) sharply increase in frequency
• For:– business intelligence– Triggering creation or changes to content
Query Spike Alerting
Conclusions• Search is important• Organisations benefit when someone takes
responsibility for effective search – the SearchMaster.
• Academic research into evaluation needs careful translation for use in enterprise search tuning.
• Further tools are needed to overcome poor queries and missing content.
Thanks to Mike Swanson of Oxfam Australia for the Ned Kelly line.