1 can people collaborate to improve the relevance of search results? florian eiteljörge...
TRANSCRIPT
1
Can People Collaborate to Improve the relevance of Search Results?
Florian Eiteljö[email protected]
June 11, 2013Florian Eiteljörge
Outline
• Web search & social search techniques• Phase one: Study setup & results• Phase two: Study setup & results• Discussion
June 11, 2013Florian Eiteljörge
Florian Eiteljörge
Web Search
• Search engines heavily used on internet• studies indicate: 50% of web search sessions fail
• Idea: use social search techniques to improve web search
June 11, 2013
Florian Eiteljörge
Social search techniques
• Idea• people search for something and give (implicit) feedback by clicking on result
items• Most clicked items seem to be mostly relevant – so they will be ranked higher
next time.• Problem
• users tend to click on the top result items• popular sites get even more popular, even if there are new high-quality pages
that would be more relevant ("rich-get-richer" phenomenon)
June 11, 2013
Florian Eiteljörge
What is the paper about?
Authors had three hypothesis related to social search techniques:• H1: Users will prefer to rate results at the top of the result lists, whether
the results are randomized, or in the order that Google presents them.• H2: Users explicit relevance rankings are not biased by the rank of the
result list [while implicit feedback is biased]• H3: For some types of queries people's collaborative effort can produce
better ordering of search results.
The authors developed a search engine environment to capture user respond by presenting Google's top ten results in randomized order to test the above hypothesis
June 11, 2013
Florian Eiteljörge
Study setup – phase one (rating)
145 participants were invited by mail to rate search results for their relevance
• participants had the possibility to rate any number of results of preselected queries in the most frequent categories (shopping, health, technology, business, computers, arts)
• participants were free to choose categories and queries they wanted to rate• the result items were presented in random order• Google-like result item layout• relevance was measured on a 4-point scale: highly relevant, relevant, don’t
know, not relevant• after rating queries, each participant was asked to answer a short survey to
determine how experience in searching affects the relevance perception
June 11, 2013
Florian Eiteljörge
Results
June 11, 2013
first bar: percent of selection of the item for ratingsecond bar: percent of times when item was rated as highly relevant
Florian Eiteljörge
Results – phase one
June 11, 2013
• participants preferred to rate the first two items (H1 confirmed)• participants explicit feedback not biased in general (H2 mostly confirmed)• feedback for the first item is biased: rated highly relevant in 70% of the
times (even if participants were told the order is randomized)
Florian Eiteljörge
Study setup – phase two (evaluation)
20 participants were invited to choose if they prefer the results based on the explicit user-feedback or the Google-results
• the invited participants self-identified themselves as novice searchers• both result-lists were displayed side-by-side• the new ranking was created with the following formula:
score = 3 x highly-relevant-count + 2 x relevant-count + don’t-know-count + (-1) x not-relevant-count
June 11, 2013
Florian Eiteljörge
Results – phase two
• in some categories the users rate result items very different from Googlee.g.: shopping (digital cameras, walking shoes) – a mean difference in ranking of 4.2
• in some categories users agree with the Google rankinge.g.: Business (Microsoft Bid for Yahoo, Online Advertisement) – a mean difference of 0.8
• 70% of the participants rated the user-based ordering higher than the Google-ordering; these participants chose to rate queries in the categories shopping, computers and arts
• the other 30% preferred the Google-ranking while choosing to rate queries of the categories business and technology
June 11, 2013
Florian Eiteljörge
Conclusion
• people prefer the top result items• explicit feedback is not biased in general• in some categories the Google-ranking is very inconsistent to the users
ranking
June 11, 2013
Florian Eiteljörge June 11, 2013
Discussion
Florian Eiteljörge
Presentation based on
Morris MR, Horvitz E. SearchTogether: an interface for collaborative web search. Symposium on User Interface Software and Technology. 2007:3-12http://www.grouplens.org/system/files/p283-agrahri.pdf
June 11, 2013