hypersearching the web, chakrabarti, soumen presented by ray yamada
Post on 05-Jan-2016
217 Views
Preview:
TRANSCRIPT
Hypersearching the Web, Chakrabarti, Soumen
Presented By
Ray Yamada
Overview
• Why Do We Care?
• Purpose of The Paper?
• Solution by Clever Project
• Pros / Cons of the Paper
• Further Research
Why Do We Care?
• Web Link Analysis is crucial for efficient Crawling and Ranking algorithms
• Crawling: Google Sitemap Submission, Yahoo Directory
• Ranking: Relevant Result
Purpose of The Paper?
• To Overcome These Challenges:
– Its Size & Growth
– Its Content Types
– Language Semantics
– New Language
– Staleness of Results
– SPAM
– And More…
Solution: Hyperlinks, Hyperlinks, Hyperlinks…
• Can Think of the Web as a Directed Graph
• Node = Web page (URL)
• Edge = Hyperlink
Solution: HITS Algorithm
• Hyperlink-Induced Topic Search (HITS)– A.k.a. Hubs and Authorities
• Hubs – Highly-valued lists for a given query– Ex. Yahoo Directory, Open Directory Project and Bookmarking
sites.
• Authorities – Highly endorsed answers to the query– Ex. New York Times, Huffington Post, Twitter
• It is possible for a webpage to be both Hub and Authority– Ex. Restaurant Review Blogs
Solution: HITS Algorithm Cont…
• For each page p, we assign it two values hub(p) and auth(p)
• Initial Value: For all p, hub(p) = 1, auth(p) = 1 (or any predetermined number)
• Authority Update Rule: For each page p, update auth(p) to be the sum of the hub scores of all pages that point to it.
• Hub Update Rule: For each page p, update hub(p) to be the sum of the authority scores of all pages that point to it.
• Normalize and Repeat
Solution: HITS Algorithm Cont…
Hub(p) Num of Links Raw Score 0.249 3 0.747 0.321 4 1.284 0.181 2 0.362 0.123 2 0.246 0.088 2 0.176 0.015 1 0.015 0.018 2 0.036 0.003 1 0.003 0.003 1 0.003
Sum: 1.00 2.872 Authority Pages (q) Raw Score Auth(q) SJ Merc News 0.57 0.198 Wall St. Journal 0.57 0.198 New York Times 0.874 0.304 USA Today 0.59 0.205 Facebook 0.123 0.043 Yahoo! 0.121 0.042 Amazon 0.024 0.008
Sum: 1.000
Calculation
Pros:
– Accurately addresses concerns and challenges we currently deal with
– Great introduction to search engine algorithm
– Briefly covered many topics (Breadth)
Cons:
– Some materials are out of date (1999)
– Ex. Google vs. Clever Project
– Lack of Depth
– Ex. Normalization of Hub and Auth values
Further Research: HITS Algorithm – Extreme Cases
• Large-in-small-out sites
– High Auth(p)
– No Problem
• Small-in-large-out sites
– High Hub(p)
– Problem
Further Research: HITS + Relevance Scoring Method
• Vector Space Model (VSM)
– Documents and queries are represented by vectors
– Term Frequency
• Okapi Measurement
– Term Frequency + Document Length
• Cover Density Ranking (CDR)
– Phrase Similarity (How close terms appear)
Further Research: HITS + Relevance Scoring Method
• Use Cosine Relevance Test
Price
Car
Further Research: HITS + Relevance Scoring Method
• Three-Level Scoring Method (TLS)
– Manual Evaluation of Relevance• Relevant Links = 2 points
• Slightly Relevant Links = 1 point
• Inactive Links + Error Links (404, 603) = 0 point
• Irrelevant Links = 0 point
– Order of query terms matters
Further Research: Co-citation Graph
• Regular Link Graph:
• Co-citation Graph:
What’s Next?
• Google’s New Search Index: Caffeine
– Announced June 8th, 2010
– Up to 50% fresher results
– Twice as fast
• Real Time Search– Twitter / Facebook
http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html
References
• Chakrabarti, Soumen; Dom, Byron; Kumar, S. Ravi; Raghavan, Prabhakar; Rajagopalan, Sridhar & Tomkins, Andrew. (1999). "Hypersearching the Web" [Article]. Scientific American, June1999, ():.
• Longzhuang Li , Yi Shang , Wei Zhang, Improvement of HITS-based algorithms on web documents, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511514]
• Henzinger, M. (2001). Hyperlink analysis for the Web. IEEE Internet Computing, 5(1), 45-50.
• Kleinberg, Jon (1999). "Authoritative sources in a hyperlinked environment" (PDF). Journal of the ACM 46 (5): 604–632. doi:10.1145/324133.324140.
• von Ahn, Luis (2008-10-19). "Hubs and Authorities" (PDF). 15-396: Science of the Web Course Notes. Carnegie Mellon University. Retrieved 2008-11-09.
Q & A
top related