web searching with multiple correct answers
DESCRIPTION
Web Searching with Multiple Correct Answers. Steve O’Hara, Google Inc., California Tom Bylander, Univ. of Texas San Antonio June 2 - 4, 2014, Thessaloniki, Greece Web Intelligence, Mining and Semantics. Factual Error Found On Internet!. - PowerPoint PPT PresentationTRANSCRIPT
Web Searching with Multiple Correct
Answers
Steve O’Hara, Google Inc., CaliforniaTom Bylander, Univ. of Texas San Antonio
June 2 - 4, 2014, Thessaloniki, GreeceWeb Intelligence, Mining and Semantics
“LONGMONT, CO—The Information Age was dealt a stunning blow Monday, when a factual error was discovered on the Internet. The error was found on TedsUltimateBradyBunch.com, a Brady Bunch fan site that incorrectly listed the show's debut year as 1968, not 1969.”
http://www.theonion.com/articles/factual-error-found-on-internet,102/
Factual Error Found On Internet!
“What is the atomic mass of Copper?” “What is the diameter of Pluto?”
Factoid Questions
# Value Units Miles Delta Min Max1 2300 KM 1,429 31.1 1,398 1,4602 2390 KM 1,485 3.1 1,482 1,4883 2170 MILES 2,170 5.0 2,165 2,1754 1416 MILES 1,416 0.5 1,416 1,4175 1444 MILES 1,444 0.5 1,444 1,4456 1370 MILES 1,370 5.0 1,365 1,3757 1500 MILES 1,500 50.0 1,450 1,5508 2390 KM 1,485 3.1 1,482 1,4889 2390 KM 1,485 3.1 1,482 1,488
10 1429 MILES 1,429 0.5 1,429 1,43011 2300 KM 1,429 31.1 1,398 1,46012 2274 KM 1,413 0.3 1,413 1,41313 1865 MILES 1,865 2.5 1,863 1,868
# Value Units Miles Delta Min Max14 2274 KILO-
METERS 1,413 0.3 1,413 1,41315 1860 MILES 1,860 5.0 1,855 1,86516 1860 MILES 1,860 5.0 1,855 1,86517 2320 KM 1,442 3.1 1,438 1,44518 1430 MILES 1,430 5.0 1,425 1,43519 2274 KILO-
METERS 1,413 0.3 1,413 1,413
20 2274 KILO-METERS 1,413 0.3 1,413 1,413
21 2270 KM 1,411 3.1 1,407 1,41422 1485 MILES 1,485 2.5 1,483 1,48823 1400 MILES 1,400 50.0 1,350 1,45024 300000
0 METERS 1,864 310.7 1,553 2,17525 1400 MILES 1,400 50.0 1,350 1,450
Select “Best” Answer
The two candidate answers of 1400 miles receive the most Support
The four candidate answers of 2274 km (1413 miles) give the most Support
What is the Diameter of Pluto?
zoom
Numeric Correlation
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0
50
100
150
200
250
300
350
400
450
Reliability for Numeric Questions
Rightiness
Ran
ked
Sup
port
What is the population of New York
City? What is the area of Canada? When did Elizabeth Taylor get
married? What is the capital of Bolivia? Who coaches the Hoyas men’s
basketball team?
Motivating Questions
Numeric Hypothesis
99
200
301
400
500.1
100
400
403
497
500
300
300.9398.6
601 600
City Populations
City, State City MSA
Abilene, Texas 118,117 166,416Akron, Ohio 198,402 701,456
Albuquerque, New Mexico 552,804 898,642Wichita Falls, Texas 103,931 150,261
Wichita, Kansas 384,445 625,526
Wilmington, North Carolina 108,297 369,685
Some Candidate Answers
(for the population of Akron, Ohio)
th a population of 217,074 (2000 census).<div/
n, OH. Population: 214,556. Compare Akron to T
ent from 69,000 to 210,000. By the mid 20'
. 1920, Population 208,435 . Warren G. Harding
with approximately 207,934 people within the c
as a population of 207934 people this year. Ak
Total population, 200,181 ...<div/><div><span
as a population of 200,000. Through the 1900s,
any) was producing 200,000 "Diamond" brand lam
ad a population of 199,110. The Akron Metropol
OH, population is 199,110.<div/><div><span st
Candidate Answer Distribution
0
5
10
15
20
25
Num
ber o
f Can
dida
te A
nsw
ers
Population of Akron, Ohio (zoomed)
Target answers: 198,402 city, 701,456 MSA
Abilene Results
Rank
CandidateAnswer Support City
RightinessMSA
Rightiness1 112,205 4.600 0.431 0.1622 114,757 4.572 0.510 0.169
Rank CandidateAnswer Pair
Paired Suppo
rt
City Rightines
s
MSA Rightine
ss1 106,654 - 108,157 4.273 0.356 0.1482 975,619 -
1,005,656 3.394 0.018 0.027
Rank
CandidateAnswer Support
CityRightine
ss
MSA Rightine
ss1 116,966 6.889 0.656 0.1752 1,005,656 5.021 0.018 0.026
Single Score(0.431/1 + 0.169/2) / 2
0.258
Paired Score(0.356 + 0.148) / 2
0.252
Combined Score(0.656 + 0.026) / 2
0.341
Target: 118,117 Cityand 166,416 MSARan
kCandidate
Answer Support CityRightiness
MSA Rightiness
1 117,063 2.000 0.668 0.1752 11,928,500 2.000 0.001 0.002
Baseline Score(0.668 + 0.002) / 2
0.335
City Population Results
Baselin
e Results
Single Result
s
Paired Result
sCombined Results
Mean Score 0.233 0.297 0.291 0.319Improveme
nt - 31% 29% 41%
Multi-Valued Named Entities
Ann
Bob
Carl
Dan
Eva
Anna
Dana
Dana
Eve
Eve
Carl
CarlDana
Fred Fred
Adapt Levenshtein’s Algorithm to whole words
Count #insertions, #deletions and #changes Instead of character matching, do whole word
matching Which uses regular Levenshtein’s edit distance
Handles typos Handles middle names Does not handle Bill / William
Proper Name Matching
Two Levels of Searches
SearchMovie
Directors
SearchMovieStars
MovieDirectors
MovieStars
Directorsand Stars
Movie Target Answers
Movie Director(s) Stars
Ben-Hur William Wyler Charlton Heston, Jack Hawkins,Stephen Boyd, Haya Harareet
Gone with the Wind Victor Fleming Clark Gable, Vivien Leigh,
Thomas Mitchell, Barbara O'Neil
Time of Violence Ludmil StaikovIossif Surchadzhiev, Rousy
Chanev,Ivan Krystev, Anya Pencheva
The English Patient
Anthony Minghella
Ralph Fiennes, Juliette Binoche,Willem Dafoe, Kristin Scott
ThomasSchindler's List Steven Spielberg Liam Neeson, Ralph Fiennes,
Ben Kingsley, Caroline Goodall
GWTW Results
Support Candidate Answer Result37 Victor Fleming correct21 George Cukor wrong12 Sam Wood (skip)
Support Candidate Answer Result10 Vivien Leigh correct5 Clark Gable (skip, correct)3 Victor Fleming (skip, director)3 Evelyn Keyes (skip)2 Rhett Butler (skip)2 Rand Brooks (skip)
MovieDirectors
MovieStars
Of 500 movies, 32 instances of co-occurring
pairs 23 additional correct answers 5 duplicate correct answers 4 incorrect answers:
Movie Results
Movie Errors and RationalizationJFK Tom Hanks is the director of a second JFK
movie.Lee Harvey Oswald was the assassin.
The Fighter The movie is about Mickey Ward.V for
VendettaAlan Moore is the author of the book.
Is Wikipedia Always Right?
“The number of factual errors shows just how unreliable it can be to use the online resource as a sole means of digging up information.
Yet millions base everything from school homework to corporate presentations using facts and figures they have gleaned from the site.
A study into, specifically, company information on the massively popular website discovered 60 per cent of articles had factual errors.”
Source: www.dailymail.co.uk/sciencetech/article-2131458/Up-articles-Wikipedia-contain-factual-errors.html#ixzz2sDrWl4RH