“towards building a cognitive system to fight for national college admission challenge”

Towards Building a Cognitive System toFight for National College Admission Challenge

Yansong FengJoint work with

Kun Xu, Songfang Huang, Dongyan Zhao

Peking UniversityIBM China Research Lab

December 1, 2016

Feng et al. (PKU) Question Answering December 1, 2016 1 / 25

Pass the Exam: A New AI Challenge

The Todai Robot ProjectNational Institute of Informatics andcollaboratorsNational Center Test for UniversityAdmissions ( 2016)Entrance Exam of University ofTokyo (2021)

Japanese, Social Science, Math,Physics

The Project Aristo and EuclidThe Allen Institute for ArtificialIntelligenceElementary School:

Science andMath

High School:

Geometry



The Todai Robot ProjectNational Institute of Informatics andcollaboratorsNational Center Test for UniversityAdmissions ( 2016)Entrance Exam of University ofTokyo (2021)Japanese, Social Science, Math,Physics

The Project Aristo and EuclidThe Allen Institute for ArtificialIntelligenceElementary School:

Science andMath

High School:

Geometry



The Todai Robot ProjectNational Institute of Informatics andcollaboratorsNational Center Test for UniversityAdmissions ( 2016)Entrance Exam of University ofTokyo (2021)Japanese, Social Science, Math,Physics

The Project Aristo and EuclidThe Allen Institute for ArtificialIntelligenceElementary School: Science andMathHigh School: Geometry


The GaoKao Challenge

Gaokao in ChinaNational College Entrance ExaminationChinese, Math, English, History, Geography, Politics, Physics,Chemistry, Biologyover 9,400,000 students in 2016

The China Gaokao ChallengePrompt research in Artificial IntelligenceTeam: national research institutes, universities and companiesReal National College Entrance ExaminationsChinese, History, Math,


The GaoKao Challenge

Gaokao in ChinaNational College Entrance ExaminationChinese, Math, English, History, Geography, Politics, Physics,Chemistry, Biologyover 9,400,000 students in 2016

The China Gaokao ChallengePrompt research in Artificial IntelligenceTeam: national research institutes, universities and companiesReal National College Entrance ExaminationsChinese, History, Math, Geography


Is that Difficult?

Which is the correct ranking of provinces according to theiraverage altitutes, from highest to lowest?

Xiang, Liao, Ning

→ Hunan, Liaoning, Ningxia

Tai, Lu, Su

→Taiwan, Shandong, Jiangsu

Qing, Yue, Jin

→Qinghua, Guangdong, Shanxi

Gui, Gan, Yu

→ Guangxi, Gansu, Henan

1 Knowledge:short names of provincesaverage altitudes of provinces

2 Reasoningrelative comparisons of provinces’ altituderanking

Not very challenging?


Is that Difficult?

Which is the correct ranking of provinces according to theiraverage altitutes, from highest to lowest?

Xiang, Liao, Ning → Hunan, Liaoning, NingxiaTai, Lu, Su →Taiwan, Shandong, JiangsuQing, Yue, Jin →Qinghua, Guangdong, ShanxiGui, Gan, Yu → Guangxi, Gansu, Henan

1 Knowledge:short names of provincesaverage altitudes of provinces

2 Reasoningrelative comparisons of provinces’ altituderanking



What about this one?

Missouri River Valley is an important agricultural area of theUnited States. Aerial figure 1 shows the winter of the MissouriRiver, where the white part is snow.1. Why is the farmland on the peninsula shaped as circular?

their watering approachrugged terraintheir farming approachlack of farmland


What about this one?

2. Could you have a guess what is the main crop in this area?Winter wheatCornRicePotato

1 Knowledge:agricultureclimatelongitude and latituderead maps and images

2 Reasoningrelationship among those factorsfind analogiescommon sense knowledge



What We Need?

1 Solid knowledge:every aspects about the Syllabus

2 Math3 Reasoning:

logical inferenceuse common sense knowledgetextual entailment

A practical starting point: Answering Factoid Questions withKnowledge

Information Retrieval Based Question AnsweringAnswering Factoid Questions with Structured Knowledge BaseAnswering Factoid Questions with both Structured andUnstructured KBs


What We Need?

1 Solid knowledge:every aspects about the Syllabus (Knowledge Bases)

2 Math3 Reasoning:

logical inferenceuse common sense knowledge (a little bit)textual entailment

A practical starting point: Answering Factoid Questions withKnowledge

Information Retrieval Based Question AnsweringAnswering Factoid Questions with Structured Knowledge BaseAnswering Factoid Questions with both Structured andUnstructured KBs


The Task

What else did the director of the movie Interstellar direct ?

fb:m.0fkf28 fb:object.type fb:film.film

fb:m.0fkf28 fb:film.film.directed_by ?x[ ]select ?y

?x fb:film.director.fim ?y?y fb:object.type fb:film.film

Inception, The Dark Knight Rises

The Dark KnightBatman Begins

…..


The Task

What else did the director of the movie Interstellar direct ?




Inception, The Dark Knight Rises

The Dark KnightBatman Begins

…..

��

� ��


Question Answering over Structured Knowledge Bases

GoalAnswer Natural Language Questions against Structured KnowledgeBases


Related Work

Information Retrieval CommunityNatural Language Processing Community

Semantic Parsing BasedPCCG style: (Zettlemoyer and Collins, 2005, Cai and Yates, 2013;Kwiatkowski et al. 2013, Reddy et al., 2014)Syntactic parsing style: (Clarke et al., 2010, Liang et al., 2011,Berant et al. 2013, Berant and Liang, 2014, Xu et al., 2014)

Information Extraction Based(Yao and van Durme 2014, Bao et al., 2014, Yih et al., 2015, Donget al., 2015)Deep Learning, End2End style(Bordes et al., 2014a, 2014b, Yang et al., 2014, Bordes et al.,2015, Zhang et al., 2016 )


Semantic Parsing Based Methods

Challenges:1 Convert questions into proper meaning representations2 Ground the meaning representation into a database query

Previously:1 Search space is huge2 Difficult to adapt to other KBs


Motivation

1 Meaning representation should be KB-independent

[what] did �the director of] [the movie] [Interstellar]else �movies] [direct]

2 Separation of meaning representation and instantiation




ns:Interstellar dbo:type dbo:filmdbp:director ?x[ ]

select ?y

?y

?y

ns:Interstellar

dbp:director ?x

dbo:filmdbo:type


Framework

what else movies did the director of the movie Interstellar direct

Parsing

[what] did �the director of] [the movie] [Interstellar]variable relation category entity

else �movies] [direct]category relation

Instantiation





Phrase Dependency Graph



NodeA phrase with a semantic label l ∈ {entity, category, variable, relation}

EdgeA predicate-argument dependency between phrasesunary predicatebinary predicate


Structure Prediction

Input: a natural language questionOutput: a phrase dependency graph

A pipeline framework to predict the structure1 Phrase Detection

what did the director of the movie Interstellar

variable relation category entity

else movies direct

category relation

2 Phrase Dependency Parsing




Instantiation



1 Converting Phrase Dependency Graph into Structured Queries2 Instantiating Structured Query against KB


Applying Rules



variable category variablecategory

rule#1 rule#1 variablerelationrelationentityrule#8

?y type movies type ?x?x ?y

moviesInterstellar Interstellardirect

the director of

[ ]select ?y

type moviesInterstellar

?xInterstellar the director of?x ?ydirect

?y type movies


Probabilistic Model

[ ]select ?y

type moviesInterstellar

?xInterstellar the director of?x ?ydirect

?y type movies

Qind Qd

ns:Interstellar dbo:type dbo:filmdbp:director ?x[ ]

select ?y

?y

?y

ns:Interstellar

dbp:director ?x

dbo:filmdbo:type

Q∗d = arg max P(Qd |Qind)

P(Qd |Qind) =n∏

i=1

P(sdi |sindi )P(odi |oindi )P(pdi |pindi )


P(sdi |sindi )P(odi |oindi )Freebase Search API ⇒ wikipedia ID ⇒ DBpedia Entity

P(pd |pind)We construct the co-occurrence matrix from the patty relationphrase dataset which includes 1,631,530 relation phrases


Results on QALDs

Question Answering over Linked Data

Processed Right Partial Recall Precision F-12014 40 34 6 0.71 0.72 0.722015 42 26 7 0.72 0.74 0.73

Nice for longer/complex sentencesEfficient: around 0.33 sec per sentence

Consistent performances0.76 of F-1 on Free9170.41 of F-1 on WebQuestions


Results on QALDs



First Place in CLEF QALD 4 and 5




Results on QALDs






Results on QALDs



Nice for longer/complex sentencesEfficient: around 0.33 sec per sentenceConsistent performances0.76 of F-1 on Free9170.41 of F-1 on WebQuestions


Further Improvement

Semantic interpretation for superlatives

Nile is the longest river in the world.

Keys:the target: Nilethe comparison set: all rivers in the worldthe comparison dimension: the length of a river

→/geography/river/lengththe ranking order: descending


A Little More Extraction

For simple sentences:

Who does michael keaton play in cars

Who

michael keaton cars

Michael Keaton

The Merry GentlemanPenthouse North

cvt1Cars

cvt2

Chick Hicks

film

starring

starring

filmdirect direct by

ctv3spouse_s

Caroline McWilliams

spousespouse_s

spouse6/5/1982 from

1/29/1990

to

Marriage

type of unioncharacter

portrayer

film

character

starring

charactercharacter

Question:

Star Graph:

Freebase Graph:

cvt3

film

actor

Leona Elizabeth Loftus George A. Douglas

parentschild


A Little More Extraction

For simple sentences:

Join Entitly Linking and Relation Extractiongives 0.49 (+0.06) F1 on WebQuestions.


With Hybrid Knowledge Base Resources

a little bit complex...Where should a visitor see in Germany ?



a little bit complex...Where should a visitor see in Germany ?What is the most popular crop during 1900s in USA ?



a little bit complex...Where should a visitor see in Germany ?What is the most popular crop during 1900s in USA ?Who did Shaq first play for ?



Either subjective, or hard to map against existing Structured KBs



Either subjective, or hard to map against existing Structured KBsUsing both structured knowledge bases and texts, e.g.,Wikipedia or existing community QA archives.



who did shaq first play for

KB-QA

Entity Linking Relation Extraction

Joint Inference

shaq: m.012xdfshaq: m.05n7bpshaq: m.06_ttvh

sports.pro_athlete.teams..sports.sports_team_roster.teambasketball.player.statistics..basketball.player_stats.team

……

m.012xdf sports.pro_athlete.teams..sports.sports_team_roster.team

Los Angeles Lakers,Boston Celtics,Orlando Magic,

Miami Heat

Freebase



Answer Refinement

Los Angeles Lakers,Boston Celtics,Orlando Magic,

Miami Heat

Freebase

Shaquille O'Neal

O'Neal signedas a free agent with the Los Angeles Lakers

Shaquille O'Neal

O'Neal played for the Boston Celtics in the 2010-11 season before retiring

Shaquille O'Neal

O'Neal was drafted

in the 1992 NBA draftby the Orlando Magic with the first overall pick

Los Angeles Lakers Boston Celtics Orlando Magic

O’Neal was drafted by the OrlandoMagic with the first overall pick in

the 1992 NBA draft

O’Neal played for the Boston Celtics in the 2010-11 season before retiring

O’Neal signed as a free agent with the Los Angeles Lakers

Refinement Model

+- -

Orlando Magic

Wikipedia Dump



TextualRelation Extraction

Triple SolverTriple Solver

who is the front man of the band that wrote Coffee & TV

Question Decomposition< ans, is the front man of, var1 >

< var1 , is a , band >< var1 , wrote , Coffee & TV >

< var1 , wrote, Coffee & TV >

Triple Solver

Entity Linking

Multi-Channel Neural NetworkParaphrase Model

Wikipedia Dump

TextualKB

KB-basedRelation Extraction

DBpediaFreebase

DBpedia Lookup

Coffee & TV

Bitter_Coffee_(Iranian_video_series)

Irish_Coffee_(TV_series)Coffee & TV

influencedByassociatedMusicalArtist

associatedBandwriter

front man ofis written by

lead vocalist of

Open Information Extractor

wrote

is the front man of

Joint Inference

Damon Albarn


Conclusion

At this point...still a lot to do for GaoKaobut, a flexible QA frameworkWith multiple resources, e.g., structured knowledge bases,Wikipedia, text books, exercises, even news papers, etc.Our collaborations with IBM China Research Lab

contribute to the Watson Competitionscontribute to a Multi-Modal QA system (with Vision China)

The KeysFrom natural languages to knowledge basesInference over structured knowledgeAnswer with common-sense knowledgeUnderstand various images, tables, figures, diagrams...


Conclusion



The KeysFrom natural languages to knowledge bases → on the wayInference over structured knowledgeAnswer with common-sense knowledgeUnderstand various images, tables, figures, diagrams...


Conclusion



The KeysFrom natural languages to knowledge bases → on the wayInference over structured knowledge → challengingAnswer with common-sense knowledgeUnderstand various images, tables, figures, diagrams...


Conclusion



The KeysFrom natural languages to knowledge bases → on the wayInference over structured knowledge → challengingAnswer with common-sense knowledge → still missingUnderstand various images, tables, figures, diagrams... → stillmissing


Thanks & Questions


“towards building a cognitive system to fight for national college admission challenge”

Technology