ibm watson technical deep dive swiss group for artificial intelligence and cognitive science

© 2013 IBM Corporation

Swiss Group for Artificial Intelligence and Cognitive Science Intelligent Systems and Applications Workshop 2014, University of BaselWatson Technical Deep Dive

@RomeoKienzler, IBM Innovation Center Zurich

© 2013 IBM Corporation2

(part of) my role @IBM● Accelerate cognitive computing

● In Switzerland● Through

● Academia● Startups/ISV's● Cloud

Watson in the cloud: bit.ly/go4bluemix


What Watson is not● Search Engine● Database System● HAL9000


What Watson is● Cognitive System (Marketing)● Combination of

● Information Retrieval● NLP

● Structured + Unstructured Data !● Runs on UIMA● Based on supervised learning


What is a parser?● Annotate sentence with

● Tags● Relationships

● Probabilistic (e.g. Stanford)● Rule based

● (E) Slot Grammar


Slot Grammar● Simplified● Lexicalist character

● High focus on words● Low focus on structure

● Assign words to slotsI can resist everything except temptation● Subject (I)● Verb (can resist)● Object (everything except temptation)


PAS - Builder● Predicate-Argument Structure● Downstream to ESG● Reduces complexity of ESG“John opened Bill's door (with his key)

John's key opened Bill's doorBill's door openedBill's door was opened (by John)”

OPEN (John door key) | | |

Agent Theme Instrument

Many ESG trees reduce to same PAS


Relationships● Relationship Extractor● Combination of

● Manual pattern specifications~30 types, high precision

● Statistical methods~7000 types, low precision

● SVM's on DBPedia/Wikipedia


Relationships (2)“The Screwtape Letters” from a senior devil to an under devil are by this man better known for children’s books

author(“this man”,“The Screwtape Letters”)


Ingestion


Ingestion● Corpus creation● Input format: TREC

(Text Retrieval Conference)● Multiple HTML pages in one

HDFS file● Parallel ingestion process

(LiteScale)


Dictionary● started w/ Wikipedia copus● Keyword → Text structure● Transformation of free text

● into Keyword → Text● optimization objective


Knowledge Expansion● Follow links in content● Identify content keywords and link

to new content● → generate more content in

Keyword → Text form


Question Analysis


Question Analysis● Named entity recognition

● Type identification /Extract focus● ESG/PAS

● Relationship detection


Question Analysis1) Extract focus2) Map to LAT3) Broad Type classification4) Detect if special handling is

needed (e.g. nested question)


Query Decomposition


Query Decomposition● Keyword identification● LAT (Lexical Answer Type)

● IBM Pat. US20120078890 for confidence estimation of LAT

● optimization objective: choosing keywords out of nontrivial set of words based on ML


Query DecompositionIn 1894 C.W. Post created his warm cereal drink Postum in this Michigan city● Focus: this Michigan City● LAT: Michigan● Keywords: 1894, C.W. Post,

created, warm, cereal, drink, Postum, Michigan, City


Query Decomposition


Primary Search


Primary Search● Lucene and Indri search engine● Preprocessing generated

keyword->text based documents● Keyword associated with found

document added to candidate answer list


Hypothesis generation


Supporting Evidence Retrieval

Unlike most sea animals, in the sea horse this pair of sense organs can move independently of one another

Question decomposition:Which [sense organ] of [Sea Horse] move independently?

Hypothesis generation:A Sea Horse can move its eyes independently.A Sea Horse can move its ears independently.A Sea Horse can move its skin independently.A Sea Horse can move its nose independently.A Sea Horse can move its tung independently.


http://angelalmassey.com/SHC/about.html


Supporting Evidence● Generated Candidate Answer is

● ESG'd● PAS'd● searched against corpus● LATs used to determine whether

a candidate answer is an instance of the answer types


Supporting Evidence


Scoring


Scoring● Optimization objective

(confidence estimation framework)● Relational (PRISMATIC, Dbpedia)● Taxonomic,Geospacial● Temporal, Source Reliability● Gender, Name consistency● Passage Support● Theory consistency


Scoring challenges● Feature significance different for

● Different questions ● Different question classes

● Very heterogeneous features● Normalization problem● Missing features

● Class imbalance


Merging and ranking


Merging and ranking1. John Fitzgerald Kennedy 2. Kennedy, 3. JFK● Different Scores● Merge to canonical form

● Morphological● Pattern-based● Table Lookup

● Partially generated from Wikipedia disabiguation pages


ExampleMYTHING IN ACTION: One legend says this was given by the Lady of the Lake & thrown back in the lake on King Arthur’s death.

● Watson merged sword + Excalibur to “sword” (canonical form)

● Preserved relation● more_specific(sword)->Excalibur


ML in Ranking● Experiments with logistic regression, support

vector machines, linear and nonlinear kernels, ranking SVM, boosting, single and multilayer neural nets, decision trees, locally weighted learning

● Finally: regularized logistic regression


Normalization● Q set of all candidate answers● Feature x_ij

● j feature, i answer● missing values imputed


Ranking● Based on training set n > 10K● IBM SPSS Modeler


Evidence Sources


Automatic Learning● Read through text semantically● Statistically rank annotated text● generate new knowledge

● Inventors patent inventions 0.8● officials submit resignations 0.7● people earn degrees at schools 0.9● fluid is a liquid 0.6● liquid is a fluid 0.5● vessels sink 0.7● people sink 8-balls (0.5) (in pool/0.8)


Next steps● “Jeopardy!” - Watson was

● Open domain● Large training set

● New “Watsons” are● Closed domain● Small, but growing training set


Demo● Bit.ly/go4bluemix


References[1] Jeffrey Kabot, “Deep Parsing”[2] Richard Nordquist, “slot and filler”[3] The Journal of Research and Development, Vol 56, 2012

ibm watson technical deep dive swiss group for artificial intelligence and cognitive science

Internet