row-less universal schema · tech pioneer bill gates microsoft, the seattle-steve balmer, will take...
TRANSCRIPT
Row-less Universal Schema
Patrick Verga and Andrew McCallum
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Relation Types
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Melinda, Bill
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
structured textual 50k columns
2
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Entity Types
pers
on
locat
ion
orga
niza
tion
per:w
orke
r pe
r:scie
ntist
loc
:city
org:n
on-p
rofit
Melinda
Steve
MicrosoftBill
SeattleDallas
USAHawaii
Obama
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
coun
try
prov
iden
ce
chair
man
pr
esid
ent
athl
ete
polit
ician
he
ad-o
f-sta
te
structured textual
3
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Relation Typesstructured textual 50k columns
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
4
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###~y
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
5
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
6
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
7
Pattern Encoder
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
####
###
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
8
January 15, 2000
Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattle-headquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas.Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
JoeJane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
9
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
####
###
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Pattern Encoder
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
####
###
####
###
####
###
####
###
####
###
####
###
####
###
10
JoeJane
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
Joe Jane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
11
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
####
###
spousequery:
####
###
#################################################
#####################
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
vectorembeddingdistributedrepresentation
ofsemantics
learnedparameters
#######
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
12
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
####
###
spousequery:
####
###
Aggregation Function
#################################################
#####################
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Melinda, BillMicrosoft, Seattle
Melinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
13
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
####
###
spousequery:
####
###
Aggregation Function
####
###
####
###
####
###
#################################################
#####################
####
###
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane
Joe NewEntity worked with his wife Jane NewEntity on relation extraction.
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
####
###
14
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
spousequery:
Aggregation Function
Melinda, Bill
spou
se
per:c
o-wo
rker
mar
ried t
o
####
###
####
###
####
###
Query Independent1. Mean Pool
2. Max Pool
#######
####
###
####
###
####
###
####
###
####
###
####
###
#################################################
#####################
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane##
####
###
####
#
####
###
####
###
####
###
####
###
####
###
####
###
####
###
15
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
Aggregation Function
Melinda, Bill
Query Dependent
co-founderquery: #######
spou
se
per:c
o-wo
rker
mar
ried t
o
####
###
####
###
####
###
-0.1 0.025.0
1. Max Column
####
###
####
###
####
###
####
###
#######
####
###
####
###
####
###
#################################################
#####################
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane##
####
###
####
#
####
###
####
###
####
###
####
###
####
###
####
###
####
###
16
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
Aggregation Function
Melinda, Bill
Query Dependent
co-founderquery: #######
spou
se
per:c
o-wo
rker
mar
ried t
o
####
###
####
###
####
###
-0.1 0.025.0
1. Max Column
2. Attention
####
###
####
###
####
###
####
###
.006 .008.986
####
###
####
###
####
###
#######
####
###
####
###
####
###
#################################################
#####################
####
###
Gates moved his family into their 55,000-
square-foot, $54 million house on the shore
of Lake Washington, just outside Seattle.
CEO
chair
man
pr
esid
ent
leade
r-of
head
-of
head
-of-s
tate
m
arrie
d to
spou
se
born
-in
frien
dpe
r:co-
work
er
per:l
ives-
in
org:t
op-m
embe
rs
org:m
embe
r
~x
~y
Pattern Encoder
Steve, Microsoft
Bill, MicrosoftBill, Steve
Microsoft, SeattleMelinda, Dallas
Bill, SeattleObama, Hawaii
Obama, US
Joe Jane##
####
###
####
#
####
###
####
###
####
###
####
###
####
###
####
###
####
###
17
Joe NewEntity worked with his wife Jane
NewEntity on relation extraction.
####
###
####
###
####
###
Aggregation Function
Melinda, Bill
Training
spou
se
per:c
o-wo
rker
mar
ried t
o
####
###
####
###
####
###
#######
Entity Type Prediction
• Freebase + Clueweb
• 1.1k FB types, 5k textual types, ~140k entities
• MAP over 707 test types
• 100 negatives randomly generated for each positive
18
19
/education/academic, /music/artist,/music/group member, /people/person
/sports/pro athlete,/sports/sports award winner, /people/person /award/award winner, /tv/tv actor
/baseball /baseball player
/architecture /engineer
/computer/computer scientist
engineer, /book/author, sir/projects/project focus , /people/person
Example Max Columns
Model MAP
Entity Embeddings
3.1
Mean Pool 35.5
Max Pool 31.0
Attention 54.5
Max Relation 54.7
Model MAP
Entity Embeddings 54.8
Mean Pool 39.5
Max Pool 32.6
Attention 55.7
Max Column 55.4
20
Entity Type Prediction
seen at train
unseen at train
Relation Extraction
• Subset of FB15k-237
• Freebase + Clueweb
21
Model MRR Hits@10
Entity Embeddings
5.2 11.9
Mean Pool 18.1 35.8
Max Pool 20.8 40.3
Attention 29.8 49.7
Max Relation 28.5 48.1
Model MRR Hits@10
Entitiy Pair Embeddings 31.9 51.7
Mean Pool 25.9 45.9
Max Pool 29.6 49.9
Attention 31.9 51.7
Max Column 31.7 51.9
22
Relation Extraction
seen at train
unseen at train
Model MRR Hits@10Entity Embeddings
5.2 11.9
5.2 11.9
Attention 29.8 49.7
27.8 51.0
Max Relation 28.5 48.1
29.6 54.2
Model MRR Hits@10
Entity Embeddings 31.9 51.7
Entity Embeddings-LSTM 33.4 54.4
Attention 31.9 51.7
Attention-LSTM 30.0 53.4
Max Column 31.7 51.9
Max Column-LSTM 30.8 54.8
23
Relation ExtractionRowless + Columnless
seen at train
unseen at train
Future Work
24
• Improvements to Attention model
• Discern between irrelevant and counter evidence
Melinda, Bill
spou
sepe
r:co-
work
erm
arrie
d to
####
###
####
###
####
###
think
s is a
wful
####
###
hatesquery:
• Row-less Universal Schema forgoes explicit row representations for aggregation over observed columns
• Performs as well as old model using fewer parameters
• Naturally extends to unseen entities
• Aggregation based on query-specific attention outperforms query-independent aggregations
25
Summary
Thank you!