executable semantic parsing - tel aviv universityjoberant/teaching/class1_index.pdf · shrdlu...

Executable Semantic Parsing

ESSLLI

Aug 22, 2016

Instructor: Jonathan Berant

Oct 2014

1

1B dollars please

2

Why is this an important language problem?

what is the second biggest city in california that is not near a river?

3

Why is this an important language problem?

what is the second biggest city in california that is not near a river?

• You can not memorize this

3

Why is this an important language problem?

what is the second biggest city in california that is not near a river?

• You can not memorize this

• Understand language atoms

• Understand how to compose them

• Generalize to any utterance

• Collect evidence from multiple sources

3

Executable semantic parsing

Which states’ capitals are also their largest cities by area?

4

Executable semantic parsing

Which states’ capitals are also their largest cities by area?

semantic parsing

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

4

Executable semantic parsing

Which states’ capitals are also their largest cities by area?

semantic parsing

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

execute logical form

Arizona,Hawaii,Idaho,Indiana,Iowa,Oklahoma,Utah

4

Executable semantic parsing

Which states’ capitals are also their largest cities by area?

semantic parsing

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

execute logical form

Arizona,Hawaii,Idaho,Indiana,Iowa,Oklahoma,Utah

Execute query against knowledge-base

4

Executable semantic parsing

Send a reminder to all students who haven’t turned in their homework.

5

Executable semantic parsing

Send a reminder to all students who haven’t turned in their homework.

semantic parsing

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

5

Executable semantic parsing

Send a reminder to all students who haven’t turned in their homework.

semantic parsing

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

execute logical form

[50 emails sent]

5

Executable semantic parsing

Send a reminder to all students who haven’t turned in their homework.

semantic parsing

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

execute logical form

[50 emails sent]

Execute command against personal assistant

5

Executable semantic parsing

[utterance: user input]

semantic parsing

[program]

execute logical form

[behavior: user output]

6

Executable semantic parsing

[utterance: user input]

semantic parsing

[program]

execute logical form

[behavior: user output]

Execution connects language with the world

6

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

7

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

8

Personal assistants everywhere!

Show me photos from Utah last august

Find videos I took at Ive’s birthday party

What movies are playing today?

. . .

9

10

Personal assistants everywhere!

Alexa, Play some Christmas music

Alexa, turn the bedroom lights on

Alexa, what’s my commute?

. . .

11

Personal assistants everywhere!

Can you help me order flowers for my mom’s birthday

Where’s the best place to go hiking in the bay area

Is there a dog friendly beach nearby?

Remind me to congratulate Tanya the next time she calls

. . .

12

Personal assistant everywhere!

13

Question answering

14

Robot control

15

Research assistants

Which country has the highest CO2 emissions?

Which had the highest increase since last year?

What fraction is from the five countries with highest GDP?

16

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

17

semantic

parsing

language logic

learning

18

semantic

parsing

language logic

learning

But executable!

18

Non-executable semantic parsing

The boy wants to go

[Banarescu et al., 2013]

19

Non-executable semantic parsing

The boy wants to go

No environment to execute semantic representation

[Banarescu et al., 2013]

19

Semantics

Sentiment analysis, Semantic Role Labeling, Named Entity Recognition, ...

20

Semantics

Sentiment analysis, Semantic Role Labeling, Named Entity Recognition, ...

Semantic parsing: Full meaning representation

20

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

21

SHRDLU [1971]

Person: Pick up a big red block.

Computer: OK.

Person: Grasp the pyramid.

Computer: I don’t understand which pyramid you mean.

Person (changing their mind): Find a block which is taller than the oneyou are holding and put it into the box.

Computer: By ”it”, I assume you mean the block which is taller thanthe one I am holding.

Computer: OK.

22

SHRDLU [1971]

Person: Pick up a big red block.

Computer: OK.

Person: Grasp the pyramid.

Computer: I don’t understand which pyramid you mean.

Person (changing their mind): Find a block which is taller than the oneyou are holding and put it into the box.

Computer: By ”it”, I assume you mean the block which is taller thanthe one I am holding.

Computer: OK.

Person: What does the box contain?

Computer: The blue pyramid and the blue block.

Person: What is the pyramid supported by?

Computer: The box.

Person: How many blocks are not in the box?

Computer: Four of them.

Person: Is at least one of them narrower than the one which I told youto pick up?

Computer: Yes, the red cube.22

Other grammar-based approaches

Lunar (1972):

List the rocks that contain chromite and ulvospinel

What minerals have been identified in the lunar samples?

What is the average analysis of Ir in rock S10055?

Give me the K / Rb ratios for all lunar samples

Chat-80 (1982):

Does Afghanistan border China?

How many countries does the Danube flow through?

What is the capital of each country bordering the Baltic?

Which country is bordered by two seas?

Woods et al., 1972; Warren and Pereira, 1982

23

Other grammar-based approaches

Lunar (1972):

List the rocks that contain chromite and ulvospinel

What minerals have been identified in the lunar samples?

What is the average analysis of Ir in rock S10055?

Give me the K / Rb ratios for all lunar samples

Chat-80 (1982):

Does Afghanistan border China?

How many countries does the Danube flow through?

What is the capital of each country bordering the Baltic?

Which country is bordered by two seas?

Problem: does not scale!

Woods et al., 1972; Warren and Pereira, 1982

23

Statistical semantic parsing

Learn from data:

Input:

What’s California’s capital? Capital(California)

How long is the Mississippi river? RiverLength(Mississippi)

... ...

[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]

24

Statistical semantic parsing

Learn from data:

Input:

What’s California’s capital? Capital(California)

How long is the Mississippi river? RiverLength(Mississippi)

... ...

Output:

countries in asia ⇒ Type.Country u ContainedBy.Asia

[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]

24

Statistical semantic parsing

Learn from data:

Input:

What’s California’s capital? Capital(California)

How long is the Mississippi river? RiverLength(Mississippi)

... ...

Output:

countries in asia ⇒ Type.Country u ContainedBy.Asia

What are the problems with this form of supervision?

[Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; ...]

24

Breadth

Depth

information retrieval

semantic parsing

25

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

26

Goals

Be able to go and build a semantic parser

Learn about modern techniques and challenges in semantic parsing

27

Plan

Class 1: Representation

• Lambda calculus

• Lambda DCS

28

Plan

Class 1: Representation

• Lambda calculus

• Lambda DCS

Class 2: Compositionality

• Grammars

• Lexicons

• Derivations

• Modelling

28

Plan

Class 3: Learning

• Traning from logical forms

• Traning from denotations

• Additional training signals

29

Plan

Class 3: Learning

• Traning from logical forms

• Traning from denotations

• Additional training signals

Class 4: Search

• CKY

• Beam-search

• Agenda-based parsing

29

Plan

Class 5: Advanced topics

• Intermediate representations

• Building semantic parsers overnight

• Deep learning

30

About me

31

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

32

Expressing language with logical form

How to handle various linguistic phenomena

• entities, relations

• Multi-arg events

• Comparatives, superlatives

• Quantifiers, determiners

Goal: Intuition for how to write down logical forms

33

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

34

Worlds

35

Worlds

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

35

Worlds

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

Which states’ capitals are also their largest cities by area?

Send a reminder to students that did not submit homework

35

Worlds

µx.Type.USState u Capital.argmax(Type.City u ContainedBy.x,Area)

∀x ∈ (Student u ¬Submit.Homework) : Remind(x)

Which states’ capitals are also their largest cities by area?

Send a reminder to students that did not submit homework

World determines relevant language

35

Worlds

• Query a DB

Mountains in Italy

36

Worlds

• Query a DB

Mountains in Italy

• Update a DB

Add a meeting from 8am-10am Monday to Friday

36

Worlds

• Query a DB

Mountains in Italy

• Update a DB

Add a meeting from 8am-10am Monday to Friday

• Execute in an environment

Bring me the cup on the table

36

Querying a DB

Type

BarackObama Person

MichelleObama Person

Honolulu CityTown

Hawaii USState

ContainedBy

Honolulu Hawaii

Honolulu UnitedStates

Chicago Illinois

Chicago UnitedStates

Profession

BarackObama Politician

BarackObama Author

MichelleObama Lawyer

PlaceOfBirth

BarackObama Honolulu

MichelleObama Chicago

Marriage

BarackObama Event8

MichelleObama Event8

GreaterThan

2 1

8 3

... ...

startDate

Event8 1992.10.03

Event13 2001.13.02

37

Graph DB

BarackObama

Person

Type

Politician

Profession

1961.08.04

DateOfBirth

HonoluluPlaceOfBirth

Hawaii

ContainedBy

City

Type

UnitedStates

ContainedBy

USState

Type

Event8

Marriage

MichelleObama

Spouse

Type

FemaleGender

1992.10.03

StartDate

Event3

PlacesLived

Chicago

Location

Event21

PlacesLived

Location

ContainedBy

Subset of Freebase (2B facts)

38

Representations

Simply-typed lambda calculus (Carpenter, 1998)

• General formalism for natural language semantics

Lambda-DCS (Liang, 2013)

• Natural language interface for databases

39

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

40

Simply-typed lambda calculus

• Constants

• Variables

• Lambda terms

• Function application

adapted from Yoav Artzi

41

Simply-typed lambda calculus

Each type has a set of constants and variables

Constants

Chicagoe

Event8ev

City〈e,t〉

PlaceOfBirthOf〈e〈e,t〉〉

42

Simply-typed lambda calculus

Each type has a set of constants and variables

Constants

Chicagoe

Event8ev

City〈e,t〉

PlaceOfBirthOf〈e〈e,t〉〉

Variables

xe, ye

aev, bev

f〈e〈e,t〉〉, g〈e〈e,t〉〉

42

Simply-typed lambda calculus

Function application

t: (PlaceOfBirthOf(BarackObama))Chicago)

t: City(Honolulu)

43

Simply-typed lambda calculus

Function application

t: (PlaceOfBirthOf(BarackObama))Chicago)

t: City(Honolulu)

Lambda terms

〈e, t〉: λx.President〈e,t〉(xe)

〈e, t〉: λx.President(x) ∧ Lawyer(x)

〈e〈e, t〉〉: λx.λyPlaceOfBirthOf(x, y)

Logical operators are defined as higher-order constants

43

Simply-typed lambda calculus

Function application and beta reduction

t : λx.City(x)(Honolulu)

beta

t : City(Honolulu)

44

Simply-typed lambda calculus

Function application and beta reduction

t : λx.City(x)(Honolulu)

beta

t : City(Honolulu)

t : λx.(λy.PlaceOfBirthOf(x, y)(BarackObama))(Chicago)

beta

t : PlaceOfBirthOf(BarackObama,Chicago)

44

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

45

Lambda-DCS

• Can be viewed as syntactic sugar for lambda-calculus

• Concise logical forms that mirror natural langauge

46

Logical forms: entity

proper nouns

lambda-dcs Chicago

lambda calculus λx.x = Chicago

denotation {Chicago}graph template Chicago

[Liang, 2013]

47

Logical forms: join

Verbs and prepositions

HasBirthplace.Chicago

λx.HasBirthplace(x,Chicago)

{MichelleObama}o ChicagoHasBirthplace

48

Logical forms: join

Verbs and prepositions

HasBirthplace.Chicago

λx.HasBirthplace(x,Chicago)

{MichelleObama}o ChicagoHasBirthplace

ChildOf.HasBirthplace.Chicago

λx.∃yChildOf(x, y) ∧HasBirthplace(y,Chicago)

{NatashaObama, MaliaObama}o yChildOf ChicagoHasBirthplace

48

Logical forms: join

Verbs and prepositions

HasBirthplace.Chicago

λx.HasBirthplace(x,Chicago)

{MichelleObama}o ChicagoHasBirthplace

ChildOf.HasBirthplace.Chicago

λx.∃yChildOf(x, y) ∧HasBirthplace(y,Chicago)

{NatashaObama, MaliaObama}o yChildOf ChicagoHasBirthplace

BirthDateOf.ChildOf.HasBirthplace.Chicago

λx.∃y∃zBirthDateOf(x, y) ∧ChildOf(y, z) ∧HasBirthplace(z,Chicago)

{06.10.2001, 07.04.1998}o yBirthdateOf zChildOf ChicagoHasBirthplace

48

Logical forms: intersection, union, negation

Profession.Lawyer uHasBirthplace.Chicago

λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)

{MichelleObama}o

Lawyer

Profession

Chicago

HasBirthplace

49

Logical forms: intersection, union, negation

Profession.Lawyer uHasBirthplace.Chicago

λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)

{MichelleObama}o

Lawyer

Profession

Chicago

HasBirthplace

Profession.Lawyer tHasBirthplace.Chicago

λx.Profession(x,Lawyer) ∨HasBirthplace(x,Chicago)

{MichelleObama,BarackObama}o

Lawyer

Profession

o

Chicago

HasBirthplace

49

Logical forms: intersection, union, negation

Profession.Lawyer uHasBirthplace.Chicago

λx.Profession(x,Lawyer) ∧HasBirthplace(x,Chicago)

{MichelleObama}o

Lawyer

Profession

Chicago

HasBirthplace

Profession.Lawyer tHasBirthplace.Chicago

λx.Profession(x,Lawyer) ∨HasBirthplace(x,Chicago)

{MichelleObama,BarackObama}o

Lawyer

Profession

o

Chicago

HasBirthplace

¬Profession.Lawyer

λx.¬Profession(x,Lawyer)

{MaliaObama,NatashaObama}49

Logical forms: n-ary relations

Where did Barack Obama live in 1985?

Where did Barack Obama live?

50

Logical forms: n-ary relations

Where did Barack Obama live in 1985?

Where did Barack Obama live?

Do we need a ternary predicate?

λx.PlaceLivedBy(x,BarackObama)

λx.PlaceLivedBy(x,BarackObama, 1985)

50

Logical forms: n-ary relations

Where did Barack Obama live in 1985?

Where did Barack Obama live?

Do we need a ternary predicate?

λx.PlaceLivedBy(x,BarackObama)

λx.PlaceLivedBy(x,BarackObama, 1985)

Neo-Davidsonian semantics: Event node modified by several arguments

o Event8LivedLocation BarackObamaPersonLiving

1985

StartLiving

LivedLocation.(PersonLiving.BarackObama u StartLiving.1985)

λx.∃e.LivedLocation(x, e) ∧PersonLiving(e,BarackObama) ∧ StartLiving(e, 1985)

{Chicago} 50

Logical forms: higher order functions

Operate on sets of entities (not entities)

count(Type.USState)

count(λx.Type(x,USState))

{50}

Similar for min, max, sum

51

Logical forms: higher order functions

Operate on sets of entities (not entities)

count(Type.USState)

count(λx.Type(x,USState))

{50}

Similar for min, max, sum

argmax(Type.USState,Area)

argmax(λx.Type(x,USState), λx.λy.Area(x, y))

{Alaska}

Similar for argmin

51

Logical forms: anaphora

those whose children influenced Barack Obama

HasChild.HasInfluence.BarackObama

λx.∃y.HasChild(x, y) ∧HasInfluence(y,BarackObama)

52

Logical forms: anaphora

those whose children influenced Barack Obama

HasChild.HasInfluence.BarackObama

λx.∃y.HasChild(x, y) ∧HasInfluence(y,BarackObama)

those whose child influenced them

µx.HasChild.HasInfluence.x

λx.∃y.HasChild(x, y) ∧HasInfluence(y, x)

52

Logical forms: lambda abstraction

How to construct compositional binary relations?

53

Logical forms: lambda abstraction

How to construct compositional binary relations?

Person who has the most children

argmax(Type.Person,Rev[λx.count(Rev[HasChildren].x)])

argmax(λx.Type(x,Person), λx.count(λy.HasChildren(x, y)))

53

Logical forms: lambda abstraction

How to construct compositional binary relations?

Person who has the most children

argmax(Type.Person,Rev[λx.count(Rev[HasChildren].x)])

argmax(λx.Type(x,Person), λx.count(λy.HasChildren(x, y)))

In some cases lambda calculus is more transparent

53

Some things we skip

Determiners and quantification

54

Dynamic environments

Database updates

Add a meeting with Tom from 2 to 4pm

Add(Type.Meeting uAttendee.Tom u StartTime.1400 uEndTime.1600)

[Artzi and Zettlemoyer, 2013]

55

Dynamic environments

Database updates

Add a meeting with Tom from 2 to 4pm

Add(Type.Meeting uAttendee.Tom u StartTime.1400 uEndTime.1600)

Executing instruction

[Artzi and Zettlemoyer, 2013]

55

Outline

• Introduction (30 min.)

– Applications

– Background

– Short history

– Goals and plan

• Representation (30 min.)

– Worlds

– Simply-typed lambda calculus

– Lambda-DCS

• Exercise (30 min.)

56

Exercise

http://esslli2016.unibz.it/?page id=356

or

cd /usr/local/sempre

./run @mode=simple-lambdadcs

(loadgraph esslli 2016/geo880.kg)

57

Exercise

a.b (a b)

a u b (and a b)

rank-m top-n for unary and binary (argmax m n u b)

count (count u)

numbers (number n)

µx.u (mark x u)

Rev(u) (rev u)

Type type.object.type

USState location.us state

City location.citytown

MajorCity location.major citytown

Area location.location.area

Size location.location.size

Population location.location.population

Capital location.us state.capital

Border location.us state.border

Traverse location.river.traverse

58

Exercise

Write logical forms for the following utterances:

state with the largest area

Top 5 cities by area

Major cities with at least 140,000 inhabitants

How many states have capitals with at least 200,000 inhabitants

State bordering Oregon and Washington

Second highest mountain in California

Which states’ capitals are also their largest cities by population

State that is traversed by the most rivers

59

Solution

(argmax 1 1 (type.object.type location.us state) location.location.area)

(argmax 1 5 (type.object.type location.citytown) location.location.size)

(and (type.object.type location.major citytown) (loca-tion.location.population (> (number 140000))))

(count (location.us state.capital (and (type.object.type location.capital)(location.location.population (> (number 200000))))))

(and (type.object.type location.us state) (and (location.us state.borderstate.oregon) (location.us state.border state.washington)))

(argmax 2 1 (and (type.object.type location.mountain) (loca-tion.location.containedby state.california)) location.location.elevation)

(and (type.object.type location.us state) (mark x (loca-tion.us state.capital (argmax 1 1 (location.location.containedby(var x)) location.location.population))))

(argmax 1 1 (type.object.type location.us state) (reverse (lambda x(count (location.river.traverse (var x))))))

60

Summary

• Key technology for conversational interfaces

61

Summary

• Key technology for conversational interfaces

• Expressing language constructs with logical forms

61

Summary

• Key technology for conversational interfaces

• Expressing language constructs with logical forms

• But how do we construct logical forms automatically?

61