physical causality of action verbs in grounded language understanding

Physical Causality of Action Verbs in Grounded Language Understanding

Sep 11 (Sun.), 2016#SNLP2016

4

1

• : (Tsutsumi Fukumu / Bao Han)

• Twitter : @levelfour_

• 8 9 Preferred Networks

•

2

• (action verb) Change of State (CoS) /

(physical causality)

• physical causality (

)

• ( ) grounding

3

Physical Causality

• ( )

( )

NumberOfPieces = “one becomes many”

• 4 semantic role causality

agent = / patient = / source = / destination = (cf.) https://ja.wikipedia.org/wiki/%E4%B8%BB%E9%A1%8C%E5%BD%B9%E5%89%B2

4

https://ja.wikipedia.org/wiki/%E4%B8%BB%E9%A1%8C%E5%BD%B9%E5%89%B2

Physical Causality

bounding box

• agent( ) =

• patient( ) =

• source( ) = http://aclweb.org/anthology/P/P16/P16-1171.pdf

5

CoS / Physical Causality•

(Dixon, et al., 2006)

6

CoS / Physical Causality

http://aclweb.org/anthology/P/P16/P16-1171.pdf

7

semantic role

INPUT:

OUTPUT: semantic role

8

movietrack γ1

track γ2

track γ3

CoS 1

CoS 2

CoS 3

visual detector

captionsemantic role λ1

semantic role λ2

VC-model

VC

λ1 → γk1 λ2 → γk2

INPUTOUTPUT

N.B. VC = Verb Causality9

• TACoS (Regneri, et al., 2013) —

http://www.coli.uni-saarland.de/~regneri/docs/TACoS.pdf10

•

Amazon Mechanical Turks CoS

1800

• (

)

(Yang, et al., 2013) (Milan, et al., 2014)

+

11

CoS

movietrack γ1

track γ2

track γ3

CoS 1

CoS 2

CoS 3

visual detector


semantic role λ2

VC-model

VC

λ1 → γk1 λ2 → γk2

INPUTOUTPUT

12

CoS• CoS

+

• CoS rule-based


13

Verb Causality

movietrack γ1

track γ2

track γ3

CoS 1

CoS 2

CoS 3

visual detector


semantic role λ2

VC-model

VC

λ1 → γk1 λ2 → γk2

INPUTOUTPUT

14

Verb Causality• : semantic role λi γk

semantic role

(Yang, et al., 2016)

• 2 ( VC-model )

1. VC-Knowledge —

2. VC-Learning — CRF

15

1 (VC-Knowledge)• patient( )

grounding

( )

• agent( ), source( ),

destination( ) grounding

patient

rule-basedhttp://aclweb.org/anthology/P/P16/P16-1171.pdf

16

1 (VC-Knowledge)• verb causality vector c(v)

1. v patient CoS

one-hot vector

2. → c(v)

• causality detection vector d(γi) γi CoS one-hot vector

• c(v) d(γi) γi patient

17

1 (VC-Knowledge)

• verb causality vector

•

18

2 (VC-Learning)• (Tellex, et al., 2011), (Yang, et al., 2016) (

) CRF

λ1

φ1

γ1

λ2

φ2

γ2

λ3

φ3

γ3

Agent(hand) Patient(the knife) Source(the drawer)“get the knifefrom the drawer”

19

2 (VC-Learning)γi(∈ Γ) … / λi … semantic role

(∈{0,1}) … i

λ1

φ1

γ1

λ2

φ2

γ2

λ3

φ3

γ3


�i

20

2 (VC-Learning)p(�|⇤,�, v) = 1

Z

Y

i

i(�i,�i,�, v)

i(�i,�i,�, v) = exp

X

l

wlfl(�i,�i,�, v)

!

i grounding i

fl wl

wl

21

2 (VC-Learning)

λ1

φ1

γ1

λ2

φ2

γ2

λ3

φ3

γ3


1(�1,�1, �1, �2) 2(�2,�2, �2) 3(�3,�3, �2, �3)

(cf.) PRML 8.4

× ×/

22

2 (VC-Learning)

2

• semantic role λi γi

• v causality detection d(Γ)

• fl φi

i(�i,�i,�, v) = exp

X

l


!

23

2 (VC-Learning)

λ1

φ1

γ1

γ2

1(�1,�1, �1, �2, v) = exp(w1f1(�1,�1, �1))⇥ exp(w2f2(�1, �1, �2, v))

24

2 (VC-Learning)

•

λi, Γ, v

φi wl

p(Φ = 1) p(Φ ≠ 1)

p(�|⇤,�, v) = 1

Zexp

X

i

X

l


!

(given)(not given)

25

2 (VC-Learning)

•

wl Φ 1

Λ, v Γ

(patient → source → destination → agent beam search )

p(�|⇤,�, v) = 1

Zexp

X

i

X

l


!trained

inputfixed

estimate

26


27


VC-Learning (Tellex, et al., 2011), (Yang, et al., 2016)

• (Tellex, et al., 2011) —

(Tellex, et al., 2011) SDC( )

4 semantic role

(Yang, et. al. 2016)

• (Yang, et al., 2016) — semantic role

( ) physical

causality

28

• semantic role CoS / physical causality

physical causality

• Patient CoS

• 4 semantic role(Agent / Patient/ Source / Destination)

grounding

29

• [1] Gao, et al., 2016. Physical Causality of Action Verbs in Grounded Language Understanding. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.

• [2] Dixon, et al., 2006. Adjective Classes: A Cross-linguistic Typology. Explorations in Language and Space C. Oxford University Press.

• [3] Regneri, et al., 2013. Grounding action descriptions in videos. Transactions of the Association for Com- putational Linguistics (TACL), 1:25–36.

• [4] Yang, et al., 2013. Detection of manipulation action consequences (mac). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 2563–2570.

• [5] Milan, et al., 2014. Continuous energy minimization for multi- target tracking. Pattern Analysis and Machine Intel- ligence, IEEE Transactions on, 36(1):58–72.

• [6] Yang, et al., 2016. Grounded semantic role labeling. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics, San Diego, CA.

• [7] Tellex, et al., 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Ad- vancement of Artificial Intelligence (AAAI).

30

physical causality of action verbs in grounded language understanding

Science