coqa: a conversational question · 2019-06-05 · chelsea's star striker didier drogba has...

CoQA: A Conversational Question Answering Challenge

1

Danqi Chen* Christopher D. ManningSiva Reddy* =

2

Question Answering

Chelsea's star striker Didier Drogba has been suffering from malaria since Tuesday, but the Ivory Coast player will still line up for the London derby against Fulham on Wednesday.

Drogba started on the bench for Chelsea's 2-0 defeat to Liverpool on Sunday due to a suspected virus…

2

Question Answering



Q: Who had malaria? A: Didier Drogba

2

Question Answering



Q: Who had malaria? A: Didier Drogba

Q: Where is Didier from? A: Ivory Coast

3

………………………………..…………

4

Conversational Question Answering



4




Q: When?

4




Q: Against who?

Q: When?

4




Q: Is he going to play with malaria?

Q: Against who?

Q: When?

4




Q: Is he going to play with malaria?

Q: Against who?

Q: When?

A: Yes

A: Fulham

A: Wednesday

5

• 127,000 questions and answers in English

• 8,000 conversations (15 turns on average)

• Collected from 7 genres: Wikipedia, News, Literature, Children's stories, English exams, Reddit, Science articles

6

CoQA: Three Main Goals

7

CoQA goal #1: conversational questions

The Virginia governor’s race is shaping up to be a foregone conclusion. Democrat Terry McAuliffe, the longtime political fixer and moneyman, hasn’t trailed in a poll since May…

Q: What are the candidates running for? A: Governor

7



Q: What are the candidates running for? A: GovernorQ: Where?

7



Q: What are the candidates running for? A: GovernorA: VirginiaQ: Where?

7



Q: What are the candidates running for? A: GovernorA: Virginia

Q: Who is the democratic candidate? A: Terry McAuliffe

Q: Who is his opponent?

Q: Where?

8

CoQA goal #2: free-form answers

Jessica is turning 80. Her granddaughter Annie is coming over in the afternoon and Jessica is very excited to see her. Her daughter Melanie and Melanie’s husband Josh were coming as well.

Q: How many are visiting?

Q: Who?

A: Three

A: Annie, Melanie and Josh

9


Span overlap found 67%

No span overlap found 33%

9


Span overlap found 67%

No span overlap found 33%

Yes/No 79%

One word edit 8% Q: what is the cost to end users? R: The service is free to users A: It is freeMultiple edits 6%

Counting 5%

Other 2%

10

CoQA goal #3: diverse genre

Train != Test

11

Conversational vs. isolated questions

SQuADCoQA

12

CoQA vs. QuAC

[Choi et al. 2018]

Wikipedia, extractive answers

7 genres, free-form answers with

extractive rationale

12

CoQA vs. QuAC

[Choi et al. 2018]

questioner and answerer sees the passage

only answerer sees the passage




12

CoQA vs. QuAC

[Choi et al. 2018]



answer length: avg = 2.7 answer length: avg = 15.1




12

CoQA vs. QuAC

[Choi et al. 2018]



answer length: avg = 2.7 answer length: avg = 15.1

human performance: 88.8 F1 human performance: 74.7 F1




13

Other datasets

[Saeidi et al. 2018]

[Elgohary et al. 2018]

14

Linguistic phenomena

70%

30%

IsolatedCoreference

15

Linguistic phenomena

27%

43%

30%

Lexical MatchParaphrasingPragmatics

Did he keep the puppy?

R: He walked the puppy home A: Yes

Generate free text as answers

Highlight spans as answers

16

Learning models

Generate free text as an answer

Context Vector

<start>

Atte

ntio

n

Dis

tribu

tion

...

Enco

der

Hid

den

Stat

es

Decoder

Hidden States

The Virginia governor ’s race … <q> What … <q> Where

Passage

weighted sum

Answer Generation

Dialog history and current question

…

…

Vocabulary D

istributiona zoo

Pvocab

[See et al. 2017]

18

What are the candidates … Where ?

…


q

Highlight a text span as an answer

[Chen et al. 2017]

18


…


…

Passage

...

The Virginia governor ’s race … …. hasn’t trailed since May

P

q


[Chen et al. 2017]

18


…


…

Passage

...

The Virginia governor ’s race … …. hasn’t trailed since May

P

q


p(answer) = maxi,j

p(starti) p(endj)

…

[Chen et al. 2017]

19

Highlight first and then generate

Question: Did she plan to have any visitors?

Generate: Yes

Highlight: Her granddaughter Annie was coming over

(discriminator + generator)

20

Evaluation

average F1 score of word overlap

Question: What did James do? Predicted: Took the puppy home Gold: Took him home

21

Results

Generate Highlight Highlight + Generate

Human

avg

F1 S

core

21

Results


Human

avg

F1 S

core

44.1

21

Results


Human

avg

F1 S

core

52.644.1

21

Results


Human

avg

F1 S

core 65.1

52.644.1

21

Results


Human

avg

F1 S

core

88.8

65.152.6

44.1

22

Results: analysis

Question: Does this help or hurt their memory of the event?Only Generator: help

22

Results: analysis


Question: Does this help or hurt their memory of the event?Only Highlighter: Answer is not a span

22

Results: analysis


Question: Does this help or hurt their memory of the event?

Generator: hurt

Highlighter: this obsession may prevent 1their brains from remembering

Question: Does this help or hurt their memory of the event?Only Highlighter: Answer is not a span

23

Importance of conversation history

10

87

0 1 2 all

Models Humans

History =

23


10

87

0 1 2 all

Models Humans

19.9

61.5

History =

23


10

87

0 1 2 all

Models Humans

79.8

19.9

66.261.5

History =

23


10

87

0 1 2 all

Models Humans

85.379.8

19.9

6666.261.5

History =

23


10

87

0 1 2 all

Models Humans

86.485.379.8

19.9

64.36666.261.5

History =

24

Amazing progress since launch

BERT

• Do models actually resolve coreferences, or do they exploit shortcuts?

• Unsupervised and low-resource scenario

• Transfer CoQA to other dialogue problems

25

What’s next?

GPT-2

• CoQA combines challenges from both dialogue and reading comprehension

• Grounded evaluation for dialogue models

• Contains several reasoning phenomena

• An avenue for transfer learning in dialogue

26

Conclusions

https://stanfordnlp.github.io/coqa/

https://stanfordnlp.github.io/coqa/

27

Acknowledgements

28


We asked the answerer to first highlight a piece of text (as the rationale) and then edit the text to generate a natural answer

28



Rationale: Jessica is turning 80Question: How old would she be?

Answer: 80

28



Rationale: Jessica is turning 80Question: How old would she be?

Answer: 80

CoQA NarrativeQA

Human agreement 88.8 59.0

why? why what?

decide that's not a good question

29

Data collection setup #8

30

Data collection setup #10

did they go? So we went over and the girls had a wonderful time

Did Tina catch butterfly, yes or no please? She did everything she could to try and catch that pretty butterfly but it kept flying way to high for her to catch

"Yes or no?" "No she did not"

• Answerer: abstractive and extractive

• Questioner: as you type feedback

• In-window chat interface for discussion

• Pay no matter what!

• Be nice

31

Data collection setup

• Stay away from boring stories

• Not more than one entity

• No coreferences

• No multiple events

35

Stories

A gramophone record (phonograph record in American English) or vinyl record, commonly known as a "record", is an analogue sound storage medium in the form of a flat polyvinyl chloride (previously shellac) disc with an inscribed, modulated spiral groove. The groove usually starts near the periphery and ends near the center of the disc.…

36

Boring story from SQuAD

Everton were founder members of the Premier League in 1992, but struggled to find the right manager. Howard Kendall had returned in 1990 but could not repeat his previous success, while his successor, Mike Walker, was statistically the least successful Everton manager to date…..

37

Good story from SQuAD

38

Answer dependent

38

Answer dependent

“I think the flow of this one turned out really awkward. I got a lot of irrelevant/nonsense questions.” — turker

39

Disagreements

• Pragmatics/common-sense

40

Linguistic Phenomena

coqa: a conversational question · 2019-06-05 · chelsea's star striker didier drogba has...

Documents