extractive summarization using inter- and intra- event relevance wenjie li, wei xu, mingli wu,...

Extractive Summarization Extractive Summarization using Inter- and Intra- Event using Inter- and Intra- Event

RelevanceRelevance

Wenjie li, Wei Xu, Mingli Wu, Chunfa Yuan, Qin LuH.K. Polytechnic Univ. & Tsinghua Univ. , CHINA

2

Introduction

How to define concepts How to judge the importance of the concepts

Extract concepts from documents (e.g. keywords, entities, events)

Identify the “most important” concepts

Create summaries by selecting sentences Create summaries by selecting sentences according to what concepts they containaccording to what concepts they contain

Extractive Summarization

3

Introduction (Cont’d)Motivation

Events contain important information Documents describe more than one similar or related

event Most existing event-based summarization explore the

importance of the events independently, or need syntactic analysis.

What we suggest Semi-structure event extracted with shallow NLP Event-relevance based summarization

Determine salient concepts using event relevance with graph ranking algorithm

What sorts of event relevance are better in what case Intra-event relevance (direct relationship) Inter-event relevance (indirect relationship)

4

Related WorkEvent-based Summarization

Judge topic as sub-events by human. Determine sentence relevance to each sub-event (Daniel et al., 2003)

Atomic events, based on co-occurrence statistics of named entity relations. (Filatova and Hatzivassiloglou, 2004)

Employ distribution of discourse entities to improve summary coherence (Barzilay and Lapata, 2005)

Event-centric method. Need syntactic analysis of sentence (Vanderwende, 2004)

Summarization with graph ranking algorithm( e.g. PageRank)

Sentence similarity according to term vectors (Mihalcea, 2005)

Sentence are linked if they share similar events (Yoshioka and Haraguchi, 2004)

Importance of the verbs and nouns constructing events was weighted as individual nodes. Need syntactic analysis (Vanderwende, 2004; Leskovec, 2004)

5

Event DefinitionComments

Events are collections of activities together with associated entities

It’s more appropriate to consider events at sentence level, rather than document level

Not all verbs denote event happening Semantic similarity or relatedness between action

words should be taken into account

Solution Semi-structure event:

Take advantages of statistical techniques from the IR community and structured information from the IE community

Avoid the complexity of deep semantic and syntactic processing

6

Event Definition (Cont’d)Who did What to Whom When and Where

Event = event term + associated named entitiesDefinition: event term

Verbs and action nouns appearing at least once between two named entities

Characterize actions or incident occurrences Roughly relate to “did What”

Definition: associated named entities Named entities connect to event term 4 types of named entities + high frequency nouns:

Person, Organization, Location, Date Convey the information of “Who”, “Whom”, “When”

and “Where”

7

Event MapEvents are related with one another semantically,

temporally, spatially, causally or conditionally

Graph structure Nodes: event terms (ET) & named entities (NE)

Words in either their original form or morphological variations are represented with a single node regardless of how many times they appear

Represent concepts rather than instances Advantages: (vs. event/sentence node)

− Be convenient to analyze the relevance among event terms and named entities either by semantic or distributional similarity

− Allow concept extraction, further conceptual compression

Links: undirected All event terms and named entities involved can

be explicitly or implicitly related

8

Event Map (Cont’d)– An ExampleSegment of a text from DUC2004 on “antitrust

case against Microsoft”

S1: The Justice Department and the 20 states suing Microsoft believe that the tape will strengthen their case because it shows Gates saying he was not involved in plans to take what the government alleges were illegal steps to stifle competition in the Internet software market.

S2: It showed a few brief clips of a point in the deposition when Gates was asked about a meeting on June 21, 1995, at which, the government alleges, Microsoft offered to divide the browser market with Netscape and to make an investment in the company, which is its chief rival in that market.

S3: In the taped deposition, Gates says he recalled being asked by one of his subordinates whether he thought it made sense to invest in Netscape.

S4: But in an e-mail on May 31, 1995, Gates urged an alliance with Netscape.

S5: The contradiction between Gates' deposition and his e-mail, though, does not of itself speak to the issue of whether Microsoft made an illegal offer to Netscape.

9

Event Map (Cont’d) – An Example

event term (ET)

named entity (NE)

10

Event Map (Cont’d)Weighted graph

We integrate the strength of the connections between nodes into event graph model in terms of the relevance defined from different perspectives

Relevance between nodes:PageRank – graph ranking algorithm

To calculate the significance of node according to and the structure of graph

Focus of our research How to derive according to intra- or/and inter- event

relevance

( , )i jr node node( )iw node

( , )i jr node node

j

j 1,2, ...t j

( )( ) (1 ) ( )

( , )ii

w nodew node d d

r node node

Where ( 1, 2,..., , ) are the nodes linking to

is a dampening factor, set to 0.85 experimentallyj inode j t j i node

d

11

Intra- & Inter- Event RelevanceRelevance Matrix

matrix element - relevance between nodes (ETs, NEs)

Event Term (ET) Named Entity (NE)

Event Term (ET) R(ET, ET) R(ET, NE)Named Entity (NE) R(NE, ET) R(NE, NE)

( , ) ( , )( , ) ( , )R ET ET R ET NE

RR NE NE R NE NE

( , )i jr node node

12

Intra- & Inter- Event Relevance (Cont’d)

Relevance Matrix

Intra-event relevance Direct relevance, explicit in the text (event map) To measure connections between actions and

arguments Symmetry



( , ) ( , )TR NE ET R ET NE

13


Relevance Matrix

Inter-event relevance Indirect relevance, need to be derived from external

resource or overall event distribution To measure how an event term/named entity connect

to another event term/named entity



14


How to determine Intra-event relevance

Intra-event relevance can be simply established by counting how many times and are associated

( , ) ( , )TR NE ET R ET NE

jneiet

( , ) ( , )Document i j i jr et ne freq et ne

15


How to determine Inter-event relevance

Event term relevance – R(ET, ET) 1.Semantice relevance from WordNet

− We use WordNet::Similarity to measure the relatedness of concept (event terms in our case) and choose lesk metric.

−

2. Topic-specific relevance from documents− Assumption: if 2 events are concerned with the

same participant, location or time, these 2 events are interrelated with each other in some ways

− Event term relevance then can be derived from the number of named entities they share.

−

( , ) and ( , )R ET ET R NE NE

( , ) | ( ) ( ) |Document i j i jr et et NE et NE et

( , ) ( , ) ( , )WordNet i j i j i jr et et similarity et et lesk et et

16


How to determine Inter-event relevance Named entities relevance – R(NE, NE)

1. Named entity relevance from documents− Named entity relevance then can be derived from

the number of event terms they share−

2. Named entity relevance from clustering− We proposed a clustering algorithm based on

words

−

( , ) | ( ) ( ) |Document i j i jr ne ne ET ne ET ne

1, , are in the same cluster( , )

0, otherwise i j

Cluster i j

ne ner ne ne

Location Person Date OrganizationMississippi Professor Sir

Richard SouthwoodFirst six months of last year

Long Beach City Council

Mississippi River

Sir Richard Southwood

Last year San Jose City Council

Richard Southwood City Council

17


How to determine Inter-event relevance Named entities relevance - R(NE, NE)

3. Named entity relevance from sentence pattern− Named entity relevance then can be revealed by

sentence context.− Example of Sentence patterns

− Window-based: Neighboring named entities are usually relevant

−

1, , are within a specified window size( , )

0, otherwise i j

Pattern i j

ne ner ne ne

<Person>, a-position-name of <Organization>, dose something.<Person> and another <Person> do something.

18

Extractive SummarizationSystem Overview

Recognize 4 types of named entities , nouns,

verbs by GATE

Determine event terms(w/ stem, w/o stop-word)

Extract eventsExtract events

Derive event relevance

Determine salience of concept with PageRank

Select out sentences containing salient

concept

19

Evaluation Task: Automatic text summarization Data:

DUC 2001 multi-document summarization task 30 English document sets Summary of 50, 100, 200, 400 word length Each set includes 10.3 documents, 602 sentences, 216

event terms, 148.5 named entities Evaluation Metric: ROUGE

Automatic evaluation Based on N-gram co-occurrence Comparing with human judgments

Method: To focus on efficiency and potential of event-relevance

based approach Without other features, such as sentence position,

headline, publication dates, etc. Simple greedy strategy to extract the most salient

sentences. Without avoidance of sentence redundancy, only

remove same sentences

20

Evaluation(1)Intra-event relevance – R(ET, NE) & R(NE, ET)

High Frequency Noun Some frequently occurring nouns, such as

“hurricane”, “euro”, are not marked by general NE taggers. But they indicate persons, organizations, locations or important objects.

A noun is considers as a frequent noun when its frequency is larger than 10.

5% improvement with high frequency nouns

R(ET,NE) NE w/ High Frequency Nouns

NE w/o High Frequency Nouns

ROUGE-1 0.33320 0.34859ROUGE-2 0.06260 0.07157ROUGE-W 0.12965 0.13471

21

Evaluation(2)Inter-event relevance R(ET, ET) & R(NE, NE)

R(ET, ET) 2 approaches:

− Semantic relevance from Word-Net− Topic-specific relevance from document number of ET’s shared named entities

Example result of event terms pairs with highest relevance “abort”-”confirm” (semantics, antonymous)

“vote”- “confirm” (associated, causal) “Document” outperforms “WordNet” by 4% Reason: WordNet may introduce non-necessary relatedness

in the topic-specific documents

R(ET,ET) Semantic relevance from Word-Net

Topic-specific relevance from document

ROUGE-1 0.32917 0.34178ROUGE-2 0.05737 0.06852ROUGE-W 0.11959 0.13471

22

Evaluation(3)Inter-event relevance R(ET, ET) & R(NE, NE)

R(NE, NE)

Example result of named entities pairs with highest relevance “Louisiana”-”Florida” (something may happen in both places)

“Florida”- “Andrew” (may happen about Andrew in Florida) Best result is “Document”- derive NE relevance

according to the numbers of shared event terms Reason: relevance derived from clustering and

neighborhoods can also be discovered by

R(NE,NE) Relevance from

Documents

Relevance from Clustering

Relevance from Window-based Context

ROUGE-1 0.35212 0.33561 0.34466ROUGE-2 0.07107 0.07286 0.07508ROUGE-W 0.13603 0.13109 0.13523

( , )Document i jr ne ne

23

Evaluation(4)Event relevance

Integration of R(ET, NE), R(ET, ET) and R(NE, NE) Different length of summary: 50,100,200,400 words Baseline:

“Event-based Extractive Summarization”, Elena Filatova & Vasileios Hatzivassiloglou, 2004

DUC-2001, 200 words summary, ROUGE-1 about 0.3

Significant improvement comparing with baseline Event-based approaches prefer longer summaries

ROUGE-1 50 100 200 400

R(NE,NE) 0.22383 0.28584 0.35212 0.41612R(ET,NE) 0.22224 0.27947 0.34859 0.41644R(ET,ET) 0.20616 0.26923 0.34178 0.41201

R(ET,NE)+R(ET, ET)+ R(NE,NE)

0.21311 0.27939 0.34630 0.41639

24

Conclusion & Future Work Extract event according to actions and associated

named entities Use event to denote concept Use Inter- and Intra- event relevance to rank event events-based summarizer gives good performance on

news documents

Improve event representation to build a more powerful event-base summarization system

Text compression technique based on concept What features of a document set preferring event-based

approaches (beyond news domain) Influence of IE performance(e.g. POS tagger, NE tagger)

25

Thank You Very Much

Do you have any questions?