GraphRel: Modeling Text as Relational Graphs for Joint Entity andRelation Extraction

Tsu-Jui FuAcademia Sinica

s103062110@m103.nthu.edu.tw

Peng-Hsuan LiAcademia Sinica

jacobvsdanniel@gmail.com

Wei-Yun MaAcademia Sinica

ma@iis.sinica.edu.tw

Abstract

In this paper, we present GraphRel, anend-to-end relation extraction model whichuses graph convolutional networks (GCNs) tojointly learn named entities and relations. Incontrast to previous baselines, we consider theinteraction between named entities and rela-tions via a relation-weighted GCN to better ex-tract relations. Linear and dependency struc-tures are both used to extract both sequentialand regional features of the text, and a com-plete word graph is further utilized to extractimplicit features among all word pairs of thetext. With the graph-based approach, the pre-diction for overlapping relations is substan-tially improved over previous sequential ap-proaches. We evaluate GraphRel on two pub-lic datasets: NYT and WebNLG. Results showthat GraphRel maintains high precision whileincreasing recall substantially. Also, GraphReloutperforms previous work by 3.2% and 5.8%(F1 score), achieving a new state-of-the-art forrelation extraction.

1 Introduction

Extracting pairs of entity mentions with seman-tic relations, i.e., triplets such as (BarackObama,PresidentOf, UnitedStates), is a central task in in-formation extraction and allows automatic knowl-edge construction from unstructured text. Thoughimportant and well-studied, three key aspects areyet to be fully handled in an unified framework:

• End-to-end joint modeling of entity recognitionand relation extraction;• Prediction of overlapping relations, i.e., rela-

tions that share a common mention;• Consideration of the interaction between rela-

tions, especially overlapping relations.

Traditionally, a pipelined approach is used tofirst extract entity mentions using a named entityrecognizer and then predict the relations betweenevery pair of extracted entity mentions (Zelenko

et al., 2003; Zhou et al., 2005; Chan and Roth,2011). Joint entity recognition and relation ex-traction models (Yu and Lam, 2010; Li and Ji,2014; Miwa and Sasaki, 2014; Ren et al., 2017)have been built to take advantage of the close in-teraction between these two tasks. While showingthe benefits of joint modeling, these complicatedmethods are feature-based structured learning sys-tems and hence rely heavily on feature engineer-ing.

With the success of deep neural networks, NN-based automatic feature learning methods havebeen applied to relation extraction. These methodsuse CNN, LSTM, or Tree-LSTM on the word se-quence between two entity mentions (Zeng et al.,2014; dos Santos et al., 2015), the shortest de-pendency paths between two entity mentions (Yanet al., 2015; Li et al., 2015), or the minimalconstituency sub-tree spanning two entity men-tions (Socher et al., 2012) to encode relevant infor-mation for each pair of entity mentions. However,these methods are not end-to-end joint modelingof entities and relations. They assume entity men-tions are given and are expected to degrade signifi-cantly in performance when a named entity recog-nizer is needed in the pipeline for real world usage.

Another challenge for relation extraction is howto take into account the interaction between re-lations, which is especially important for over-lapping relations, i.e., relations sharing com-mon entity mentions. For example, (Barack-Obama, PresidentOf, UnitedStates) can be in-ferred from (BarackObama, Governance, United-States); the two triplets are said to exhibit Enti-tyPairOverlap. Another case is that the formertriplet could also be inferred from (BarackObama,LiveIn, WhiteHouse) and (WhiteHouse, Presiden-tialPalace, UnitedStates), where the latter two aresaid to exhibit SingleEntityOverlap. Althoughcommon in knowledge base completion, such in-teraction, whether via direct deduction or indirect

evidence, is particularly difficult for joint entityrecognition and relation extraction models, whereentities are not present in the input. Indeed, al-though Zheng et al. (2017) propose a strong neu-ral end-to-end joint model of entities and relationsbased on an LSTM sequence tagger, they have tocompletely give up overlapping relations.

In this paper, we propose GraphRel, a neu-ral end-to-end joint model for entity recognitionand relation extraction that is the first to handle allthree key aspects in relation extraction. GraphRellearns to automatically extract hidden features foreach word by stacking a Bi-LSTM sentence en-coder and a GCN (Kipf and Welling, 2017) de-pendency tree encoder. Then GraphRel tags entitymention words and predicts relation triplets thatconnect mentions, where is the 1st-phase predic-tion.

To gracefully predict relation triplets while tak-ing into account the interactions between them,we add a novel 2nd-phase relation-weighted GCNto GraphRel. Already guided by both entityloss and relation loss, the 1st-phase GraphRelextracts node hidden features along dependencylinks while establishing a new fully connectedgraph with relation-weighted edges. Then, by op-erating on the intermediate graph, the 2nd-phaseGCN effectively considers the interaction betweenentities and (possibly overlapping) relations be-fore the final classification for each edge. WithGraphRel, our contribution is threefold:

• Linear and dependency structures, as well asimplicit features among all word pairs of thetext, are considered by our method;• We perform end-to-end, joint modeling of en-

tities and relations while considering all wordpairs for prediction;• The interaction between entities and relations is

carefully considered.

We evaluate the method on two public relation ex-traction datasets: NYT and WebNLG. The exper-imental result shows that GraphRel substantiallyimproves the overlapping relations over previouswork, and achieves a new state-of-the-art on bothdatasets.

2 Related Work

The BiLSTM-GCN encoder part of our model re-sembles the BiLSTM-TreeLSTM model proposedby Miwa and Bansal (2016), as they also stack

a dependency tree on top of sequences to jointlymodel entities and relations. They use Bi-LSTMon each sentence for automatic feature learning,and the extracted hidden features are shared by asequential entity tagger and a shortest dependencypath relation classifier. However, while introduc-ing shared parameters for joint entity recognitionand relation extraction, they must still pipeline theentity mentions predicted by the tagger to formmention pairs for the relation classifier.

Instead of trying to classify each mention pairas in previous work, Zheng et al. (2017) for-mulate relation extraction as a sequential taggingproblem (NovelTagging) as with entity recogni-tion. This allows them to model relation extractionby an LSTM decoder on top of a Bi-LSTM en-coder. However, while showing promising resultson the NYT dataset, their strength comes from fo-cusing on isolated relations and completely giv-ing up overlapping relations, which are relativelyrare in the dataset. In comparison, the proposedGraphRel gracefully handles all types of relationswhile being end-to-end and jointly modeling en-tity recognition.

Zeng et al. (2018) then propose an end-to-endsequence-to-sequence model for relation extrac-tion. They encode each sentence by a Bi-LSTM,and use the last encoder hidden state to initial-ize one (OneDecoder) or multiple (MultiDecoder)LSTMs for dynamic decoding relation triplets.When decoding, triplets are generated by selectinga relation and copying two words from the sen-tence. The seq2seq setup partially handles inter-action between triplets. However, interactions be-tween relations are only unidirectionally capturedby considering previous generated triplets with acompulsory linear order when generating a newone. Instead, in this paper, we propose propa-gating entity and relation information on a wordgraph with automatically learned linkage by ap-plying 2nd-phase GCN on top of the LSTM-GCNencoder.

Recently, considering dependency structure byGCN has been used in many natural languageprocessing (NLP) tasks. Marcheggiani and Titov(2017) applies GCN on word sequences for se-mantic role labeling. Liu et al. (2018) encode longdocuments via GCN to perform text matching. Ce-toli et al. (2016) combine RNN and GCN to rec-ognize named entities. There are also some works(Peng et al., 2017; Zhang et al., 2018; Qian et al.,

Figure 1: Graph Convolutional Network (GCN)

2019; Luan et al., 2019) about considering depen-dency structure of word sequence for relation ex-traction. In our proposed GrpahRel, we not onlystack Bi-LSTM and GCN to consider both linearand dependency structures but also adopt a 2nd-phase relation-weighted GCN to further model theinteraction between entities and relations.

3 Review of GCN

As convolutional neural network (CNN), GraphConvolutional Network (GCN) (Kipf and Welling,2017) convolves the features of neighboring nodesand also propagates the information of a node toits nearest neighbors. Shown in Fig. 1, by stackingGCN layers, GCN can extract regional features foreach node.

A GCN layer retrieves new node features byconsidering neighboring nodes’ features with thefollowing equation:

hl+1u = ReLU

∑v∈N(u)

(W lhlv + bl

) ,

where u is the target node andN (u) represents theneighborhood of u, including u itself; hlv denotesthe hidden feature of node v at layer l;W and b arelearnable weights, mapping the feature of a nodeonto adjacent nodes in the graph; and h ∈ Rf ,W ∈ Rf×f , and b ∈ Rf , where f is the featuresize.

4 Methodology

The overall architecture of the proposed GraphRelwhich contains 2 phases prediction is illustratedin Fig. 2. In the 1st-phase, we adopt bi-RNN andGCN to extract both sequential and regional de-pendency word features. Given the word features,we predict relations for each word pair and the en-tities for all words.

Then, in 2nd-phase, based on the predicted1st-phase relations, we build complete relational

graphs for each relation, to which we apply GCNon each graph to integrate each relation’s informa-tion and further consider the interaction betweenentities and relations.

4.1 1st-phase PredictionAs the state-of-the-art text feature extractor(Marcheggiani and Titov, 2017; Cetoli et al.,2016), to take into account both sequential and re-gional dependencies, we first apply bi-directionalRNN to extract sequential features and then use bi-directional GCN to further extract regional depen-dency features. Then, based on the extracted wordfeatures, we predict the relation for each word pairalong with the word entity.

4.1.1 Bi-LSTMWe use the well-known long short-term memory(LSTM) (Hochreiter and Schmidhuber, 1997) asour bi-RNN cell. For each word, we combine theword embedding and part-of-speech (POS) em-bedding as the initial feature:

h0u =Word(u)⊕ POS(u),

where h0u represents the initial feature of word u,and Word(u) and POS(u) are the word and POSembedding of word u, respectively. We use pre-trained word embeddings from GloVe (Penningtonet al., 2014), and the POS embedding is randomlyinitialized for training with the whole GraphRel.

4.1.2 Bi-GCNSince the original input sentence is a sequence andhas no inherent graph structure to speak of, as Ce-toli et al. (2016), we use dependency parser to cre-ate a dependency tree for the input sentence. Weuse the dependency tree as the input sentence’s ad-jacency matrix and use GCN to extract regionaldependency features.

The original GCN was designed for undirectedgraphs. To consider both incoming and outgoingword features, we follow Marcheggiani and Titov(2017) and implement bi-GCN as

→hl+1u = ReLU

∑v∈→N(u)

(→W

lhlv+

→bl)

←hl+1u = ReLU

∑v∈←N(u)

(←W

lhlv+

←bl)

hl+1u =

→hl+1u ⊕

←hl+1u ,

Figure 2: Overview of GraphRel with 2nd-phase relation-weighted GCN.

where hlu represents the hidden features of word u

at layer l,→N (u) contains all words outgoing from

word u, and←N (u) contains all words incoming to

word u, both including word u itself. W and b areboth learnable convolutional weights.

→W ,

→b and

←W ,

←b also represent the outgoing weight and in-

coming weight, respectively. We concatenate bothoutgoing and incoming word features as the finalword feature.

4.1.3 Extraction of Entities and RelationsWith the extracted word features from bi-RNN andbi-GCN, we predict the word entity and extractthe relation for each word pair. For the word en-tity, we predict for all words according to the wordfeatures over 1-layer LSTM and apply categoricalloss, denoted as eloss1p, to train them.

For relation extraction, we remove the depen-dency edges and do prediction for all word pairs.For each relation r, we learn weight matrices W 1

r ,W 2

r , W 3r and calculate the relation tendency score

S as

S(w1,r,w2) =W 3rReLU

(W 1

r hw1 ⊕W 2r hw2

),

where S(w1,r,w2) represents the relation tendencyscore for (w1, w2) under relation r and (w1, w2)refers to the word pair. Note that S(w1,r,w2)

should be different from S(w2,r,w1). For wordpair (w1, w2), we calculate all of the pair’s rela-tion tendency scores, including non-relation, anddenote it as S(w1,null,w2). We apply the soft-max function to S(w1,r,w2), yielding Pr(w1, w2),which represents the probability of each relation rfor (w1, w2).

Since we extract relations for each word pair,our design includes no triplet count restrictions.By investigating the relations for each word pair,

Figure 3: Relation-weighted graph for each relation.

GraphRel identifies as many relations as possi-ble. With Pr(w1, w2), we can also calculate therelation categorical loss here, denoted as rloss1p.Please note that though both eloss1p and rloss1p

will not be used as final prediction, they arealso good auxiliary loss for training 1st-phaseGraphRel.

4.2 2nd-phase Prediction

The extracted entities and relations in 1st-phase donot take each other into account. To consider inter-action between named entities and relations, andto take account implicit features among all wordpairs of the text, we present a novel 2nd-phaserelation-weighted GCN for further extraction.

4.2.1 Relation-weighted Graph

After 1st-phase prediction, we build completerelation-weighted graph for each relation r wherethe edge of (w1, w2) is Pr(w1, w2), as shown inFig. 3.

Then, 2nd-phase adopts bi-GCN on each re-lation graph which considers different influenceddegrees of different relations and aggregates as thecomprehensive word feature. The process can be

represented as

hl+1u = ReLU

(∑v∈V

∑r∈R

Pr (u, v)×(W l

rhlv + blr

))+hlu,

where Pr(u, v) represents the edge weight (theprobability of word u to word v under relationr). Wr and br means the GCN weight under re-lation r. V includes all words and R containsall relations. Note that the complete bi-GCN alsotakes both the incoming and outgoing situationsinto account. The bi-GCN in 2nd-phase furtherconsiders the relation-weighted propagations andextracts more sufficient features for each word.

With the newer word features from 2nd-phase,we perform named entity and relation classifica-tion again for more robust relation prediction. Thelosses for these are denoted as eloss2p and rloss2p.

4.3 Training DetailWe use two kinds of loss in GraphRel: entity lossand relation loss, both of which belong to cate-gorical loss. For entity loss, we use the conven-tional (Begin, Inside, End, Single, Out) taggingfor the ground-truth labels. Each word belongs toone of the five classes. The ground-truth entity la-bel for eloss1p and eloss2p are the same; we usecross-entropy as the categorical loss function dur-ing training.

For relation loss, we feed in a one-hot relationvector as the ground truth of Pr(w1, w2) for eachword pair (w1, w2). Since we predict relationsbased on word pairs, the ground truth should like-wise be based on word pairs. That is, word Unitedhas a HasPresident relation to both word Barackand word Obama, as does word States. We be-lieve that this word-pair-based relation represen-tation provides GraphRel with the information itneeds to learn to extract relations. The ground-truth relation vector for rloss1p and rloss2p are thesame. As entity loss, we also use cross-entropy asthe categorical loss function during training.

For both eloss and rloss, we add an additionaldouble-weight for those in-class entity or relationterms. Finally, the total loss is calculated as thesum of all entity loss and relation loss:

lossall = (eloss1p + rloss1p) + α (eloss2p + rloss2p) ,

where α is a weight between loss of 1st-phaseand 2nd-phase. We minimize lossall and train thewhole GraphRel in an end-to-end manner.

4.4 Inference

During inference, the baseline prediction methodis head prediction, where a relation triplet such as(BarackObama, PresidentOf, UnitedStates) is ex-tracted if and only if BarackObama, UnitedStatesare both identified as entity mentions and Presi-dentOf is the most probable class of P(Obama,States).

Another baseline extraction method that mightbe more stable is average prediction, where allword pairs between an entity mention pair aretaken into account and decide a relation with max-imum averaged probability.

Finally, we propose a threshold predictionmethod, where all word pairs of an entity men-tion pair are still taken into account but in an in-dependent fashion. For example, if 2 of the 4 dis-tributions have PresidentOf as the most probableclass, then the triplet (BarackObama, PresidentOf,UnitedStates) is extracted only if 2/4 = 50% > θwhere θ is a free threshold parameter. This way,users can select their preferred precision and re-call trade-off by adjusting θ. In the experiments,if not specified, threshold inference with θ = 0 isused.

5 Experiments

In this section, we present the experimental resultsof the proposed GraphRel. We first describe im-plementation details, the datasets, and the base-lines we compare with. Then we show the quan-titative results for two datasets, conduct detailedanalyses, and different categories of named enti-ties. Finally, we demonstrate the improved effectof 2nd-phase via a case study.

5.1 Experimental Settings

In our implementation, we chose the pre-trainedGloVe (300d) as a fixed word embedding. Theword embedding was then concatenated with atrainable POS embedding (15d) as the final inputembedding for each word. The POS tag for eachword and the dependency tree for whole sentenceswas retrieved from spaCy (Honnibal and Johnson,2015).

We use bi-LSTM with 256 units and 2-layerbi-GCN with 256 feature size in 1st-phase. Forthe 2nd-phase, the relation-weighted bi-GCN is1-layer, also with a feature size of 256. Duringtraining, we set the LSTM dropout rate to 0.5, thelearning rate to 0.0008, and the loss weight α to3. We train GraphRel using the Adam (Kingma

MethodNYT WebNLG

Precision Recall F1 Precision Recall F1NovelTagging 62.4% 31.7% 42.0% 52.5% 19.3% 28.3%OneDecoder 59.4% 53.1% 56.0% 32.2% 28.9% 30.5%

MultiDecoder 61.0% 56.6% 58.7% 37.7% 36.4% 37.1%GraphRel1p 62.9% 57.3% 60.0% 42.3% 39.2% 40.7%GraphRel2p 63.9% 60.0% 61.9% 44.7% 41.1% 42.9%

Table 1: Results for both NYT and WebNLG datasets.

CategoryNYT WebNLG

Train Test Train TestNormal 37013 3266 1596 246

EPO 9782 978 227 26SEO 14735 1297 3406 457All 56195 5000 5019 703

#Relation 24 246

Table 2: Statistics of dataset.

and Ba, 2015) optimizer and implement it underPyTorch.

5.1.1 DatasetWe use the NYT (Riedel et al., 2010) andWebNLG (Gardent et al., 2017) datasets to eval-uate the proposed method. As NovelTagging andMultiDecoder, for NYT, we filter sentences withmore than 100 words and for WebNLG, we useonly the first sentence in each instance in our ex-periments. The statistics of NYT and WebNLG isdescribed in Table. 2.

We divided relation triplets into three cate-gories: Normal, EntityPairOverlap (EPO), andSingleEntityOverlap (SEO). The counts for eachcategory are also shown in Table 2. Since one en-tity belonged to several different relations, Entity-PairOverlap and SingleEntityOverlap were moredifficult tasks. We discuss the result for differentcategories in the detailed analysis.

5.2 Baseline and Evaluation Metrics

We compared GraphRel with two baselines: Nov-elTagging (Zheng et al., 2017) and MultiDe-coder (Zeng et al., 2018). NovelTagging is a se-quence tagger which predicts both entity and rela-tion classes for each sentence word. MultiDecoderis a state-of-the-art method that considers relationextraction as a seq-seq problem and uses dynamicdecoders to extract relation triplets. The results forboth baselines come directly from the original pa-

pers.As two baselines, we adopted the standard F1

score to evaluate the results. The predicted tripletswere seen as correct if and only if the relation andthe head of the two corresponding entities were thesame as the ground truth.

5.3 Quantitative Results

Table 1 presents the precision, recall, and F1 scoreof NovelTagging, MultiDecoder, and GraphRelfor both the NYT and WebNLG datasets. OneDe-coder, proposed in MultiDecoder’s original pa-per, uses only a single decoder to extract relationtriplets. GraphRel1p is the proposed method butonly 1st-phase, and GraphRel2p is the completeversion, which predicts relations and entities afterthe 2nd-phase.

For the NYT dataset, we see that GraphRel1-hopoutperforms NovelTagging by 18.0%, OneDe-coder by 4.0%, and MultiDecoder by 1.3% interms of F1. As it acquires both sequential andregional dependency word features, GraphRel1-hopperforms better on both precision and recall, re-sulting in a higher F1 score. With relation-weighted GCN in 2nd-phase, GraphRel2p, whichconsiders interaction between name entities andrelations, further surpasses MultiDecoder by 3.2%and yields a 1.9% improvement in comparisonwith GraphRel1p.

Similar results can be found on the WebNLGdataset: GraphRel1p outperforms baseline F1scores by 3.6%, and GraphRel2p further improves2.2% upon GraphRel1p. From the NYT andWebNLG results, we show that GCN’s regionaldependency feature and 2nd-phase prediction bothaid relation prediction in terms of precision, recall,and F1 score.

NovelTagging and MultiDecoder both use a se-quential architecture. As NovelTagging assumesthat an entity belongs to a single relation, preci-sion is high but recall is low. MultiDecoder uses a

Figure 4: Results (F1 score) by named entity category.

Figure 5: Results (F1 score) by sentence triplet count.

dynamic decoder to generate relation triplets. Be-cause of the innate restrictions on RNN unrolling,the number of triplets it can generate is limited.However, for GraphRel, as we predict relations foreach word pair, we are free of that restriction. Webelieve that GraphRel is the most balanced methodas it maintains both high precision and high recall,yielding higher F1 scores.

5.4 Detailed Analysis

To further analyze the proposed GraphRel, wepresent the results under different types of triplets,different inference methods, the improvementover name entity recognition, and different num-bers of GCN layer used.

5.4.1 Different Types of TripletsWe first investigate the results under different en-tity categories. Fig. 4 presents the results for bothNYT and WebNLG datasets.

For GraphRel, as we predict relations for allword pairs, all words can have relations withother words: thus entity overlap is not a problem.Though MultiDecoder tries to use a dynamic de-

coder, the result shows that GraphRel surpassesthem in all entity categories. For instance, on theWebNLG dataset, GraphRel1p outperforms Mul-tiDecoder by 3.5% on the normal class, 2.9% onthe EPO class, and 3.4% on the SEO class. AndGraphRel2p further improves GraphRel1p for eachclass.

We also compare the results given differentnumbers of triplets in a sentence, as illustrated asFig. 5. The x-axis represents 1, 2, 3, 4, or morethan 5 triplets in a sentence. Because of the sin-gle decoder, OneDecoder performs well for sin-gle triplets, but performance drops drastically formore triplets in a sentence. As with the experimentfor different entity categories, GraphRel1p andGraphRel2p both outperform the baselines underall numbers of triplets in a sentence. GraphRel1poutperforms MultiDecoder by 7.5% for more than5 triplets in a sentence and GraphRel2p further sur-passes MultiDecoder by 11.1% on NYT.

5.4.2 Inference Methods

We compare the two baseline inference methods,head and average, and the threshold method under

Sentence GraphRel1p GrapRel2p

Agra Airport is in India where (Agra Airport, location, India) (Agra Airport, location, India)one of its leaders is Thakur. (India, leader name, Thakur) (India, leader name, Thakur)

In Italy, the capital is Rome and (Italy, captical, Rome) (Italy, captical, Rome)A.S. Gubbio 1910 is located there. (A.S. Gubbio 1910, ground, Italy)Asam pedas (aka Asam padeh) is (Asam pedas, alias, Asam padeh) (Asam pedas, alias, Asam padeh)

from the Sumatra and Malay (Asam pedas, region, Malay Peninsula) (Asam pedas, region, Malay Peninsula)Peninsula regions of Malaysia. (Asam pedas, country, Malaysia) (Asam padeh, region, Malay Peninsula)

(Asam pedas, country, Malaysia)(Asam padeh, country, Malaysia)

Table 3: Case Study for Graph1p and GraphRel2p.

Figure 6: Results by different decision thresholds.

Method NYT WebNLGGraphRel1p 88.8% 89.1%GraphRel2p 89.2% 91.9%

Table 4: F1 score of entity recognition for GraphRel.

different θ. Fig. 6 shows their results when appliedto GraphRel2p on NYT and WebNLG.

It can be seen that the threshold inferencemethod efficaciously adjusts the trade-off betweenprecision and recall with different choices of θ. Byreducing the threshold from θ = 0.8 to θ = 0, therecall is significantly increased by 1.8% and 1.4%respectively on NYT and WebNLG, with only amarginal 0.6% loss of precision. The effective-ness of the proposed threshold method then leadsto the best performance on both datasets, surpass-ing both the head and average ones.

5.4.3 Improvement over Entity Recognitionand Different Numbers of GCN Layer

From Table. 4, GraphRel2p can surpass 1st-phaseby 0.4% and 2.8% for entity recognition on bothNYT and WebNLG. It also shows that 2nd-phaserelation-weighted GCN is effective on not only re-lation extraction but also name entity recognition.

To confirm that our 2-layer 1st-phase added 1-†#GCN layer in 1st-phase set to 2.‡#GCN layer in 1st-phase and 2nd-phase set to 2 and 1.

Phase #GCN layer NYT WebNLG

1st-phase2 60.0% 40.7%3 60.0% 40.5%

2nd-phase†1 61.9% 42.9%2 61.6% 42.4%

3rd-phase‡ 1 61.8% 42.7%

Table 5: F1 score by different numbers of GCN layer.

layer 2nd-phase is the best setting, we investigatethe result of different numbers of GCN layer usedin both 1st-phase and 2nd-phase. Table. 5 presentsthe results of using 3 GCN layers for 1st-phase and2 layers of relation-weighted GCN for 2nd-phase.However, it shows that more GCN layers can notbring out better prediction and our (2, 1) layer set-ting should be the most suitable one for relationextraction task.

We also experiment on 3rd-phase method,adopting relation-weighted GCN again where thegraph is based on 2nd-phase’s predicited rela-tions. And it shows that our 2nd-phase is sufficientenough for relation extraction.

5.5 Case Study

Table. 3 shows the case study of our proposedGraphRel. The first sentence is an easy case andboth GraphRel1p and GraphRel2p can extract ac-curately. For the second case, although there doesnot belong to name entity, it should contain thehidden semantic of Italy. Therefore, 2nd-phasecan further predict that A.S. Gubbio 1910 groundsin Italy. The third case is an SEO class in whichGraphRel1p discovers that Asam pedas is the sameas Asam padeh, thus the latter should also locatein Malay Peninsula and come from Malaysia.

6 Conclusion

In this paper, we present GraphRel, an end-to-end relation extraction model which jointly learns

named entities and relations based on graph con-volutional networks (GCN). We combine RNNand GCN to extract not only sequential featuresbut also regional dependency features for eachword. Implicit features among all word pairs ofthe text are also considered in our approach. Wepredict relations for each word pair, solving theproblem of entity overlapping. Furthermore, weintroduce a novel relation-weighted GCN that con-siders interactions between named entities and re-lations. We evaluate the proposed method on theNYT and WebNLG datasets. The results show thatour method outperforms previous work by 3.2%and 5.8% and achieves a new state-of-the-art forrelation extraction.

ReferencesA. Cetoli, S. Bragaglia, A.D. O’Harney, and M. Sloan.

2016. Graph convolutional networks for named en-tity recognition. In Proceedings of TLT.

Yee-Seng Chan and Dan Roth. 2011. Exploitingsyntactico-semantic structures for relation extrac-tion. In Proceedings of ACL.

Claire Gardent, Anastasia Shimorina, Shashi Narayan,and Laura Perez-Beltrachini. 2017. Creating train-ing corpora for nlg micro-planners. In Proceedingsof ACL.

Sepp Hochreiter and Jrgen Schmidhuber. 1997. Longshort-term memory. Neural Computation.

Matthew Honnibal and Mark Johnson. 2015. An im-proved non-monotonic transition system for depen-dency parsing. In Proceedings of EMNLP.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In Proceedingsof ICLR.

Thomas Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutionalnetworks. In Proceedings of ICLR.

Jiwei Li, Minh-Thang Luong, Dan Jurafsky, and Eu-dard Hovy. 2015. When are tree structures neces-sary for deep learning of representations? In Pro-ceedings of EMNLP.

Qi Li and Heng Ji. 2014. Incremental joint extractionof entity mentions and relations. In Proceedings ofACL.

Bang Liu, Ting Zhang, Di Niu, Jinghong Lin, Kun-feng Lai, and Yu Xu. 2018. Matching long textdocuments via graph convolutional networks. arXivpreprint arXiv:1802.07459.

Yi Luan, Dave Wadden, Luheng He, Amy Shah, MariOstendorf, and Hannaneh Hajishirzi. 2019. A gen-eral framework for information extraction using dy-namic span graphs. In Proceedings of NAACL.

Diego Marcheggiani and Ivan Titov. 2017. Encodingsentences with graph convolutional networks for se-mantic role labeling. In Proceedings of EMNLP.

Makoto Miwa and Mohit Bansal. 2016. End-to-end re-lation extraction using lstms on sequences and treestructures. In Proceedings of ACL.

Makoto Miwa and Yutaka Sasaki. 2014. Modelingjoint entity and relation extraction with table repre-sentation. In Proceedings of EMNLP.

Nanyun Peng, Hoifung Poon, Chris Quirk, KristinaToutanova, and Wen tau Yih. 2017. Cross-sentencen-ary relation extraction with graph lstms. In Pro-ceedings of ACL.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global vectors for wordrepresentation. In Proceedings of EMNLP.

Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, andRegina Barzilay. 2019. Graphie: A graph-basedframework for information extraction. In Proceed-ings of NAACL.

Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R.Voss, Heng Ji, Tarek F. Abdelzaher, and Jiawei Han.2017. Cotype: Joint extraction of typed entities andrelations with knowledge bases. In Proceedings ofWWW.

Sebastian Riedel, Limin Yao, and Andrew McCallum.2010. Modeling relations and their mentions with-out labeled text. In Proceedings of ECML-PKDD.

Cicero Nogueira dos Santos, Bing Xiang, and BowenZhou. 2015. Classifying relations by ranking withconvolutional neural networks. In Proceedings ofACL.

Richard Socher, Brody Huval, Christopher D. Man-ning, and Andrew Y. Ng. 2012. Semantic composi-tionality through recursive matrix-vector spaces. InProceedings of EMNLP-CoNLL.

Xu Yan, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng,and Zhi Jin. 2015. Classifying relations via longshort term memory networks along shortest depen-dency paths. In Proceedings of EMNLP.

Xiaofeng Yu and Wai Lam. 2010. Jointly identifyingentities and extracting relations in encyclopedia textvia a graphical model approach. In Proceedings ofCOLING.

Dmitry Zelenko, Chinatsu Aone, and AnthonyRichardella. 2003. Kernel methods for relation ex-traction. Journal of Machine Learning Research(JMLR).

Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou,and Jun Zhao. 2014. Relation classification via con-volutional deep neural network. In Proceedings ofCOLING.

Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu,and Jun Zhao. 2018. Extracting relational facts byan end-to-end neural model with copy mechanism.In Proceedings of ACL.

Yuhao Zhang, Peng Qi, and Christopher D. Manning.2018. Graph convolution over pruned dependencytrees improves relation extraction. In Proceedingsof EMNLP.

Suncong Zheng, Feng Wang, Hongyun Bao, YuexingHao, Peng Zhou, and Bo Xu. 2017. Joint extrac-tion of entities and relations based on a novel taggingscheme. In Proceedings of ACL.

GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang.2005. Exploring various knowledge in relation ex-traction. In Proceedings of ACL.