Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2906–2919Florence, Italy, July 28 - August 2, 2019. c©2019 Association for Computational Linguistics

2906

Fine-Grained Temporal Relation Extraction

Siddharth VashishthaUniversity of Rochester

Benjamin Van DurmeJohns Hopkins University

Aaron Steven WhiteUniversity of Rochester

Abstract

We present a novel semantic framework formodeling temporal relations and event dura-tions that maps pairs of events to real-valuedscales. We use this framework to constructthe largest temporal relations dataset to date,covering the entirety of the Universal Depen-dencies English Web Treebank. We use thisdataset to train models for jointly predictingfine-grained temporal relations and event du-rations. We report strong results on our dataand show the efficacy of a transfer-learning ap-proach for predicting categorical relations.

1 Introduction

Natural languages provide a myriad of formal andlexical devices for conveying the temporal struc-ture of complex events—e.g. tense, aspect, auxil-iaries, adverbials, coordinators, subordinators, etc.Yet, these devices are generally insufficient for de-termining the fine-grained temporal structure ofsuch events. Consider the narrative in (1).(1) At 3pm, a boy broke his neighbor’s window.

He was running away, when the neighborrushed out to confront him. His parents werecalled but couldn’t arrive for two hours be-cause they were still at work.

Most native English speakers would have little dif-ficulty drawing a timeline for these events, likelyproducing something like that in Figure 1. Buthow do we know that the breaking, the run-ning away, the confrontation, and the calling wereshort, while the parents being at work was not?And why should the first four be in sequence, withthe last containing the others?

The answers to these questions likely involve acomplex interplay between linguistic information,on the one hand, and common sense knowledgeabout events and their relationships, on the other

run away

arrive

rush outconfront

callbe at work

break

5pm3pm 4pm

Figure 1: A typical timeline for the narrative in (1).

(Minsky, 1975; Schank and Abelson, 1975; Lam-port, 1978; Allen and Hayes, 1985; Hobbs et al.,1987; Hwang and Schubert, 1994). But it remainsa question how best to capture this interaction.

A promising line of attack lies in the taskof temporal relation extraction. Prior workin this domain has approached this task as aclassification problem, labeling pairs of event-referring expressions—e.g. broke or be at workin (1)—and time-referring expressions—e.g. 3pmor two hours—with categorical temporal rela-tions (Pustejovsky et al., 2003; Styler IV et al.,2014; Minard et al., 2016). The downside of thisapproach is that time-referring expressions mustbe relied upon to express duration information.But as (1) highlights, nearly all temporal dura-tion information can be left implicit without hin-dering comprehension, meaning these approachesonly explicitly encode duration information whenthat information is linguistically realized.

In this paper, we develop a novel frameworkfor temporal relation representation that puts eventduration front and center. Like standard ap-proaches using the TimeML standard, we draw in-spiration from Allen’s (1983) seminal work on in-terval representations of time. But instead of an-notating text for categorical temporal relations, wemap events to their likely durations and event pairsdirectly to real-valued relative timelines. Thischange not only supports the goal of giving a morecentral role to event duration, it also allows us tobetter reason about the temporal structure of com-

2907

plex events as described by entire documents.We first discuss prior work on temporal rela-

tion extraction (§2) and then present our frame-work and data collection methodology (§3). Theresulting dataset—Universal Decompositional Se-mantics Time (UDS-T)—is the largest temporalrelation dataset to date, covering all of the Univer-sal Dependencies (Silveira et al., 2014; De Marn-effe et al., 2014; Nivre et al., 2015) English WebTreebank (Bies et al., 2012). We use this dataset totrain a variety of neural models (§4) to jointly pre-dict event durations and fine-grained (real-valued)temporal relations (§5), yielding not only strongresults on our dataset, but also competitive perfor-mance on TimeML-based datasets (§6).1

2 Background

We review prior work on temporal relations frame-works and temporal relation extraction systems.

Corpora Most large temporal relation datasetsuse the TimeML standard (Pustejovsky et al.,2003; Styler IV et al., 2014; Minard et al., 2016).TimeBank is one of the earliest large corpora builtusing this standard, aimed at capturing ‘salient’temporal relations between events (Pustejovskyet al., 2003). The TempEval competitions build onTimeBank by covering relations between all theevents and times in a sentence.

Inter-sentential relations, which are necessaryfor document-level reasoning, have not been a fo-cus of the TempEval tasks, though at least onesub-task does address them (Verhagen et al., 2007,2010; UzZaman et al., 2013, and see Chamberset al. 2014). Part of this likely has to do with thesparsity inherent in the TempEval event-graphs.This sparsity has been addressed with corporasuch as the TimeBank-Dense, where annotators la-bel all local-edges irrespective of ambiguity (Cas-sidy et al., 2014). TimeBank-Dense does not cap-ture the complete graph over event and time rela-tions, instead attempting to achieve completenessby capturing all relations both within a sentenceand between neighboring sentences. We take in-spiration from this work for our own framework.

This line of work has been further improvedon by frameworks such as Richer Event De-scription (RED), which uses a multi-stage an-notation pipeline where various event-event phe-nomena, including temporal relations and sub-

1Data and code are available at http://decomp.io/.

event relations are annotated together in the samedatasets (O’Gorman et al., 2016). Similarly, Honget al. (2016) build a cross-document event cor-pus which covers fine-grained event-event rela-tions and roles with more number of event typesand sub-types (see also Fokkens et al., 2013).

Models Early systems for temporal relation ex-traction use hand-tagged features modeled withmultinomial logistic regression and support vectormachines (Mani et al., 2006; Bethard, 2013; Linet al., 2015). Other approaches use combined rule-based and learning-based approaches (D’Souzaand Ng, 2013) and sieve-based architectures—e.g. CAEVO (Chambers et al., 2014) andCATENA (Mirza and Tonelli, 2016). Recently,Ning et al. (2017) use a structured learning ap-proach and show significant improvements onboth TempEval-3 (UzZaman et al., 2013) andTimeBank-Dense (Cassidy et al., 2014). Ninget al. (2018) show further improvements onTimeBank-Dense by jointly modeling causal andtemporal relations using Constrained ConditionalModels and formulating the problem as an IntergerLinear Programming problem.

Neural network-based approaches have usedboth recurrent (Tourille et al., 2017; Cheng andMiyao, 2017; Leeuwenberg and Moens, 2018)and convolutional architectures (Dligach et al.,2017). Such models have furthermore been usedto construct document timelines from a set ofpredicted temporal relations (Leeuwenberg andMoens, 2018). Such use of pairwise annotationscan result in inconsistent temporal graphs, and ef-forts have been made to avert this issue by employ-ing temporal reasoning (Chambers and Jurafsky,2008; Yoshikawa et al., 2009; Denis and Muller,2011; Do et al., 2012; Laokulrat et al., 2016; Ninget al., 2017; Leeuwenberg and Moens, 2017).

Other work has aimed at modeling event du-rations from text (Pan et al., 2007; Gusev et al.,2011; Williams and Katz, 2012), though this workdoes not tie duration to temporal relations (see alsoFilatova and Hovy, 2001). Our approach com-bines duration and temporal relation informationwithin a unified framework, discussed below.

3 Data Collection

We collect the Universal Decompositional Seman-tics Time (UDS-T) dataset, which is annotated ontop of the Universal Dependencies (Silveira et al.,2014; De Marneffe et al., 2014; Nivre et al., 2015)

2908

Dataset #Events #Event-EventRelations

TimeBank 7,935 3,481TempEval 2010 5,688 3,308TempEval 2013 11,145 5,272TimeBank-Dense 1,729 8,130Hong et al. (2016) 863 25,610UDS-T 32,302 70,368

Table 1: Number of total events, and event-event tem-poral relations captured in various corpora

English Web Treebank (Bies et al., 2012) (UD-EWT). The main advantages of UD-EWT overother similar corpora are: (i) it covers text froma variety of genres; (ii) it contains gold standardUniversal Dependency parses; and (iii) it is com-patible with various other semantic annotationswhich use the same predicate extraction standard(White et al., 2016; Zhang et al., 2017; Rudingeret al., 2018; Govindarajan et al., 2019). Table 1compares the size of UDS-T against other tempo-ral relations datasets.

Protocol design Annotators are given two con-tiguous sentences from a document with two high-lighted event-referring expressions (predicates).They are then asked (i) to provide relative time-lines on a bounded scale for the pair of events re-ferred to by the highlighted predicates; and (ii) togive the likely duration of the event referred toby the predicate from the following list: instan-taneous, seconds, minutes, hours, days, weeks,months, years, decades, centuries, forever. In ad-dition, annotators were asked to give a confidenceratings for their relation annotation and each oftheir two duration annotation on the same five-point scale - not at all confident (0), not very con-fident (1), somewhat confident (2), very confident(3), totally confident (4).

An example of the annotation instrument isshown in Figure 2. Henceforth, we refer to thesituation referred to by the predicate that comesfirst in linear order (feed in Figure 2) as e1 and thesituation referred to by the predicate that comessecond in linear order (sick in Figure 2) as e2.

Annotators We recruited 765 annotators fromAmazon Mechanical Turk to annotate predicatepairs in groups of five. Each predicate pair con-tained in the UD-EWT train set was annotated bya single annotator, and each in the UD-EWT de-velopment and test sets was annotated by three.

Predicate extraction We extract predicatesfrom UD-EWT using PredPatt (White et al., 2016;

Figure 2: An annotated example from our protocol

Zhang et al., 2017), which identifies 33,935 pred-icates from 16,622 sentences. We concatenate allpairs of adjacent sentences in the documents con-tained in UD-EWT, allowing us to capture inter-sentential temporal relations. Considering all pos-sible pairs of predicates in adjacent sentences is in-feasible, so we use a heuristic to capture the mostinteresting pairs. (See Appendix A for details.)

1

A

B

C

0

0.12 0.48

0.48 0.66e1

e2

0.02 0.38

0.38 0.56e1

e2

0.30 0.48

0.480.57e1

e2

0 1

0 0.66

0.66 1e1

e2

Figure 3: Normalization of slider values

Normalization We normalize the slider re-sponses for each event pair by subtracting the min-imum slider value from all values, then dividingall such shifted values by the maximum value (af-ter shifting). This ensures that the earliest begin-ning point for every event pair lies at 0 and that theright-most end-point lies at 1 while preserving theratio between the durations implied by the sliders.Figure 3 illustrates this procedure for three hypo-thetical annotators annotating the same two eventse1 and e2. Assuming that the duration classes fore1 or e2 do not differ across annotators, the relativechronology of the events is the same in each case.This preservation of relative chronology, over ab-solute slider position, is important because, for thepurposes of determining temporal relation, the ab-solute positions that annotators give are meaning-less, and we do not want our models to be forcedto fit to such irrelevant information.

Inter-annotator agreement We measure inter-annotator agreement (IAA) for the temporal re-lation sliders by calculating the rank (Spearman)

2909

0.00

0.05

0.10

0.15

0.20

0.25

instant

seconds

minuteshoursda

ysweeks

monthsyears

decades

centuriesforever

Rel

ativ

e Fr

eque

ncy

Splitdevtrain

Figure 4: Distribution of event durations.

correlation between the normalized slider posi-tions for each pair of annotators that annotated thesame group of five predicate pairs in the develop-ment set.2 The development set is annotated by724 annotators. Rank correlation is a useful mea-sure because it tells us how much different anno-tators agree of the relative position of each slider.The average rank correlation between annotatorswas 0.665 (95% CI=[0.661, 0.669]).

For the duration responses, we compute the ab-solute difference in duration rank between the du-ration responses for each pair of annotators thatannotated the same group of five predicate pairs inthe development set. On average, annotators dis-agree by 2.24 scale points (95% CI=[2.21, 2.25]),though there is heavy positive skew (�1 = 1.16,95% CI=[1.15, 1.18])—evidenced by the fact thatthe modal rank difference is 1 (25.3% of the re-sponse pairs), with rank difference 0 as the nextmost likely (24.6%) and rank difference 2 as a dis-tant third (15.4%).

Summary statistics Figure 4 shows the distri-bution of duration responses in the training and de-velopment sets. There is a relatively high densityof events lasting minutes, with a relatively evendistribution across durations of years or less andfew events lasting decades or more.

The raw slider positions themselves are some-what difficult to directly interpret. To improve in-terpretability, we rotate the slider position spaceto construct four new dimensions: (i) PRIORITY,which is positive when e1 starts and/or ends ear-lier than e2 and negative otherwise; (ii) CONTAIN-MENT,which is most positive when e1 containsmore of e2; (iii) EQUALITY, which is largest when

2Our protocol design also allows us to detect some badannotations internal to the annotation itself, as opposed tocomparing one annotator’s annotation of an item to another.See Appendix B for further details on our deployment of suchannotation-internal validation techniques.

Figure 5: Distribution of event relations.

both e1 and e2 have the same temporal extentsand smallest when they are most unequal; and (iv)SHIFT, which moves the events forward or back-ward in time. We construct these dimensions bysolving for R in

R

2

664

�1 �1 1 1�1 1 1 �1�1 1 �1 11 1 1 1

3

775 = 2S� 1

where S 2 [0, 1]N⇥4 contains the slider posi-tions for our N datapoints in the following order:beg(e1), end(e1), beg(e2), end(e2).

Figure 5 shows the embedding of the event pairson the first three of these dimensions of R. The tri-angular pattern near the top and bottom of the plotarises because strict priority—i.e. extreme positiv-ity or negativity on the y-axis—precludes any tem-poral overlap between the two events, and as wemove toward the center of the plot, different prior-ity relations mix with different overlap relations—e.g. the upper-middle left corresponds to eventpairs where most of e1 comes toward the begin-ning of e2, while the upper middle right of the plotcorresponds to event pairs where most of e2 comestoward the end of e1.

4 Model

For each pair of events referred to in a sentence,we aim to jointly predict the relative timelines ofthose events as well as their durations. We thenuse a separate model to induce document timelinesfrom the relative timelines.

Relative timelines The relative timeline modelconsists of three components: an event model, a

2910

What to feed my been sickdog ….…. for

What to feed my been sickdog ….…. for

What to feed my been sickdog ….…. for

ELMo

Attention Attention

AttentionAttention Attention

gpred(i) gpred(j)

MLPrelMLPdur MLPdur

gdur(i) gdur(j)grel(i,j)

hours days

Tuner

Figure 6: Network diagram for model. Dashed arrowsare only included in some models.

duration model, and a relation model. These com-ponents use multiple layers of dot product atten-tion (Luong et al., 2015) on top of an embeddingH 2 RN⇥D for a sentence s = [w1, . . . , wN ]tuned on the three M -dimensional contextual em-beddings produced by ELMo (Peters et al., 2018)for that sentence, concatenated together.

H = tanh (ELMo(s)WTUNE + bTUNE)

where D is the dimension for the tuned embed-dings, WTUNE 2 R3M⇥D, and b

TUNE 2 RN⇥D.

Event model We define the model’s represen-tation for the event referred to by predicate k asgpredk 2 RD, where D is the embedding size.We build this representation using a variant of dot-product attention, based on the predicate root.

aSPANpredk = tanh

�A

SPANPREDhROOT(predk) + b

SPANPRED

�

↵predk = softmax�HSPAN(predk)a

SPANpredk

�

gpredk = [hROOT(predk);↵predkHSPAN(predk)]

where ASPANPRED 2 RD⇥D,bSPAN

PRED 2 RD; hROOT(predk)is the hidden representation of the kth predicate’sroot; and HSPAN(predk) is obtained by stacking thehidden representations of the entire predicate.

As an example, the predicate been sick fornow in Figure 2 has sick as its root, and thuswe would take the hidden representation for sickas hROOT(predk). Similarly, HSPAN(predk) would beequal to taking the hidden-state representations

of been sick for now and stacking them together.Then, if the model learns that tense information isimportant, it may weight been using attention.

Duration model The temporal duration rep-resentation gdurk for the event referred to by thekth predicate is defined similarly to the event rep-resentation, but instead of stacking the predicate’sspan, we stack the hidden representations of theentire sentence H.

aSENTdurk = tanh

�A

SENTDUR gpredk + b

SENTDUR

�

↵durk = softmax(HaSENTdurk )

gdurk = [gpredk ;↵durkH]

where ASENTDUR 2 RD⇥size(gpredk ) and b

SENTDUR 2 RD.

We consider two models of the categorical du-rations: a softmax model and a binomial model.The main difference is that the binomial modelenforces that the probabilities pdurk over the 11duration values be concave in the duration rank,whereas the softmax model has no such constraint.We employ a cross-entropy loss for both models.

Ldur(dk;p) = � log pdk

In the softmax model, we pass the duration rep-resentation gdurk for predicate k through a multi-layer perceptron (MLP) with a single hidden layerof ReLU activations, to yield probabilities pdurkover the 11 durations.

vdurk = ReLU(W(1)DURgdurk + b

(1)DUR)

p = softmax(W(2)DURvdurk + b

(2)DUR)

In the binomial distribution model, we again passthe duration representation through a MLP witha single hidden layer of ReLU activations, but inthis case, we yield only a single value ⇡durk . Withvdurk as defined above:

⇡ = �⇣w

(2)DURvdurk + b

(2)DUR

⌘

pc =

✓n

c

◆⇡n(1� ⇡)(n�c)

where c 2 {0, 1, 2, ..., 10} represents the rankeddurations – instant (0), seconds (1), minutes (2),..., centuries (9), forever (10) – and n is the maxi-mum class rank (10).

Relation model To represent the temporal re-lation representation between the events referredto by the ith and jth predicate, we again use a sim-ilar attention mechanism.

2911

aSENTrelij = tanh

�A

SENTREL [gpredi ;gpredj ] + b

SENTREL

�

↵relij = softmax⇣Ha

SENTrelij

⌘

grelij = [gpredi ;gpredj ;↵relijH]

where ASENTREL 2 RD⇥2size(gpredk ) and b

SENTREL 2 RD.

The main idea behind our temporal model is tomap events and states directly to a timeline, whichwe represent via a reference interval [0, 1]. Forsituation k, we aim to predict the beginning pointbk and end-point ek � bk of k.

We predict these values by passing grelijthrough an MLP with one hidden layer ofReLU activations and four real-valued outputs[�i, �i, �j , �j ], representing the estimated relativebeginning points (�i, �j) and durations (�i, �j) forevents i and j. We then calculate the predictedslider values sij = [bi, ei, bj , ej ]

hbk, ek

i=

h�⇣�k

⌘,�

⇣�k +

����k���⌘i

The predicted values sij are then normalized in thesame fashion as the true slider values prior to beingentered into the loss. We constrain this normalizedsij using four L1 losses.

Lrel(sij ; sij) =���(bi � bj)� (bi � bj)

���+���(ei � bj)� (ei � bj)

���+���(ej � bi)� (ej � bi)

���+

|(ei � ej)� (ei � ej)|

The final loss function is then L = Ldur + 2Lrel.Duration-relation connections We also ex-

periment with four architectures wherein the du-ration and relation models are connected to eachother in the Dur! Rel or Dur Rel directions.

In the Dur ! Rel architectures, we modifygrelij in two ways: (i) additionally concatenatingthe ith and jth predicate’s duration probabilitiesfrom the binomial distribution model, and (ii) notusing the relation representation model at all.

grelij = [gpredi ;gpredj ;↵relijH;pi;pj ]

grelij = [pi;pj ]

In the Dur Rel architectures, we use two mod-ifications: (i) we modify gdurk by concatenatingthe bk and ek from the relation model, and (ii)we do not use the duration representation model

at all, instead use the predicted relative durationek � bk obtained from the relation model, passingit through the binomial distribution model.

gdurk = [gpredk ;↵durkH; bk; ek]

⇡durk = ek � bk

Document timelines We induce the hidden doc-ument timelines for the documents in the UDS-T development set using relative timelines from(i) actual pairwise slider annotations; or (ii) slidervalues predicted by the best performing model onUDS-T development set. To do this, we assume ahidden timeline T 2 Rnd⇥2

+ , where nd is the to-tal number of predicates in that document, the twodimensions represent the beginning point and theduration of the predicates. We connect these la-tent timelines to the relative timelines, by anchor-ing the beginning points of all predicates such thatthere is always a predicate with 0 as the beginningpoint in a document and defining auxiliar variables⌧ij and sij for each events i and j.

⌧ij = [ti1, ti1 + ti2, tj1, tj1 + tj2]

sij =⌧ij �min(⌧ij)

max(⌧ij �min(⌧ij))

We learn T for each document under the relationloss Lrel(sij , sij). We further constrain T to pre-dict the categorical durations using the binomialdistribution model on the durations tk2 implied byT, assuming ⇡k = �(c log(tk2)).

5 Experiments

We implement all models in pytorch 1.0. Forall experiments, we use mini-batch gradient de-scent with batch-size 64 to train the embeddingtuner (reducing ELMo to a dimension of 256), at-tention, and MLP parameters. Both the relationand duration MLP have a single hidden layer with128 nodes and a dropout probability of 0.5 (seeAppendix D for further details).

To predict TimeML relations in TempE-val3 (TE3; UzZaman et al., 2013, Task C-relationonly) and TimeBank-Dense (TD; Cassidy et al.,2014), we use a transfer learning approach. Wefirst use the best-performing model on the UDS-Tdevelopment set to obtain the relation representa-tion (grelij ) for each pair of annotated event-eventrelations in TE3 and TD (see Appendix E for pre-processing details). We then use this vector as in-put features to a SVM classifier with a Gaussian

2912

Model Duration RelationDuration Relation Connection ⇢ rank diff. R1 Absolute ⇢ Relative ⇢ R1

softmax X - 32.63 1.86 8.59 77.91 68.00 2.82binomial X - 37.75 1.75 13.73 77.87 67.68 2.35- X Dur Rel 22.65 3.08 -51.68 71.65 66.59 -6.09binomial - Dur! Rel 36.52 1.76 13.17 77.58 66.36 0.85binomial X Dur! Rel 38.38 1.75 13.85 77.82 67.73 2.58binomial X Dur Rel 38.12 1.75 13.68 78.12 68.22 2.96

Table 2: Results on test data based on different model representations; ⇢ denotes the Spearman-correlation coef-ficient; rank-diff is the duration rank difference. The model highlighted in blue performs best on durations and isalso close to the top performing model for relations on the development set. The numbers highlighted in bold arethe best-performing numbers on the test data in the respective columns.

kernel to train on the training sets of these datasetsusing the feature vector obtained from our model.3

Following recent work using continuous la-bels in event factuality prediction (Lee et al.,2015; Stanovsky et al., 2017; Rudinger et al.,2018; White et al., 2018) and genericity prediction(Govindarajan et al., 2019), we report three met-rics for the duration prediction: Spearman correla-tion (⇢), mean rank difference (rank diff ), and pro-portion rank difference explained (R1). We reportthree metrics for the relation prediction: Spearmancorrelation between the normalized values of ac-tual beginning and end points and the predictedones (absolute ⇢), the Spearman correlation be-tween the actual and predicted values in Lrel (rela-tive ⇢), and the proportion of MAE explained (R1).

R1 = 1� MAEmodel

MAEbaseline

where MAEbaseline is always guessing the median.

6 Results

Table 2 shows the results of different model ar-chitectures on the UDS-T test set, and Table 4shows the results of our transfer-learning approachon test set of TimeBank-Dense (TD-test).

UDS-T results Most of our models are able topredict the relative position of the beginning andending of events very well (high relation ⇢) andthe relative duration of events somewhat well (rel-atively low duration ⇢), but they have a lot moretrouble predicting relation exactly and relativelyless trouble predicting duration exactly.

3For training on TE3, we use TimeBank (TB; Pustejovskyet al., 2003) + AQUAINT (AQ; Graff) datasets provided inthe TE3 workshop (UzZaman et al., 2013). For training onTD, we use TD-train and TD-dev.

Duration model The binomial distributionmodel outperforms the softmax model for dura-tion prediction by a large margin, though it hasbasically no effect on the accuracy of the relationmodel, with the binomial and softmax models per-forming comparably. This suggests that enforcingconcavity in duration rank on the duration proba-bilities helps the model better predict durations.

Connections Connecting the duration and re-lation model does not improve performance ingeneral. In fact, when the durations are directlypredicted from the temporal relation model—i.e.without using the duration representation model—the model’s performance drops by a large margin,with the Spearman correlation down by roughly 15percentage points. This indicates that constrain-ing the relations model to predict the durations isnot enough and that the duration representation isneeded to predict durations well. On the otherhand, predicting temporal relations directly fromthe duration probability distribution—i.e. withoutusing the relation representation model—resultsin a similar score as that of the top-performingmodel. This indicates that the duration representa-tion is able to capture most of the relation charac-teristics of the sentence. Using both duration rep-resentation and relation representation separately(model highlighted in blue) results in the best per-formance overall on the UDS-T development set.

TimeBank-Dense and TempEval3 Table 4 re-ports F1-micro scores on the test set of TimeBank-Dense compared with some other systems as re-ported by Cheng and Miyao (2017). We reportthese scores only on Event-Event (E-E) relationsas our system captures only those. We also com-pute the standard temporal awareness F1 score onthe test set of TempEval-3 (TE3-PT) considering

2913

DurationWord Attention Rank Freq

soldiers 0.911 1.28 69months 0.844 1.38 264Nothing 0.777 5.07 114minutes 0.768 1.33 81astronauts 0.756 1.37 81hour 0.749 1.41 84Palestinians 0.735 1.72 288month 0.721 2.03 186cartoonists 0.714 1.35 63years 0.708 1.94 588days 0.635 1.39 84thoughts 0.592 2.90 60us 0.557 2.09 483week 0.531 2.23 558advocates 0.517 2.30 105

RelationWord Attention Rank Freq

occupied 0.685 1.33 54massive 0.522 2.71 66social 0.510 1.68 57general 0.410 3.52 168few 0.394 3.07 474mathematical 0.393 7.66 132are 0.387 3.47 4415comes 0.339 2.39 51or 0.326 3.50 3137and 0.307 4.86 17615emerge 0.305 2.67 54filed 0.303 7.14 66s 0.298 4.03 1152were 0.282 3.49 1308gets 0.239 7.36 228

Table 3: Mean attention weight, mean attention rank, and frequency for 15 words in the development set with thehighest mean duration-attention (left) and relation-attention (right) weights. For duration, the words highlighted inbold directly correspond to some duration class. For relation, the words in bold are either conjunctions or wordscontaining tense information.

only E-E relations and achieve a score of 0.498.4

Our system beats the TD F1-micro scores of allother systems reported in Table 4. As a refer-ence, the top performing system on TE3-PT (Ninget al., 2017) reports an F1 score of 0.672 overall relations, but is not directly comparable to oursystem as we only evaluate on event-event rela-tions. These results indicate that our model is ableto achieve competitive performance on other stan-dard temporal classification problems.

Systems Evaluation Data F1(E-E)

CAEVO TD-test 0.494CATENA TD-test 0.519Cheng and Miyao (2017) TD-test 0.529This work TD-test 0.566

Table 4: F1-micro scores of event-event relations inTD-test based on our transfer learning experiment.

7 Model Analysis and Timelines

We investigate two aspects of the best-performingmodel on the development set (highlighted in Ta-ble 2): (i) what our duration and relation rep-resentations attend to; and (ii) how well docu-ment timelines constructed from the model’s pre-

4We do not report the temporal awareness scores (F1)of other systems on TE3-PT, since they report their metricson all relations, including timex-timex, and event-timex re-lations, and thus they are not directly comparable. For TD,only those systems are reported that report F1-micro scores.

dictions match those constructed from the annota-tions. (See Appendix F for further analyses.)

Attention The advantage of using an attentionmechanism is that we can often interpret what lin-guistic information the model is using by analyz-ing the attention weights. We extract these at-tention weights for both the duration representa-tion and the relation representation from our bestmodel on the development set.

Duration We find that words that denotesome time period—e.g. month(s), minutes, hour,years, days, week—are among the words withhighest mean attention weight in the durationmodel, with seven of the top 15 words directlydenoting one of the duration classes (Table 3).This is exactly what one might expect this modelto rely heavily on, since time expressions arelikely highly informative for making predictionsabout duration. It also may suggest that wedo not need to directly encode relations betweenevent-referring and time-referring expressions inour framework—as do annotation standards likeTimeML—since our models may discover them.

The remainder of the top words in the dura-tion model are plurals or mass nouns (soldiers,thoughts etc.). This may suggest that the plu-rality of a predicate’s arguments is an indicatorof the likely duration of the event referred to

2914

by that predicate. To investigate this possibil-ity, we compute a multinomial regression predict-ing the attention weights ↵s for each sentence sfrom the K morphological features of each wordin that sentence Fs 2 {0, 1}length(s)⇥K , whichare extracted from the UD-EWT features columnand binarized. To do this, we optimize coeffi-cients c in argcmin

PsD (↵s k softmax (Fsc)),

where D is the KL divergence. We find thatthe five most strongly weighted positive featuresin c are all features of nouns—NUMBER=plur,CASE=acc, PRONTYPE=prs, NUMBER=sing, GEN-DER=masc—suggesting that good portion of du-ration information can be gleaned from the ar-guments of a predicate. This may be becausenominal information can be useful in determin-ing whether the clause is about particular eventsor generic events (Govindarajan et al., 2019).

Relation A majority of the words with high-est mean attention weight in the relation modelare either coordinators—such as or and and—orbearers of tense information—i.e. lexical verbsand auxiliaries. The first makes sense because, incontext, coordinators can carry information abouttemporal sequencing (see Wilson and Sperber,1998, i.a.). The second makes sense in that infor-mation about the tense of predicates being com-pared likely helps the model determine relative or-dering of the events they refer to.

Similar to duration attention analysis, for rela-tion attention, we find that the five most stronglyweighted positive features in c are all fea-tures of verbs or auxiliaries—PERSON=1, PER-SON=3, TENSE=pres, TENSE=past, MOOD=ind—suggesting that a majority of the information rel-evant to relation can be gleaned from the tense-bearing units in a clause.

Document timelines We apply the documenttimeline model described in §4 to both the an-notations on the development set and the best-performing model’s predictions to obtain timelinesfor all documents in the development set. Figure7 shows an example, comparing the two resultingdocument timelines.

For these two timelines, we compare the in-duced beginning points and durations, obtaining amean Spearman correlation of 0.28 for beginningpoints and -0.097 for durations. This suggests thatthe model agrees to some extent with the annota-tions about the beginning points of events in mostdocuments but is struggling to find the correct du-

was lower than

got

rate

showed

was great

took

recommend

go

explain

Figure 7: Learned timeline for the following docu-ment based on actual (black) and predicted (red) an-notations: “A+. I would rate Fran pcs an A + becausethe price was lower than everyone else , i got my com-puter back the next day , and the professionalism heshowed was great . He took the time to explain thingsto me about my computer , i would recommend you goto him. David”

ration spans. One possible reason for poor pre-diction of durations could be the lack of a directsource of duration information. The model cur-rently tries to identify the duration based only onthe slider values, which leads to poor performanceas already seen in one of the Dur Rel model.

8 Conclusion

We presented a novel semantic framework formodeling fine-grained temporal relations andevent durations that maps pairs of events to real-valued scales for the purpose of constructingdocument-level event timelines. We used thisframework to construct the largest temporal rela-tions dataset to date – UDS-T – covering the en-tirety of the UD-EWT. We used this dataset to trainmodels for jointly predicting fine-grained tempo-ral relations and event durations, reporting strongresults on our data and showing the efficacy of atransfer-learning approach for predicting standard,categorical TimeML relations.

Acknowledgments

We are grateful to the FACTS.lab at the Universityof Rochester as well as three anonymous reviewersfor useful comments on this work. This researchwas supported by the University of Rochester,JHU HLTCOE, and DARPA AIDA. The U.S.Government is authorized to reproduce and dis-tribute reprints for Governmental purposes. Theviews and conclusions contained in this publica-tion are those of the authors and should not beinterpreted as representing official policies or en-dorsements of DARPA or the U.S. Government.

2915

ReferencesJames F Allen. 1983. Maintaining knowledge about

temporal intervals. Communications of the ACM,26(11):832–843.

James F Allen and Patrick J Hayes. 1985. A common-sense theory of time. In Proceedings of the 9th Inter-national Joint Conference on Artificial Intelligence-Volume 1, pages 528–531. Morgan Kaufmann Pub-lishers Inc.

Steven Bethard. 2013. Cleartk-timeml: A minimalistapproach to tempeval 2013. In Second Joint Con-ference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh In-ternational Workshop on Semantic Evaluation (Se-mEval 2013), volume 2, pages 10–14.

Ann Bies, Justin Mott, Colin Warner, and Seth Kulick.2012. English web treebank. Linguistic Data Con-sortium, Philadelphia, PA.

Taylor Cassidy, Bill McDowell, Nathanael Chambers,and Steven Bethard. 2014. An annotation frame-work for dense event ordering. In Proceedings of the52nd Annual Meeting of the Association for Compu-tational Linguistics (Volume 2: Short Papers), vol-ume 2, pages 501–506.

Nathanael Chambers, Taylor Cassidy, Bill McDowell,and Steven Bethard. 2014. Dense event orderingwith a multi-pass architecture. Transactions of theAssociation for Computational Linguistics, 2:273–284.

Nathanael Chambers and Dan Jurafsky. 2008. Jointlycombining implicit constraints improves temporalordering. In Proceedings of the Conference on Em-pirical Methods in Natural Language Processing,pages 698–706. Association for Computational Lin-guistics.

Fei Cheng and Yusuke Miyao. 2017. Classifying tem-poral relations by bidirectional lstm over depen-dency paths. In Proceedings of the 55th AnnualMeeting of the Association for Computational Lin-guistics (Volume 2: Short Papers), volume 2, pages1–6.

Marie-Catherine De Marneffe, Timothy Dozat, Na-talia Silveira, Katri Haverinen, Filip Ginter, JoakimNivre, and Christopher D. Manning. 2014. Uni-versal Stanford dependencies: A cross-linguistic ty-pology. In Proceedings of the Ninth InternationalConference on Language Resources and Evaluation(LREC-2014), volume 14, pages 4585–4592.

Pascal Denis and Philippe Muller. 2011. Predictingglobally-coherent temporal structures from texts viaendpoint inference and graph decomposition. InIJCAI-11-International Joint Conference on Artifi-cial Intelligence.

Dmitriy Dligach, Timothy Miller, Chen Lin, StevenBethard, and Guergana Savova. 2017. Neural tem-poral relation extraction. In Proceedings of the 15thConference of the European Chapter of the Associa-tion for Computational Linguistics: Volume 2, ShortPapers, volume 2, pages 746–751.

Quang Xuan Do, Wei Lu, and Dan Roth. 2012. Jointinference for event timeline construction. In Pro-ceedings of the 2012 Joint Conference on EmpiricalMethods in Natural Language Processing and Com-putational Natural Language Learning, pages 677–687. Association for Computational Linguistics.

Jennifer D’Souza and Vincent Ng. 2013. Classifyingtemporal relations with rich linguistic knowledge.In Proceedings of the 2013 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 918–927.

Elena Filatova and Eduard Hovy. 2001. Assigningtime-stamps to event-clauses. In Proceedings ofthe Workshop on Temporal and Spatial Informa-tion Processing-Volume 13, page 13. Association forComputational Linguistics.

Antske Fokkens, Marieke van Erp, Piek Vossen, SaraTonelli, Willem Robert van Hage, Luciano Ser-afini, Rachele Sprugnoli, and Jesper Hoeksema.2013. GAF: A grounded annotation framework forevents. In Workshop on Events: Definition, Detec-tion, Coreference, and Representation, pages 11–20.

Venkata Subrahmanyan Govindarajan, BenjaminVan Durme, and Aaron Steven White. 2019.Decomposing generalization: Models of generic,habitual, and episodic statements. arXiv preprintarXiv:1901.11429.

David Graff. The aquaint corpus of English newstext:[content copyright] Portions c� 1998-2000 NewYork Times, Inc., c� 1998-2000 Associated Press,Inc., c� 1996-2000 Xinhua News Service. Linguis-tic Data Consortium.

Andrey Gusev, Nathanael Chambers, Pranav Khaitan,Divye Khilnani, Steven Bethard, and Dan Juraf-sky. 2011. Using query patterns to learn the dura-tion of events. In Proceedings of the Ninth Inter-national Conference on Computational Semantics,pages 145–154. Association for Computational Lin-guistics.

Jerry R Hobbs, William Croft, Todd Davies, DouglasEdwards, and Kenneth Laws. 1987. Commonsensemetaphysics and lexical semantics. ComputationalLinguistics, 13(3-4):241–250.

Yu Hong, Tongtao Zhang, Tim O’Gorman, SharoneHorowit-Hendler, Heng Ji, and Martha Palmer.2016. Building a cross-document event-event rela-tion corpus. In Proceedings of the 10th LinguisticAnnotation Workshop held in conjunction with Asso-ciation for Computational Linguistics 2016 (LAW-X2016), pages 1–6.

2916

Harold Hotelling. 1936. Relations between two sets ofvariates. Biometrika, 28(3/4):321–377.

Chung Hee Hwang and Lenhart K Schubert. 1994. In-terpreting tense, aspect and time adverbials: A com-positional, unified approach. In Temporal Logic,pages 238–264. Springer.

Leslie Lamport. 1978. Time, clocks, and the orderingof events in a distributed system. Communicationsof the ACM, 21(7):558–565.

Natsuda Laokulrat, Makoto Miwa, and Yoshimasa Tsu-ruoka. 2016. Stacking approach to temporal relationclassification with temporal inference. Informationand Media Technologies, 11:53–78.

Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettle-moyer. 2015. Event detection and factuality assess-ment with non-expert supervision. In Proceedingsof the 2015 Conference on Empirical Methods inNatural Language Processing, pages 1643–1648.

Artuur Leeuwenberg and Marie-Francine Moens.2018. Temporal information extraction by predict-ing relative time-lines. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, pages 1237–1246.

Tuur Leeuwenberg and Marie-Francine Moens. 2017.Structured learning for temporal relation extractionfrom clinical records. In Proceedings of the 15thConference of the European Chapter of the Asso-ciation for Computational Linguistics, pages 1150–1158.

Chen Lin, Dmitriy Dligach, Timothy A Miller, StevenBethard, and Guergana K Savova. 2015. Multi-layered temporal modeling for the clinical domain.Journal of the American Medical Informatics Asso-ciation, 23(2):387–395.

Minh-Thang Luong, Hieu Pham, and Christopher DManning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprintarXiv:1508.04025.

Inderjeet Mani, Marc Verhagen, Ben Wellner,Chong Min Lee, and James Pustejovsky. 2006. Ma-chine learning of temporal relations. In Proceedingsof the 21st International Conference on Compu-tational Linguistics and the 44th annual meetingof the Association for Computational Linguistics,pages 753–760. Association for ComputationalLinguistics.

Christopher Manning, Mihai Surdeanu, John Bauer,Jenny Finkel, Steven Bethard, and David McClosky.2014. The stanford corenlp natural language pro-cessing toolkit. In Proceedings of 52nd AnnualMeeting of the Association for Computational Lin-guistics: System Demonstrations, pages 55–60.

Anne-Lyse Myriam Minard, Manuela Speranza, RubenUrizar, Begona Altuna, Marieke van Erp, AnneleenSchoen, and Chantal van Son. 2016. Meantime,

the newsreader multilingual event and time corpus.In Proceedings of the Tenth International Confer-ence on Language Resources and Evaluation (LREC2016). European Language Resources Association(ELRA).

Marvin Minsky. 1975. A framework for representingknowledge. The Psychology of Computer Vision.

Paramita Mirza and Sara Tonelli. 2016. Catena: Causaland temporal relation extraction from natural lan-guage texts. In Proceedings of COLING 2016,the 26th International Conference on ComputationalLinguistics: Technical Papers, pages 64–75.

Qiang Ning, Zhili Feng, and Dan Roth. 2017. A struc-tured learning approach to temporal relation extrac-tion. In Proceedings of the 2017 Conference on Em-pirical Methods in Natural Language Processing,pages 1027–1037.

Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018.Joint reasoning for temporal and causal relations. InProceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), volume 1, pages 2278–2288.

Joakim Nivre, Zeljko Agic, Maria Jesus Aranzabe,Masayuki Asahara, Aitziber Atutxa, Miguel Balles-teros, John Bauer, Kepa Bengoetxea, Riyaz Ah-mad Bhat, Cristina Bosco, Sam Bowman, GiuseppeG. A. Celano, Miriam Connor, Marie-Catherinede Marneffe, Arantza Diaz de Ilarraza, Kaja Do-brovoljc, Timothy Dozat, Toma Erjavec, RichrdFarkas, Jennifer Foster, Daniel Galbraith, Filip Gin-ter, Iakes Goenaga, Koldo Gojenola, Yoav Gold-berg, Berta Gonzales, Bruno Guillaume, Jan Haji,Dag Haug, Radu Ion, Elena Irimia, Anders Jo-hannsen, Hiroshi Kanayama, Jenna Kanerva, Si-mon Krek, Veronika Laippala, Alessandro Lenci,Nikola Ljubei, Teresa Lynn, Christopher Man-ning, Ctlina Mrnduc, David Mareek, Hctor Mart-nez Alonso, Jan Maek, Yuji Matsumoto, Ryan Mc-Donald, Anna Missil, Verginica Mititelu, YusukeMiyao, Simonetta Montemagni, Shunsuke Mori,Hanna Nurmi, Petya Osenova, Lilja vrelid, ElenaPascual, Marco Passarotti, Cenel-Augusto Perez,Slav Petrov, Jussi Piitulainen, Barbara Plank, Mar-tin Popel, Prokopis Prokopidis, Sampo Pyysalo, Lo-ganathan Ramasamy, Rudolf Rosa, Shadi Saleh, Se-bastian Schuster, Wolfgang Seeker, Mojgan Ser-aji, Natalia Silveira, Maria Simi, Radu Simionescu,Katalin Simk, Kiril Simov, Aaron Smith, Jan tpnek,Alane Suhr, Zsolt Sznt, Takaaki Tanaka, Reut Tsar-faty, Sumire Uematsu, Larraitz Uria, Viktor Varga,Veronika Vincze, Zdenk abokrtsk, Daniel Zeman,and Hanzhi Zhu. 2015. Universal Dependencies 1.2.http://universaldependencies.github.io/docs/.

Tim O’Gorman, Kristin Wright-Bettner, and MarthaPalmer. 2016. Richer event description: Integratingevent coreference with temporal, causal and bridg-ing annotation. In Proceedings of the 2nd Workshopon Computing News Storylines (CNS 2016), pages47–56.

2917

Feng Pan, Rutu Mulkar-Mehta, and Jerry R Hobbs.2007. Modeling and learning vague event dura-tions for temporal reasoning. In Proceedings of the22nd National Conference on Artificial Intelligence-Volume 2, pages 1659–1662. AAAI Press.

Fabian Pedregosa, Gal Varoquaux, Alexandre Gram-fort, Vincent Michel, Bertrand Thirion, OlivierGrisel, Mathieu Blondel, Peter Prettenhofer, RonWeiss, Vincent Dubourg, Jake Vanderplas, Alexan-dre Passos, David Cournapeau, Matthieu Brucher,Matthieu Perrot, and Edouard Duchesnay. 2011.Scikit-learn: Machine Learning in Python. Journalof Machine Learning Research, 12:2825–2830.

Matthew E Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. arXiv preprint arXiv:1802.05365.

James Pustejovsky, Patrick Hanks, Roser Sauri, An-drew See, Robert Gaizauskas, Andrea Setzer,Dragomir Radev, Beth Sundheim, David Day, LisaFerro, et al. 2003. The timebank corpus. In Corpuslinguistics, volume 2003, page 40. Lancaster, UK.

Rachel Rudinger, Aaron Steven White, and BenjaminVan Durme. 2018. Neural models of factuality.arXiv preprint arXiv:1804.02472.

Roger C Schank and Robert P Abelson. 1975. Scripts,plans, and knowledge. In Proceedings of the4th International Joint Conference on ArtificialIntelligence-Volume 1, pages 151–157. MorganKaufmann Publishers Inc.

Natalia Silveira, Timothy Dozat, Marie-CatherineDe Marneffe, Samuel R Bowman, Miriam Connor,John Bauer, and Christopher D Manning. 2014. Agold standard dependency corpus for english. InProceedings of the Ninth International Conferenceon Language Resources and Evaluation (LREC-2014), pages 2897–2904.

Gabriel Stanovsky, Judith Eckle-Kohler, YevgeniyPuzikov, Ido Dagan, and Iryna Gurevych. 2017. In-tegrating deep linguistic features in factuality pre-diction over unified datasets. In Proceedings of the55th Annual Meeting of the Association for Compu-tational Linguistics (Volume 2: Short Papers), vol-ume 2, pages 352–357.

William F Styler IV, Steven Bethard, Sean Finan,Martha Palmer, Sameer Pradhan, Piet C de Groen,Brad Erickson, Timothy Miller, Chen Lin, GuerganaSavova, et al. 2014. Temporal annotation in the clin-ical domain. Transactions of the Association forComputational Linguistics, 2:143.

Julien Tourille, Olivier Ferret, Aurelie Neveol, andXavier Tannier. 2017. Neural architecture for tem-poral relation extraction: A bi-lstm approach for de-tecting narrative containers. In Proceedings of the55th Annual Meeting of the Association for Compu-tational Linguistics (Volume 2: Short Papers), vol-ume 2, pages 224–230.

Naushad UzZaman, Hector Llorens, Leon Derczyn-ski, James Allen, Marc Verhagen, and James Puste-jovsky. 2013. Semeval-2013 task 1: Tempeval-3:Evaluating time expressions, events, and temporalrelations. In Second Joint Conference on Lexicaland Computational Semantics (* SEM), Volume 2:Proceedings of the Seventh International Workshopon Semantic Evaluation (SemEval 2013), volume 2,pages 1–9.

Marc Verhagen, Robert Gaizauskas, Frank Schilder,Mark Hepple, Graham Katz, and James Pustejovsky.2007. Semeval-2007 task 15: Tempeval temporalrelation identification. In Proceedings of the 4thInternational Workshop on Semantic Evaluations,pages 75–80. Association for Computational Lin-guistics.

Marc Verhagen, Roser Sauri, Tommaso Caselli, andJames Pustejovsky. 2010. Semeval-2010 task 13:Tempeval-2. In Proceedings of the 5th Interna-tional Workshop on Semantic Evaluation, pages 57–62. Association for Computational Linguistics.

Aaron Steven White, Drew Reisinger, Keisuke Sak-aguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger,Kyle Rawlins, and Benjamin Van Durme. 2016.Universal decompositional semantics on universaldependencies. In Proceedings of the 2016 Confer-ence on Empirical Methods in Natural LanguageProcessing, pages 1713–1723, Austin, TX. Associ-ation for Computational Linguistics.

Aaron Steven White, Rachel Rudinger, Kyle Rawl-ins, and Benjamin Van Durme. 2018. Lexicosyn-tactic inference in neural models. arXiv preprintarXiv:1808.06232.

Jennifer Williams and Graham Katz. 2012. Extractingand modeling durations for habits and events fromtwitter. In Proceedings of the 50th Annual Meet-ing of the Association for Computational Linguis-tics: Short Papers-Volume 2, pages 223–227. Asso-ciation for Computational Linguistics.

Deirdre Wilson and Dan Sperber. 1998. Pragmaticsand time. Pragmatics and Beyond New Series, pages1–22.

Katsumasa Yoshikawa, Sebastian Riedel, MasayukiAsahara, and Yuji Matsumoto. 2009. Jointly identi-fying temporal relations with markov logic. In Pro-ceedings of the Joint Conference of the 47th AnnualMeeting of the ACL and the 4th International JointConference on Natural Language Processing of theAFNLP: Volume 1-Volume 1, pages 405–413. Asso-ciation for Computational Linguistics.

Sheng Zhang, Rachel Rudinger, and BenjaminVan Durme. 2017. An evaluation of predpatt andopen ie via stage 1 semantic role labeling. In IWCS2017—12th International Conference on Computa-tional Semantics (Short papers).

2918

A Data Collection

We concatenate two adjacent sentences to forma combined sentence which allows us to captureinter-sentential temporal relations. Considering allpossible pairs of events in the combined sentenceresults into an exploding number of event-eventcomparisons. Therefore, to reduce the total num-ber of comparisons, we find the pivot-predicate ofthe antecedent of the combined sentence as fol-lows - find the root predicate of the antecedent andif it governs a CCOMP, CSUBJ, or XCOMP, fol-low that dependency to the next predicate until apredicate is found that doesn’t govern a CCOMP,CSUBJ, or XCOMP. We then take all pairs of theantecedent predicates and pair every predicate ofthe consequent only with the pivot-predicate. Thisresults into

�N2

�+M predicates instead of

�N+M2

�

per sentence, where N and M are the number ofpredicates in the antecedent and consequent re-spectively. This heuristic allows us to find a pred-icate that loosely denotes the topic being talkedabout in the sentence. Figure 8 shows an exampleof finding the pivot predicate.

Figure 8: Our heuristic finds fly as (the root of) thepivot predicate in Has anyone considered that perhapsGeorge Bush just wanted to fly jets?

B Rejecting Annotations

We design multiple checks to detect potentiallybad annotations during our data collection. A sin-gle assignment contains 5 annotations (predicate-pairs). Once an annotation is flagged by any ofthese checks, we may accept or reject the assign-ment based on our subjective opinion about theparticular case. Annotations are flagged based onthe following conditions:

B.1 Time completionOur pilot studies indicate a median time of roughly4 minutes to complete a single assignment (5 an-notations). We automatically reject any assign-

Figure 9: An example illustrating an inconsistency be-tween the annotated slider positions and the durations

ment which is completed under a minute as webelieve that it is not plausible to finish the assign-ment within a minute. We find that such annota-tions mostly had default values annotated.

B.2 Same slider values

If all the beginning points and end-points in an as-signment have the same values, we automaticallyreject those assignments.

B.3 Same duration values

Sometimes we encounter cases where all durationvalues in an assignment are annotated to have thesame value. This scenario , although unlikely,could genuinely be an instance of correct anno-tation. Hence we manually check for these casesand reject only if the annotations look dubious innature based on our subjective opinion.

B.4 Inconsistency between the sliderpositions and durations

Our protocol design allows us to detect poten-tially bad annotations by detecting inconsistencybetween the slider positions (beginning and end-points) and the duration values of events in an an-notated sentence. The annotator in Figure 9 as-signs slider values for e1 (think) as [7,60] i.e. atime-span of 53 and assigns its duration as min-utes. But at the same time, the slider values for e2(do) are annotated as [50,60] i.e. a time-span of10, even though its duration is assigned as years.This is an inconsistency as e2 has a smaller time-span denoted by the sliders but has the longer du-ration as denoted by years. We reject assignmentswhere more than 60% of annotations have this in-consistency.

2919

C Inter-annotator agreement

Annotators were asked to approximate the relativeduration of the two events that they were annotat-ing using the distance between the sliders. Thismeans that an annotation is coherent insofar as theratio of distances between the slider responses foreach event matches the ratio of the categorical du-ration responses. We rejected annotations whereinthere was gross mismatch between the categori-cal responses and the slider responses — i.e. oneevent is annotated as having a longer duration butis given a shorter slider response — but becausethis does not guarantee that the exact ratios arepreserved, we assess that here using a canonicalcorrelation analysis (CCA; Hotelling 1936) be-tween the categorical duration responses and theslider responses.

Duration Relation

dur(e1)

dur(e2)

beg(e1)

end(e1)

beg(e2)

end(e2)

CC2

CC10.5

0.0

−0.5

Figure 10: Scores from canonical correlation analysiscomparing categorical duration annotations and sliderrelation annotations.

Figure 10 shows the CCA scores. We find thatthe first canonical correlation, which captures theratios between unequal events, is 0.765; and thesecond, which captures the ratios between roughlyunequal events, is 0.427. This preservation of theratios is quite impressive in light of the fact thatour slider scales are bounded; though we hopedfor at least a non-linear relationship between thecategorical durations and the slider distances, wedid not expect such a strong linear relationship.

D Confidence Ratings

Annotators use the confidence scale in differentways. Some always respond with totally confi-dent whereas others use all five options. To caterto these differences, we normalize the confidenceratings for each event-pair using a standard ordi-nal scale normalization technique known as riditscoring. In ridit scoring ordinal labels are mappedto (0, 1) using the empirical cumulative distribu-tion function of the ratings given by each annota-tor. Ridit scoring re-weights the importance of ascale label based on the frequency of its usage.

We weight both Ldur, and Lrel by the ridit-scored confidence ratings of event durations and

event relations, respectively.

E Processing TempEval3 andTimeBank-Dense

Since we require spans of predicates for ourmodel, we pre-process TB+AQ and TD by re-moving all xml tags from the sentences and thenwe pass it through Stanford CoreNLP 3.9.2

(Manning et al., 2014) to get the correspondingconllu format. Roots and spans of predicates arethen extracted using PredPatt. To train the SVMclassifier, we use sklearn 0.20.0; Pedregosaet al. 2011. We run a hyperparameter grid-searchover 4-fold CV with C: (0.1, 1, 10), and gamma:(0.001, 0.01, 0.1, 1). The best performance oncross-validation (C=1 and gamma=0.001) is thenevaluated on the test set of TE3 i.e. TE3-Platinum(TE3-PT), and TD-test. For our purposes, theidentity and simultaneous relations in TB+AQ areequivalent when comparing event-event relations.Hence, they are collapsed into one single relation.

F Further analysis

We rotate the predicted slider positions in the re-lation space defined in §3 and compare it with therotated space of actual slider positions. We see aSpearman correlation of 0.19 for PRIORITY, 0.23for CONTAINMENT, and 0.17 for EQUALITY. Thissuggests that our model is best able to captureCONTAINMENT relations and slightly less goodat capturing PRIORITY and EQUALITY relations,though all the numbers are quite low comparedto the absolute ⇢ and relative ⇢ metrics reportedin Table 2. This may be indicative of the factthat our models do somewhat poorly on predict-ing more fine-grained aspects of an event relation,and in the future it may be useful to jointly trainagainst the more interpretable PRIORITY, CON-TAINMENT, and EQUALITY measures instead ofor in conjunction with the slider values.