cmu team-a in tdt 2004 topic tracking

Carnegie MellonSchool of Computer ScienceLanguage Technologies Institute

CMU Team-1 in TDT 2004 Workshop 1

CMU TEAM-A in TDT 2004Topic Tracking

Yiming YangSchool of Computer ScienceCarnegie Mellon University



CMU Team A

– Jaime Carbonell (PI)

– Yiming Yang (Co-PI)

– Ralf Brown

– Jian Zhang

– Nianli Ma

– Shinjae Yoo

– Bryan Kisiel, Monica Rogati, Yi Chang



Participated Tasks in TDT 2004

Topic Tracking (Nianli Ma et al.)

Supervised Adaptive Tracking (Yiming Yang et al.)

New Event Detection (Jian Zhang et al.)

Link Detection (Ralf Brown)

Hierarchical Topic Detection – not participated



Topic Tracking with Supervised

Adaptation(“Adaptive Filtering” in TREC)

On-topic

Test documents

Current document

Training documents (past)time

Off-topic

Unlabeled documents

Topic 1

Topic 2

Topic 3…

RelevanceFeedback



Topic Tracking with Pseudo-Relevance(“Topic Tracking” in TDT)

On-topic?

Test documents

Current document

Training documents (past)time

Off-topic

Unlabeled documents

Topic 1

Topic 2

Topic 3…

Pseudo-RelevanceFeedback (PRF)



Adaptive Rocchio with PRF

• Conventional version

• Improved version

)()( ||

1

|1|

1)1(

tDdj

tDdnewi

ji

dD

ddD

tp

judgment system

judgmenthuman

12/)),(cos(

1eight

newdtpw

)()( ||

1

|1|

1)1(

tDdj

tDdnewi

ji

dD

dweightdD

tp



Rocchio in Tracking on TDT 2003 Data

Without PRF 0.1437With weighted PRF0.1266

0.1437

0.1266

0.1

0.11

0.12

0.13

0.14

0.15

Without PRF With weighted PRF

Ctrk

Weighted PRF reduced Ctrk by 12%.

Ctrk: the cost of tracking, i.e., the harmonic average of miss rate and false alarm rate



Primary Tracking Results in TDT 2004

0

0.2

0.4

0.6

0.8

1

Ctrk

CMU_A

Umass

ICT

UMD

NEU



DET Curves of Methods on TDT 2004 Data

Charles’ target



Supervised Adaptive Tracking

• “Adaptive filtering” in TREC (since 1997)– Rocchio with threshold calibration strategies (Yang

et al., CIKM 2003)

– Probabilistic models assuming Gaussian/exponential distributions (Arampatzis et al, TREC 2001)

– Combined use of Rocchio and Logistic regression (Yi Zhang, SIGIR 2004)

• A new task in TDT 2004– Topics are narrower, and typically short lasting than

the TREC topics



Our Experiments• 4 methods

– Rocchio with a fixed threshold (Roc.fix)– Rocchio with an adaptive threshold using Margin-based Local

Regression (Roc.MLR)– Nearest Neighbor (Ralf’s variant) with a fixed threshold

(kNN.fix)– Logistic regression (LR) regularized by a complexity penalty

• 3 corpora– TDT5 corpus, as the evaluation set in TDT 2004– TDT4 corpus, as a validation set for parameter tuning– TREC11 ( 2002) corpus, as reference set for robustness

analysis

• 2 optimization criteria– Ctrk: TDT standard, equivalent to setting the penalty ratio for

miss vs. false alarm to 1270: 1 (approximately)– T11SU: TREC standard, equivalent to the penalty ratio of 2:1



Outline of Our Methods• Roc.fix and NN.fix

– Non-probabilistic model, generating ad hoc scores for documents with respect to each topic

– Fixed global threshold, tuned on a retrospective corpus

• Roc.MLR– Non-probabilistic model, ad hoc scores– Threshold locally optimized using incomplete

relevance judgments for a sliding window of documents

• LR– Probabilistic modeling of Pr(topic | x) – Fixed global threshold that optimizes the utility



Regularized Logistic Regression• The objective is defined as to find the optimal regression

coefficients

• This is equivalent to Maximum A Posteriori (MAP) estimation with prior distribution

• It predicts the probability of a topic given the data

F

jjj

N

ii

Tii

wwxwyysw

1

2

1

* )()exp(1log)(minarg

)2

1,(~

w

))exp(1/(1),|1( **i

Ti xwywxyp



Roc.fix on TDT3 Corpus

RF on 1.6% of documents, 25% Min-cost reduction

Base: No RF or PRFPRF: Weighted PRFMLR: Partial RFFRF: Complete RF



Effect of SA vs. PRF: on TDT5 Corpus

0.0324

0.07220.0707

0.1378

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Roc.fix NN.fix

Ctrkwith SA

with PRF

With Rocchio.fix: SA reduced Ctrk by 54% compared to PRF;With Nearest Neighbors: SA reduced Ctrk by 48%.



SATracking Results on TDT5 Corpus

0.7328

0

0.2

0.4

0.6

0.8

CMU_A

CMU_B

Umass

UMD0.0324

0

0.1

0.2

0.3

CMU_A

Umass

CMU_B

UMD

For each team, the best score (with respect to Ctrk or T11SU) of the submitted runs is presented.

Ctrk (the lower the better)

T11SU (the higher the better)



Relative Performance of Our Methods

0.0324

0.6917

0.1394

0.7328

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

opt:Ctrk opt:T11SU

Roc.fix

Roc.mlr

LR

TREC Utility (T11SU): Penalty of miss vs. f/a = 2:1TDT Cost (Ctrk): Penalty of miss vs. f/a ~= 1270:1



Main Observations

• Encouraging results: a small amount of relevance feedback (on 1~2% documents) yielded significant performance improvement

• Puzzling point: Rocchio without any threshold calibration, works surprisingly well in both Ctrk and T11SU, which is inconsistent to our observations on TREC data. Why?

• Scaling issue: a significant challenge for the learning algorithms including LR and MLR in the TDT domain.



Temporal Nature of Topics/Events

0

0.1

0.2

0.3

0.4

0.5

0.6

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

P(t

op

ic|w

ee

k)

TDT2001

TREC10

BN

TREC Topic: ElectionsTREC Topic: Elections

TDT Event: Nov. APEC MeetingTDT Event: Nov. APEC Meeting

Broadcast News Topic: Broadcast News Topic: KidnappingsKidnappings



Topics for Future Research

• Keep up with new algorithms/theories

• Exploit domain knowledge, e.g., predefined topics (and super topics) in a hierarchical setting

• Investigate topic-conditioned event tracking with predictive features (including Named Entities)

• Develop algorithms to detect and exploit temporal trends

• TDT in cross-lingual settings



References

Y. Yang and B. Kisiel. Margin-based Local Regression for Adaptive Filtering. ACM CIKM 2003 (Conference on Information and Knowledge Management).

J. Zhang and Y. Yang. Robustness of regularized linear classification methods in text categorization ACM SIGIR 2003, pp 190-197.

J. Zhang, R. Jin, Y. Yang and A. Hauptmann. Modified logistic regression: an approximation to SVM and its application in large-scale text categorization . ICML 2003 (International Conference on Machine Learning), pp888-897.

N. Ma, Y. Yang & M. Rogati. Cross-Language Event Tracking. Asia Information Retrieval Symposium (AIRS), 2004.

cmu team-a in tdt 2004 topic tracking

Documents