capturing user intent for analytical process
DESCRIPTION
Capturing User Intent for Analytical Process. Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College. Hien Nguyen Jacob Russell Adam Olson University of Wisconsin - Whitewater. Presentation at AITP meeting, February 2 nd 2010. Outline. Problem Challenges - PowerPoint PPT PresentationTRANSCRIPT
Capturing User Intent for Analytical Process
Eugene Santos Jr.John WilkinsonFei YuDeqing LiKeum Kim
Dartmouth College
Hien NguyenJacob RussellAdam Olson
University of Wisconsin - Whitewater
Presentation at AITP meeting, February 2nd 2010.
Outline
• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work
Presentation at AITP meeting, February 2nd 2010.
Outline
• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work
Presentation at AITP meeting, February 2nd 2010..
Problem
Presentation at AITP meeting, February 2nd 2010.
APEXB, Senior AnalystIran’s deskExpertise: Iran, Iraq, Middle EastNuclear weapon programs
APEXF, Junior Analyst
Iran’s desk
Expertise: assessment of Iranian
clerical support. APEX
L, Ad
visory
Analy
st
Iran’s
desk
Expe
rtise:
Iraq,
irania
n poli
tical
polic
ies
Problem
Presentation at AITP meeting, February 2nd 2010.
APEXB, Senior AnalystDoes not like to talk to anyone
APEXF, Junior Analyst
Face book, Second lifeAP
EXL,
Advis
ory An
alyst
Rush
es th
rough
every
thing
Problem
Presentation at AITP meeting, February 2nd 2010.
Modeling an analyst’s intent to improve the effectiveness of collaboration among intelligence
analysts• Capture the process instead of
topical interests• May help improve the diversity of
collaboration
Research Challenges
• Partiality and incompleteness of captured knowledge
• Dynamic information spaces, user foci, and preferences
• Uncertainty in analyst’s actions
Presentation at AITP meeting, February 2nd 2010.
Research Questions
1. What is user intent and how do we capture it
2. How do we evaluate the effectiveness of the intent model
Presentation at AITP meeting, February 2nd 2010.
Outline
• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work
Presentation at AITP meeting, February 2nd 2010.
Our Intent Model• Analyst intent = Goals + Actions + Commitment• Our user model consists of 3 components that are
designed to capture intent: – Foci: What is the working space of the analyst
and what he is concentrating on? (Goal & Commitment)
– Rationale: Why does the analyst have these foci? (Goals)
– Action: How are the analyst’s goals accomplished? (Actions)
Presentation at AITP meeting, February 2nd 2010.
Our Intent Model
Presentation at AITP meeting, February 2nd 2010.
APEXFSearch
Retain
Event database
Foci Rationale Action
Example - Rationale
Presentation at AITP meeting, February 2nd 2010.
Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?
APEXF
12/11/2007 at 11:00
Iranian_cleric_commu
nity
community
Cleric_communit
y
iranian_communit
y
Nuclear_program
Program
isa isa
isa isa
Related_to
Iranian
Related_to
Ayatollah_khamenei
isa
Related_to
Ahmadinejad
Related_to
G0
…
Example - Foci
Presentation at AITP meeting, February 2nd 2010.
Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?
APEXF
12/11/2007 at 11:00
G0iranian_cleric_community
nuclear_programayatollah_khamenei
…Commitment level: 1.0
Example - Action
Presentation at AITP meeting, February 2nd 2010.
Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?
APEXF
12/11/2007 at 11:00 Bayesian networkHMM
1
2
3.88 .91
.82
.06
.06
.01
.06
.03
.17
Examining Evidence
Searching for Evidence
Going through Documents
Example
Presentation at AITP meeting, February 2nd 2010.
Search: Is iranian clerical community supporting Iran's nuclear program?
12/11/2007 at 11:10
Search: Are there clerics who oppose Iran's nuclear program?
APEXF12/11/2007 at 11:05
Example - Rationale
Presentation at AITP meeting, February 2nd 2010.
APEXF
Iranian_cleric_communit
y
community
Cleric_community
iranian_community
Nuclear_program
Program
isa isa
isa isa
Related_to
Iranian
Related_to
Ayatollah_khamenei
isa
Related_to
Ahmadinejad
Related_to
G0 G1 G2
G3Iran_nuclear_porogram
isa
….
Example - Foci
Presentation at AITP meeting, February 2nd 2010.
APEXF
G0iranian_cleric_community
nuclear_programayatollah_khamenei
…Commitment level: 0.33
G1
clericIran_nuclear_program
Iran…
Commitment level: 0.5
G2
clericIran_nuclear_program
Iran…
Commitment level: 0.66
G3 …
Example - Action
Presentation at AITP meeting, February 2nd 2010.
APEXF
1
2
3 .91
.82
.06
.06
.01
.06
.03
.17
Examining Evidence
Searching for Evidence
Going through Documents
HMMBayesian networks
.88
Example - Intent
• Goal:– Iranian clerics and nuclear weapons
program• Actions:
– Searches• Commitment:
– F(0.33, 0.5, 0.66)
Presentation at AITP meeting, February 2nd 2010.
APEXF
Outline
• Problem• Motivation• Our User Intent Model• Preliminary evaluation• Future work
Presentation at AITP meeting, February 2nd 2010..
Evaluation
• Do we capture user intent more precisely compared to a simple interest list approach?
• Do we capture a user’s intent earlier in the analytic process compared to the interest-based approach?
Presentation at AITP meeting, February 2nd 2010.
Evaluation
• Testbed:APEX 07, 8 intelligence analysts
• Tasks: • Where does the Iranian clerical
community stand on Ayatollah Khamenei ?• President Ahmadinejad's policies with
regards to Iran’s civilian and military nuclear program?
• Database: contain 5613 events, 8 types of actions.
Presentation at AITP meeting, February 2nd 2010.
Evaluation
Presentation at AITP meeting, February 2nd 2010.
Retain
Acces
s
Search
StartA
pp
Discard
Asses
s
MakeH
ypoth
esis
Assoc
iateE
vid...
0
0.1
0.2
0.3
0.4
0.5
0.6
Probability of Each Action in APEX Dataset
APEXBAPEXCAPEXEAPEXFAPEXHAPEXKAPEXLAPEXP
Do we capture user intent more precisely?
• Analysts: APEXL- APEXC, APEXE-APEXH APEXL- APEXK, APEXF-APEXB.
• Procedure: Create a user model for each user using 25 consecutive events on December 11, 2007.
Precision =
• Cgoals: a goal node that is found in both intent models.
Presentation at AITP meeting, February 2nd 2010.
CgoalsCgoalsrelevant _
Do we capture user intent more precisely?
Presentation at AITP meeting, February 2nd 2010.
APEXBF APEXEH APEXKL APEXLC0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
InterestIntent
P-value=0.0396<0.05
Whether we capture a user’s intent earlier?
Presentation at AITP meeting, February 2nd 2010..
• Analysts: APEXF – APEXB– APEXB: nuclear weapon program and Iran.– APEXF: grand Ayatollah
Intent (t=5)
Interest (t=8)
Future work
Presentation at AITP meeting, February 2nd 2010.
• Research and implementation:– Generation of goal descriptions– Integrating between content-based and
action-based information– Link between common intent and
collaboration effectiveness• Evaluation:
– Prediction of action sequences– Define measures of collaboration
effectiveness
Acknowledgement
Presentation at AITP meeting, February 2nd 2010.
• This research is funded by CASE program (IARPA) grant #FA8750-07-2-0219 DOD-AFRL (2007-2008). Thanks to the CASE technical advisory team for making the data available, giving constructive feedback and supporting this research.
Related work
Presentation at AITP meeting, February 2nd 2010.
• Eugene Santos Jr. and Hien Nguyen. “Modeling Users for Adaptive Information Retrieval by Capturing User Intent”. Book Chapter. In M. Chevalier, C. Julien, C. Souledupuy (Eds.). Collaborative and Social Information Retrieval and Access: Techniques for Improved User Modeling. 2009.
• Hien Nguyen and Eugene Santos Jr. (a) “An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval”. In Proceedings of the 2007 International Conference on Artificial Intelligence. Las Vegas, NV. June 2007.
• Hien Nguyen and Eugene Santos Jr. (b) “Effects of prior knowledge on the effectiveness of a hybrid user model for information retrieval”. In Proceedings of the Homeland Security and Homeland Defense VI conference. Orlando, FL. March 2007. Vol. 6538.
• Hien Nguyen, Eugene Santos Jr., Aaron Schuet, and Nathan Smith. “Hybrid User Model for Information Retrieval”. In Technical Report of Modeling Others from Observations workshop at AAAI-2006 conference.
• Hien Nguyen, Eugene Santos Jr., Qunhua Zhao and Hua Wang. “Capturing User Intent for Information Retrieval”. In Proceedings of the 48th Annual Meeting for the Human Factors and Ergonomics Society (HFES-04), New Orleans, LA. 2004. Pages 371-375
Past undergraduate research project
Presentation at AITP meeting, February 2nd 2010.
• - Hybrid Approach to Word Sense Disambiguation. Jacob Russell, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. (nominated by UWW Undergraduate Research Committee to be presented at CUR on the Hill).
• Implementation of intent-based user model for business domain. Adam Olson, Nathan Lutterman, LQ McDonald III, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI.
• Web-based user model services for analytic process. 2008. Nathan Lutterman, Adam Olson, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Maryland, MD.
• Simulations of network defects. 2007. Russell Jacob, Nathan Smith, Nicholas Orcust, Hien Nguyen, Jim Harris, Benette Harris. Poster presented at Undergraduate research day, April 4, 2007 and National Conference for Undergraduate research (NCUR 2007), April 12-14. CA.
• Evaluation of Hybrid User Model for Information Retrieval. 2006. Aaron Schuett, Nathan Smith, and Hien Nguyen. Poster presented at National Conference of Undergraduate Research 2006, Raleigh, NC.
Current undergraduate research project
Presentation at AITP meeting, February 2nd 2010.
• Roulette game – Application of Modeling under Uncertainty. Richard Lilja, Hien Nguyen
• Intent model for supporting medical trauma treatment. LQ McDonald, Adam Olson, Hien Nguyen