understanding the impact of the role factor in collaborative information retrieval

Understanding the Impact of the Role Factor in CIRLynda Tamine and Laure Soulier

1

Collaborative search in context

333

Collaborative Search

Complex, exploratory or fact-finding tasksBibliographic, medical, e-Discovery, academic search…

What?

Requirement or setup need Shared interestsInsufficient knowledge Division of labor

Why?

Groups vs. Communities

Who?

Synchronous vs. Asynchronous

When?

Colocated vs. Remote

Where?

Crowd-sourcing User mediationImplicit vs. Explicit intent System mediation

How?


Collaborative Search Models

Collaborative Search | Objectives | Design | Results | Implications | Conclusion

Users' activities in social search [Evans and Chi, 2010]

AFTE

RBE

FORE

DU

RIN

G

● Refining the information need● Structuring the task guidelines

● Sense-making process : Querying Reading Extracting Exchanging...

Organizing/Distributing search outcomesAssessing collective relevance

Collaborative search practices [Morris, 2008]

Communication channels [Gonzalez-Ibanez et al., 2013]

Other behavioral models [Hyldegärd, 2009 ; Karunakaran et al., 2013 ; Shah, 2012]

5

Collaborative Search Paradigms


Sharing of knowledge[Foley and Smeaton, 2009]

Division of labor[Kelly and Payne, 2013]

Awareness[Dourish and Bellotti, 1992]

Role-based division of labor

Document-based division of labor

Communication and shared workspace

Ranking-based on relevance judgments

Collaborator's actions

Spatio-temporal context

Shared information

need Doc2 Yes, good

Doc2 Yes, good

6

Role-based Collaborative Search


DescriptionNaturalness Collaboration mediation Flexibility level

Exchange Interfaces Algorithmic approaches Role Algorithmic

User-drivencollaboration

Fixed roles [Imazu et al., 2011] = ++ + -- - - Freely-negociated roles [Morris et al., 2008] ++ ++ + -- + -

System-mediated collaboration

Fixed roles leveraged by algorithmic mediation [Pickens et al., 2008, Shah et al., 2010]

- = + ++ - +

User-driven system-mediated collaboration

Fixed roles : - freely negotiated- dynamically mined- leveraged by algorithmic mediation[Soulier et al., 2014]

+ = + ++ + +

Role taxonomy [Golovchinsky et al., 2009]

● Prospector/Miner● Gatherer/Surveyor● Domain A expert/Domain B expert● Search expert/Search novice

→ Analyzing the impact of the role factor in collaborative search

7

Study Objectives

8

Research Questions and Hypothesis

Understanding differences in users' behaviors in role-oriented and non-role-oriented collaborative search sessions:

● RQ1 : How do user search behaviors differ ?

● RQ2 : If any, does the coordination between collaborators differ ?

● RQ3 : Do users with assigned roles respect the guidelines? If no, why ?

● RQ4 : Which effect on the search effectiveness?

Research Hypothesis:• Behaviors of collaborators might be complementary w.r.t. division of labor policies

[Spence et al., 1995]

• Differences in collaborators' behaviors represent complementary signals [Soulier et al., 2014]


Study Design

10

Participants

75 pairs of users (undergrad and Ph.D. students between 18 and 30)● Already know each other● Already worked on collaborative project together● Experience in browsing the web and using search engines

Compensation● Material compensation ($20)● Additional gift ($50) for three most effective groups


System

Open-source Coagmento collaborative search system


Tasks

Exploratory search taskThe mayor of your countryside village must choose between building a huge industrial complex or developing a nature reserve for animal conservation. As forest preservationists, you must raise awareness about the possibility of wildlife extinction surrounding such an industrial complex. Yet, before warning all citizens, including the mayor, you must do extensive research and collect all the facts about the matter. Your objective is to create a claim report together, outlining all the possible outcomes for wildlife should the industrial complex be built. Your focus is on wildlife extinction. You must investigate the animal species involved, the efforts done by other countries and the association worldwide to protect them and the reasons we, as humans, must protect our environment in order to survive. You must identify all relevant documents, facts, and pieces of information by using bookmarks, annotations, or saving snippets. If one document discusses several pieces of useful information, you must save each piece separately using snippets. Please assume that this research task is preliminary to your writing, enabling you to provide all relevant information to support your claims in your report.

3 settings associated to 3 TREC topics:● W/oRole (any role guidelines) – topic 408i (tropical storm)

● PM (prospector/miner) – topic 392i (robotics)

● GS (gatherer/surveyor) – topic 347i (wildfire extinction)


User Study Workflow and Data

Step 1 : Sign-up questionnaire ● Users' characteristics● Users' habits in search browsing (search level and role assignment)● Users' habits in collaborative work

Step 2 : The training step ● System tutorial● Short time-period to test the system

Step 3 : The 30-minute search task ● Real task in which search logs where collected

Step 4 : Post-task questionnaire ● Opinion about CIS● Difficulty level of the task


npq Average number of visited pages by query

dt Average time spent between two visited pages

nf Average number of relevance feedback (snippets, annotations & bookmarks)

qn Average number of submitted queries

ql Average number of query tokens

qo Average number of shared tokens among successive queries

nbm Average number of exchanged messages

14

Results

15

Users’ Behavioral Differences


Modeling collaborators’ search behaviors [Soulier et al., 2014]

Bookmarked document

…

…

Annotated document

Annotated document

q2

q1

q3

q4

Snipped document

Shared information

need

● Avoiding noisy search actions

● Behaviors change

Search feature-based representation Temporal-based representation

Su1(t) (wu1, f1

(t) ,...,wu1, fn

(t) )

Su2(t) (wu2 , f1

(t) ,...,wu2 , fn

(t) ) Su1(t)

wu1, f1

(1) ... wu1, fn

(1)

... ... ...

wu1, f1

(t) ... wu1, fn

(t)

Su2(t)

wu2 , f1

(1) ... wu2 , fn

(1)

... ... ...

wu2 , f1

(t) ... wu2 , fn

(t)

16



Users with assigned roles significantly behave differently

Differences between W/Role and W/oRole settings● W/oRole spent longer time on pages and submit more queries

● W/Role annotated/bookmarked/snipped more

Query overlap explained by the role guidelines

Communication seems to be similarly distributed among scenarios

Objective: Analyzing the impact of the role factor on users' behavior

Slow and exhaustive evaluation style common among less (skill-based) experienced users

17



Mostly coordinative and content messages

● Coordination more important for W/oRole

● Content more important for W/Role●

Users in PM scenario need to coordinate more than those in GS one

Objective: Analyzing the impact of the role factor on communication channels

- Bridging the gap between individual and collaborative perception of the task- Role negotiation for W/oRole

Some roles might lead to ambiguity

18

Users’ Division of Labor Strategies


f1

f2

f3

f4

f1

f2

Δf3

f4

*Δf1

Δf2

*Δf3

*Δf4

Difference significance test (Kolmogorov-Smirnov)

Δf1

f3

Δf4

Δf3

Δf1

Δf4

1 0.3 -0.5

0.3 1 -0.8

-0.5 -0.8 1

✔ Step 1: Identifying search behavior differences ✔ Step 2: Characterizing users’ roles

Correlations on search behavior differences for:- Highlighting search skill oppositions- Identifying in which each collaborator is the most

effective

Objective: Analyzing the impact of the role factor on users' search strategies

Investigating and contrasting complementarities of intra-group users' behavior [Soulier et al., 2014]

● Differences between users' behaviors

● Negative correlations between users' differences

19

Users' division of labor strategies


Prospector-Miner Gatherer-Surveyor Without Role

Objective: Analyzing the impact of the role factor on users' search strategies

Higher number of significant correlated pairwise features in W/oRole

Exhibiting significant differences and reach convergence● PM: 2 of the 4 pairwise features varied between negative/null correlations

● GS: after 20-25 min

● W/oRole : after 15 min

Roles seem weakened the interdependence between users:- no coordination stability/role drift for PM- encounter cohesion for GS

Higer flexibility for W/oRole

20

Users' division of labor strategies


Task generally perceived as difficult

A trend for easy in the W/oRole scenario, not assessed by the Chi-Square test

Task difficulty mainly related to page content, organization, and role ● Users in W/oRole feel that organization is more difficult

● Role guidelines might be difficult to follow

Objective: Analyzing the impact of the role factor on search task perception

21

Search Effectiveness


Formula [Shah and Gonzalez-Ibanez, 2011]

Precision

Recall

F-measure

Objective: Analyzing the impact of the role factor on the search task effectiveness

P(g)=RelevantCoverage(g)

Coverage (g)

R(g)=RelevantCoverage (g)

GroundTruth

F (g)=2⋅P(g)⋅R(g)

P(g)⋅R (g)

Effectiveness higher for W/oRole only for the precision measure● More likely to spend more time on pages

● Less involved in making assessments

● Less communicative about the document content

But, users in both settings able to identify relevant pages

More successful in discarding irrelevant pages

22

Design Implications

23

Mining Latent Evolving Roles

Findings● Users in W/Role settings found difficult to follow their role guidelines● Users in W/oRole setting were able to coordinated even though coordination relied

on intensive coordination

Opened perspectives● Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :

Mining users' unalbelled and latent roles in a « just in time fashion » Reinjecting these unlabelled and latent roles in CIR algorithms


Providing Task-based Role Template


Findings● Task coordination messages are important for both settings

Opened perspectives● Designing content-based role template adapted to the search task type

Enhancing functional roles (PM, GS, ...) Reducing costs of organizing division of labor search strategies

Detecting collaborators' search intent and knowledge differences Refining users' browsing behaviors by distributing information nuggets among

collaborators Guiding collaborators in a « step-by-step »-oriented search according to their knowledge

25

Mining Latent Evolving Roles

Findings● Users in W/Role settings found difficult to follow their role guidelines● Users in W/oRole setting were able to coordinated even though coordination relied

on intensive coordination

Opened perspectives● Enhanced the user-driven system-mediated approach [Soulier et al., 2014] by :

Mining users' unalbelled and latent roles in a « just in time fashion » Reinjecting these unlabelled and latent roles in CIR algorithms


Enhancing Role-based Awareness


Findings● Users in W/Role settings found difficult to follow their role guidelines● Roles seems to reduce coordination-based communication costs

+ Fixed roles might be required for some tasks (e-Discovery, medical, ...)

Implications● Improving awareness w.r.t. role drift

Identifying indicators associated to each pair of roles Designing interfaces detecting rôle drift and monitoring users' support in order to better

follow role specifications

27

Conclusion

28

Conclusion

User-study of collaborative and exploratory search tasks constrained (or not) by role guidelines

– Users' search behaviors

– Users' division of labor strategies

– Search effectiveness

Quantitative and qualitative analysis highlighting : – Users without prior roles adopt a slow and exhaustive search

– Users without prio roles were more likely to broaden their complementarities and structure their role relatively early

– Roles seems to limit the precision of the search results

Discussion about promising perspective in CIR/CIS

Limitations of the work :

– Collaboration in dyads

– Exploratory search tasks


30

See you on Friday!

Leif Azzopardi Jeremy Pickens Tetsuya Sakai Laure Soulier Lynda Tamine

Thank you for your attention

Do you have a question?

31

understanding the impact of the role factor in collaborative information retrieval

Technology