efficient multiple-click models in web search
DESCRIPTION
Efficient Multiple-Click Models in Web Search. Fan Guo 1 , Chao Liu 2 and Yi-Min Wang 2 1 Carnegie Mellon University 2 Microsoft Research Feb 11, 2009. Overview. Web Search Activity. Search Engine Presentation. User Interaction Logs. Models for Leveraging User Feedbacks. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/1.jpg)
Fan Guo1, Chao Liu2 and Yi-Min Wang2
1 Carnegie Mellon University2 Microsoft Research
Feb 11, 2009
![Page 2: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/2.jpg)
Overview
Web Search Activity
User Interaction
Logs
Search Engine
Presentation
Models for Leveraging
User Feedbacks
![Page 3: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/3.jpg)
Overview
Web Search Activity
User Interaction
Logs
Search Engine
Presentation
Models for Leveraging
User Feedbacks
Logging
![Page 4: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/4.jpg)
User Implicit FeedbacksA very rich set of candidate features (Agichtein et al., SIGIR’06)
![Page 5: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/5.jpg)
User Implicit FeedbacksPractical consideration
Recording effort (how much tweaking?)Noise elimination (e.g., dwelling time?)Would the answer change after scaling up to
10x? 100x?
![Page 6: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/6.jpg)
Click LogsQuery term
cmuDocument URL
www.cmu.edu Position/Rank
1Clicked or Not
Yes
Query termcmu
Document URLwww.cmu.edu
Position/Rank1
Clicked or NotYes
Query termcmu
Document URLwww.cmu.edu
Position/Rank1
Clicked or NotYes
Query termcmu
Document URLwww.ri.cmu.ed
u Position/Rank
10Clicked or Not
No
![Page 7: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/7.jpg)
Click LogsQuery term
cmuDocument URL
www.cmu.edu Position/Rank
1Clicked or Not
Yes
Time stamp01/19/2009,
4:00:01pmSession ID
...User ID
...
Query termcmu
Document URLwww.cmu.edu
Position/Rank1
Clicked or NotYes
Query termcmu
Document URLwww.cmu.edu
Position/Rank1
Clicked or NotYes
Query termcmu
Document URLwww.ri.cmu.ed
u Position/Rank
M = 10Clicked or Not
No
![Page 8: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/8.jpg)
A Snapshot
![Page 9: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/9.jpg)
Overview
Web Search Activity
User Interaction
Logs
Search Engine
Presentation
Models for Leveraging
User Feedbacks
Cleanup
![Page 10: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/10.jpg)
Overview
Web Search Activity
User Interaction
Logs
Search Engine
Presentation
Models for Leveraging
User Feedbacks
Modeling
![Page 11: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/11.jpg)
Clicks are biased
Images are copied from (Joachims et al., 2007) in ACM TOIS
![Page 12: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/12.jpg)
Examination HypothesisA web document must be examined before
clicked.
Relevance is defined as the conditional probability of click upon examination.
P(Click) = P(Examination) * Relevance, formulated in (Richardson et al., WWW’07)
![Page 13: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/13.jpg)
Most Simple Click ModelVariable Definition
Ei: binary r.v. for examination on position i;
Ci: binary r.v. for click on position i;
rdi: relevance for document di on position i.
![Page 14: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/14.jpg)
Most Simple Click ModelIndependent Click Model
P(Ei = 1) = 1
P(Ci=1) = rdi
Therefore we have
![Page 15: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/15.jpg)
Cascade HypothesisProposed in (Craswell et al., WSDM’08)Strict Linear order of examination and click
Ei = 0 => Ei+1 = 0
Cascade Model
![Page 16: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/16.jpg)
Cascade ModelModeling the first click (Craswell et al., WSDM’08)
P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rdi
P(E1=1) =1, P(Ei+1=1|Ei=0) = 0
P(Ei+1=1|Ei=1, Ci=0) = 1
P(Ei+1=1|Ei=1, Ci=1) = ?
examination hypothesiscascade hypothesis
modeling a single click
![Page 17: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/17.jpg)
Dependent Click ModelModeling multiple clicks
P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rdi
P(E1=1) =1, P(Ei+1=1|Ei=0) = 0
P(Ei+1=1|Ei=1, Ci=0) = 1
P(Ei+1=1|Ei=1, Ci=1) = λi
examination hypothesiscascade hypothesis
modeling 1+ click(s)
![Page 18: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/18.jpg)
Dependent Click Modelλi (1 ≤ i < 10) are global user behavior
parameters.The full picture
![Page 19: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/19.jpg)
DCM AlgorithmsTwo sets of unknown values for estimation
Query-specific: document relevance rGlobal: user behavior parameter λ
Approximate Maximum-Likelihood LearningMaximizing a lower bound of log-likelihoodWould be exact if the chain length is infinite
![Page 20: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/20.jpg)
Estimation Formulae
empirical CTR measured before last clicked position
empirical probability of “clicked-but-not-last”
DCM Algorithms
![Page 21: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/21.jpg)
DCM ImplementationKeep 3 counts for each query-document pair
Then
![Page 22: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/22.jpg)
Overview
Web Search Activity
User Interaction
Logs
Search Engine
Presentation
Models for Leveraging
User Feedbacks
Application
![Page 23: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/23.jpg)
Click Model ApplicationsUser-Perceived Relevance
A good candidate feature for ranking, but…As input for existing ranking algorithms v.s. human judgment
![Page 24: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/24.jpg)
Click Model ApplicationsSearch Engline Evaluation
Summary of the overall user experienceExpected clickthrough rate
Browsing Behavior Statistics
Application in Sponsored SearchBuilding block for auction/pricing mechanism(Kempe and Mahdian, WINE 2008) (Aggarwal et al., WINE
2008)
(Guo et al., WSCD’09)
![Page 25: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/25.jpg)
![Page 26: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/26.jpg)
Click Data SetCollected in 2 weeks in July 2008.Discard query sessions with no clicks.178 most frequent queries removed.
![Page 27: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/27.jpg)
Click Data SetAfter preprocessing:
110,630 distinct queries4.8M/4.0M query sessions in the training/test
set
Training Time: ~7 min
![Page 28: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/28.jpg)
Evaluation CriteriaTest Data Log-likelihood
Given the document impression in the test setCompute the chance to recover the entire click
vectorAveraged over different query sessions
![Page 29: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/29.jpg)
Test Data Log-LikelihoodICM: -1.401, DCM: -1.327 (26.5% chance)
![Page 30: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/30.jpg)
Evaluation CriteriaPredicting First/Last Clicks
Given the document impression in the test setDraw 100 click vector samples (with 1+ clicks)Compute the corresponding RMS error
![Page 31: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/31.jpg)
First/Last Clicked Position
![Page 32: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/32.jpg)
Examination/Click Distribution
![Page 33: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/33.jpg)
Difference by User Goals
From (Guo et al., WSCD’09)on a different data set
![Page 34: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/34.jpg)
An AlternativeUser Browsing Model (Dupret et al., SIGIR’08)Examine probability depends on
Preceding clicked positionDistance to preceding clicked position
Comparing with DCM:Allow jump in examination (no longer strictly
linear)Parameter set is an order of magnitude largerMore expensive algorithms (iterative, EM-like)
![Page 35: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/35.jpg)
ConclusionClick Models:
A principled way of integrating user click dataAll current models are based on examination
hypothesisShould be scalable and incremental
DCM:built upon cascade model, for multiple-clicksintroduce a set of position-dependent to
characterize the examine-next probabilityprovides a fast, simple, yet effective solution
![Page 36: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/36.jpg)
Open ProblemsTrade-off in effectiveness/efficiency
Evaluation (test bed and test metric)
Click-model relevance v.s. human-judged relevance
Click model for universal search?
![Page 37: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/37.jpg)
Acknowledgement
Chao Liu Yi-Min Wang
Ethan Tu Li-Wei He Nick Craswell
![Page 38: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/38.jpg)
Acknowledgement
Christos Faloutsos Lei Li
![Page 39: Efficient Multiple-Click Models in Web Search](https://reader036.vdocuments.net/reader036/viewer/2022070404/56813b21550346895da3dae5/html5/thumbnails/39.jpg)
Thank you!