interactively optimizing information retrieval systems as a dueling bandits problem icml 2009 yisong...
TRANSCRIPT
![Page 1: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/1.jpg)
Interactively Optimizing Information Retrieval Systems as
a Dueling Bandits Problem
ICML 2009
Yisong Yue Thorsten Joachims
Cornell University
![Page 2: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/2.jpg)
Learning To Rank
• Supervised Learning Problem– Extension of classification/regression– Relatively well understood– High applicability in Information Retrieval
• Requires explicitly labeled data– Expensive to obtain– Expert judged labels == search user utility?– Doesn’t generalize to other search domains.
![Page 3: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/3.jpg)
Our Contribution
• Learn from implicit feedback (users’ clicks)– Reduce labeling cost– More representative of end user information needs
• Learn using pairwise comparisons– Humans are more adept at making pairwise judgments– Via Interleaving [Radlinski et al., 2008]
• On-line framework (Dueling Bandits Problem)– We leverage users when exploring new retrieval functions– Exploration vs exploitation tradeoff (regret)
![Page 4: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/4.jpg)
Team-Game Interleaving
1. Kernel Machines http://svm.first.gmd.de/
2. Support Vector Machinehttp://jbolivar.freeservers.com/
3. An Introduction to Support Vector Machineshttp://www.support-vector.net/
4. Archives of SUPPORT-VECTOR-MACHINES ...http://www.jiscmail.ac.uk/lists/SUPPORT...
5. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/
1. Kernel Machines http://svm.first.gmd.de/
2. SVM-Light Support Vector Machine http://ais.gmd.de/~thorsten/svm light/
3. Support Vector Machine and Kernel ... Referenceshttp://svm.research.bell-labs.com/SVMrefs.html
4. Lucent Technologies: SVM demo applet http://svm.research.bell-labs.com/SVT/SVMsvt.html
5. Royal Holloway Support Vector Machine http://svm.dcs.rhbnc.ac.uk
1. Kernel Machines T2http://svm.first.gmd.de/
2. Support Vector Machine T1http://jbolivar.freeservers.com/
3. SVM-Light Support Vector Machine T2http://ais.gmd.de/~thorsten/svm light/
4. An Introduction to Support Vector Machines T1http://www.support-vector.net/
5. Support Vector Machine and Kernel ... References T2http://svm.research.bell-labs.com/SVMrefs.html
6. Archives of SUPPORT-VECTOR-MACHINES ... T1http://www.jiscmail.ac.uk/lists/SUPPORT...
7. Lucent Technologies: SVM demo applet T2http://svm.research.bell-labs.com/SVT/SVMsvt.html
f1(u,q) r1 f2(u,q) r2
Interleaving(r1,r2)
(u=thorsten, q=“svm”)
Interpretation: (r2 Â r1) ↔ clicks(T2) > clicks(T1)
Invariant: For all k, in expectation same number of team members in top k from each team.
NEXTPICK
[Radlinski, Kurup, Joachims; CIKM 2008]
![Page 5: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/5.jpg)
Dueling Bandits Problem
• Continuous space bandits F – E.g., parameter space of retrieval functions (i.e., weight vectors)
• Each time step compares two bandits– E.g., interleaving test on two retrieval functions– Comparison is noisy & independent
![Page 6: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/6.jpg)
Dueling Bandits Problem
• Continuous space bandits F – E.g., parameter space of retrieval functions (i.e., weight vectors)
• Each time step compares two bandits– E.g., interleaving test on two retrieval functions– Comparison is noisy & independent
• Choose pair (ft, ft’) to minimize regret:
• (% users who prefer best bandit over chosen ones)
T
tttT ffPffP
1
1)'*()*(
![Page 7: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/7.jpg)
T
tttT ffPffP
1
1)'*()*(
•Example 1•P(f* > f) = 0.9•P(f* > f’) = 0.8•Incurred Regret = 0.7
•Example 2 •P(f* > f) = 0.7•P(f* > f’) = 0.6•Incurred Regret = 0.3
•Example 3•P(f* > f) = 0.51•P(f* > f) = 0.55•Incurred Regret = 0.06
![Page 8: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/8.jpg)
Modeling Assumptions
• Each bandit f 2F has intrinsic value v(f)– Never observed directly– Assume v(f) is strictly concave ( unique f* )
• Comparisons based on v(f)– P(f > f’) = σ( v(f) – v(f’) )– P is L-Lipschitz
– For example: )exp(1
1)(
xx
![Page 9: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/9.jpg)
Probability Functions
![Page 10: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/10.jpg)
Dueling Bandit Gradient Descent
• Maintain ft
– Compare with ft’ (close to ft -- defined by step size)
– Update if ft’ wins comparison
• Expectation of update close to gradient of P(ft > f’)– Builds on Bandit Gradient Descent [Flaxman et al., 2005]
![Page 11: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/11.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 12: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/12.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 13: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/13.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 14: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/14.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 15: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/15.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 16: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/16.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 17: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/17.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 18: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/18.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 19: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/19.jpg)
δ – explore step size γ – exploit step sizeCurrent pointLosing candidateWinning candidate
Dueling Bandit Gradient Descent
![Page 20: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/20.jpg)
Analysis (Sketch)
• Dueling Bandit Gradient Descent– Sequence of partially convex functions ct(f) = P(ft > f)
– Random binary updates (expectation close to gradient)
• Bandit Gradient Descent [Flaxman et al., SODA 2005]
– Sequence of convex functions – Use randomized update
(expectation close to gradient)
– Can be extended to our setting
(Assumes more information)
![Page 21: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/21.jpg)
Analysis (Sketch)
• Convex functions satisfy
– Both additive and multiplicative error– Depends on exploration step size δ – Main analytical contribution: bounding multiplicative error
*)()(*)()( xxxcxcxc
![Page 22: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/22.jpg)
Regret Bound
• Regret grows as O(T3/4):
• Average regret shrinks as O(T-1/4)– In the limit, we do as well as knowing f* in
hindsight
T
tttT ffPffP
1
1)'*()*(
RdLTT 102E 4/3
δ = O(1/T-1/4 )γ = O(1/T-1/2 )
![Page 23: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/23.jpg)
Practical Considerations
• Need to set step size parameters– Depends on P(f > f’)
• Cannot be set optimally– We don’t know the specifics of P(f > f’)– Algorithm should be robust to parameter settings
• Set parameters approximately in experiments
![Page 24: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/24.jpg)
00.10.20.30.40.50.60.70.80.9
1
10 570
1130
1690
2250
2810
3370
3930
4490
5050
5610
6170
6730
7290
7850
8410
8970
9530
Aver
age
Regr
et
Regret Comparison DBGD vs BGD
DBGD
BGD 1
BGD 2
• 50 dimensional parameter space• Value function v(x) = -xTx• Logistic transfer function• Random point has regret almost 1
More experiments in paper.
![Page 25: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/25.jpg)
Web Search Simulation
• Leverage web search dataset– 1000 Training Queries, 367 Dimensions
• Simulate “users” issuing queries– Value function based on NDCG@10 (ranking measure)– Use logistic to make probabilistic comparisons
• Use linear ranking function.
• Not intended to compete with supervised learning– Feasibility check for online learning w/ users– Supervised labels difficult to acquire “in the wild”
![Page 26: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/26.jpg)
• Chose parameters with best final performance• Curves basically identical for validation and test sets (no over-fitting)• Sampling multiple queries makes no difference
0.480.5
0.520.540.560.58
0.60.62
063
0000
1260
000
1890
000
2520
000
3150
000
3780
000
4410
000
5040
000
5670
000
6300
000
6930
000
7560
000
8190
000
8820
000
9450
000
Trai
ning
NDC
G @
10Web Simulation Results
Sample 1
Sample 10
Sample 100
Ranking SVM
![Page 27: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/27.jpg)
What Next?
• Better simulation environments– More realistic user modeling assumptions
• DBGD simple and extensible – Incorporate pairwise document preferences– Deal with ranking discontinuities
• Test on real search systems– Varying scales of user communities– Sheds on insight / guides future development
![Page 28: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/28.jpg)
Extra Slides
![Page 29: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/29.jpg)
Active vs Passive Learning
• Passive Data Collection (offline)– Biased by current retrieval function
• Point-wise Evaluation– Design retrieval function offline– Evaluate online
• Active Learning (online)– Automatically propose new rankings to evaluate– Our approach
![Page 30: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/30.jpg)
Relative vs Absolute Metrics
• Our framework based on relative metrics– E.g., comparing pairs of results or rankings– Relatively recent development
• Absolute Metrics– E.g., absolute click-through rate– More common in literature – Suffers from presentation bias– Less robust to the many different sources of noise
![Page 31: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/31.jpg)
What Results do Users View/Click?
[Joachims et al., TOIS 2007]
![Page 32: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/32.jpg)
![Page 33: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/33.jpg)
Analysis (Sketch)
• Convex functions satisfy
– We have both multiplicative and additive error– Depends on exploration step size δ – Main technical contribution: bounding multiplicative error
*)()(*)()( xxxcxcxc
T
tttt ffPffP
1
*)()(E
Existing results yields sub-linear bounds on:
![Page 34: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/34.jpg)
Analysis (Sketch)
• We know how to bound
• Regret:
• We can show using Lipschitz and symmetry of σ:
T
tttT ffPffP
1
1)'*()*(
LTffPffPT
ttttT
1
*)()(E2E
T
tttt ffPffP
1
*)()(E
![Page 35: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/35.jpg)
More Simulation Experiments
• Logistic transfer function σ(x) = 1/(1+exp(-x))• 4 choices of value functions
• δ, γ set approximately
![Page 36: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/36.jpg)
TR
![Page 37: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/37.jpg)
NDCG• Normalized Discounted Cumulative Gain• Multiple Levels of Relevance
• DCG:– contribution of ith rank position:
– Ex: has DCG score of
• NDCG is normalized DCG – best possible ranking as score NDCG = 1
)1log(
12
i
iy
45.5)6log(
1
)5log(
0
)4log(
1
)3log(
3
)2log(
1
![Page 38: Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem ICML 2009 Yisong Yue Thorsten Joachims Cornell University](https://reader035.vdocuments.net/reader035/viewer/2022081414/55146661550346414e8b5b96/html5/thumbnails/38.jpg)
Considerations
• NDCG is discontinuous w.r.t. function parameters– Try larger values of δ, γ– Try sampling multiple queries per update
• Homogenous user values– NDCG@10– Not an optimization concern– Modeling limitation
• Not intended to compete with supervised learning– Sanity check of feasibility for online learning w/ users