Learning Conference Reviewer Assignments

Adith Swaminathan

Guide :

Prof. Soumen Chakrabarti

Department of Computer Science andEngineering,Indian Institute of Technology, Bombay

Future Work (from BTP1)

Given WWW2010’s assignments, learn Affinity_Param, Topic_Param and Irritation

Citations as edge featuresLoad-Constrained Partial AssignmentsBetter estimation of Assignment Quality

Background

Conference Reviewer-Paper Assignment as a Many-Many-matching [1]

Minimum Cost Network Flow (MCF)

Conference Reviewer Assignment

Set of Reviewers, R, max #papers = L_i Set of Papers, P, min #reviews = K

Assumption : Only require #reviews, not quality

Suppose we have cost function A_ij(y) for <R_i, P_j>

ILP -> Assumption -> MCF

Two problems

Integer Linear Programs are NP-Hard!– Relax?– More assumptions?

How to determine A_ij?– M * N ~ 10000– Multimodal clues

ILP -> Assumption -> MCF

Enforce structure on A_ij– Better model multimodality– Fewer parameters to fix

“Learn” A_ij using Structured Learning Techniques

A_ij = wT Φ(R_i, P_j, y_ij)

Ramifications of Structured Costs

Costs decompose over <R_i, P_j> pairs– Decomposable Preference Auction– Polynomial Algorithms for DPAs [2]

Restricted notion of optimality– Per-reviewer/Per-paper constraint could be

combinatorial– Stability?

ILP -> Assumption -> MCF

Minimum Cost Network Flow

Directed graph G=(V,E), capacities u(E)>= 0, costs c(E)

Nodes have numbers b(V) : Sum(b(V)) = 0

Task : Find a function f: E->R+ which satisfies the b-flow at minimum cost

Successive Shortest Path Algorithm

Node features and Edge features

Profile

Topics

Reviewer

Topics

Contents

Paper

Affinity

Bid

Topic Overlap

Cites

The Loss Function

L_ij = w_1 * exp(-Affinity_ij) + w_2 * [[1 – Topic_Overlap_ij]] + w_3 * Bid_Cost

Bid_Cost = Potential(R_i, P_j, y_ij)

Irritation (I) and Disappointment (D) needs to be set

Assignment Quality Measures

Number of Bids Violated?– Not a reliable measure.

+ve Bids Violated–ve Bids ViolatedAssignments satisfying Topic MatchConfidence?

Confidence == Quality?

Very sparse– Fewer than 5% observed– Extrapolated Confidence?

Reliable– Bids as a precursor of Confidence [3]

– Confidence-Augmented Loss?

Learning w’s

Transductive Ordinal Regression– Assume : Assignments are independent (Naïve)– Heuristic : Augment observed dataset– Extrapolate observed Confidence [4]

– Learn w over extrapolated dataset

Support Vector Machine for Structured Outputs– Cast as soft-margin SVM formulation [5]

– Upper-bound objective with a convex fn (Optimality?)– Minimize, using Cutting Plane (Approximate)

Transductive Ordinal Regression [6]

SVM Struct. [7]

Loss Augmented Inference ~ Most Violated ConstraintLoss is decomposable -> Modified MCF

PA

RA

: P

aper

Ass

ign

men

t to

R

evie

wer

s A

pp

arat

us

Results

Bimodal Behaviour

•Reviewer either gets few or L_i papers•Load Penalties [8]

•Introduce more parameters•Infer using modified MCF•Learning parameters?

•Load Rebalancing•Tradeoff between MCF optimum and old assignment

Penalise Reviewer Loads

Load Constrained Assignments

Avenues for Future Work

•Document Modelling for Affinity Scores

•Objective Assignment Evaluation

•Transitive Citation Scores

•Load Penalty Parameter Estimation

References

1. The Conference Paper Assignment Problem, J. Goldsmith, R.H. Sloan, 2007

2. MultiAgent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Y. Shoham, K. Leyton-Brown, 2009

3. Automating the Assignment of Submitted Manuscripts to Reviewers, S.T. Dumais, J. Nielson, 1992

4. Semisupervised Regression with cotraining algorithms, Z. Zhou, M. Li, 2007

References – contd.

5. Learning structured prediction models : A Large Margin Approach, B. Taskar, et al, 2005

6. Ologit : Ordinal Logistic Regression for Zelig, G. King, et al, 2007

7. SVM Learning for Interdependant and Structured Output Spaces, I. Tsochantaridis, et al, 2004

8. Word Alignment via Quadratic Assignment, S. Lacoste-Julien, et al, 2006