Jen-Tzung Chien, Meng-Sung Wu

Minimum Rank Error Language Modeling

• Introduction• Language model for information retrieval• Minimum rank error model• Experiments• Conclusion

Outline

• the language model is useful for investigating the linguistic regularities in queries and documents for information retrieval

• But the accuracy of classifying queries into the relevant documents is not concerned with the ranks of the retrieved documents

• MCE training is also used in IR. In the MCE procedure, the expected loss function is minimized with probabilistic descent algorithm for optimal Bayes risk

Introduction

• With MCE, the rate of misclassification is reduced. But rank result is still not consist with the performance measure, i.e. AP

• The minimum rank error (MRE) language model is established by a gradient descent algorithm to obtain discriminative retrieval for training queries with minimum expected rank error loss

Introduction

• the ranking is calculated by the likelihood function using the -gram language model

• Given a text document , the set of ML n-gram parameters

Language model for information retrieval

• the document terms are often too few to train reliable ML model. Many words are unseen in the document, leading to zero probabilities in many n-gram events

• the smoothed language model is obtained by linear interpolation of the document and background models

Language model for information retrieval

• MCE is a training method based on Bayes decision theory. This method can reduce misclassification by minimize the expect loss with three step procedure

• First, a misclassification measure is defined

• Second, the misclassification measure is normalized as the classification error loss function ranging between 0 and 1 by the sigmoid function given as follows

Minimum Classification Error Model

• The third step is to measure the classification performance by calculating the expected loss due to the observed queries and document models

• through the descent algorithm, the parameter set can be update with iterative procedure and learning rate

Minimum Classification Error Model

• Receiver Operating Characteristic (ROC) is one kind of measure which consider the true positive rate and false positive rate; Area Under ROC Curve(AUC) gives a value for the ROC curve

Information Retrieval Measures

• Average Precision (AP) Versus Rank Error

• Minimum Rank Error (MRE) Model

• Implementation and Interpretation

Minimum rank error model

• The information retrieval model can be estimated by optimizing the AP, but the minimization of the expected AP loss function is mathematically intractable

• So we develop the rank error loss function instead of the classification error loss

Average Precision (AP) Versus Rank Error

• The MRE training procedure assures the model discriminability in sense of minimizing the ambiguity in the ranking problem

• Like in MCE training, three step procedure is performed to estimate the MRE language model

• First we define the misranking measure for a relevant document

Minimum Rank Error (MRE) Model

• The rank error loss function is calculated by substituting the misranking measure into sigmoid function. And the expect rank error is calculated over the entire training set including all query and their relevant documents

Minimum Rank Error (MRE) Model

• The document model is iteratively updated by the descent algorithm

• Considering a logarithm bigram in document model, the differentials are calculated by

Minimum Rank Error (MRE) Model

• The figure below shows the procedure of MRE language model training for information retrieval

Implementation and Interpretation

• MRE and MCE are derived as the discriminative learning algorithms from the same Bayes decision theory, but they are different by two aspects

• In performance metrics– MRE minimizes the Bayes rank risk based on the

rank error loss function– MCE minimizes the Bayes risk due to classification

errors

• In use of training data– The MRE model uses queries and their

corresponding document lists as training samples– MCE considers all irrelevant documents in a rank list

Implementation and Interpretation

Experiments

Experiments

• Most classification systems are based on minimization of classification errors, and thus do not reflect the ranking performance of retrieval systems

• This paper focuses on the ranking problem, and presents a new discriminative retrieval model. The experiment results also shows MRE retrieves more relevant documents with high ranks than MCE

Conclusion