investigation of various factorization methods for large recommender systems g. takács, i....

Investigation of Various Factorization Methods for Large Recommender

Systems

G. Takács, I. Pilászy, B. Németh and D. Tikk

www.gravityrd.com

10th International Workshop on High Performance Data Mining (in conjunction with ICDM)

Pisa, December 15th 2008

Content

Problem definition Approaches Matrix factorization

Basics, BRISMF, Semipositive, Retraining Further enhancements

Transductive MF, Neighbor based correction Experimental results

Collaborative filtering

Problem definition I.

1 4 3

44

2 44

Problem definition II.

The phenomenon can be modeled by the random triplet (U, I, R).

A realization of the phenomenon (u, i, r) means that the u-th user rated the i-th item with value r.

user id (range: {1, …, M})item id (range: {1, …, N})rating value (range: {r1, …, rL})

Problem definition III.

The goal: predict R from on (U, I). Error criterion: mean squared error (RMSE). The task is nothing else than the classical

regression estimation. Classical methods fail because of the unusual

characteristics of the predictor variables.

Content

Problem definition Approaches Matrix factorization

Basics, BRISMF, Semipositive, Retraining Further enhancements

Transductive MF, Neighbor based correction Experimental results

Approaches

Matrix factorization: approximates the rating matrix by the product of two lower-rank matrices.

Neighbor based approach: defines similarity between the rows or the columns of the rating matrix.

Support based approach: characterizes the users based on the binarized rating matrix.

Restricted Boltzmann machine: models each user by a stochastic, recurrent neural network.

Global effects: cascades 1-variable predictors.

Content

Problem definition Approaches Matrix factorization

Basics, BRISMF, Semipositive, Retraining Further enhancements

Transductive MF, Neighbor based correction Experimental results

Matrix Factorization (MF)

Idea: approximate the rating matrix as the product of two lower-rank matrices

R ≈ P ∙ Q

Problem: huge number of parameters (e.g. 10

million), R is partially unknown. Solution: incremental gradient descent.

P: user feature matrix (M x K)

Q: item feature matrix: (K x N)R: rating matrix (M x N)

MF sample - learning

Q

P1 4 3

44

2 44

R1.2

1.2

0.4

-0.5

0.9

-0.4

1.4 0.8 -1.3 -0.1 0.5

-0.2 0.5 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.2

1.2

0.4

-0.5

0.9

-0.4

1.4 0.8 -1.3 -0.1 0.5

-0.2 0.5 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.4

0.9

-0.4

1.3 0.8 -1.3 -0.1 0.5

-0.1 0.5 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.4

0.9

-0.4

1.3 0.8 -1.3 -0.1 0.5

-0.1 0.5 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.2

1.2

0.4

-0.3

0.9

-0.4

1.3 0.9 -1.3 -0.1 0.5

-0.1 0.4 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.2

1.2

0.4

-0.3

0.9

-0.4

1.3 0.9 -1.3 -0.1 0.5

-0.1 0.4 -0.4 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.4

1.3 0.9 -1.3 -0.0 0.5

-0.1 0.4 -0.4 1.5 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.4

1.3 0.9 -1.3 -0.0 0.5

-0.1 0.4 -0.4 1.5 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.1

0.4

-0.2

0.8

-0.4

1.3 0.9 -1.2 -0.0 0.5

-0.1 0.4 -0.3 1.5 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.1

0.4

-0.2

0.8

-0.4

1.3 0.9 -1.2 -0.0 0.5

-0.1 0.4 -0.3 1.5 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.4

1.3 0.9 -1.2 0.1 0.5

-0.1 0.4 -0.3 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.4

1.3 0.9 -1.2 0.1 0.5

-0.1 0.4 -0.3 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.5

-0.2

0.9

-0.3

1.5 0.9 -1.2 0.1 0.5

0.0 0.4 -0.3 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.5

-0.2

0.9

-0.3

1.5 0.9 -1.2 0.1 0.5

0.0 0.4 -0.3 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.2

1.5 0.9 -1.1 0.1 0.5

0.0 0.4 -0.2 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.4

-0.2

0.9

-0.2

1.5 0.9 -1.1 0.1 0.5

0.0 0.4 -0.2 1.6 0.3

MF sample - learning

Q

P1 4 3

44

2 44

R1.1

1.2

0.5

-0.2

0.9

-0.1

1.5 0.9 -1.1 0.1 0.6

0.0 0.4 -0.2 1.6 0.2

After a while...

MF sample - learning

Q

P1 4 3

44

2 44

R1.4

0.9

2.5

1.1

1.9

-0.3

1.5 2.1 1.0 0.7 1.6

-1.0 0.8 1.6 1.8 0.0

MF sample - prediction

Q

P1 4 3

44

2 44

R1.4

0.9

2.5

1.1

1.9

-0.3

1.5 2.1 1.0 0.7 1.6

-1.0 0.8 1.6 1.8 0.0

-0.5 3.5

4.9 1.1

3.3 2.4

1.5

BRISMF

Enhancements on the previous model: User and item Biases (offsets). Regularization.

We can call this Biased Regularized Incremental Simultaneous MF (BRISMF).

This is a very effective MF variant indeed. Leaving out any of these characteristics

(B, R, I, S) leads to inferior accuracy.

Semipositive MF

It is useful to put a nonnegativity constraint on the user feature matrix P.

There are many possible ways to implement this (e.g. PLSA, alternating least squares).

Our solution: if a user feature becomes negative after the update, then it is set to zero.

Reset User Features

Disadvantage of BRISMF: user features updated at the beginning of an epoch may be inappropriate at the end of the epoch.

Solution: 1) Reset user features at the end of the training. 2A) Retrain user features. 2B) Retrain both user and item features.

R P

Q

P'

Content

Problem definition Approaches Matrix factorization

Basics, BRISMF, Semipositive, Retraining

Further enhancements Transductive MF, Neighbor based correction

Experimental results

Transductive MF

How is it possible to use the Netflix Qualifying set in the correction phase?

We use the following simple solution:

Fast and Accurate NB Correction I.

Neighbor based (NB) methods can improve the accuracy of factor models, but conventional NB methods are not scalable.

Is it possible to integrate the NB approach into the factor model without losing scalability?

Fast and Accurate NB Correction II.

Where sjk is (normalized scalar product based similarity):

OR (normalized Euclidean distance based similarity)

NB Correction sample

Q

P1 4

R1.4 1.6

1.5 2.1 1.0 2.2 1.6

-1.0 0.8 1.6 0.7 0.0

4.20.5 4.2

4.1

Similarity: 0.2, Error: -0.5

Similarity: 0.8, Error: +0.2

Correction: -0.1

Content

Problem definition Approaches Matrix factorization

Basics, BRISMF, Semipositive, Retraining Further enhancements

Transductive MF, Neighbor based correction

Experimental results

Results I.

Method Name of our methodOur Probe 10 Our Quize Bell et al's QuizeSimple MF BRISMF#10000.8938 0.8939 0.8998Retrained MF BRISMF#1000UM0.8921 0.8918 N/AMF with neighbor correctionBRISMF#1000UM,S10.8905 0.8904 0.8953

MethodName of our

methodOur Probe10 Our Quiz Bell et al's Quiz

Simple MFBRISMF

#10000.8938 0.8939 0.8998

Retrained MFBRISMF

#1000UM0.8921 0.8918 N/A

MF with neighbor correction

BRISMF

#1000UM/S20.8905 0.8904 0.8953

Results II.

Method Name of our methodOur Probe 10 Our Quize Bell et al's QuizeSimple MF BRISMF#10000.8938 0.8939 0.8998Retrained MF BRISMF#1000UM0.8921 0.8918 N/AMF with neighbor correctionBRISMF#1000UM,S10.8905 0.8904 0.8953

Epoch Training Time (sec) RMSE

1 120 0.9188

2 200 0.9071

3 280 0.9057

4 360 0.9028

5 440 0.9008

6 520 0.9002

Thanks!

?