a joint model of feature mining and sentiment analysis for product review rating

Jorge Carrillo de Albornoz - ECIR 2011

1

A Joint Model of Feature Mining and Sentiment Analysis for Product Review RatingJorge Carrillo de AlbornozLaura PlazaPablo Gervás Alberto Díaz

Universidad Complutense de Madrid

NIL (Natural Interaction based on Language)


2

MotivationProduct review forums have become

commonplaceReviews are of great interest

◦Companies use them to exploit their marketing-mix

◦ Individuals are interested in others’ opinions when purchasing a product

Manual analysis is unfeasibleTypical NLP tasks:

◦Subjective detection◦Polarity recognition◦Rating inference, etc.


3

MotivationTraditional approaches:

◦Term frequencies, POS, etc.◦Polar expressions

They do not take into account:◦The product features on which the

opinions are expressed◦The relations between them


4

HypothesisHumans have a conceptual

model of what is relevant regarding a certain product

This model influences the polarity and strength of their opinions

It is necessary to combine feature mining and sentiment analysis strategies to◦Automatically extract the important

features◦Quantify the strength of the opinions

about such features


5

The HotelReview Corpus25 reviews from 60 different hotels

(1500 reviews)Each review:

◦The city◦The reviewer nationality◦The date◦The reviewer category◦A score in 0-10 ranking the opinion◦A free-text describing, separately,

what the reviewer liked and disliked


6

The HotelReview CorpusNo relation between the score and the

text describing the user opinion:

Two annotators◦Excellent, Good, Fair, Poor and Very poor◦Good, Fair and Poor

After removing conflicting judgments =1000 reviews

Good location. Nice roof restaurant - (I have stayed in the baglioni more than 5 times before). Maybe reshaping/redecorating the lobby.Noisy due to road traffic. The room was extremely small. Parking awkward. Shower screen was broken and there was no bulb in the bedside light.


7

The HotelReview Corpus

Download: http://nil.fdi.ucm.es/index.php?q=node/456


8

Automatic Product Review RatingStep I: Detecting Salient Product Features

◦ Identifying the features that are relevant to consumers

Step II: Extracting the User Opinion◦ Extracting from the review the opinions

expressed on such featuresStep III: Quantifying the User Opinions

◦ Predicting the polarity of the sentences associated to each feature

Step IV: Predicting the Rating of a Review◦ Translating the product review into a Vector of

Feature Intensities (VFI)


9

Step I: Detecting Salient Product FeaturesObjective: Identifying the product

features that are relevant to consumers Given a set of reviews R={r1, r2, …, rn}:

1. The set of reviews is represented as a graph Vertices = concepts Edges = is a + semantic similarity relations

2. The concepts are ranked according to its salience and a degree-based clustering algorithm is executed

3. The result is a number of clusters where each cluster represent a product feature


10

Step II: Extracting the User Opinion on Each Product FeatureObjective: Locating in the review all

textual mentions related to each product feature1. Mapping the reviews to WordNet concepts2. Associating the sentences to feature

clusters: Most Common Feature (MCF): more WordNet

concepts in common All Common Features (ACF): every feature

with some concept in common Most Salient Feature (MSF): the sentence is

associated to the highest score feature


11

Step III: Quantifying the User OpinionsObjective: Quantifying the opinion

expressed by the reviewer on the different product features

Classifying the sentences of each review into positive or negative

Any polarity classification system may be used

Our system:◦ Concepts rather than terms◦ Emotional categories◦ Negations and quantifiers

12

Step IV: Predicting the Rating of a ReviewObjective: Aggregate all previous

information to provide an overall rating for the review1. Mapping the product review to a VFI2. A VFI is a vector of N+1 values representing

the detected features and the other feature3. Two strategies for assigning values to the

VFI: Binary Polarity (BP): the position in the VFI of the feature

assigned to each sentence is increased or decreased in one according to the polarity of the sentence

Probability of Polarity (PP): the feature position is increased or decreased with the probability calculated by the classifier


[-1.0, 0.0, 0.0, 0.0, …,-1.0, 0.0, 0.0,1.0, 0.0, 1.0, 0.0, …., 1.0]


13

Experimental SetupHotelReview corpus: 1000 reviewsDifferent feature sets:

◦Feature set 1: 50 reviews 24 feature clusters and 114 concepts



Baselines:◦Carrillo de Albornoz et al. (2010)◦Pang et al. (2002)


14

Experiment 1Objectives:

1. To examine the effect of the product feature set

2. To determine the best heuristic for sentence-to-feature assignment (Most Common Feature, All Common Features and Most Salient Feature)

Task: Three classes classification (Poor, Fair and Good)

We use the Binary Polarity strategy for assigning values to the VFI vector


15

Experiment 1 - Results

Method

Feature set 1 Feature set 2 Feature set 3MCF

ACF MSF

MCF

ACF MSF

MCF

ACF MSF

Logistic 69.8

67.7

69.8

70.4

67.4

70.8

69.1

67.4 70

LibSVM 69 67.1

69.2 69 67.

8 69.2

68.8

67.7 69

FT 66.8

64.2

66.8

66.3

65.2

68.6

68.4

65.8

68.4

Average accuracies for different classifiers, using different feature sets and sentence-

to-feature assignment strategies


16

Experiment 1 - DiscussionFeature set 2 reports the best results

for all classifiersAccuracy differs little across different

feature sets and increasing the number of reviews used for extracting the features does not always improve accuracy

This is due to the fact that users are concerned about a small set of features which are also quite consistent among users


17

Experiment 1 - DiscussionThe Most Salient Feature (MSF)

heuristic for sentence-to-feature assignment produces the best outcome

The Most Common Feature (MCF) heuristic reports very close results

But the All Common Features (ACF) one behaves significantly worse

It seems that only the main feature in each sentence provides useful information for the task


18

Experiment 1IObjectives:

1. To check if the Probability of Polarity strategy produces better results than the Binary Polarity strategy

2. Test the system in a 5-classes prediction taskTasks:

◦Three classes classification (Poor, Fair and Good)

◦Five classes classification (Very Poor, Poor, Fair, Good and Excellent)

We use the Feature set 2 and the MSF strategy for these experiments


19

Experiment 1I - Results

Method 3-classes

5-classes

Logistic 71.7 46.9LibSVM 69.4 45.3FT 66.9 43.7Carrillo de Albornoz et al. [9] 66.7 43.2

Pang et al. [4] 54.2 33.5Average accuracies for different classifiers in the 3-classes and 5-classes

prediction task.


20

Experiment 1I - DiscussionThe Probability of Polarity behaves

significantly better than the Binary Polarity strategy

It allows to captures the degree of negativity/positivity of a sentence, not only its polarity

It is clearly not the same to say The bedcover was a bit dirty than The bedcover was terribly dirty


21

Experiment 1I - DiscussionThe results in the 5-classes

prediction task are considerably lower than in the 3-classes task

This was expected:1. The task is more difficult2. The borderline between Poor-Very poor

and Good-Excellent instances is fuzzyOur system significantly outperforms

both baselines in all tasks


22

Conclusions and Future WorkThe system performs significantly better

than previous approachesThe product features have different impact

on the user opinionUsers are concerned about a relatively small

set of product featuresThe salient features can be easily obtained

from a relatively small set of product reviews and without previous knowledge

Differences between the various Weka classifiers are not marked


23

Conclusions and Future WorkError propagation of the sentence

polarity classifierError assigning sentences to features

◦Not enough information: Dirty. Stinky. Unfriendly. Noisy

◦Co-reference problem: Anyway, everybody else was nice

To evaluate the system over other domains

To translate the system to other language


24

Thank you!Any question?

a joint model of feature mining and sentiment analysis for product review rating

Documents

albornoz ecir

product feature jorge

language1jorge carrillo

reviewsjorge carrillo

themjorge carrillo

featuresjorge carrillo

reviewseach review

joint model of feature