detecting spam reviews on the chinese online...

24
Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin Ryu, Fumito Masui, Michal Ptaszynski Department of Computer Science, Kitami Institute of Technology, Japan 1

Upload: others

Post on 03-Jun-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao

Shinshin Ryu, Fumito Masui, Michal Ptaszynski Department of Computer Science,

Kitami Institute of Technology, Japan

1

Page 2: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Outline

Introduction Data set Review Analysis Classification Experiment Conclusions

2

Page 3: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Introduction

・Various online shopping sites ・Spam reviews ・“Internet Water Army” in China

3

Page 4: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・Related Work:

review content spammer’s behavior

Introduction

→the lack of publicly available Chinese data sets

・Related Chinese research problem:

4

→the low accuracy and reliability of training data→the insufficient amount of public user information on Chinese shopping sites …

Page 5: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Chinese fake review detection method →stealth marketing →14 product parameters →SVM (Support Vector Machine)

・In our research:

Introduction

5

Page 6: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Outline

Introduction Data set Review Analysis Classification Experiment Conclusions

6

Page 7: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Data set・Data Set

https://world.taobao.com/

total review: 200 fake review: 31.5% test data: 20 training data: 180 →10-fold cross-validation

7

Page 8: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・Labeling

Web

Questionnaire

evaluator (60)

group A (30)

group B (30)

Data set

8

kappa value = 0.52

Page 9: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Outline

Introduction Data set Review Analysis Classification Experiment Conclusions

9

Page 10: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Design, Appearance, Quality, Consistency, Size, Tailoring, Fabric, Color fading, Reputation, Price, Staff, Express delivery, Satisfaction, Recommendation

5 very satisfied4 satisfied3 neither2 dissatisfied1 very dissatisfied

Review Analysis

・Product Parameters:

10

Page 11: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・The scoring results on apparel productsReview Analysis

11

Page 12: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Parameter ProportionRecommendation 0.83

Reputation 0.61Staff 0.61

Appearance 0.53… …

Fabric 0.42Size 0.36Price 0.20

Quality 0.00

・The ratio of fake reviews in non-confidence interval

4 highest

Review Analysis

12

95%

Page 13: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Reputation

StaffRecommendation

Appearance

Reputation

StaffRecommendation

Appearance

・The score details of highest 4:

Fake Normal

125 4Review Analysis

13

Page 14: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・The features of product parameters:A.Recommendation →“快快入手”(to buy quickly), →“强烈推荐”(highly recommended)

Review Analysis

14

B. Quality & Price →“物美价廉”,“物超所值”(really good and cheap)C. Staff →“服务态度”(attitude of the staff and services)

D. Size →the numbers

Page 15: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Outline

Introduction Data set Review Analysis Classification Experiment Conclusions

15

Page 16: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Classification Experiment

・SVM (Support Vector Machine) ・two experiments →experiment 1: word & product parameters →experiment 2: product parameters

16

Page 17: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Case Feature1 Bag-of-Words2 Staff, Quality&Price , Size, Recommendation *3 Staff, Quality&Price, Size, Recommendation4 Quality&Price, Size, Recommendation5 Staff, Size, Recommendation6 Staff, Quality&Price, Recommendation7 Staff, Quality&Price, Size8 Staff, Quality&Price9 Staff, Size10 Staff, Recommendation11 Quality&Price, Size12 Quality&Price, Recommendation13 Size, Recommendation14 Staff15 Quality&Price16 Size17 Recommendation

Classification Experiment

17

Page 18: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・experiment 1(word & product parameter)

Classification Experiment

Accuracy 0.68Precision 0.68

Recall 1.00

Occurrence TF TF_IDF

Case

Accuracy

18

Page 19: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・experiment 2 (product parameter)

Classification Experiment

Accuracy 0.73Precision 0.72

Recall 1.00

Case

Accuracy

19

Occurrence TF TF_IDF

Page 20: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Case Feature Accuracy12 Quality & Price、Recommendation 77.0010 Staff、Recommendation 76.853 Staff、Quality & Price、Size、Recommendation 76.404 Quality & Price、Size、Recommendation 76.306 Staff、Quality & Price、Recommendation 76.005 Staff、Size、Recommendation 75.8513 Size、Recommendation 74.8717 Recommendation 74.732 Staff、Quality & Price、Size、Recommendation* 72.407 Staff、Quality & Price、Size 69.408 Staff、Quality & Price 69.2016 Size 68.819 Staff、Size 67.3311 Quality & Price、Size 66.00

・Accuracy of experiment 2 in a descending order Classification Experiment

20

Recommendation + α

→Weight : Occurrence

Page 21: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Outline

Introduction Data set Review Analysis Classification Experiment Conclusions

21

Page 22: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Conclusions

・The four features of product parameters: →Recommendation, Quality & Price, Staff, Size

22

・Two experiments: →experiment 1 (word & product parameters) →experiment 2 (product parameters) showed higher results →”Recommendation + α” had higher performance

Page 23: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

・Future work →Automatically increase the data and the feature set →Considering of the different levels of features

Conclusions

23

Page 24: Detecting Spam Reviews on the Chinese Online …arakilab.media.eng.hokudai.ac.jp/~ptaszynski/data/IWMST...Detecting Spam Reviews on the Chinese Online Shopping Site TaoBao Shinshin

Thank you

24