Collaborates with:•

Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features

Image from:https://www.ratchetandwrench.com/articles/5045-dealing-with-customer-complaints

Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei and Jimmy Lin

Who are we?

RSVP.ai is a Canadian startup based in Waterloo, Ontario

that aims to build deep natural language understanding

systems to facilitate seamless dialogues between humans

and machines.

2

One of the largest Chinese e-commerce company. As of the first quarter of 2018, its platform has

301.8 million active users.

!3

Hi, Agent ID 1234 from JD.com, happy to assist

you！

I’ve told you the address SEVERAL TIMES already!!! But you’ve wasted my time

by making me go back to the original address.

Hi, can you please provide me with the order number?

I’ll look into this for you!

The same problem has happened on many orders. I want to protect

my rights as a consumer!!!

!4Image from: http://www.customerexperienceinsight.com/customers-said-you-suck-handling-complaints/

Customer Complaint Escalation

!5

Customers

Tough-to-Please Customers

Company’s Agents

Consumer Protection Bureau

~70%

~30%

~75%: Text ~25%: Others

Detecting Customer Complaint Escalation• 300+ complaints every day!

• Bad customer service experience causes serious brand damage to JD.com.

• Importance of real time detection system

!6

Online classification problem over dialogue

Hard problem! <0.01% complaints!1 week to 1 month13%

1 day to 1 week41%

<1 day47%

How long does it takes for the customer complaints after they talk to the agents?

Model

!7

Tf-idf vectors

Manually-engineered featuresNeural Network!

Hierarchical Attention Network

!8Yang et al. 2016. Hierarchical attention networks for document classification. (NAACL 2016)

Too Complicated! We simplify!

!9

Time lineBefore Now

Dialogue StreamC A C A C A CA

W1 W2 Wn

……

……VTF−IDF

LSTM LSTM LSTM……Wtf−idf

btf−idf

a1

a2

an

Encoded Dialogue Representation External Feature ftf−idf

Softmax

FANBase

Watt

FANTF-IDF

FANfull

Model Framework• 😡 😵 😤

• ……

• ???

• !!!

• # of sentences / words

• # of words in two term dictionaries (complaint, 12315, customer right, ……)

• Dialog Sentiment

•

Model Framework

!10

Time lineBefore Now

Dialogue StreamC A C A C A CA

W1 W2 Wn

……

……VTF−IDF

LSTM LSTM LSTM…… Watt

Wtf−idf

btf−idf

a1

a2

an

Encoded Dialogue Representation External Feature ftf−idf

Softmax

!11

Evaluation Metrics

Recall@K =# of Detected Complainted Customers in Top K

# of Customer Complaints

!12

Experimental Setup

• Comparison with Baselines

• Effect of Negative Samples

• Results over An Entire Week

• Online Deployment Results

!13

Comparison with BaselinesLR-dict

LR-all

LightGBM

fastText

CNN

LSTM

FAN-base

FAN-tf-idf

FAN-full

0 6.5 13 19.5 26

Recall@5000

!14

10M

5M

1M

0.1M

0 6.5 13 19.5 26

Recall@5000

Effect of Negative Samples

# of negative samples

!15

0

10

20

30

40

June 17th June 18th June 19th June 20th June 21st June 22nd June 23rd

Average

Recall@5000

Results over An Entire Week

!16

0

7.25

14.5

21.75

29

Oct 8th Oct 9th Oct 10th Oct 11st Oct 12nd Oct 13rd Oct 14th

Average

Recall@5000

Online Deployment Results

!17

Lesson Learned

• Start with simple models.

• Don’t start over. Always reuse existing solutions.

• If NN cannot provide enough capacity, try manually-engineered features!

!18

Q & A

Thank you!