modeling and analyzing ivr systems, as a special case of self-services

1

Nitzan Carmeli

23.06.2014

MSc Advisors: Prof. Haya Kaspi, Prof. Avishai Mandelbaum

Industrial Engineering and ManagementTechnion

Empirical Support: Dr. Galit Yom-Tov

o Interactive Voice Response (Voice Response Unit)

2

IntroductionIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

o Agents salaries are typically 60%-80% of the overall operating

budget in call centers

3




o Properly designed IVR system increase customer satisfaction

increase loyalty

decrease staffing costs

3




o Properly designed IVR system increase customer satisfaction

increase loyalty

decrease staffing costs

Customers self-serve

Increase profit

3


o Call Center with an IVR system

4

Introduction

IVR

servers

Abandonm

ent

Success

Introduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

Returns

P

?

?

o Methodologies for evaluating IVRso Quantifying IVR usability and cost-effectiveness

o Agent time being saved by handling the call, or part of the call, in the IVR

o Original reasons for calling vs. experience

(by analyzing end-to-end calls)

Suhm B. and Peterson P. (2002, 2009)

5

Literature ReviewIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary






5

Literature Review

30% completed service in IVR

Only 1.6% completed it successfully







o Designing and optimizing IVRso Human-Factor-Engineering

o Mainly comparing broad (shallow) designs and narrow (deep) designs.

Examples: Schumacher R. et al. (1995), Commarford P. et al. (2008) and many more.

5

Literature Review

30% completed service in IVR

Only 1.6% completed it successfully


6

Literature ReviewIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

Broad (shallow) design:

Narrow (deep) design:

0

123456

0

1

2

3

4

56

o Reducing IVR service timeso The IVR menu as a service tree –

reducing the average time to reach a desired service

Salcedo-Sanz S. et al. (2010)

o Stochastic search in a Foresto Models of stochastic search in the context of R&D projects

o Optimality of index policy.

Denardo E.V., Rotblum U.G., Van der Heyden L. (2004)

Granot D. and Zuckerman D. (1991)

7

Literature Review

1

( )M

i ii

Minimize f T t p


o Modeling customer flow within an IVR system

o Using EDA to estimate model parameters and identify usability

problems: has actually inspired our theoretical model

o Using optimal search solutions and empirical analysis to compare

alternative designs and infer design implications

8

Research GoalsIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

9

Search ModelIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

Adapted from an actual IVR of a commercial enterprise

10

o Notations

( , ) - rooted tree representing the IVR systemG V E

s

ab

cde

fg


10

o Notations


- set of tree leaves, representing IVR servicesM V

s

ab

cde

fg


10

o Notations



- Tree edges, representing IVR menu optionsE

s

ab

cde

fg

- root vertex, representing the IVR mains menu


10

o Notations



s - root vertex, representing the IVR main menu

s

ab

cde

fg

( ) - set of immediate successors of vertex iA i ( ) ={c,d,e} A a



10

o Notations



s - root vertex, representing the IVR main menu

s

ab

cde

fg

( ) - set of immediate successors of vertex iA i

( ) - immediate predecessor of vertex ipre i( ) pre c a



o Customer searcho Customers pay for each unit of time they spend in the IVR

o Customers receive reward when reaching a desired service

o Customers may look for more than one service

o Customers have finite patience

11


o Customer searcho At each stage the customer can choose:

1. Terminate the search and leave the system

12

s

ab

cde

fg


Search Model



2. Terminate the search and opt-out to an agent

12

s

ab

cde

fg


opt-out

opt-out

opt-out

12

s

ab

cde

fg


opt-out

opt-out

opt-out




3. Move forward to one of i’s immediate successors,

( )j A i

12

s

ab

cde

fg


opt-out

opt-out

opt-out

( )j A i





4. Move backward to i’s immediate predecessor,

( )pre i

12

s

ab

cde

fg


opt-out

opt-out

opt-out

( )j A i





4. Move backward to i’s immediate predecessor,

5. Move backward to the root vertex,

( )pre i

s

o IVR flow animation

13


Search Model

o States

14

, ,nS i T N

current vertex

Total time spent in the serach

: 1 vector representing repeated visits to each vertex

Where

i

T

N V


o Model Parameters - Edges

15

( ),t - exploration time of edge ( , ),

given that vertex i was visited ( ) times before

N ii j e i j

N i

Where

( ) - number of previous visits to vertex iN i


Search Model

, ,nS i T N ( )1 ,, , ( )N i

n i jS j T t N i

00:04 00:10 00:16 00:22 00:28 00:34 00:40 00:46 00:52 00:58 01:04 01:10 01:16

0.000

0.500

1.000

1.500

2.000

2.500

3.000

3.500

4.000

4.500

5.000

5.500

ID time before Play Account BalanceTotal for November2008 to June2009,All days

Total 1 2 3 4 [5 - 10)

[10 - 15) [15 - 20) [20 - 30) [30 - 40) [40 - 50) [50 - 100)

Time(mm:ss) (1 sec. resolution)

Rel

ativ

e fr

eque

ncie

s %

16


o Model Parameters - Edges

17

( ) ( ), , , (memoryless)N i N ii j i j i jP P T t T P t

- customer patience

exp( ), assumed


Search Model

18

o Model Parameters - Leaves

- success probability of vertex iiP

- reward earned from successful completion of vertex iir

( ) - time spent in vertex i given that we have been successful in visiting itsert i

( ) - time spent in vertex i given that we have not been successful in visiting itFt i

( ) . .

( ) . . 1ser i

iF i

t i w p PT

t i w p P


18

o Model Parameters - Leaves

o

- success probability of vertex iiP

- reward earned from successful completion of vertex iir

( ) - time spent in vertex i given that we have been successful in visiting itsert i

( ) - time spent in vertex i given that we have not been successful in visiting itFt i

=1

r =0, t (i)=0, t (i)=0 i

i ser F

P

( ) . .

( ) . . 1ser i

iF i

t i w p PT

t i w p P


For non-leaves vertices, \ :i V M

o Search candidates

19

1A sequence ( ,..., ) of vertices in V will be called a nU i i candidate

1

A candidate will be called if it

starts with with , and if every two sequential vertices is : , ,

satisfies one of the following conditions:

1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U s a c f c a d


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U fs a c c a d


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U ds fa c c a


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U ds fa ac c


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U ds a f ac c


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U ds a c f c a


Search Model

o Search candidates

19


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s

s

ab

cde

fg

Example: U s a c f c a d


Search Model

o Search candidates

20


1




1.

k k

Defini U proper

s

tion

U i i

1

1

1

( )

( )

,

k k

k k

k k

i A i

i pre i

i s i s


Search Model

- The set of all candidatesadmissible - The set of all candidatesproper

o Search candidates

21

( ) - Net revenue earned by fathoming candidate X U U

Expectation over: patience, success probabilities, time in vertices and edges

( ) ( )R U E X U


Search Model

o Search candidates

21

( ) - Net revenue earned by fathoming candidate X U U

Expectation over: patience, success probabilities, time in vertices and edges

( ) ( )R U E X U


Search Model

*Our goal is to find a candidate that maximizes ( ) ; U R U U

o Search candidates

22

A candidate could be dominated by another

candidate (could not be optimal) if it has at least one of the

following properties:

1. There is at least one leaf which appears mo

1. Proposit proper U

i

n

M

io

re than once in

2. The candidate has a subsequence ( ) ( ), where

3. The candidate has a subsequence , where

4. The candidate ends with

5. The candidate has a subsequence

U

U pre i i pre i i M

U i s i M

U i M

U

( )

6. The candidate has a subsequence ,

where ( , ) ( )

i pre i i

U i s direct sequence to j

L i j pre i


Search Model

o Search candidates

23

' \

The set of all candidates

D

admissible

The set ' is finite and each of its

candidates is a finite pa

2.

th

Proposition

D - The set of all candidates with at least

one of the properties of 1 Propositi

pro r

on

pe


Maximal Tree Algorithm

24


Search Model

Spanning the tree with forward tran

0. Initialization: ( , )

1.

2.

3. Updating w

Spanning the tree with backward trans

ith the set of new vertices,

if , else, re

ition

si

s

ti

o s

t

n

T T

T

T

T V E G

T M

M Stop

urn to step 1.

o Maximal Tree Algorithm

25


Recall: , ,nS i T N

( ) current vertex

( ) vector representing formor er backward transitionsdered

u U

base u

index u

:U T s

a

c

f

( )fc

( , )f ca

( , )f cd

26

A candidate is (in ') it is represented

by a path starting from the root, ,

.

T T

proper U admissible iff

U T

Theorem

V E

The set ' = \D and the output of Maximal Tree Algorithm are equivalent


o Backward induction – Index calculation

z27

MDP

payment for time spent in vertex

discounted reward from vertex

payment for time spe

( )( )

0

(

nt in

)

0

( ) - discounted revenue earned from vertex

. .

( )

ser

ser

F

T

t vt v s

v v

t vs

r v v V

r e ce ds w p P

r v

ce ds

vertex

. . 1 vw p P



28

MDPIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary

*

* *

*

( ) - discounted revenue earned from the tree with root vertex

( ) ( )

In the special case where : ( ) ( )

T

T

r v v V

R v E r v

v M r v r v


29

,

,

,

,

, ,

*

- discounted revenue earned earned from edge ( , ) ,

given that we successfully reached vertex after units of time

( )u u v

u u v

u u v

u

Tu v

u

u v u v

t tt t s

t tt

r u v E

u t

R E r

E e R v ce ds

,u u v u

u

sut t t

t

ce ds t


MDP


30

,

*,

We get:

( ) ( ) 1 1u

u v

tu v t

c cR e R v

L

,

* *,

While in , it is enough to calculate:

( ) ( ) 1 1u vu v t

u

c cR R v

L


MDP


31


MDP

* *,

( )

*

( )( )*

,( )

0* *, ( )( )

( )*,

( )0

max 0 : ( ) ( )

Set ( )

max . .

max 0 : ( )

max . . 1

ser

ser

F

F

u vv A u

t ut u s

u u v uv A u

u v t uv A ut u s

u v uv A u

if R Set R u E r u

u Stop

r R e ce ds w p P

if R Set r u

R e ce ds w p P

* *

* *,

( )

( ) =E ( )

( ) arg max u vv A u

Set R u r u

Set u R

32


( , )

IVR original tree

G V E

32


( , )

IVR original tree

G V E

( , )

Extended tree representing

possible customer actions within the

IVR system

T TT V E

32


( , )

IVR original tree

G V E

( , )

Extended tree representing

possible customer actions within the

IVR system

T TT V E

Calculating indices

using backward induction to

find optimal search policy

o A medium bank

o About 450 working agents per day (~200 during peak hours)

o Call by call data from May 1st, 2008 to June 30th, 2009

o Overall Summary of calls

33

System DescriptionIntroduction Literature Review Theoretical Model Empirical Analysis Model Implications Summary and Discussion

Case Study

IVR Calls Total % Out of total Average per weekday

# Served only by IVR 27,709,543 61.6% 50,937

# Requesting also agent service 17,280,159 38.4% 31,765


o A medium bank

o About 450 working agents per day (~200 during peak hours)

o Call by call data from May 1st, 2008 to June 30th, 2009

o Overall Summary of calls

33


Case Study

IVR Calls Total % Out of total Average per weekday

# Served only by IVR 27,709,543 61.6% 50,937

# Requesting also agent service 17,280,159 38.4% 31,765


10% of IVR calls – Identification only!

34


Case Study

34


Case Study

Requested by less than 2% of

the calls

o Estimating abandonments

35


Case Study


35

Threshold timeThreshold time

Threshold time

Case StudyIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary


36


Case Study

00:01 00:11 00:21 00:31 00:41 00:51 01:01 01:11 01:21

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

ILBank IVR subevent duration, Query, Current Account - Recent Account ActivityTotal for May2008 to March2009,All days

Fitting Mixtures of Distributions

Empirical Total Lognormal Lognormal Lognormal Lognormal Lognormal Lognormal

Time(mm:ss) (1 sec. resolution)

Rel

ativ

e fr

eque

ncie

s %

Parameter Estimates

Components MixingProportions(%)

Scale Shape Mean StandardDeviation

1. Lognormal 5.19 1.34180 0.35 4.07 1.492. Lognormal 8.20 0.83213 0.08 2.30 0.183. Lognormal 39.05 3.05912 0.45 23.53 11.034. Lognormal 26.03 3.95811 0.20 53.43 10.855. Lognormal 11.80 4.04316 0.06 57.11 3.476. Lognormal 9.73 4.51403 0.31 95.92 30.95

o Example - Comparing different designs

(based on frequent services in our database)

37


Model ImplicationsIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary


38



Service rType 1

tser (mean, sec)

tF(mean, sec)

P

Play account balance 200c 21 6 0.75

Account Summary 0 51 5 0.765

Recent Account Activity 400c 45 3 0.87

Account Activity Today 0 50 3 0.78

Selected Securities 0 43 3 0.7

Stock Exchange Israel 0 16 7 0.775

Transfer between Account 0 17 3 0.6

Credit Card Vouchers 600c 49 3 0.915

,

0.05 NIS/sec

=0.05

=0.004 average customer patience 250 sec

t , t exponentially distributed

t exponentially distributed for every ,ser F

i j

c

i j


o Design A - original:

Optimal candidate :

Customer revenue:

39



Play Account Balance

Recent Account Activity

: Main menu Main Menu Checking account

Main menu General Activ Credit Ci ati rd Vouchere ss

U

( ) 3.38R U


o Design B - deep:

Optimal candidate:

Customer revenue:

40



Play Account Balance

Recent Account Activity

Credit Card Vouc

: Main Menu

Main Menu M1 M2

M2 M3 M4 M5

M he s6 r

U

( ) 2.74R U


o Design C - shallow:

Optimal candidate:

Customer revenue:

41



Play Account Balance Recent Account Activity: Main Menu Main Menu

M Credit ai Can Me rd Vouchersnu

U

( ) 3.61R U

o Comparing different designs – Organization point of view

Example, assume:o 10,000 calls per day

o Cost per unit of time of communication = 0.003 NIS/sec

o Cost per second of agent call = 0.03 NIS/sec (based on UG project)

42



Design A - Original

Design B – Deep

Design C - Shallow

Customer revenue 3.38 2.74 3.61

Cost of communication

4,736 5,606 4,826

Revenue from saved agent time

30,564 30,564 30,564

Total organization profit

25,828 24,958 25,738

o How to design an IVR tree to encourage the use of certain services?

43




o Customer learning?

43





o Do customers navigate optimally?

o Services with low demand – lack of interest or lack of patience?

43







o Sensitivity to customer patience?

o Anticipating long waiting in agent queue – does it affect customer

behavior?

43







o Sensitivity to customer patience?

o Anticipating long waiting in agent queue – does it affect customer

behavior?

o Using IVR and state information to control customers behavior

examples: return later, use IVR for service

43



o Stochastic models of customer flow within the IVRo Comparing IVR designs

o Insights for better designs

o Supplementing previous knowledge in HFE, CS fields

o Data analysis as a basis for theoretical models

o Can be easily modified to other self-service systems

44


Summary and Future researchIntroduction Literature Review Research Goals Search Model MDP Case Study Model Implications Summary






44



Example: Web search engines

Hassan A. et al. (2010)






o Interesting open questions:o How does one recognize an abandonment from self-service when “seeing” one?

o Interrelation between customer-IVR interaction and customer-agent interaction?

44



Example: Web search engines

Hassan A. et al. (2010)

Thank you ,Questions?

modeling and analyzing ivr systems, as a special case of self-services

Documents

literature review30

ivr menu

ivr systems

ivrsquantifying ivr

special case of

ivr service times

designed ivr system

costeffectivenessagent