skim at sawtooth software conference 2012: analyzing menu-based conjoint modeling data
Post on 09-Jun-2015
1.149 Views
Preview:
DESCRIPTION
TRANSCRIPT
expect great answers
Menu-Based Choice modeling (MBC): a practitioner’s comparison of different methodologies
Sawtooth Software Conference, March 2012
Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder
Menu-based choice modeling – the next big thing
Sawtooth Software has recently launched its new Menu Based
Choice modeling software. Although the idea of build-your-own
exercises has been around for a while, the launch of a new tool from
Sawtooth Software usually causes a lot of excitement and uptake of
use.
As practitioners, we at SKIM want to be ready for the avalanche of
projects, so we started to look into pros and cons of several analysis
approaches.
2
Menu-based choice modeling – the next big thing
Look around
Menu-based choices are everywhere and
are becoming increasingly common
3
4
Whopper $3.50 California W. $ 4.50
Omega3 $3.75 Chicken Deli $ 3.50
Cheddar $0.50 American cheese
$ 0.75
Crispy Onions
$1.50 Bacon
$1.50
Curly fries $1.25
French fries $1.05
✔ ✔
✔
✔
✔
Total price $ 8.50
Menu-based choice exercises are found in areas where combining items matters
• Menu optimization in fast food/branded restaurant chains
• Telecom services bundling
• BYO computers (e.g. Dell)
• Optional features pricing optimization in automotive market
• Add-on services in the financial and insurance services industry
Menu-based Choice Modeling exercises deliver item-level forecasts of performance in these markets
It can deliver:
• Demand curves on an item level among many items
• Forecast revenue and find the optimal price for all items on the menu
• Measure uptake and decide whether to add a new item to your portfolio
• Cross-effects price sensitivity and cannibalization effects
• Does decreasing the price of single items hurt full menu sales?
• Most often chosen combinations and their prices
• Suggesting which items to bundle
• Insight in budget constraints
• How many items can we stuff in a bundle before we exceed the decision
maker’s budget?
6
At SKIM, we’re practitioners. We would like to understand how MBC works in our practice
In particular, we would like to better understand the analysis
procedure. At first sight, we loved the beta version of the Sawtooth
Software tool, but we wanted to investigate more.
So we developed an alternative analysis approach, and applied it to
a study into the consumer’s willingness to pay for features of a
notebook computer.
This presentation contains a comparison of results on aspects of
internal validity.
7
SKIM’s Menu Based Choice exercise
We apply the approach to a study into
consumer features of computer notebook
8
9
No glare screen
Wireless speakers
Easy keys
On-screen keyboard
spotlight
DVORAK keyboard
External battery
indicator
High quality touch-
screen
External radio w/
speakers
Universal plug for
US, EU, UK
3D-ready HD
webcam
Eye scanner
Gold-plated jack
Technical advancement has
brought new vistas of safety
and security and today it is
very easy to make your
laptops and notebooks safe
and secure with technologies
such as fingerprint readers,
face recognition, eye
scanners etc.
Eye recognition ensured only you can access your laptop
through a fast and accurate scan of the retina.
The laser scanner is conveniently positioned on top of the
screen, next to the webcam
This pilot application had the following specifications:
• 12 consumer
features
(2 levels: On/Off)
• 12 price attributes
(3 levels)
• 1 notebooks core
attribute (3 levels)
• 1 none option
9 choice tasks:
• 7 random tasks
• 2 hold out tasks
to estimate
predictive validity
10
There are 3 price
levels per feature,
varied in accordance
with an orthogonal
research design
26 attributes
7
9 Choice tasks
3 price levels
Sample size: 1408
There are various models to analyze MBC data:
As presented in Bryan Orme’s paper
“Menu-Based Conjoint Modeling Using Traditional Tools” :
• Exhaustive Alternatives Model
• Serial Cross Effect Model
Both models have drawbacks that we thought we could solve using
SKIM’s method:
• Choice Set Sampling Model
11
Exhaustive Alternatives Model All possible ways to choose options are included
in the choice set.
• This model formally recognizes and predicts the
combinatorial outcomes of menu choices.
• The dependent variable is the choice of a
combination using a single logit-based (MNL)
model
• All possible combinations of options are coded as
one attribute where each level is a combination :
• e.g. with 3 on/off options, this attribute would have
2^3=8 levels
• Price: one price attribute for each option (or one
total price attribute)
12
Drawbacks
The number of possible
combinations grows
exponentially with the
number of options (2ⁿ
dichotomous choices),
transcending into a
problem of computational
feasibility.
Serial Cross-effect Model
The choice of each option is modeled
separately
• The dependent variable is the single
choice of a feature
• N different logit models predicting the
choice of option X as a function of:
• Price of option X
• All other significant cross effects
13
Drawbacks
Only significant
cross-effects should
be included -
meaning they have
to be detected
beforehand
We thought of solving it by introducing a ‘hybrid’ approach: Choice Set Sampling approach
• Like in the Exhaustive Alternatives Model, we consider the full choice set
with all possible combinations of options. However:
• we code each feature and its price as separate attributes (instead of a unique
attribute with all combinations as levels);
• we use importance sampling* – we consider a random sample from the set of
all chosen combinations
• Similarly to the Serial Cross-Effect Models, we also consider whether a
respondent chose an option at various price points.
14
* See importance sampling Ben-Akiva and Lerman (1985)
Coding the “sampling of alternatives” approach
1. In our model there are a total of 3*2^12=12888 possible combinations. However, “only” 4560 were chosen at least once.
Note: » Each feature is either included in the combination (1) or not (2)
» Option prices are alternative specific
2. We draw a random sample from this choice set. It is basically still a single logit-based (MNL) model where the dependent variable is the choice of the combination.
15
Combination # Core Feature 1 Price 1 Feature 2 Price 2 Feature 3 Price 3 ... Feature 12 Price 12 Choice
1 3 2 0 1 3 2 0 ... ... ... 0
2 1 2 0 2 0 1 2 ... ... ... 1
3 2 1 1 1 3 2 0 ... ... ... 0
... ... ... ... ... ... ... ... ... ... ... ...
4560 3 2 0 1 3 2 0 ... ... ... 0
Coding the “sampling of alternatives” approach
3. Each task is codified with 33 concepts/combinations drawn from
the sub-sample, with:
• The chosen alternative in each task
• 32 combinations randomly sampled from the choice set of all chosen
combinations
16
CASEID Task# Concept# Core Feature 1 Price1 Feature 2 Price2 ... Response
1 1 1 1 1 2 1 3 ... 0
1 1 2 1 2 0 1 3 ... 1
1 1 3 1 2 0 2 0 ... 0
... ... ... ... ... ... ... ... ... ...
1 1 33 2 2 0 1 3 ... 0
Coding the “sampling of alternatives” approach
4. In addition, our model is “hybrid” because we add extra dummy tasks for each respondent:
• For each choice task, we add 12 dummy tasks, one per feature
• We check whether a feature has been chosen at a specific price point
• No explicit modeling of cross effects between features
• This coding contains the information that respondent 1 in task 1 chooses feature 1 at price points 1, while she does not choose feature 2 at price point 3, and so on. Therefore we embed a price barrier in our model which amplifies accuracy in price sensitivity estimation.
17
CASEID Concept# Core Feature 1 Price1 Feature 2 Price2 Feature 3 Price3 ... Response
1 1 1 1 1 2 0 2 0 ... 1
1 2 1 2 0 2 0 2 0 ... 0
1 1 1 2 0 1 3 2 0 ... 0
1 2 1 2 0 2 0 2 0 ... 1
1 1 1 2 0 2 0 1 1 ... 0
1 2 1 2 0 2 0 2 0 ... 1
Analysis steps of SKIM’s Choice Set Sampling approach
• Using this setting we run HB estimation, so we can estimate utilities
for:
• Each feature (present/not present; 12 utility values and their mirrors)
• Each price level for each feature (3*12 utility values)
• None option (nothing is chosen; one utility value)
• We build a simulator in Excel, based on either Share of Preference
(SoP) or Share of First Choice (SoFC) with which we have:
• Single feature choice prediction
• Combinations choice prediction
18
Serial-Cross effect model
• Using Sawtooth Software’s MBC we build 12 different models for
each feature choice
• We could not find any significant cross-effects between the
features, both using counts and aggregate logit
• We use HB estimation and we simulate:
• Single Feature Choice predictions using Draws and Point Estimates
• Combinations choice predictions using Draws, Point Estimates and
Weighted Draws.
19
All approaches can be used to answer the same business question
That’s why we compare the approaches to see:
• Which approach delivers the highest validity?
And because as practitioners, we often find ourselves dealing with
demanding clients and strict deadlines, so that we don’t just need
approaches that work but that are also efficient and as easy to apply:
• Which approach is most efficient to a practitioner?
20
Which approach has the highest validity?
The results - Choice Set Sampling vs Serial Cross-Effect model
21
Results 1: Single Features Choice Predictions The Hold-out choice tasks suggest a similar performance
22
Hold-out 1 Hold-out 2
R-Squared MAE R-Squared MAE
Serial
Cross-Effect
Model
HB, Point Estimates 0.991 0.9% 0.990 1.0%
HB, Draws 0.991 0.9% 0.992 1.1%
Choice Set
Sampling Model
HB, First Choice 0.987 1.6% 0.989 1.7%
HB, Share of Preference 0.984 1.5% 0.981 1.5%
Both approaches have a very low MAE on the hold-out tasks
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10
Option11
Option12
Observed Serial cross-effect model (Draws) Choice set sampling model (SoP)
23
Fre
q.
of
ch
oic
e
Holdout task - 1
No structural consistency in errors
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
5.0%
5.5%
6.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10
Option11
Option12
Choice set sampling model (First Choice, MAE = 1.5%) Serial cross-effect model (Draws, MAE = 0.9%)
24
Ab
so
lute
err
or
Holdout task - 1
• Hit rate: % of respondents for which the choice on the option was predicted correctly
• 2 holdout tasks x 1408 respondents = 2816 observations for the hit rate
25
86.8% 91.1%
86.5% 84.4% 87.5%
84.9% 86.6% 86.5% 90.9% 89.0% 87.0% 87.2%
86.4% 90.7%
85.8% 84.1% 86.5% 85.1% 87.0% 86.5% 90.2% 88.9% 86.8% 86.7%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10
Option11
Option12
Choice set sampling model (First Choice) Serial cross-effect model (Weighted Draws)
Hit r
ate
The individual hit rate is almost the same across the two hold out tasks
Both models fit individual choices of combinations
41.0%
63.2% 66.2% 67.9%
83.6% 86.4%
41.3%
63.6% 66.2% 67.7%
83.5% 86.2%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
All 12 optionchoices predicted
correctly
At least 11choices predicted
correctly
At least 10choices predicted
correctly
At least 9 choicespredicted correctly
At least 7 choicespredicted correctly
At least 8 choicespredicted correctly
Choice set sampling model (First Choice) Serial cross-effects model (Weighted Draws)
26
Result 2: Feature combination predictions
So we can conclude that both approaches are viable tools for MBC analyses
Both models
• Are able to predict accurately hold-out choice tasks on aggregate
level
• Are extremely effective to predict individual choices of single
options and combinations
So both models are viable tools for analyzing MBC data.
• But which one is the most effective for practitioners?
27
Which approach is most efficient to a practitioner?
So both approaches work and it comes down to efficiency.
28
Choice Set Sampling approach – Benefits and Drawbacks
Benefits
• One model to estimate, one
model to simulate
• No need to make a call on
which cross-effects to
include
• Explicitly predicts choice of
combinations
Drawbacks
• Complex procedure: time
consuming set up for estimation
• Simulations are computationally
intensive
• Simulators are not very handy
tools for clients
29
Serial Cross Effect Model – Benefits and drawbacks Benefits
• Dedicated software
available
• Explicit inclusion of cross-
effects in the model
• Easy simulation tools
Drawbacks
• Learning curve of
understanding how to interpret
the significance of cross /
interaction effects and their
inclusion in the model – it
takes art and craft to build an
accurate model
• Once cross-effects are
included in the model, they
hold for all respondents
30
To conclude: Sawtooth Software’s Serial Cross-effect model is the practitioner’s choice
• We would recommend using Sawtooth Software’s Serial Cross-effect
model and software package,
• After the initial learning, it’s an easy to apply and time-effective solution, thanks
to its dedicated software
• One just needs to invest in the learning curve of making the call about the
significance and meaning of interaction/cross effects
• If you want to use the Choice Set Sampling model, be prepared to invest
time to create dedicated tools
31
twitter.com/
skimgroup
facebook.com/
skimgroup linkedin.com/
company/skim
youtube.com/
skimvideos skimgroup.com
contact us or follow us online!
Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder
www.skimgroup.com | +31 10 282 3535
top related