duncan stuart - festival of newmr - 2010

KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)

SLIDE 1

12/5/10 1 Speaker Duncan Stuart, Kudos OrganisaPonal Dynamics, New Zealand Part 1: Session 3: Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (London)

BUMP BUMP -‐ A New Metric Inspired by Neural Networks


SLIDE 2

Background

1.  A quick framing discussion – the context of where this comes from.

2.  A swi[ overview of Neural Networks. What they are, how they work.

3.  A couple of examples (disguised) of things we’ve done with NNs....and then the BUMP discovery. We weren’t looking for it – but there it was.

4.  QuesPons. Duncan Stuart


SLIDE 3

HOW THIS PRESENTATION FITS IN WITH NEW MR


SLIDE 4 SLIDE 4

Where NN analysis fits in to New MR

This schemaPc describes what’s possible in market research.

Old MR basically lives in the lower le[. The discovery and analyPc dimension is preay basic. We ask simplisPc quesPons, then we treat these to basic analysis – o[en merely descripPve. Pies, bars and crosstabs.

Consequently (and also independently) the types of decision that come out of this research are also preay basic.

The value fronPer is around about where the middle curve is. Analyse and Plan.


SLIDE 5 SLIDE 5

Old MR lives in the boaom le[. New MR in the top right.

This schemaPc describes what’s possible in market research.

New MR is about pushing the fronPer further. Anyone with SurveyMonkey, and email list and Excel can do most of the things back in the red square. Old MR has become commodiPsed. More to the point, if everybody does it, it then doesn’t offer the user any great compePPve advantage.

New MR is about using available technology, and some professional brains, to operate in a much richer arena. Instead of the somewhat descripPve “what is” research, it takes us into the world of “what if.”


SLIDE 6 SLIDE 6

The world of modelling.

•  Acknowledges that the world is complex. For example all consumers won’t respond en masse in the same direcPon.

•  Acknowledges that consumer aftudes make up only part of the story. For example the economy may tank. The context may change.

•  Acknowledges that business decisions always involve risk.

•  Old MR doesn’t really work with these assumpPons. –  It usually works with mean scores, or simple linear regressions: in others kind

of treaPng the market as homogenous.

–  Research designs stay within the paradigm of market research. Did you see the ad? How strongly did you prefer Brand C?

–  We are asked to make a call. This pack or that? Launch or not launch? And we don’t usually employ very sophisPcated means of evaluaPng those decisions.


SLIDE 7

WHAT ARE NEURAL NETWORKS?


SLIDE 8 SLIDE 8

Neural Networks

•  NNs are a computer emulaPon of the mind. A computer learning process that searches for answers through thousands (maybe millions) of trial and error calculaPons.

–  A very typical example is the use by banks, to ascertain the risks in mortgage lending. Rather than have the bank manager make a decision, big banks have a NN system.

–  A typical result is that your local bank manager gets it wrong 12% of the Pme while NNs, working blindly off the data, get it wrong only 7% of the Pme.

–  In other words 41% fewer defaults.


SLIDE 9 SLIDE 9

Regular analyPcs are not so predicPve.

–  Regular analyPcs tend to simplify and are engineered around averages and simple regressions. They employ a few variables (lifestage, credit raPng and current income) and cannot really cope with, say 80 variables which interact with each other. (Variable X is condiPonal upon Variables Y and Z)

–  For example in the bank loan applicaPon do you have a fatal disease? 99.9% of people say no. So as a variable it might be discarded as useless. But if you DO have a fatal disease...what’s your credit risk now?

–  Or some variables are condiPonal upon others: do you have life insurance? may be totally irrelevant unless you answered yes to QuesPon 10.

–  In other words many variables are not important unless they are. A linear formula (regressions, correlaPons and other techniques) don’t really tell us what’s going on case by case. They explain the average story, but not case by case.

–  NN’s get over this problem...


SLIDE 10 SLIDE 10

Neural networks can handle complexity.

•  Computer learning – with the NN trying itera5on aOer itera5on, walking around the data landscape tweaking the algorithm (maybe millions of 5mes) un5l it can find no more accurate way of explaining the data. With computers, massive non-‐linear, mul5variate algorithms are no big problem.

•  Several outputs. 1.  A ranked list of the variables that have

greatest “acPvaPon” or effect on the predicted behaviour. (Great when you pile 80 variables into the mix.)

2.  Every respondent or case gets a unique ac#va#on score.


SLIDE 11 SLIDE 11

Let’s show a simple example. This was energy drinks.

AcPvaPon score too low. Probability that respondent does not buy Brand X.

Threshold

AcPvaPon score higher than 0.72. They probably Do buy Brand X.

1.  We used tracking data – training the NN on 1,000 respondents and trying the results on another 1000. 2.  There were 80+ variables to determine who and who doesn’t drink energy drinks. Variables include

demographic, aftude, lifestyle interests, surfing, nightclubs, etc. 3.  We went home. The NN worked overnight and by morning gave every respondent a simple acPvaPon score

based on the complex algorithm it had generated.

Tested 92% accurate

•  At first we used the ac8va8on score as the sort of measure you use with cross tabs. Our interest was in what it was that drove ac8va8on.

•  But then....


SLIDE 12 SLIDE 12

Then for some reason we ploXed the distribu5on of ac5va5on scores for the respondents.

Threshold

We got a bell curve that leaned over to the right. Don’t forget, this has been derived from a hugely complex mulPvariate process. We let NN do the interrogaPon...

ACTIVATION SCORE

AcPvaPon score too low. Probability that respondent does not buy Brand X.

AcPvaPon score higher than 0.72. They probably Do buy Brand X.

Tested 92% accurate


SLIDE 13 SLIDE 13

Well this was fascina5ng...a whole heap of people live near the threshold!

Threshold ACTIVATION SCORE

A big groundswell of people live right next to the Berlin Wall.

They look very “BUMP-‐able”


SLIDE 14 SLIDE 14

Bump. How close are users to the Berlin Wall? How easily could they be bumped over?

Berlin Wall 0.72

Low Bump Index. Blue populaPon lives miles from the wall.

A zillion dollar campaign won’t convert them.

High Bump Index. Grey populaPon is

very bumpable. All they need is a

nudge.


SLIDE 15 SLIDE 15

A bi-‐polar Bump situa5on....Perhaps your market is more polarised?

Berlin Wall 0.72

Note to marketers – don’t waste your 8me on these Low Bump

people!

Note to marketers – focus on these High Bump people.!


SLIDE 16 SLIDE 16

Now we can play what-‐if?

•  IdenPfy the people who live near the wall.

•  Start tweaking the variables that most drive their acPvaPon score.

•  (For example in the energy drinks case we examined what would happen if they got a job, or if they suddenly lost interest in night-‐clubbing, or if they suddenly rated Brand Z as parPcularly cool.)

•  And by tweaking the data and then re-‐running the algorithm we could see how the acPvaPon scores changed – case by case: how many people jumped over the line? Or not. (Or jumped back the wrong way!)

Berlin Wall


SLIDE 17 SLIDE 17

Some things I’ve learned about this approach.

•  Generally very accurate and trustworthy. 90%+ accuracy is preay standard.

•  Tells us which variables are driving the ac5va5on scores. Which are worth “playing with” in our modelling.

•  Usually reveals that if ‘what-‐if’ happens, then there are two or more countervailing trends. In other words it let’s us see complexity.

•  And by delivering us a meta-‐measure such as BUMP Index we can s5ll think in simple terms even if the algorithms that gives us the ac8va8on scores are gigan8cally complex. It helps us zoom in on some very useful informaPon but in a very explainable way. E.g. 20% of the market is just half a step from buying your product, and here’s who they are.


SLIDE 18 SLIDE 18

Summary

As next generaPon researchers I hope we’re consciously moving upwards and rightwards on this model.

Neural Networks help us quite easily work with the rich, mulP-‐variate, curvilinear nature of forecasPng – and BUMP, apart from being a cool liale measure in itself, shows us that the outer fronPers of research value can be reached and communicated quite simply.


SLIDE 19

DUNCAN STUART AND RAY POYNTER

Q & A


SLIDE 20

Thank you!

Duncan Stuart FMRSNZ

duncan@kudos-‐dynamics.com

Telephone 64 9 366 0620

www.kudos-‐dynamics.com

Funds from all projects go directly to the language school we built and support in Siem Reap Cambodia. www.savong.com

duncan stuart - festival of newmr - 2010

Marketing