Accelerated Testing

for Retention and CRM Programs

Brian Fichtel

Senior Director, Customer Retention

Hearst Magazines

Gordon Bell

President, LucidView

Managing Partner, Artestry gbell @ lucidview.com

1

Overview of Retention / CRM programs

• Multiple effort subscription renewal series utilizing a mix

of direct mail, e-mail, telephone and on-magazine efforts

• Generally segment by original acquisition source

(e.g. Direct Mail, Internet) and subscriber tenure

• Series runs for as long as 9 months

• Retention efforts are highly responsive and highly

profitable but have small quantities compared to

Acquisition Direct Mail

• All of the above makes testing a long, challenging

process with limited testing opportunities

2

Control Renewals / Retention Program

4 months

before expiration

- 3 months

- 2 months

- 1 month

Subscription

expiration

1 month after

expiration

+ 2 months

+ 3 months

+ 4 months

3

CRM Testing: Real-World Challenges

4 months

before expiration

- 3 months

- 2 months

- 1 month

Subscription

Expiration

+ 2 months

+ 3 months

+ 4 months

Subscription segments

Direct mail Internet Prior

Renewals

20 U.S. Magazines

Car

and D

river

Cosm

opolita

n

Countr

y L

ivin

g

ELLE

E

squir

e

Food N

etw

ork

M

agazin

e

Good

Housekeepin

g

Mari

e C

lair

Opra

h

Magazin

e

Popula

r M

echanic

s

Redbook

Road &

Tra

ck

Sevente

en

Tow

n &

Countr

y

Vera

nda

Wom

an’s

Day

Monthly expire groups

Month 1

Month 2

Month 3

Month 4

4

“Business Case” for Multivariate Testing

• Many creative, offer and segmentation test ideas but

small retention quantities that limit testing opportunities

• Started with

41 ideas to test…

Envelope size Stronger copy pushing payment

Postage Laser magazine title on all letters

Return slip position More aggressive in promotion of RAB discount

Branding Increase urgency around expire

Format Attached mail

Offer and payment info Same form with all efforts

Urgency Postcard with drive-to-web

Magazine benefits copy Offer expression - $0.97 vs. rounded up (9.97 vs. 10)

Guarantee No discount for late effort High segments

Payment chit Incentive - extra issues vs. pricing

Highlight discount Credit card renewal option

Change price 2- and 3-year renewal option

Incentive for early renewal Hearst Rewards (earn points)

Late-series incentive Renew now for digital copy/app

Multi-year offer Special offer upon cancel request

Compress contact stream Add a chit - gift/cross-sell subscription

More e-mail Premium with 2- or 3-year term

Start renewal cycle earlier Shorter term offer (6 issues) offer

Add telemarketing efforts Sweepstakes entry for early renewal

E-mail beforehand Magnitude of price change (small/medium)

Decrease time between efforts

5

“Business Case” for Multivariate Testing

• Multivariate testing provided opportunity to test many

ideas in a short period of time

– Enabling results in 9 months instead of 3 years

• Multivariate testing also provided an opportunity to

gain insights about test element interactions

that’s not possible with traditional A/B testing

6

Test Design: Universe of Possibilities

2-level

Test elements

Optimal designs • Complete freedom

• Cost = complexity

and error if assumptions

are wrong

Taguchi designs • Use many levels

• Bold assumption that

interactions don’t exist

Orthogonal test designs

• Full factorial

• Fractional factorial

• Plackett-Burman

• Centerpoints

• CCD (etc.)

7

A/B split-run

tests

Selected designs

“Mosaic” Test Design

• Two test designs

1. 11 creative changes

2. 6 elements of the offer and contact strategy

• Why?...

a) Number of test elements

b) Magnitude of impact (creative changes tend to have a smaller effect)

c) Value of interactions – mix of contacts and offer elements tend to have

larger, more valuable interactions

• Choices

1. 12-recipe test for 11 creative elements ( = fewer package versions)

2. 16-recipe test for 6 offer and creative elements ( = more interactions)

8

CRM Testing: Real-World Challenges

4 months

before expiration

- 3 months

- 2 months

- 1 month

Subscription

Expiration

+ 2 months

+ 3 months

+ 4 months

Subscription segments

Direct mail Internet Prior

Renewals

20 U.S. Magazines C

ar

and

Dri

ver

Cosm

opolita

n

Countr

y L

ivin

g

ELLE

E

squir

e

Food N

etw

ork

M

agazin

e

Good

Housekeepin

g

Mari

e C

lair

Opra

h

Magazin

e

Popula

r M

echanic

s

Redbook

Road &

Tra

ck

Sevente

en

Tow

n &

C

ountr

y

Vera

nda

Wom

an’s

Day

Monthly expire groups

Month 1

Month 2

Month 3

Month 4

Test ideas 4

segmentation models

11 creative elements

(2,048 possible combinations)

6 elements of the offer and

contact strategy (96 combos)

9

Test Elements

Test Elements - Creatives (-) Control (+) New idea

A Envelope size Large Small

B Postage Metered Indicia

C Return slip position Top of form Bottom of form

D Branding Magazine branding Hearst corporation branding

E Format Letter (with signature) Statement (no signature)

F Offer and payment info On right side Move to left side

G Urgency Current urgency copy Different urgency copy

H Magazine benefits copy Control Stronger benefits

J Guarantee None Add guarantee copy

K Payment chit No Add chit

L Highlight discount No Yes

Test Elements - Offer and Strategy (-) Control (+) New idea

A Change price Control +/- 20%

B Incentive for early renewal No Yes, offer incentive in 2nd effort

C Late-series incentive Discount Premium

D Multi-year offer Only once Add multi-year option to additional efforts

E Compress series timing Control (every 4 weeks) Compress (every 2-3 weeks) and add efforts at end

F More e-mail efforts Control More e-mail following DM efforts

Brainstormed 41 new ideas selected 17 test elements (split into 2 test designs)

10

“Mosaic” Test Designs

Envelo

pe s

ize

Posta

ge

Retu

rn s

lip p

ositio

n

Bra

ndin

g

Form

at

Off

er

and p

aym

ent

info

Urg

ency c

opy

Magazin

e b

enefits

copy

Guara

nte

e c

opy

Paym

ent

chit

Hig

hlig

ht

dis

count

(late

r

Recipe A B C D E F G H J K L Sample

1 – – – – – – – – – – – 205632 + + + – + + – + – – – 205883 + – + + – + – – – + + 204804 – – – + + + – + + – + 204715 + + – + + – + – – – + 204666 + – – – + + + – + + – 204577 + – + – – – + + + – + 205818 – + + – + – – – + + + 206339 – – + + + – + + – + – 20569

10 – + – – – + + + – + + 2045511 – + + + – + + – + – – 2051612 + + – + – – – + + + – 20528

Creative Test

Change p

rice

Incentive f

or

early r

enew

al

Late

-series incentive

Multi-year

off

er

Com

pre

ss s

eries t

imin

g

More

e-m

ail

eff

ort

s

Recipe A B C D E F Sample

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

Offer & Strategy Test

11

Creative Test (control)

12

Creative Test (2 versions)

E+

F+

D+

C+

G+

H+

13

Mosaic Creative Test – Results

-0.10%

-0.16%

-0.23%

+0.39%

+0.58%

-0.73%

-0.81%

-1.22%

-1.22%

-1.48%

-3.59%

0.0% 1.0% 2.0% 3.0%

B-: Postage

H-: Magazine benef its copy

C-: Return slip position

F+: Of fer and Payment Info

K+: Payment Chit

E-: Format

G-: Urgency

L-: Highlight discount (no)

A-: Envelope size (large)

J-: Guarantee (no)

D-: Branding (magazine)

Effect (avg RR of test as % change vs. control)

Hearst Magazines CreativeTest: Renewal Rate|Effects| as % change versus the control

Elements: sign of effect (optimal setting)

Four significant effects:

D-: Brand letters with the magazine title

(not Hearst Magazines)

J- and L-: Skip the guarantee language

and highlighting the discount

A-: Keep the larger envelope

14

Mosaic Offer & Strategy

Test – Design

3-levels of price means that each A+

Recipe is divided into two groups:

Half with a 20% drop in price

Half with a 20% price increase

Change p

rice

Incentive f

or

early r

enew

al

Late

-series incentive

Multi-year

off

er

Com

pre

ss s

eries t

imin

g

More

e-m

ail

eff

ort

s

Recipe A B C D E F Sample

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

15

Recipe A B C D E F Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4 Model 1 Model 2 Model 3 Model 4

1 – – – – – – 20,4372 + + – – – – 19,8803 – – – + – + 20,5854 + + + – + – 19,9455 + + + + + + 20,0816 – + – + + – 20,4677 – – + – + – 20,3908 + – + – – + 19,9799 + – + + – – 20,083

10 + – – + + – 20,09011 – + – – + + 20,19012 + + – + – + 19,98813 – + + + – – 20,51614 – – + + + + 20,55715 + – – – + + 19,95116 – + + – – + 20,170

Change p

rice

Incentive f

or

early r

enew

al

Late

-series incentive

Multi-year

off

er

Com

pre

ss s

eries t

imin

g

More

e-m

ail

eff

ort

s

Segment 1 Segment 2 Segment 3

Total Sample Size

Mosaic Offer & Strategy Test – analytical plan

16

Offer & Strategy Test –

Execution: recipes 1 vs. 5

- 4 months

- 3 months

- 2 months

- 1 month

Expiration

+ 1 month

+ 2 months

+ 3 months

+ 4 months

+ 5 months

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

$ $

17

Mosaic Offer Test – Results

0.07%

0.11%

0.12%

0.17%

0.25%

0.35%

0.36%

0.65%

0.67%

+0.86%

+1.22%

+1.56%

-2.43%

+3.04%

-5.86%

0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0%

AE

ACF

AC

ACD

CD

AB

AD

AF

CF

E+: Compress contact stream (yes)

F+: More e-mail (yes)

A+: Change price (yes)

D-: Multi-year of fer (no)

B+: Incentive for early renewal (yes)

C-: Late-series incentive (discount)

Effect (avg RR of test as % change vs. control)

Hearst Magazines Billing & Renewals Test: Renewal Rate|Effects| as % change versus the control (optimal = 6.7% lift)

Elements: sign of effect (optimal setting)

All 6 elements are significant

(but no interactions)

C-, B+: Keep the later discount,

but add and early-renewal incentive

E+, F+: Add more contacts – more e-mails

and compressed timing

D-: Skip the multi-year offer

A+: Change price (but how?)

18

-50%

0%

50%

100%

150%

200%

250%

-20% Control +20%

% c

ha

ng

e in

re

ve

nu

e v

s. c

on

tro

l

A: Price

Effect (A): Price change(change in Revenue versus control)

Magazine #1

Magazine #2

Magazine #3

Mosaic Offer Test – Effect of Price…

To maximize revenue

• For magazine #1,

increase price 20% (Revenue increases 19%

but response is 9% lower)

• Increase price 20%

for magazine #2 (Revenue increases 23%

but response is 22% lower)

• For magazine #3, remain

at the control price

19

-50%

0%

50%

100%

150%

200%

250%

-20% Control +20%

% c

ha

ng

e in

re

ve

nu

e v

s. c

on

tro

l

A: Price

Effect (A): Price change(change in Revenue by Segment)

#1 - Cancel risk

#1 - Loyal

#2 - Cancel risk

#2 - Loyal

#3 - Cancel risk

#3 - Loyal

… Depends upon segmentation

Differences by

Segmentation Model

• “Loyal” customers (defined by a combination

of statistical models)

Respond well

to higher prices

• Subscribers who are a

“Cancel risk” are better

served at the control or

lower price…

20

Because… Need to consider response

(not shown) along with revenue

Renewal rate drops up to 26%

for this high-risk segment

And we need to deliver promised rate base

Plus renewals are lower cost

than new subscribers

Summary of Mosaic Tests

• Tested 17 marketing-mix elements at once

• Quantified 10 significant elements

– Analyzing main effects, interactions, and “curvature”

in the price effect

• Pinpointed differences by segment, title, and

statistical model

• Tested in 9 months what would require more

than 3½ years with A/B splits

21

Conclusions & Insights

• Benefits of multivariate testing for Hearst retention marketing

­ Identified creative, offer, segmentation elements with most leverage

with some rolled out immediately, others fine-tuned via more tests

­ Low-leverage test elements removed from future test consideration

­ Additional tests to “fine-tune” high-leverage elements led to roll outs

New control series that outperformed existing control series of the

last 7 years (response lifts as high as 15%)

New model-based offer and segmentation strategy based on a

subscriber’s likelihood to renew or cancel

22

Risks & Recommendations

• Risks of multivariate testing for Retention efforts

­ Testing more elements increases risk of negative impact

­ Normally low risk test elements could be high risk when mixed with

other test elements to form counter-intuitive recipes needed for test

­ Control creative may not accommodate variety of recipes required

for testing, which may force additional creative changes and

response risk

• Recommendations on when to use multivariate testing

­ Statistically significant A/B test volumes can’t accommodate

all test elements

­ There’s a lot to test and results are needed quickly

­ The organization supports bold, high risk/reward testing

23

Accelerated Testing

for Retention and CRM Programs

Brian Fichtel

Senior Director, Customer Retention

Hearst Magazines

Gordon Bell

President, LucidView

Managing Partner, Artestry gbell @ lucidview.com

24