going beyond ab testing - unbounce · going beyond ab testing . how to connect customers to...

Going Beyond AB Testing

How to Connect Customers to

Experiences

1. Decision Rules as a Framework

2. Picking Rules -> Optimization

3. Multi-Armed Bandit

twitter: @mgershoff

Blog: www.conductrics.com/blog

Overview

twitter: @mgershoff

Bandity Tools

twitter: @mgershoff

Bandity Tools

twitter: @mgershoff

Bandity Tools

twitter: @mgershoff

Bandity Tools

Decision Rules

Decision Rule

IF [THIS]

Decision Rule

IF [THIS]

THEN [THAT]

A Logic that links

Observations to Actions

Why Decision Rules?

Decision Rule: General Form

Conductrics twitter: @mgershoff Blog: www.conductrics.com/blog

If [Observed Data]

Then [Causal Action]

Decision Rule: General Form

If [Anyone]

Decision Rule: Simple Test

If [Anyone]

Then [A or B]

Decision Rule: Simple Test

If [Unbounce User]

Decision Rule: Targeted Test

If [Unbounce User]

Then [ or ]

Decision Rule: Targeted Test

Two Types of Data

• Day Part

• Weekend/Weekday

• Mobile/DeskTop

• Browser Type

• User Age

• Geo/Census

• Weather

• Tenure/RFM Score

Observed Data

• The Button – Ha!

• Price

• Sales Offers

• Shipping Type

• Layout/UX

• Which Products

• Suggested

Quantities

Causal Action

Learning Efficiently

Apply ‘Best’ Rules

Optimization is Balance

How do we pick the rules?

But …

If THIS Then THAT

If THIS Then THAT If THIS Then THAT

If THIS Then THAT

If THIS Then THAT If THIS Then THAT

If THIS Then THAT

Learn From Domain Experts …

* Image Source http://en.wikipedia.org/wiki/Anna_Wintour

…and from Experiments

* Image Source http://en.wikipedia.org/wiki/Long_Ashton_Research_Station

22 matt@conductrics.com; www.conductrics.com

Multi-Armed Bandit Experiment

Which is Better to Experiment?

"DevonLarratt-TarasIvakin" by Above99 - Own work. Licensed under CC BY 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:DevonLarratt-TarasIvakin.jpg#/media/File:DevonLarratt-TarasIvakin.jpg

Source: http://www.startrek.com/article/ten-for-ward-10-favorite-kirk-fight-scenes

Experimentation Review

The A/B

Anyone

Simple AB: Data Collection

Anyone

Simple AB: Evaluation Phase

1) Calculate Average/Means of A & B

2) Calculate Spread/Variance in A & B

Mean(B) – Mean(A) t=

4) Use evidence to make a selection

(Or favorite Test)

Simple AB: Evaluation Phase

Simple AB: Apply Rule

If [Anyone]

Offer B Then

How Does it Work?

Search in Most

Promising Areas

A measure of Uncertainty

Mean - 2*Std +2*Std

Recall Confidence Interval

Mean +Bonus

Score each option using the upper

portion of the interval as a Bonus

Upper Confidence as Bonus

Conversion Rate

5% 10%

1) Use upper portion of CI as ‘Bonus’

Select A

2) Select based on highest ‘Score’

0 5% 10%

1) Selecting Action ‘A’ reduces uncertainty

bonus (because more data)

Select C

2) Action ‘C’ now has highest score

Conversion Rate

When MAB?

• Headlines

• Flash Sales

Adaptive/Bandit AB Test

Is Learning Perishable?

Yes! Not Really

• Site Redesign

• Many Options

• Many Tests

• Personalized

• Top ‘k of M

Complex Problem?

Yes! Not Really

• Few Options

• Few Tests

• No/Simple Targeting

• Single Option

Pfizer Use Case The ‘IF‘ Part

• Three Different User Segments

• Wyeth Pharma Search Segment

• King Pharma Search Segment

• All others Segment

Conductrics Confidential

Pfizer Use Case

The ‘Then’ Part

• Best Content Topic for each Segment

• Best Navigation UI

The ‘IF‘ Part

• Wyeth Pharam Search Segment

Pfizer Use Case

The ‘Then’ Part

• Best Content Topic for each Segment

• Best Navigation UI

The ‘IF‘ Part

• Wyeth Pharam Search Segment

The ‘How’ Part

• Test Bandit Approach

Pfizer Use Case

Topics

1)Popular: Default 2) Wyeth Related 3) King Related

Investor Navigation

1) NavA: Default

2) NavB: Box

3) NavC: Tab

Create Agent

Create Segments

Wyeth Interested

King Interested

Top Option 8 times more often than the default

Results: Wyeth Segment

Tab Box Tab Box Tab Box Tab Box

Wyeth King

Bandit Random

Confidence Intervals MAB & AB C

Wyeth King

Bandit Random

High Value Option-> Smaller CIs

Wyeth King

Bandit Random

Low Value -> Higher Uncertainty C

Wyeth King

Bandit Random

Low Value -> Higher Uncertainty C

All Others

Wyeth Searchers

King Searchers

Content Options

Navigation Options

• For the segments - the content selection had a greater impact

• For ‘all others’ – the Navigation selection had the most impact

Nav or Topics: Impact

1. Automation

2. Changing World/Perishable

3. Complexity / Scale

When to Use Bandits

See Russell & Norvig ‘s Artificial Intelligence: A Modern Approach

Optimization

Just 5 Parts

Measureable Goals

Ability to ‘Sense’ the Environment

Ability to Act/Execute

Observe Outcomes

Play ‘Better’ Options More Often

If Then Receive

Learn Decision Logic

Wake up. We are Done!

Twitter:mgershoff Email:matt.gershoff@conductrics.com

Image source: http://www.washingtonpost.com/blogs/london-2012-olympics/wp/tag/soccer/ (Andrew Medichini — Associated Press)

going beyond ab testing - unbounce · going beyond ab testing . how to connect customers to...

Documents

going beyond

bsc going beyond traditional metrics

going beyond poverty

the benefits of going beyond conventional therapies for adhd...

going beyond grammar ii

going beyond iso

jonah - going beyond

going beyond simple sample size calculations: a ... ·...

going beyond our club (pdf)

going beyond leed

going beyond evaluation buzzwords

going beyond simple social

going beyond the expected

going beyond code

makeup behind the mirror; going beyond delisting? { a · pdf...

going beyond google analytics

going beyond the degree_li_sm

going beyond the gate

going beyond the music

going beyond grammar