A/B Testing

for Mobile AppsTechniques and Opportunities

Roberto Carli, Apptimize

1

Aristotle~340 BC

Galileo1589

Aristotle~340 BC

Galileo1589

Aristotle~340 BC

What is A/B Testing?

Using randomized, controlled

experiments to improve an

outcome of interest

The golden standard for

customer-facing innovation

1948

1994

2001

2004

2013

Agenda

What to test

in Mobile

Making the

right

decision

A/B testing

architecture

1 2 3

What Works Well

Call to

Action

Engagement

UI

Call to Action

Original Variant A Variant B

40

%

29

%

Engagement UIOriginal Variant

14

6%Fewer profiles

“liked”

11%Loss in revenue

over a year

26%Loss in growth

month over month

Other Things to Test

15

Login Screen Flows

Menu & Features Page layout Message & Copy

Upgrade &

Purchase

Agenda

What to test

in Mobile

Making the

right

decision

A/B testing

architecture

1 2 3

Flow of A/B Testing

What to Measure

Reaction Funnel Long-Term(Engagement

& Retention)

Variant: no skip!Original Reaction Metrics

Additional Signups

Lost Users

Variant: no skip!Original Funnel

Sign In / Sign Up

Item Detail

Add to Cart

Checkout

Purchase

Variant: no skip!Original Long Term

Return Rate

Number of Sessions

Time in app

Etc.

Agenda

What to test

in Mobile

Making the

right

decision

A/B testing

architecture

1 2 3

Mobile – Why It’s Hard

23

Instant At Least

3 Weeks

Programmatic A/B Testing

Dynamic Variables

Real-Time

Orig = UPGRADE NOW

Var A = UNLOCK FOR

LIFETurn on Var

A

Declare Var CTA;

Default = UPGRADE

NOWSet CTA =

UNLOCK FOR

LIFE

Edit UILabel

-> UNLOCK FOR LIFE

One Time

Agenda

What to test

in Mobile

Making the

right

decision

A/B testing

architecture

1 2 3

Questions?