a/b testing mobile apps at reigndesign

8/14/2019 A/B Testing Mobile Apps at ReignDesign

1/22


2/22

1 - About A/B Testing

What is A/B Testing? (pg. 2)Examples of A/B Testing for Web (pg. 3)

Benefits of A/B Testing for Apps & Games (pg. 6)

2 - Case Study: Spot the Difference With Friends!

About Spot the Difference With Friends! (pg. 8)Driving More IAP Conversions (pg. 9)

Optimizing Level Difficulty (pg. 10)

Time Penalties: A Hypothesis Reversed (pg. 11)

3 - Using Splitforce

Getting Started with Splitforce (pg. 15)

Launching an Experiment (pg. 16)

Interpreting Results (pg. 19)

Splitforce, Inc. - http://www.splitforce.com- All Rights Reserved.
http://www.splitforce.com/http://www.splitforce.com/


3/22

1 - About A/B Testing



4/22

What is A/B Testing?

A/B testing is the practice of publishing multiple versions of your

product at the same time in order to determine which version isdriving more desirable user behavior.

A/B testing is a decision-making system used to affect incrementalimprovements to your product over time, and is practiced byGoogle, Amazon, Facebook, Zynga, Netflix and more.

By collecting and analyzing data on how new features or designsare affecting user behavior, you can rollout changes to yourproduct that have been proven to improve your key businessmetrics.



5/22

Examples of A/B Testing for Web

Question: Which of the following lead generation formsresulted in a higher completion rate?

Version A Version B

Answer: Version B resulted in a 114% improvement in the rate

at which users who saw this form completed it and clicked thesearch button.

Even subtle changes such as changing the text of a selling point, asin this example, can have a dramatic impact on your businessbottom line. In this case, the improvement in the completion rateled to more 114% more leads and revenue for this business.

Oftentimes, observed user behavior can contrast heavily withwhat we believe will occur based on intuition. In this case, manymarketers may be surprised that the version with the wordFREE underperformed. A/B testing helps validate suchassumptions to make smarter business decisions.



6/22

In 2008, the Obama for President campaign raised an additional$60 million through A/B testing this website landing page:

Obamas web analytics team testing four different buttons:



7/22

And six different media:

And this was the winning combination:

+40%increase in conversion

rate________

2.9 millionadditional donators

________

$60 millionvalue of additional

donations



8/22

Benefits of A/B Testing for Apps & Games

Web optimization and A/B testing is great. But for release-

software where new code cant be pushed everyday, A/B testingrequires a new approach. Lets look at some of the benefits:

Drive More Revenue

Test different price points, offers, conversionfunnels, call-to-actions and more tooptimize the rate at which free usersbecome paying customers and increaseaverage revenue per user, average checkout

size and customer lifetime value.

Increase User Stickiness

Improve the average session times,

play counts, or virality of your gameby testing anything from userinterface layouts to game mechanics,and in-app messages to pushnotifications.

Improve ROI

Maximize the return on your customeracquisition spend and optimize post-download user activation rates by testingdifferent signups flows, tutorials, or access

to beginner features.



9/22

2 - Case Study: Spot the

Difference With Friends!



10/22

About Spot the Difference With Friends!

Spot the Difference With Friends! is the #1 Kids

Game in the UK, Singapore, Philippines, andMalaysia. It is produced and published byReignDesign, a Shanghai-based app studio.

Players canchallenge theirFacebookfriends to spot

the differencebetween setsof picturesagainst theclock.

And engagewith the largercommunity by

trying to getto the top ofthe leadersboard.

ReignDesign worked with Splitforce to run a series of A/B testselements of user experience and rules of the game, resulting inincreased ad revenues and in-app purchase conversion rates.



11/22

Driving More In-App Purchase (IAP) Conversions

With the increasing popularity of free-to-play and users clear

preference for ad-free apps, IAPs have become a popular way forgame and app publishers to earn revenue.

In ReignDesigns game, new players are given 6 stars, which act asa virtual currency that is spent during gameplay. One star is spentfor each round of play. When players run out of stars, they can nolonger play the game until they buy more.

Initially, ReignDesign had included an in-app notification whichencouraged users to go to the in-app store and buy more starswhen they tapped on the Play button but had no stars.

This in-app purchase funnel was not performing well, with about0.5% of users actually converting to paid customers. ReignDesigndecided to test another funnel, in which the in-app notificationwas removed and users were sent directly to the in-app store.



12/22

Original New

The new funnel (LaunchBuy) converted free users into paid user

at 1.12%, representing 125% increase in ReignDesigns IAPconversion rate.

Optimizing Level Difficulty

When designing a social game, lots of questions are tough toanswer without data.

In particular, ReignDesign wasnt initially sure how to set theprogression of level difficulty. Make the game too easy, and peoplemight get bored. Make it too hard, and people might becomefrustrated.

Instead of guessing at the optimal difficulty levels, ReignDesigntested three different variations of game difficulty progression:

Variation 1: Easy - Medium - Hard (EMH)

Variation 2: Easy - Medium - Medium (EMM)

Variation 3: Easy - Easy - Medium (EEM)



13/22

In order to measure the difference in user behavior whenexposed to the different level difficulty progressions, ReignDesignimplemented quantified goals to log the total number of levelsplayed by users for each variation.

The result: users that were exposed to the third variation (Easy -

Easy - Medium) played an average of 1.76% more levels thanusers exposed to the baseline.

This had a direct impact on ReignDesigns ad revenues, as ads areshown in between levels and gameplay.

Time Penalties: A Hypothesis Reversed

In the single-player version of ReignDesigns game, a 3-secondtime penalty is deducted when players make an error.



14/22

The game design team believed that this time penalty woulddiscourage users from playing more in the multiplayer version, butdecided to test and see if that was indeed the case.

After running atest comparingtime penaltyversus no timepenalty in themultiplayerversion, the dataclearly showed

that users witha time penaltyplayed 6.3%more rounds

than users without.



15/22

In this case, running an A/B test prevented ReignDesign fromremoving a feature that they believed would have a negativeimpact on user engagement. Through testing they discovered that

the time penalty feature was actually good for engaging users,which was a result contrary to their initial business intuition.

The increases in play time as a result of these tests were reflectedin additional ad revenue earned from the Spot the Difference

With Friends! game since launch.



16/22

3 - Using Splitforce



17/22

Getting Started with Splitforce

In order to A/B test your app or game with Splitforce, you will

first need to setup a connection between your app and ourservers by integrating the Splitforce library.

Step 1:Download and install the Splitforce library for either iOS(http://spl it force.com/ios/download) or Unity (http://splitforce.com/unity/download).

Step 2: Setup the library in your apps global header (.pch) filefor iOS, or in the Assets directory of your Unity project.

Step 3: Initialize the library using your unique Splitforce App IDand App Password.

For iOS:

For Unity:

http://splitforce.com/unity/downloadhttp://www.splitforce.com/http://www.splitforce.com/http://splitforce.com/unity/downloadhttp://splitforce.com/unity/downloadhttp://splitforce.com/unity/downloadhttp://splitforce.com/unity/downloadhttp://splitforce.com/ios/downloadhttp://splitforce.com/ios/download


18/22

Launching an Experiment

Once youve setup a line of communication between your app

and Splitforce, you can start configuring and publishingexperiments.

Splitforce works by replacing hard-coded elements in your appwith dynamic elements that can be served configuration data on-

the-fly from the Splitforce servers.

In this example, we have setup two variations testing both the

button text and the level of gravity in the app:

You can also track different types of experiment goals, includingrate goals (like tap-thru or conversion rates), timed goals (likeaverage session lengths) or quantity goals (like play counts). In thisexample, we set a rate goal for tap-thru on a button:



19/22

The experiment and goal code snippets are generated accordingto the experiment settings. Copy and paste experiment snippetsto replace the hard-coded elements in your app where theexperiment will occur.

iOS experiment snippet:

Unity experiment snippet:



20/22

Copy and paste goal snippets where the goal is actuallyhappening.

iOS goal snippet:

Unity goal snippet:

Thats it! You are ready to compile and submit a fully testableapp to the app stores.

If you want to setup a new variation, you can do so at any time inthe future without having to alter your code, recompile orresubmit to the app store through the experiment setup

interface.

When new users launch your app for the first time, the Splitforcelibrary will place them into a cohort and show the user variationdata for that cohort. Variation data will then be cached locally on



21/22

the users device, and they will always see that variation until youdeclare a winner for the experiment.

At the end of each code snippet is a default variation which youcan configure to show users who are not connected or have aslow connection the first time they launch your app. Users in thedefault variation cohort will also be tracked, and the defaultvariation data will be persistently applied to them in the same way

that non-default variation data is applied to other cohorts.

Interpreting Results

For every A/B test that you run with Splitforce, you will have aresults dashboard including a set of analytics that will help youunderstand how your test variations compare in terms of drivingdesirable user behavior:

1 - Goals:number of times a goal is reached.

2 - Unique Conversions:number of unique users whoreached a goal at least once.

3 - Sessions: number of times your app was run.

4 - Unique Users: number of users who ran your app.

5 - Goals per Session: number of goals reached / thenumber of sessions run

6 - Goals per Unique User: number of goals reached / thenumber of users who ran your app



22/22

7 - Average Session Time: amount of time users spendrunning your app per session on average

8 - User Retention Rate: number of DAU on average as apercent of total unique users

9 - Observed Improvement: percent increase in themetric for one variation over the baseline

10 - Chance to Beat Baseline: probability that one

variation will outperform the baseline over the long-term

The metrics that are displayed in your test results page will bedependent on what kind of goals you are tracking.

The results page also allows you to declare any given variation asa winner and roll it out to your entire user base. Itsrecommended that you do not set any variation as a winner untilstatistically significant results have been collected for the test, asindicated by the green color of the Chance to Beat Baseline.

_____

Sign up at http://splitforce.comto get started testing today!

S li f I h // li f All Ri h R d
http://splitforce.com/http://www.splitforce.com/http://splitforce.com/http://splitforce.com/

a/b testing mobile apps at reigndesign

Documents