appiterate - saif ignition
DESCRIPTION
Appiterate held a workshop at SAIF Ignition (held at Bangalore on 20th Sep 2013) on A/B testing for mobile apps. It had a look at pre-existing practices in web A/B testing and also extensively covered mobile A/B testing.TRANSCRIPT
TEXT
SAIF Ignition
Bangalore
20th Sep 2013
A/B testing for mobile apps
WHAT IS A/B TESTING?
Version ‘A’ vs Version ‘B’
That’s how the name A/B testing came about.
How does it work?
WEB A/B TESTING - Etsy
• In-house A/B testing solution
• Results are sometimes surprising!
• Infinite scroll is presumed to be a better experience as you see more items faster
• Visitors clicked on
• fewer results (-11%)
• favourited less (-8%)
• fewer items purchased (-22%)
INFINITE SCROLL
Infinite
Scroll
Source – Design for continuous experimentation, http://www.slideshare.net/danmckinley/design-for-continuous-experimentation
WEB A/B TESTING
• What went wrong?
• Changed multiple things at once
• They should have broken-down the feature change into two hypothesis testing validations:
• More items is better
• Faster is better
INFINITE SCROLL
WEB A/B TESTING
• Broke-down the process into testing multiple short measurable tests
• Decrease the number of items in dropdown
• Default to ‘all items’
• Search bars for in-shop and in-page search
• Rich auto-suggest
• .. and many more such tests!
SEARCH DROPDOWN
WEB A/B TESTING
• Most people did not notice the major feature changes
• Search dropdown was subsequently withdrawn
• Iterative ‘Design – Develop – Measure’
SEARCH DROPDOWN
Design Develop Measure
Infinite Scroll
Search Dropdown
Design Develop Measure
MOBILE A/B TESTING – Why is it needed?
• Increase conversion, revenue, engagement metrics
• User-segmented and targeted app feature launches
• Most mobile apps struggle to make money - faster iterations will genuinely help developers in monetizing their apps
MOBILE A/B TESTING – Challenges
• Complex coding involved
• Delays in app store updates -
• slowing the pace of innovation
• Simplicity and ease of web A/B testing not prevalent on mobiles -
• thus devoid of 'iterative improvements‘
• key growth drivers on the web
WHY A/B TEST MOBILE APPS?
• Everyone A/B tests in real-life:
• People try two or more pieces of clothing before buying
• You have a test drive of two shortlisted cars
• The HiPPO (the Highest Paid Person’s Opinion) method of decision making
• Decisions based on such opinions, hunches, guesses and conjectures
• Use data in decision making
• A/B testing on mobile apps is a great way to increase conversions and improving your key mobile metrics.
WHAT TO A/B TEST IN NATIVE MOBILE APPS?
If it is there in your app, it can be A/B tested!
Example use cases:
• Call to Action buttons (these include buttons like ‘buy now’, ‘add to cart’, ‘sign up’)
• Set any value to the text, size, colors, fonts or its position
• App features – Test engagement metrics by having different number of free levels in games
• User experience and app navigation sequence tests
• Entire screen themes/layouts
• Different page opens on tap of a button
• Different ways of social media credential-based sign-ins
Aim was to increase
conversion on the following:
•Review button – User
engagement metric
•Price link – Increases
revenue in the form of
referral fees.
VARIATIONS: Control Variation
DESCRIPTION: A review button with longer
text No visual element (arrow) beside the price
A review button with shorter text A small arrow beside the
price
DISTRIBUTION: 50% 50%
RESULTS:
Price link clicks on 18.1% of screen opens
Price link clicks on 22.4% of screen opens
Longer text button got clicked 1.4% times of screen
opens
Shorter text button got clicked 1.1% times of screen opens
The app lets users compare prices of multiple cross-category products from online stores.
Hypotheses:
•A short or a long text on a
‘review’ button
•Impact of a small visual
element (arrow) beside the
price
Price link with a visual element resulted in an increase of 23.4% in revenues for the app!
The longer text on the review button is better at driving engagement by 23.1%
CASE STUDY: PRICE COMPARISON APP
TIME FOR DEPLOYMENT & ITERATION fast slow
in-code test (normal)
PR
E S
ET
UP
TIM
E
•Integrate SDK •Pre-specify tests •Pre-specify attribute values
mo
re
less
COMPARING TEST TYPES (1/3)
TIME FOR DEPLOYMENT & ITERATION fast slow
in-code test (normal)
data driven test
PR
E S
ET
UP
TIM
E
•Integrate SDK •Pre-specify tests •Pre-specify attribute values
mo
re
less
COMPARING TEST TYPES (2/3)
•Integrate SDK •Pre-specify tests •Change attribute values live
•Bypass app store updates •Drag-n-drop visual interface •Fast and easy deployment
wysiwyg
TIME FOR DEPLOYMENT & ITERATION fast slow
in-code test (normal)
data driven test
PR
E S
ET
UP
TIM
E
•Integrate SDK •Pre-specify tests •Change attribute values live
•Integrate SDK •Pre-specify tests •Pre-specify attribute values
mo
re
less
COMPARING TEST TYPES (3/3)
Live Tracking of User’s Interaction with the App
Content Management System Feature Launches and Segmented Rollouts
NEW FEATURES
“If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps innovation is 1% invention and 99% iteration.”
www.appiterate.com