concept validation testing jessica richardson digital optimization consultant, accenture...
TRANSCRIPT
Concept Validation Testing
Jessica Richardson Digital Optimization Consultant, Accenture
Concept Validation:
• Testing to see if the concept makes an impact, and validate the priority
• Change that would normally require a large dev effort, but use testing to prototype the concept without actually implementing
• The design does not have to be perfect, just good enough to get the idea across
How are we organized?
AB TeamLocated: Bangalore
AB Lead(Yours Truly)
Client Product Teams
106 tests per year32% win rate
Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-150
2
4
6
8
10
12
14
16
0%
5%
10%
15%
20%
25%
30%
35%
40%
14%13%
0% 0%
13%
10%
33%
22%
29%
11%
38%36%
Concept TestingOptimization Design/UX Decision New idea % New Idea Tests
Organized as Product Teams
Lesson 1:"Free" and “easy” doesn’t always mean you should build it.
“Let’s run a pilot”
Will this additional information increase add to cart?
Result: Add to Cart Rate = not statistically different
Product Category Diff in Add to Cart
Laptop +9.7%
Monitor -11.3%
Switch Inconclusive
Keyboard/Mouse -6.6%
Software Inconclusive
Printer -36.3%
Lets look at it by product category….
https://www.usertesting.com/videos/KwA8aJB3b9lqc5QFQVd6xQ/clips/714750?back=study_1516692
We asked a few users......
No one expected to see the content that was actually behind the +
icons.
Users felt that the + icons had random
information.
Not one user said they would buy the product
based on the info in icons.
Lesson 2: Real customer feedback is invaluable.
Run User Testing
Discover User Issue
Develop Hypothesis
Users are leaving the site to find more information about the product, like videos.
Providing videos to show the look and feel, will reduce exit rates due to users leaving the page in user testing.
1. Where are we going to get these videos?
2. How are we going to source this?
3. The dev effort to build is high……
“Let’s run a pilot to prove if its worth building at all. We can manually source the videos for a subset of SKUs.”
Potential solution = a lot of questions
The Test…
-31% drop in bounce rate
-27% drop in exit rate
Lesson 3: Powerful testing can be done when you work closely with your dev team.
Client vs. vendor recommendations: What will increase revenue gains?
Results: Total categories increased but the overall amount of items sold wasn’t different
compared to the original
What’s missing?
Round 2: Added “add to cart” buttons
+150% increase in items sold
+83% lift in Sales
Lesson 1:"Free" and “easy” doesn’t always mean you should build it.
Lesson 2: Getting real customer feedback is invaluable.
Lesson 3: Powerful testing can be done when you work closely with your dev team.