mastering the art of experimentation - tech circus€¦ · the lean ux process - jeff gothelf. a/b...

MASTERING THE ART OF EXPERIMENTATION

Chris Gibbins - Director of UX & Optimisation

Biglight.co.uk | @cjgibbins | @biglightbuzz

1

The future belongs to brands that provide remarkable customer experiences.

We exist to help them do just this.

SOME OF OUR CUSTOMER EXPERIENCE OPTIMISATION CLIENTS

WORKSHOP AGENDA

15:40

15:50

16:10

16:20

16:55

17:20

18:00

START

INTRODUCTION TO EXPERIMENTATION

THE EXERCISE

- CUSTOMER PROBLEM 1

- CUSTOMER PROBLEM 2

- CUSTOMER PROBLEM 3 (If we have time)

END

What about you?

What comes to mind when you hear the word

“TESTING” “EXPERIMENTATION”

INTRODUCTION TO EXPERIMENTATION

CREATIVE UX

INSIGHT

EXPERIMENTATION

1 Learn what really works for users

4 Iterate as you can’t get everything right in your first attempt

5 Measure the ROI of all our work

2 Be more adventurous in our ideas / innovate

3 Fail fast6 Become better UXers and Designers

EXPERIMENTATION IS KEY TO CONTINUAL IMPROVEMENT OF THE UX

“The most important single thing is to focus obsessively on the customer. Our goal is to be earth’s most customer-centric company.”

Jeff Bezos, CEO Amazon

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day,”

Jeff Bezos, CEO Amazon

OPTIMISATION / PERSONALISATION TOOLS

AB

C

Off-the-shelf solutions e.g. Develop your own e.g.

LIVE EXAMPLE

Control The original website

Variation 1UI enhancements to filters

Variation 2Horizontal filter layout

THE LEAN UX PROCESS - JEFF GOTHELF

A/B Testing live on the

website

Roll-out winning experiences with a much greater degree of confidence

+Calculate the uplift and ROI

Design & Build the A/B test variations

Several concepts at

a time

OUR PROCESSES ARE NOT THAT DIFFERENT

Last 4 months of UX and Optimisation activities for a client with 3 websites

35 A/B tests completed

Each one had on average of 3 design variations plus the control

12 significant winners (33% win rate)

TYPICAL PROGRAMME SHOWING HOW WE ITERATIVELY IMPROVE THE CUSTOMER EXPERIENCE

The Experimentation Exercise

Explore and investigate the customer problem making sure that you are clear on what the issues are.

Start sketching out ideas to solve the customer problem. Try to come up with at least 8 in total

Refine down to 4 ideas / variations which could be tested as an experiment. Be extra brave with the 4th ‘wildcard’ one!

1) Customer problem 2) Crazy Eights 3) Experiment design

10 mins

15 mins 10 mins

THE EXERCISE - 3 CUSTOMER PROBLEMS TO SOLVE

4) Present back

A couple of you to talk through the ideas

10 mins

IDENTIFYING CUSTOMER PROBLEMS AND OPPORTUNITIES

IDENTIFYING CUSTOMER PROBLEMS AND OPPORTUNITIES

Long product listing pages and too much choice make it hard for users to find their ideal product.

The customer problem in detail:

- Website example: COS www.cosstores.com

- Usability testing has shown time and again that users struggle when there is too much choice on very long product listing page

- User Interviews have revealed a need for more direction and also more inspiration to help users choose especially when there are a large number of products in a category

- In store research has revealed how staff members guide users to a smaller range of products that better fit the users’ needs.

The challenge

- How do we improve the overall experience on the product listings pages to help users more easily find their ideal products?

CUSTOMER PROBLEM 1.

90x Men’s Knitwear items

Takeaways- If you UXer or Product Designer who wants to make a measurable

difference, you really need to embrace online experimentation (e.g.

A/B testing)

- Push yourself and go beyond the obvious solution… be more

adventurous and you might just stumble across a solution that 10x

better for users.

- Be a little cautious of the phrase ‘best practice’. Often it’s used by

people wanting to push through their solution!

THANK YOU

Biglight.co.uk | @cjgibbins | @biglightbuzz

23