userzoom webinar: how to conduct web customer experience benchmarking

February 2010

www.userzoom.com

Our speakers for today’s webinar are:

1. Alfonso de la Nuez, Founder & CMO at UZ

2. Kim Oslob, Sr. Research & Product Strategy Director at

UZ

3. Ania Rodriguez, Principal at Key Lime Interactive

Welcome everyone!

Copyright 2010. Key Lime Interactive and UserZoom. All Rights Reserved.

www.userzoom.com

1. What do we mean by UX Benchmarking? Why is it

important?

2. How to conduct UX Benchmarking studies

3. Results you must have

4. Examples

5. UserZoom and Key Lime Interative at a glance

Five things we’d like to talk to you about today

Copyright 2010. Key Lime Interactive and UserZoom. All Rights Reserved.

www.userzoom.com

What is UX Benchmarking & why is it important?

Definition of Benchmarking:

The process of comparing the business

processes and performance metrics

including cost, cycle time, productivity, or

quality to another that is widely considered

to be an industry standard benchmark or

best practice.

It provides a snapshot of the performance

of your business and helps you understand

where you are in relation to a particular

standard.

www.userzoom.com


Definition of UX:

It’s about how a person feels about using a system.

It highlights the experiential, affective, meaningful

and valuable aspects of HCI and product ownership,

but it also covers a person’s perceptions of the

practical aspects such as utility, ease of use and

efficiency of the system.

User experience is subjective in nature, as it is

about an individual’s feelings and thoughts about

the system. User experience is dynamic, because it

changes over time as the circumstances change.

www.userzoom.com


Ok… So how do we define UX Benchmarking?

www.userzoom.com


Common UX research challenges

• Difficulties obtaining objective, statistically significant, actionable

data

• Obtaining idea adoption from stakeholders

• Geographic/demographic limitations

• Research is costly and time-consuming

www.userzoom.com


Some may wonder: Is UX Benchmarking…

?

www.userzoom.com


Not really.

Here’s how we’d define UX Benchmarking:

• The process of comparing a website’s performance, specifically in

relation to UX metrics, including among others effectiveness,

efficiency, overall satisfaction and quality perception, to another that

is widely considered to be an industry standard benchmark or best

practice

• UX benchmarking provides a snapshot of the performance of your

website and helps you understand where you are in relation to a

particular standard.

www.userzoom.com


Why is it now more important than ever before?

It’s all aboutcompetitiveness, productivity, &efficiency

www.userzoom.com


‘You can’t manage what you can’t measure’

Attributed to Peter Drucker, Mgt Guru

The question is:

How are you actually measuring UX and defining your benchmark?

www.userzoom.com

1. What is UX Benchmarking & why is it more important than

ever?



4. Examples



www.userzoom.com

How to conduct UX Benchmarking studies

www.userzoom.com


10 must-dos with UX Benchmarking

1. Define measurable & comparable UX-related KPIs (key performance indicators)

2. Select competitors to benchmark against

3. Define target audience

4. Identify common site activities/tasks (between-subjects design)

5. Conduct research using scenario-based testing

6. Gather both quantitative & qualitative data

7. Use the quantitative data to test for statistically significant differences between

means or proportions

8. Use qualitative data to support your findings and understand the why

9. Research at least twice a year to measure design changes and trends in

expectations

10.Use the right research solution/tool

www.userzoom.com


Graph from Bill Albert, Bentley University

www.userzoom.com


Our solution: UZ Self-Serve Edition

• On-demand software application that empowers businesses to manage & conduct online UX research

• Feature-rich, versatile* software solution, designed for task-based surveys (or unmoderated remote usability testing)

• Allows researchers to manage the whole study, including: create script, launch, collect data and analyze results

*Also good for online surveys, card-sorting and voice of customer surveys

16

www.userzoom.com

Once you have

- defined measurable & comparable UX-related KPIs (key performance

indicators)

- selected competitors to benchmark against

- defined target audience

Let’s walk through the process

Create a study script11

Launch the study, collect data, monitor progress22

Analyze results33

… the 3-step process is fairly simple

www.userzoom.com

How UserZoom works

Creating a study script using the UZ Editor 11

Creating a study is EASY

• Test structure built in

• Preview options

• Advanced scripting

• Ask questions depending on users’ online behavior

• Integrated user recruitment capabilities

• Pre-testing available

www.userzoom.com

Feature-rich, advanced scripting capabilities:

High-level of flexibility

Advanced task validation criteria (by question, by page interception, by

time...)

Advanced questionnaires tool (all type of questions: Choice, scales,

open...)

Insert multimedia (images, videos, flash, applets, sounds...)

Manage advanced logic/branching & conditions

Behavioral questionnaires (by interception)

Randomization: tasks, answers...

Piping capabilities

Mandatory questions

Time-out options

Sample & data quality control

Multi-lingual studies support (including Asian languages).

…and more

How UserZoom works


www.userzoom.com

How UserZoom works


Partners of

Choose the user recruitment yourself

• Panel

• Intercept

• Private mailing list

www.userzoom.com

Launch the study, collect data, monitor progress

How UserZoom works

22

• Hundreds of users can be tested

• Participation in the natural context…

• …from geographically spread locations

• No human moderation needed

• Our browser bar connects users with our secure servers

www.userzoom.com


ever?



4. Examples



www.userzoom.com

Analyze data real-time using UZ Analytics

Results to look for

33

Effectiveness + efficiency ratios

www.userzoom.com

Analyze data using UZ Analytics

How UserZoom works

33

Qualitative feedback collected. Easy visualization through ‘text clouds’

www.userzoom.com

Analyze data using UZ Analytics

How UserZoom works

33

www.userzoom.com


ever?



4. Examples of industry benchmark studies



www.userzoom.com

Example 1: Healthcase industry study (USA)

www.userzoom.com

Example: Healthcase industry study (USA)

www.userzoom.com

Study Design Introduction:

Based on what you know, how negative or positive is your image of PROVIDER?

Which of the following brand attributes, if any, do you associate with PROVIDER?

Post-task:

Self-Reported Success

Ease of Use

Satisfaction with amount of time to complete task

What did you learn that you previously did not know?

What did you like?

What did you dislike?

Problems or frustrations while completing task(s)

Wrap-up:

Overall satisfaction

Likelihood to purchase

Likelihood to recommend : (Net Promoter Score )

Post-task brand image

Post-study brand attributes

Security and Privacy Likes/Dislikes with site

IntroductionIntroduction

ScreenerScreener

N=200

AetnaAetna

n-=50

BCBSBCBS

Wrap-upWrap-up

Task 1Task 1

Task 2Task 2

KaiserPermanente

KaiserPermanente

UnitedUnited

Task1:Task1:

Task 2Task 2

Task 1Task 1

Task 2Task 2

Task 1Task 1

Task 2Task 2

n-=50 n-=50 n-=50

www.userzoom.com

Brand Perception: Pre/Post

All sites had a positive lift in brand perception.

+22%

+14%

+2%

+20%

www.userzoom.com

Brand Pillar Metrics:Pre vs. Post Differences

United showed a 26% increase on the ‘helpful’ attribute and maintained “professional” while other sites had a decrease on these two attributes.

All sites showed a sharp increase in the “rude” attribute.

www.userzoom.com

Overall Satisfaction

Panelists were most satisfied with Blue Cross Blue Shield and Aetna’s site when researching and getting rates for Medicare Supplemental Insurance.

Differences of 17% or greater are significant at 90% CI

www.userzoom.com

Net Promoter Score (NPS)

0 1 2 3 4 5 6 7 8 9 10 NPSAetna

12% 8% 4% 2% 10% 12% 14% 8% 18% 8% 4%-50%

BCBS 8% 4% 6% 10% 2% 8% 14% 10% 4% 10% 24% -18%KP 18% 6% 8% 4% 4% 6% 6% 16% 8% 8% 16% -28%United

10% 4% 6% 6% 4% 22% 16% 8% 10% 8% 6%-54%

When asked if they would recommend PROVIDER to a colleague or friend, all sites had relatively low NEGATIVE NPS with Aetna and United Healthcare being the lowest.

Detractors Promoters

NPS% of Promoters

(9s and 10s)Detractors

(0 through 6)= -

www.userzoom.com

Task 1: Learn & Explore

Aetna BCBS

Kaiser Permanente United

The success rate for the “learn and explore” task, although the frustration rate was above 40% for all sites showing room for improvement.

www.userzoom.com

Usability Metrics: Learn & Explore

Blue Cross Blue Shield had the highest mean usability scores for task 1.

www.userzoom.com

Blue Cross Blue Shield Liked Best:Learn & Explore

Ease of navigation:

“I liked the ease of finding information directly relating to the Blue Shield’s Medicare Supplement insurance.”

“it was all user friendly....I think even people who don't use web sites a lot would be able to navigate”

Comparison charts:

“clearly divided options for each plan”

“the graphic showing the differences between the supplemental plans”

www.userzoom.com

Top Frustrations: Learn & Explore

About half of the panelists on all sites had some sort of frustration with learning about Medicare Supplemental Insurance online.

Opportunities

www.userzoom.com

Example 2: Used cars websites industry study (Spain)

www.userzoom.com

Example: Used cars websites industry study (Spain)

USERS SAMPLE

• 50 users in their natural context of use (home).• Users have medium/high level of internet experience• Over 18 years old• Users use or have used Internet as a means to finding information about

how to buy a second-hand car

TESTED WEBS

• Supermotor.com• Autocity.com• Autoscout24.es• Coches.net• Segundamano.es

METHODOLOGY

• Remote testing using UserZoom technology with real end users. • ‘Task-based analysis’: users must try to solve real tasks• Field work dates: November 2008• Place: Online, users in Spain

www.userzoom.com

Tasks Description

1. Free search of a car model Users search freely on the Internet for any car model

Note: 50 users completed this task

2. Guided search for an specific car model on a selected website

The task requests users to search for an specific car model, on each of the second-hand cars web sites.

Note: 10 users per website evaluated


www.userzoom.com41

How do users search for a second-hand car on the Internet?

%

I have used search engines 61

I have surfed just by directly inserting the Website name of the page

21

I used both methods 19


www.userzoom.com

WEBSEffectivene

ss

Efficiency

Time

Clicks

Autocity.com90%

3:31' 21

Autoscout24.es 80%

2:08' 29

Coches.net90%

4:00' 57

Segundamano.es 90%

2:22' 24

Supermotor.com 60%

3:55' 45


www.userzoom.com

Easiest Website to search for a car

Autocity.com

Autoscout24.es Coches.net

Segundamano.es

Supermotor.com Media

It is easy to start the car search

5.9 6.6 6.9 6 5.5 6,2

The searcher fields are easy to fill up and understand

6 6.6 7 6.3 6.2 6,4

The searcher has the requiered options to do a search

5.3 6.6 6.8 5.5 5 5,9

Important information as price, car age, milage, model, is available

6 6.6 6.8 6.2 4.7 6,1


www.userzoom.com44

Task analisys: Main problems

Autocity.com

Autoscout24.es Coches.net

Segundamano.es

Supermotor.com Average

I have the required information to make a buying decission

5.2 5.1 6.7 5.3 5 5,5

Webpage shows car features, extras and accessories

5.3 5.7 6.5 5.2 4.8 5,5

I can clearly see the final price and included concepts

6 6.1 5.4 5.9 5.8 5,8

Photographs help to have a clear idea of the car.

6 6.2 6.6 5.9 4.2 5,8

I have all the information to contact the seller

5.5 6.1 6.4 5.5 5.6 5,8

Too much advertising quantity

4.9 5.4 6.1 5.5 4.2 5,2


www.userzoom.com

Example 3: Continuous Benchmarking (Marketing + Usability)

www.userzoom.com

Continuous benchmarking (Marketing + Usability)

Online scorecard

www.userzoom.com


www.userzoom.com 48


www.userzoom.com 49


www.userzoom.com 50


www.userzoom.com


ever?



4. Examples



www.userzoom.com

UserZoom at a glance

Leading online user experience research software company

Developes a proprietary on-demand software application for online

research

Built for & by UX & marketing professionals

Offers a cost-effective, time-saving methodology

Has 9 years of experience in UX research & consulting

Has offices in Sunnyvale (CA), London (UK) and Barcelona (Spain)

www.userzoom.com

Key Lime Interactive at a glance

Experienced market research and usability professionals Over 10 years of consulting experience

testing Internet 500 websites and providing actionable results

Most consultants have Masters in Human-Computer Interaction, Psychology, Industry Engineering or related field

Cross-Industry Expertise Ability to scorecard against competition Ability to suggest cross-industry

recommendations to improve ease of use

Independent Third Party Impartial with no-hidden agenda Ability to work with business team and

developers to “get everyone onboard”

Technology-agnostic Can work with any solution or recommend

the best solution to meet your specific needs

Expert Reviews

www.userzoom.com

What makes us proud?

www.userzoom.com

Questions?

www.userzoom.com

Would you like a demo?

Contact us and we’ll offer you a chance to see and try out our technology

Do you have any questions?We’re sure you have many questions. Don’t hesitate to contact us and we’ll be glad to answer them.

USA Office 440 N. Wolfe Rd. - Sunnyvale, CA 94085 (USA) · [t] +1 (408) 524 7445

UK Office AylesburyHouse, 17-18 AylesburyStreet, London EC1R 0DB(UK) · [t] + 44 (0)20 7193 2171

Spain Offices Av. Diagonal 419, 3º 2ª - 08008 Barcelona - T. +34 93 414 75 54 - F. +34 93 368 42 95 Gran Vía 62, 7º Izda. - 28013 Madrid - T. +34 91 542 62 52 - F. +34 91 298 08 92

Contact us at [email protected]

Members of