webinar: retail ux benchmarking
DESCRIPTION
In this webinar we covered the results of a recent Retail UX Benchmark Study that included Nordstrom, Neiman Marcus, Saks Fifth Avenue and Lord & Taylor. Nate also gave tips on how to create your own successful competitive benchmarking study.TRANSCRIPT
Webinar: Retail UX Benchmarking: Which Brand Provides the Best Website Experience?
Nate Colker| UserZoom12/13/2012
Intros
Alfonso de la Nuez
Co-Founder & Co-CEO
Nate Colker
UX Researcher
Twitter Hashtag #uzwebinar
Introduction Study Design Results Tips
• Defining UX Benchmarking
• Luxury department store Competitive Industry Study
• Tips for building your own competitive benchmarking study
Overview
Introduction Study Design Results Tips
Define Benchmarking
“Benchmarking is the process of comparing one's business processes and performance metrics to industry bests or best practices from other industries.”
http://en.wikipedia.org/wiki/Benchmarking
Introduction Study Design Results Tips
UX Benchmarking is the process of comparing a website’s (or any digital product) performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception, to another that is widely considered to be an industry standard benchmark or best practice
Definining UX Benchmarking & Metrics
Task Completion
Task Time Satisfaction
Task ConfidenceTask Level - Single Ease Question Test Level - System Usability ScaleNet Promoter Score
Number of ClicksNumber of Page Views
Introduction Study Design Results Tips
Define measurable & comparable UX-related KPIs
Jeff Sauro, Measuringusability.com
Introduction Study Design Results Tips
Why do we Care about Measuring User Experience?Ordering a Pizza Online (1994)
http://articles.chicagotribune.com/1994-08-22/news/9408230031_1_pizza-hut-santa-cruz-operation-internet
Introduction Study Design Results Tips
Why do we Care about Measuring User Experience?Ordering a Pizza Online (2012)
Introduction Study Design Results Tips
Introduction Study Design Results Tips
The World of User Experience Testing
http://www.useit.com/alertbox/user-research-methods.html Chart by Christian Rohrer
Unmoderated Usability Testing
Introduction Study Design Results Tips
Unmoderated Usability Testing
Introduction Study Design Results Tips
Case Study: 4 Brands
Introduction Study Design Results Tips
Study Summary
• Between subjects design• 50 Participants per brand (200 total)• 2 Tasks + survey questions• Randomized task order• Recruitment criteria
– Must shop online– Shopped at luxury department stores
• Total study length : ~14 minutes• Used a panel vendor• 3-4 days to field the study (11/20-11/23)
Introduction Study Design Results Tips
Task Descriptions
Task 1 Objective: measure findability of a common apparel item (“scavenger hunt task”)
Task DescriptionPlease find the price of a grey wool dress by Eileen Fisher. Please note down the price, as you will be asked for it later.
Task 2 Objective: Access desirability and perceived quality of the sales section (“open task”)
Task DescriptionPlease take your time to explore the sale section and look for things that are of interest to you. Once you feel you have spent an adequate amount of time exploring this section, click on the “I’m Finished” button. We will then ask you some questions about your experience with the sale section of Nordstrom.com.
Introduction Study Design Results Tips
Task 1 Completion Rates: Find a dress
Nordstrom Neiman Marcus Saks Lord & Taylor0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
72%
54%
74%
84%
Task Success Rate
The completion rates on Nordstrom, Saks and Lord & Taylor were significantly better than on Neiman Marcus (90% Confidence Level)
Introduction Study Design Results Tips
Disaster Rates (Highest Confidence but Failure)
Nordstrom Neiman Marcus Saks Lord & Taylor0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
72%
54%
74%84%
20%
32%
10%
12%8%
14% 16%4%
DisasterFailureSuccess
Introduction Study Design Results Tips
Task-Level Ease
Nordstrom Neiman Marcus Saks Lord & Taylor1
2
3
4
5
6
7
4.8
3.9
6 5.9
Very
Diffi
cult
Very
Eas
y
Overall, how difficult or easy was it to find the price of a grey wool dress?
Introduction Study Design Results Tips
Too many choices. Small print beneath the pictures.
I couldn’t find it and wasted a lot of time trying.
I never found the grey wool dress therefore I couldn't know the cost. This is obviously a failure at the task asked of the survey. I gave myself the lowest score possible, not for lack of trying thought!
I could not find the dress. I searched under eileen as the designer. I typed in grey wool dress in the search box and only two childrens dresses came up. I looked under sale items too and found three dresses by this designer non of which were wool.
I put Eileen Fisher grey wool dress in search bar and search query came up with "can not find Eileen Fisher grey wool dress" so I put in search Eileen Fisher wool dress and I found dresses none were grey.... I then looked through other pages and did not see a grey dress.
Search didn’t work - came back with not found, didn’t have intuitive drop-downs, under designers didn’t have dresses as option. No description said was a grey wool dress, closest was gray 3/4 length dress. Took to long to weed through options looking for dresses by Eileen.
Briefly describe why you chose the number you did.
Nordstrom
Neiman Marcus
Open-ended responses from participants who thought the task was difficult (1, 2 or 3)
Introduction Study Design Results Tips
The Saks website kept sending me to Saks Direct - it took a number of attempts (7? 8?) to get to dresses, then find the red wool dress assigned. It was very frustrating. Had the site worked well, it should have been easy.
Number of steps to complete task
It was not a difficult site to navigate, but I would rather narrow down a category rather than to search for a specific item using the search box
Its the only grey dress that said it was wool. it kept bringing up all different thing unrelated.
Because i could not figure out which brand of grey wool dress it was.
It took time to find the right dress, and I’m not completely sure it was the right one
Briefly describe why you chose the number you did.
Saks Fifth Avenue
Lord & Taylor
Open-ended responses from participants who thought the task was difficult (1, 2 or 3)
Introduction Study Design Results Tips
Task Time: Completed tasks only
Nordstrom Neiman Marcus Saks Lord & Taylor0
50
100
150
200
250
300
350
236
293
146126
Sec
ond
s
Introduction Study Design Results Tips
Completion time on Lord & Taylor was significantly lower than it was on Nordstrom & Neiman Marcus (90% Confidence Level)
Post task questions
Nordstrom Neiman Marcus
Saks Lord & Taylor1
2
3
4
5
6
7
5.5
4.7
6.3 6.2
5.2
4.4
6.25.9
5.1
4.1
66.2
It was easy to navigate the websiteIt was clear how to start finding the price of the wool dressI am satisfied with the amount of time it took to complete the task
Str
ong
ly D
isa
gree
Str
ong
ly A
gree
Introduction Study Design Results Tips
Task 2
Evaluate the sales section
Please take your time to explore the sale section and look for things that are of interest to you. Once you feel you have spent an adequate amount of time exploring this section, click on the “I’m Finished” button. We will then ask you some questions about your experience with the sale section of Nordstrom.com
Introduction Study Design Results Tips
Sale Section Evaluation
How satisfied are you with the number of items that were on sale?
Nordstrom Neiman Marcus Saks Lord & Taylor1
2
3
4
5
6
7
5.55.2
4.95.3
Not
at
all S
atis
fied
Ver
y S
atis
fied
Introduction Study Design Results Tips
Sale Section Evaluation
How satisfied are you with the sale prices?
Nordstrom Neiman Marcus Saks Lord & Taylor1
2
3
4
5
6
7
3.9
3.33
3.7
Not
at
all S
atis
fied
Ver
y S
atis
fied
Introduction Study Design Results Tips
Sale Section EvaluationS
tron
gly
Dis
agr
eeS
tron
gly
Agr
ee
Nordstrom Neiman Marcus
Saks Lord & Taylor1
2
3
4
5
6
7
6.2
5.76
6.2
5.75.4
5.1
5.5
4.8
4 3.9
4.5It was easy to navigate the sale sectionI found some items that I likedI would consider checking out the sale section again in the fu-ture
Introduction Study Design Results Tips
Final Questionnaire: What’s your attitude towards the brand?
Nordstrom Neiman Marcus
Saks Lord & Taylor1
2
3
4
5
6
7
4.8 4.84.4
4.1
4.9
4.34.6
4.8
BeforeAfter
Introduction Study Design Results Tips
Final Questionnaire:Brand Perception
1. Trendy 30% 62% +32%
2. Good Value 8% 26% +18%
3. Friendly 12% 30% +18%
4. Exciting 18% 32% +14%
Pre Post1. Expensive 60% 80% +20% 2. Trendy 34% 48% +14% 3. Trustworthy 24% 18% -6%
4. Customer 20% 8% -12% Focused
Pre Post
Nordstrom Neiman Marcus
1. Trendy 50% 62% +12%
2. Expensive 80% 88% +8%
3. Exciting 16% 12% -4%
4. Trustworthy 28% 12% -16%
Pre Post
Saks Fifth Avenue
1. Expensive 48% 82% +34%
2. Trendy 22% 52% +30%
3. Exciting 8% 26% +18%
4. Good Quality 42% 58% +16%
Pre Post
Lord & Taylor
Introduction Study Design Results Tips
Final Questionnaire
Nordstrom NeimanMarcus
Saks Fifth Avenue
Lord & Taylor
The website is easy to use 5.8 5.1 6.2 6.3
I feel comfortable purchasing from this website
5.2 4.3 5.1 5.2
I find the website to be attractive 5.5 5.3 5.5 5.6
The website has a clean and simple presentation 5.7 5.3 5.1 5.9
I will likely visit this website in the future 4.5 3.6 3.9 4.4
On a scale from 1 to 7Introduction Study Design Results Tips
Final Questionnaire:Net Promoter Score
0 1 2 3 4 5 6 7 8 9 10 NPS
Nordstrom10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6%
-38%
Neiman Marcus
8% 10% 10% 4% 8% 18% 10% 12% 18% 6% 6%-
56% Saks Fifth Avenue
12% 2% 10% 0% 2% 18% 16% 10% 10% 6% 14%
-40%
Lord & Taylor10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6%
-38%
Detractors Promoters
NPS% of Promoters
(9s and 10s)Detractors
(0 through 6)= -
Introduction Study Design Results Tips
Final Questionnaire:What did you like the most about the site?
Nordstrom Neiman Marcus“I really liked that you can change the colors of different items without being forced to open that particular item, taking you to another screen.”
“That I could click on the pictures and see the back of the garment.”
“The search function quickly found what I was looking for.”
“I like how the pictures of the apparel would show front and back when I hovered the mouse over the picture.”
Introduction Study Design Results Tips
Saks Fifth Avenue Lord & Taylor
“The amount of items I could see on one window. I hate hitting next over and over again.”
“Garment being worn by a moving person”“Easy to find anything. love the search/browse by size, color etc.”
“It was easy to navigate and find products I was looking for.”
“Search options. By Designer By Color By Price. Very Easy and nice to use.”
Final Questionnaire:What did you like the least about the site?
Nordstrom
Saks Fifth Avenue
Neiman Marcus
Lord & Taylor
“Small print beneath the pictures.”“Hard to search compared to other sites I have been on.”
“The models are ultra slim, it can be discouraging for someone who isn’t a size 2 or size 0.”
“The filters by designer only includes size color etc. not dresses, tops etc.”
“Drill through for finding the dress was difficult.”“It was difficult to navigate when searching for something specific.”
Introduction Study Design Results Tips
“The prices were a little steep.”
“They were very expensive. I also felt that for the price you would think they would have more trendy items of clothing. I didn’t find anything thatInterested me.”
“A little too expensive to buy more than one or two items.”
“The prices, but that is not the website”
We have an exciting announcement about a new feature…
Exciting announcement of new feature
Some Final Tips for Creating your own Benchmark
Task Selection
Standardized design
Randomize Conditions
Cleaning the Data
Introduction Study Design Results Tips
Selecting the Right Tasks
A Balance of User Data and Business Goals
What are users coming to the sites to do (core tasks) ?
Are there any tasks that are required to use your site?
What drives revenue on your site?
Introduction Study Design Results Tips
Selecting the Right Tasks
Introduction Study Design Results Tips
A Balance of User Data and Business Goals
What are users coming to your site to do (core tasks) ?
- Browsing new sale items
Are there any tasks that are required to use your site?
- Creating an account?
What drives revenue on your site?
- Completing a purchase
Standardize Tasks Across Competitors
Introduction Study Design Results Tips
Introduction Case Study Results Analysis
Randomize Task Order
Screener Pre-task questions
Task 1
Task 2
Final QuestionsRandomized
Control for ordering effects• Participants become more proficient as they use the site etc.
There are exceptions, sometimes tasks must be completed in
order
Clean the Data
Watch out for Cheaters
• Very Low task times
• 0 Clicks
• Nonsense Answers
• Enable Quality
Controls
Introduction Study Design Results Tips
http://en.wikipedia.org/wiki/File:Cheating.JPG by Hariadhi
Take this with you today:
• Triangulate, use multiple metrics
• Experimental design
• Clean data before analysis
• Want to learn more?
Thanks so much for your attention!
Time for Q&A
Follow us on Twitter @userzoom