Cost-Benefit Analysis for HCI

University of Michigan Human Factors Engineering Short Course

2011

Mark W. NewmanUniversity of Michigan School of Information

mwnewman@umich.edu

Learning Objectives

• How to argue for the value of user-centered design– General arguments for incorporating usability methods– Cost/benefit projections for specific projects

User-Centered Design

• Involve users throughout process– At beginning: Needs assessment

• e.g., contextual inquiry– Throughout: Iterative design & evaluation

• e.g., user testing, heuristic evaluation– After deployment: Feedback for next version

• e.g., surveys, log analysis

• Incorporates User Experience Research and Interaction Design

Design

Build

Test

You Don’t Need User-Centered Design

You Don’t Need User Centered Design

• … if you’re a genius

Alternatives to UCD1

• I’m a user, and I can use it• This thing is so new and different there are no way

users can evaluate it• This technology is so awesome that people will be

willing to learn it• We’ll just release it and fix any problems in the next

release• Users just want features, so we need to devote all

resources to implementation

1 These are not recommended

The Bottom Line

General Arguments for UCD

• External ROI– Increased sales– Decreased support– Increased perception of

value– Early identification of

problems– (Loyalty)

General Arguments for UCD

• Internal ROI– Increased productivity– Decreased support– Decreased errors– Decreased training

costs– Early identification of

problems– (Morale)

Specific Arguments: Costs vs. Benefits• Benefits

– How much will we save?– How much will we make?– What risks will be reduced?

• Costs– What resources will be required?

• Human effort• Recruiting, compensation, equipment, etc.

– How long will it take to complete?• Effects on time-to-market• Lost opportunities

Metrics for Cost/Benefit

• ROI*

– ROI > 0: profit– ROI < 0: loss– ROI = 1: 100% profit (return is 2x investment)

• Payback– Time to recoup initial investment– Payback time = Vi / Vp (expressed in time period p)

*: we are ignoring the fact that Vf could generate interest if not invested here.

High-level example

• You believe a user-centered design project will result in a sales increase of 5%– Current sales are $100,000 per month– The project will take 6 months and cost $50,000

• What is the payback period?– Vi = $50,000; Vp = $5,000

– Payback = Vi/Vp = 10 months

• What is the ROI after 1 year? 3 years?– Vf_1yr = 60,000; Vf_3yr = $180,000

– Vf_1yr/Vi – 1 = 0.2 (20%); Vf_3yr/Vi – 1 = 2.6 (260%)

Usability ROI: Evidence

• 53 redesign projects with before-and-after metrics– 66 measurements in all

Metric Class Improvement

Sales/conversion rate (20 cases)

87%

Traffic/visitor count (13 cases)

96%

User performance (14 cases)

119%

Feature use (10 cases) 223%

Overall average 114%

(Note: 8 outliers with extreme improvements were removed)

Usability ROI: Decline over time

• Data represents two studies: 2000 & 2006– 2000: 135%– 2006: 83% – Excludes huge gains (5 measures 900%+ & 3 measures ∞%)

• Possible reasons for lower numbers in 2006– Low hanging fruit has been picked– Usability budgets have not increased as web has gotten

“better”

(Aggregate: 114%)

Has the web gotten better? Avg sales conversion rate in 2000: 1% Avg sales conversion rate in 2006: 2%

Has the web gotten better? Avg sales conversion rate in 2000: 1% Avg sales conversion rate in 2006: 2%

Different Benefits

Estimating the Costs

• What work will be done?– Selecting methods– Estimating amount of effort for each method

• How much does it cost?– How much do people get paid?– Are there other costs?

Example Project: Baconfreak.com

• Daily Visitors: 10,960• Conversion rate: 1%• Average sale: $28.40• The plan: We’re going

to rework the site to have a “modern look”

Development Plan

• Designer will make info architecture, page layouts, and graphics

• Software developers (3) will design and implement the functionality

• Database manager will redesign database• QA people (2) will test for bugs• Project manager will keep things on track• It will take 6 months:

– 1 month for up front system design, site architecture, page layouts– 5 months for implementation of website code and database changes– 2 months for QA at end, overlapping with development

Your Proposal

• Add 2 months at the front to provide for:– Interviews/Contextual inquiry– Survey– Comparative analysis

• Hire Usability Specialist for 8 months (total) to oversee– Initial Needs-Finding– Low-fi prototype and test– Heuristic evaluation(s)– User test(s)– Web analytics setup

Original Development Plan

Design

Development

QA

Original Development Plan

Inte

rview

s

Compa

rativ

e Ana

lysis

Surve

y

Lo-fi

pro

totyp

e te

st

Heuris

tic E

valua

tion

User t

est #

1

User t

est #

2

Analyt

ics p

roce

ss

Design

Development

QA

UX Research

Added Costs

• Existing costs– Project manager for 6 months– Designer for 3 months– 3 Developers for 5 months– 2 QA people for 2 weeks

• Added costs– 8 months of UX Researcher salary

*http://www.simplyhired.com/a/salary/search/q-user+experience+researcher/l-ann+arbor,+mi

Revised Budget

Role FTE Salary Overhead Cost

Project Manger

6/12 80,000 50% 60,000

Designer 3/12 80,000 50% 30,000

Developers 15/12 (3@5mo)

90,000 50% 179,550

Database Mgr 5/12 70,000 50% 43,750

QA Engineers 4/12 (2@2mo)

60,000 50% 29,700

Total 343,000

UX Researcher

8/12 80,000 50% 80,400

Total 423,400 (23%)

Projected Benefits

• User-centered process yields – Conversion increase of 87%– Traffic increase of 96%

• Generously assume that the non-UX development process would yield 50% increase in each

• What is the ROI after 1 year?• What is the payback period?

Previous W/O UX W/UX

Conversion rate 1% 1.5% 1.87%

Visitor Count 10,960 16,440 20,500

Avg sale $28.40 $28.40 28.40

Daily revenue $3,113 $7,003 $10,887

Yearly revenue $1,136,090 $2,556,256 $3,973,806

1 year value (Vf)

0 $1,420,166 $2,837,716

Investment (Vi) 0 $343,000 $423,400

ROI (1 year) 0 3.1 5.7

Payback .24 yr (3 mo) .15 yr (7.5 wk)

Usability Budgets: Best Practices

• Conducted at “Usability Week 2006” conferences in NY, SF, Sydney, London

• Based on 143 projects – Avg project size: 9 person-years– Avg usability investment: 6 person-months (~1040 hrs)

– Median usability investment: 10%– Mean usability investment: 17%– Regression model:

(units: person months) (note: budget grows roughly as square root.)

Other Considerations

• Value of incorporating usability earlier into process– Gilb’s $1 - $10 - $100 rule

• Cost of fixing problems increases by an order of magnitude as you progress from design – development – release

Serious Risks: Accuracy Matters

• 5,330 voters double-punched Gore & Buchanan

– 2,908 Gore/McReynolds– 1,631 Bush/Buchanan

• 3,407 (0.76%) voted for Buchanan– pre-election polls

predicted ~600• Bush won Florida by 537

votes

Although Democrats are listed 2nd, they

are the 3rd hole

Although Democrats are listed 2nd, they

are the 3rd holePunching the 2nd hole votes for the Reform

party

Punching the 2nd hole votes for the Reform

party

Horizontal lines lead the eye to the wrong holes,

causing additional confusion

Horizontal lines lead the eye to the wrong holes,

causing additional confusion [Tognazzini 2001]

Serious Risks: Therac 25

• Radiation therapy machine– Electron mode: low power, short burst– X-ray mode: high power, longer duration,

concentrated on target

Serious Risks: Therac 25

• 6 accidents; 4 fatalities (85-87)• Usability flaws + Software bugs resulted in huge

doses of radiation (100x expected amts)– Minor malfunctions were common (~40 per day)– Easy to restart after failure (Press “P”)– Common path through system led to error state

(“Cursor Up” to change modes, followed quickly by “B” to start treatment)

– Operators unable to interpret cryptic “Malfunction 54” message

[Levenson 95]

UCD and Risks

• UCD is helpful for certain kinds of risks:– Helps anticipate expectations, skills, existing practices of

users– Avoid likely misconceptions and sources of error– Reduce engineering risks: feature creep, fluid requirements– Reduce likelihood of user rejection

• However, UCD does not protect against rare events– Formal analysis methods are recommended for safety

critical features!

Thimbleby 2007, Boehm 1991

Parting Thoughts: The Long View

• Wherever possible– Collect metrics before and after UCD-based redesign– Institute an ongoing measurement process– Insist on similar metrics for other projects

Parting Thoughts: The Method Muddle

• User-centered design is our best defense against getting it wrong (and best chance of getting it right!)

• This lecture treats user-centered design as a black box• Different methods are good for different things• During the rest of this course, ask:

– What risks does this method help minimize?• Wrong requirements, learnability, poor performance, likelihood of

errors, user resistance– Where could this method fit into an iterative process?

• Early, middle, late, post-deployment

References

• Nielsen, J., & Gilutz, S. (2008) Usability return on investment, 3 ed. available at http://www.NNgroup.com/reports/roi

• Bias, R. G. and Mayhew, D. J. (2005) Cost Justifying Usability: An update for the Internet Age. San Francisco, CA: Morgan Kaufman.

• Usability First: Usability ROI: Case Studies. http://www.usabilityfirst.com/roi/studies.txl [retrieved 7/1/08].

• Nielsen, J. Return on Investment for Usability (Jakob Nielsen’s Alertbox). http://www.useit.com/alertbox/roi-first-study.html [retrieved 7/1/08]

• Nielsen, J. Usability ROI Declining, but Still Strong (Jakob Nielsen’s Alertbox). http://www.useit.com/alertbox/roi.html [retrieved 7/1/08].

• Nielsen, J. Do Government Agencies and Non-Profits Get ROI from Usability? http://www.useit.com/alertbox/government-nonprofit.html [retrieved 7/1/08].

• Usability Professionals Association. Resources for Selling Usabilityhttp://www.upassoc.org/usability_resources/usability_in_the_real_world/selling_usability.html. [retrieved 7/1/08].

• Donoghue, K. (2002) Built for Use: Driving Profitability Through the User Experience. McGraw Hill.

References

• Thimbleby, H. 2007. User-centered methods are insufficient for safety critical systems. In Proceedings of the 3rd Human-Computer interaction and Usability Engineering of the Austrian Computer Society Conference on HCI and Usability For Medicine and Health Care (Graz, Austria). A. Holzinger, Ed. Lecture Notes In Computer Science. Springer-Verlag, Berlin, Heidelberg, 1-20.

• Boehm, B. W. 1991. Software Risk Management: Principles and Practices. IEEE Software 8(1). January 1991

• Levenson, N. Safeware: System Safety and Computers. Addison-Wesley. 1995. (contains a chapter on the Therac-25 accidents)

• Tognazzini, B. The Butterfly Ballot: Anatomy of a Disaster. AskTog, January, 2001. http://www.asktog.com/columns/042ButterflyBallot.html

These slides

http://mwnewman.people.si.umich.edu/courses/hfsc2011

Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html

What People DoWhat People Do

What People SayWhat People Say

Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html

Why & How to Fix

Why & How to Fix

How Many & How

much

How Many & How

much

Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html

Why & How to Fix

Why & How to Fix

How Many & How

much

How Many & How

much

Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html

Other considerations:

•Should I employ users at all? •Where should method go in this process?•What prerequisites (e.g. prototypes) do I need for the method?

Other considerations:

•Should I employ users at all? •Where should method go in this process?•What prerequisites (e.g. prototypes) do I need for the method?

Rohrer, C. http://www.useit.com/alertbox/user-research-methods.html