David Rosen, User Experience Analyst

Try Before You Buy:

User Experience Testing in

Your RFP Process Can

Save You Time and Money

WHO IS THIS GUY?

David Rosen

User Experience Analyst

University of Minnesota

Twitter: @DRosen_UX

Hashtag: #heweb14

Track hashtag: #UAD8

WHAT’S AN RFP?

Requirement Points Vendor 1 Vendor 2 Vendor 3

Does the sandwich have ham

on it?

3 Yes (3) Yes (3) No (0)

What condiments are available

for the sandwich?

3 Mayo,

Mustard (2)

Mustard,

Jalapenos (1)

All condiments

(3)

Can the sandwich be toasted? 2 Yes (2) No (0) Yes (2)

How quickly will the sandwich

be made?

5 10 minutes

(3)

5 minutes (5) 15 minutes (1)

Sandwich RFP

#UAD8

EXPERIENCE CHECK

http://www.cnet.com/au/news/supercomputer-runs-longest-simulation-of-brain-activity-to-date/

#UAD8

HOW DID WE GET THIS STUFF?

Some common reasons software vendors are chosen:

• They were the least expensive

• They gave the most polished demonstration of their product

• They checked the most boxes on the RFP

• They were one of the committee member’s favorites before the process began

• We have already been using their product

#UAD8

https://creativecommons.org/licenses/by-nc/2.0/legalcode

“It’s a very sobering feeling to be up in

space and realize that one’s safety factor

was determined by the lowest bidder on a

government contract.” Alan Shepard

https://www.flickr.com/photos/barry_b/76055201/

#UAD10

REENACTMENT OF THAT MEETING

#UAD8

“No”

http://corporate.ford.com/our-company/heritage/company-milestones-news-

detail/commemorating-150-years-of-henry-ford-visionary

#UAD8

ANOTHER CHALLENGE

https://www.flickr.com/photos/stevendepolo/3797233888/

Going with the

product as-is• Customization is hard

to maintain and

expensive

#UAD8

NOT “VANILLA”

https://www.flickr.com/photos/ginnerobot/3156117675/

#UAD8

IMPLEMENTING “VANILLA”

https://www.flickr.com/photos/stevendepolo/3797233888/

“Vanilla” can be

tasty…

if you let your

users test it

#UAD8

AVOIDING NON-FAT, UNFLAVORED ICE MILK

Case Study:

Survey Tool

#UAD8

ASKING USERS IF THEY WANT FEATURES

“Who Wants a Pony?” image from http://voiceofthemonkey.com/wp-content/uploads/2011/11/Who-Wants-A-Pony.jpg

#UAD8

THE OLD TOOL LOOKED LIKE THIS

#UAD8

GETTING INCLUDED

https://www.flickr.com/photos/glynlowe/7183121469/

#UAD8

VENDOR DEMONSTRATIONS

#UAD8

NO MORE SMOKE AND MIRRORS

Because we were dictating the terms

of the demonstration, we could see

when the vendors were trying to skip

steps, or avoid less pleasant

interactions.

https://www.flickr.com/photos/xtream_i/316644980/

#UAD8

PREPARATION FOR USABILITY TESTING

“We’re reluctant to let you

try this without at least a

three-hour training

session”

#UAD8

PREPARATION FOR USABILITY TESTING

#UAD8

PREPARATION FOR USABILITY TESTING

#UAD8

SOMETIMES IT’S HARD TO WATCH

They made it

look so EASY!

#UAD8

GATHERING ISSUES

• Identifying pain

points

• Quantity of issues

is not a comparison

tool

#UAD8

NEXT STEPS

#UAD8

NEXT STEPS

#UAD8

NEXT STEPS

#UAD8

WHAT WE GAVE TO THE VENDOR

Recommendations

– A description of what we wanted to either understand better or have changed

Alert Training/Support

– Training and Help Desk teams will need to be aware of this issue

Prepare Vendor Request

– A request for a change in order for the solution to fit the need

Response Required

– We are expecting an explanation of this issue

Submit OIT Request

– This may require configuration on our part

Hold for Further Review

#UAD8

WHAT WE GAVE TO THE VENDOR

We understand that some of these tasks may or may not be standard,

but they were included as a means to evaluate the usability and

intuitiveness of the system. In this way, we are able to gain some

insight into configuration setting issues, areas that may require extra

training, and/or items that need to addressed by you, the vendor.

Attached is a workbook outlining the issues that were experienced

and that we would like you to respond to. Please review and enter

your response in the space provided. In the columns to the right, you

may indicate if any of these options are applicable.

#UAD8

WHAT WE HEARD FROM ONE OF THE VENDORS

“A response was required from us on each item, and since we were motivated to win the contract we made commitments to address the vast majority of issues. Now that all items targeted for the short term have been addressed, we know our product has become easier to use.”

#UAD8

WHAT WE HEARD FROM SOME OTHER VENDORS

http://memecrunch.com/meme/15QYJ/uhhhh-no

#UAD8

UX HELPED BUILD A BUSINESS CASE

#UAD8

LESSONS LEARNED

#UAD8

http://educationalaspirations.com/category/assessment-data/

LESSONS LEARNED

Vendors don’t understand

what we are doing by

“usability testing”

They also think that we’re

“difficult”

#UAD8

http://www.flickr.com/photos/79538062@N00/3732589905

LESSONS LEARNED

Access to the

finalists’ software

cannot be optional

#UAD8

LESSONS LEARNED

Vendor Demonstrations

need a strong moderator

from the selection team

#UAD8

https://www.flickr.com/photos/leshaines123/6909108380/

LESSONS LEARNED

Testing is best after

vendor demonstrations

are complete

#UAD8

IF YOU REMEMBER NOTHING ELSE

1. WE. SAVE. MONEY.

– We’ve had a preview of implementation

– Our training and help desk teams are

prepared ahead of time for known

issues

2. There are fewer surprises in a situation

where surprises are expensive

3. This makes vendors better vendors and

partners

#UAD8

Go do it!

1. Use the stuff you’d ask the

vendors anyway

2. You can do this on the cheap

with limited resources

#UAD8

BOOKS TO GET STARTED

#UAD8

THANK YOU!

Questions?

David Rosen

User Experience Analyst

University of Minnesota

Email: rose0414@umn.edu

Twitter: Drosen_UX

Web: z.umn.edu/usability