introducing: the lab notebook - a tool for managing a lean ux process

© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook

Introducing: The Lab NotebookA hypothesis-driven approach for

delivering value faster

#UXLabNotebookJulie Casanave @jcasanaveKerry Holeman @uxkerry


What’s this all about?


Project completion can easily be mistaken for project success.


A successful launch delivers measurable customer value. If users don’t adopt the solutionwe need to evaluate and make a decision.

What was designed & delivered

How it’s used


Science!


What’s the most important unanswered question?

What’s the smallest thing we can make to answer that question?

-Steve Mulder, Senior Director of Audience Insights, NPR


Create a Lab Report

Createthe

experiment

Decide what to do

next

Definethe question


Create a Lab Report

Createthe

experiment

How will we learn it?

Decide what to do

next

What do we do with what we

learned?

Definethe question

What do we want to learn?


Define the QuestionWhat do we want to learn?


Problem Statement

WHAT ISSUE NEEDS SOLVING?

“I am [customer] trying to [accomplish goal].But [insert problem] because [insert cause].

This makes me feel [emotion].

The consequence for [my team/company] is [adverse business effect]”


I am a 60-year old trying to take a retirement withdrawal online, because I need to cover the cost of a

new car.

But I don’t feel confident doing it online because I don’t understand if I’m subject to penalties or tax

consequences.

This makes me feel anxious.

The consequence for Fidelity is costly customer service calls and poor customer perception.

How might we…

“I am [customer] trying to [accomplish goal].

But [insert problem] because [insert cause].


The consequence for [my team/company] is …


Exercise

WHAT ISSUE NEEDS SOLVING?

“I am [customer] trying to [accomplish goal].But [insert problem] because [insert cause].


The consequence for [my team/company] is [adverse business effect]”


Define the ExperimentHow will we learn?


Define the Experiment

How the issue will be addressed

What we’ll build to address it

How we’ll know it works

HYPOTHESIS(solution strategy)

TESTstrategy

MEASUREMENTstrategy

+ +

“[This change] will enable [who] to achieve [what],

leading to [value].”

“The simplest way to test this is to [test

strategy].”

“We will know we’ve succeeded when we

observe [user behavior/sentiment].




TESTstrategy

MEASUREMENTstrategy

+ +

A Turbo-Tax-like withdrawal process, which guides people through a decision path, will help customers withdraw retirement funds with a higher degree of confidence, leading to fewer calls to the call center.

[This change] will enable [who] to achieve [what], leading to [value].”




TESTstrategy

MEASUREMENTstrategy

+ +

The simplest way to test this is with a moderated usability session + paper prototype which takes the customer down one sample decision-making path to completion.

The simplest way to test this is to [test strategy].




TESTstrategy

MEASUREMENTstrategy

+ +

We will know we’ve succeeded when we observe customers:1. Successfully complete the process without indicating

they’d abandon.2. Indicate that their level of confidence about

completing the transaction is higher than it is today.

We will know we’ve succeeded when we observe [user behavior/sentiment].


Exercise: Hypothesis


TESTstrategy

MEASUREMENTstrategy

+ +

“[This change] will enable [who] to achieve [what], leading to [value].”


Decide on a Test StrategySE

NTI

MEN

T(W

hat

peop

le s

ay) BEH

AVIOR

(What people do)

QUANTITATIVE(Reveals the

“what”)

QUALITATIVE(Reveals the “why”)

Surveys

Moderated Usability Study

A/B Testing

Self/Un-ModeratedUsability Study Opinion Lab Verbatims

Focus Groups Interview

Call Center Notes

Social Media Activity

Web Analytics

Emotional Reaction Study

Card Sort Treejack Study First Click Study

Eye-Tracking Study

Pre-release Post-release

PRO TIPBe biased towards behavioral methods


Think creatively about what type of experiment will give the results you need.Deployed code is only one kind of experiment.(and often the most expensive kind)


Exercise: Test Strategy


TESTstrategy

MEASUREMENTstrategy

+ +

“The simplest way to test this is to [test strategy].”


Decide How to MeasureHAPPINESS

ENGAGEMENT

ADOPTION

RETENTION

TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden

Measures of attitudes, often collected via survey• Emotional reaction• Perceived level of effort• Sentiment rating

• Level of confidence• Volume of opinion lab comments

https://www.gv.com/lib/how-to-choose-the-right-ux-metrics-for-your-product



ENGAGEMENT

ADOPTION

RETENTION


Level of user involvement•Bounce rate •Channel switching

• Page visits • Searches •Click-throughs

• Eye-tracking• Visitors




ENGAGEMENT

ADOPTION

RETENTION


Gaining new users of a product or feature•Click rates•Click-throughs

•New accounts




ENGAGEMENT

ADOPTION

RETENTION


The rate at which existing users are returning•NA for our program




ENGAGEMENT

ADOPTION

RETENTION


Efficiency, effectiveness, and error rate

•Completion speed• Error rate

•Abandonment•Completion rate

• Pathing•Re-searches




ENGAGEMENT

ADOPTION

RETENTION


• Emotional reaction• Perceived level of effort• Sentiment rating

• Bounce rate • Channel

switching

• Click rates• Click-throughs

• Not applicable

• Completion speed• Error rate


• Page visits • Searches • Click-throughs

• Abandonment• Completion rate


• New accounts

• Pathing• Re-searches

What metric matters most?




ENGAGEMENT

ADOPTION

RETENTION


• Emotional reaction• Perceived level of effort• Sentiment rating

• Bounce rate • Channel

switching

• Click rates• Click-throughs

• Not applicable

• Completion speed• Error rate


• Page visits • Searches • Click-throughs

• Abandonment• Completion rate


• New accounts

• Pathing• Re-searches

Does this experience give you ahigher or lower level of

confidence compared to today’s process?




ENGAGEMENT

ADOPTION

RETENTION


• ??

• ??

• ??

• ??

• ??

• ??

• ??

• ??

• ??

• ??

• ??

WHICH ONE MATTERS MOST?

WHAT’S YOUR SET OF METRICS?



Exercise: Measurement Strategy


TESTstrategy

MEASUREMENTstrategy

+ +

“We will know we’ve succeeded when we observe [user behavior/sentiment].


Decide what to do nextWhat do we do with what we learned?


What We Learned Major findings, unanticipated discoveries

Pivot? Or Persevere? Or Stop?At what point was the conclusion reached?

Time Lapsed Since Launch

Conclusion

What will happen next?Next steps

DD Month YYYYLaunch DateDocumenting what was learned

Outcomes How did experiment do against HEART metrics?


A disproven hypothesis is evidence.

It is not failure.“I have not failed.

I've just found 10,000 ways that won't work.”-Thomas A. Edison


And when that experiment is done?


LATHER, RINSE, REPEAT? !

? !

? !

? !

? !

? !

DESIGN RUN ITERATE

LAB NOTEBOOK

LAB REPORT

LAB REPORT LAB REPORT

LAB REPORT


The “grain of salt” slide


MITIGATE RISK• Deliver value faster• Reduce wasted time, effort,

money

FOCUSED EXPERIMENTS• Clear learning agenda• Simple test strategy• Well defined metrics


Thank You.

Julie Casanave | @jcasanaveKerry Holeman | @UXKerry

introducing: the lab notebook - a tool for managing a lean ux process

Engineering