etl may 2010

ESCAPE THE LAB

Pleasestand by

WILL!BEGIN!AT 9:00

Friday, May 7, 2010

ESCAPE THE LABLET’S!GET!STARTED!MMKAY?

Friday, May 7, 2010

Friday, May 7, 2010

I’m CydHe’s Nate

Friday, May 7, 2010

#ETLAB

Friday, May 7, 2010

remoteUSABILITY

Friday, May 7, 2010

oh

Friday, May 7, 2010

and

Friday, May 7, 2010

also

Friday, May 7, 2010

NO

Friday, May 7, 2010

time

Friday, May 7, 2010

travel

Friday, May 7, 2010

OR

Friday, May 7, 2010

lasers

Friday, May 7, 2010

HOW!TO!BEgraceful

When All Your

TECHNOLOGYBREAKS

Friday, May 7, 2010

PLEASE DON’T TRIPON!THE!CABLES

Friday, May 7, 2010

Follow the buddy system

Friday, May 7, 2010

How about you all?

Best remote research storyMost important thing to get out of today?Biggest fear or skepticism about remote research?

Friday, May 7, 2010

Go TO escapethelab.com NOW

Friday, May 7, 2010

Friday, May 7, 2010

Remote Research

Definition:

Research where the participant is not sharing physical space with the researcher.

In personRemote

% of qualitative research done remotely*

*Source: BP’s best guess

Friday, May 7, 2010

What we’ll cover today

RecruitingModeratedUnmoderatedResultsPitfalls

Friday, May 7, 2010

We shall begin...with a little experiment

Friday, May 7, 2010

EXERCISEJUMP IN THE POOL

Table groups

Pick one moderator and one participant

(they need to be in different locations)

10 minutes of moderating

Friday, May 7, 2010

MODERATOR Instructions

Your participant has a sheet of paper and a linkGoal: find out what’s hardest about presenting physical task instructions onlineHint: ask them to give you a play by play of what they are doingFollow along with your own paper

NO helping, even if you could.

Friday, May 7, 2010

It’s A RACE

First participant to make an origami giraffe is the winner

Friday, May 7, 2010

Friday, May 7, 2010

Surprises?

Friday, May 7, 2010

When does remote win?

Page 10-15

Friday, May 7, 2010

Native Environment

Cheap ethnography, not cheap lab.

Friday, May 7, 2010

Real Time

Allows self-motivated tasks.

Friday, May 7, 2010

Involved Observers

Easier to collaborate in real time.

Friday, May 7, 2010

Geography

Friday, May 7, 2010

When does remote fail?

Friday, May 7, 2010

Bandwidth

Yes, it’s still an issue.

Friday, May 7, 2010

Translation

It’s expensive and difficult to do live.

Friday, May 7, 2010

Minors

Consent is a bear.Especially if they’re under 13.

Friday, May 7, 2010

FacesWe’ve all got webcams today.

Most users don’t.Webcams are invasive.

Friday, May 7, 2010

CASE STUDY: Epic FAIL

Friday, May 7, 2010

What went wrong

Coins were way more valuable than we expectedPeople mis-assessed their English levelsBandwidth was really a problemNobody plays Habbo when their parents are home to consent

Friday, May 7, 2010

making the most

Friday, May 7, 2010

TIME-AWARE TasksTold to find purple pumps.

Doesn’t care.

Has to wear this for sister’s wedding this weekend.

Whole different interaction.

Friday, May 7, 2010

Real Time OBservers

More on page 109-111

Friday, May 7, 2010

Change on the fly

“No one seems engaged in the laptop customizer”

“Let’s add questions about who they get recommendations from”

Friday, May 7, 2010

Designing Remote Studies

Friday, May 7, 2010

Friday, May 7, 2010

plan these things

Friday, May 7, 2010

plan these things

MethodRecruitingLogistics

Friday, May 7, 2010

MODERATED must-haves

Friday, May 7, 2010


Consent

Friday, May 7, 2010


ConsentTech set-up

Friday, May 7, 2010


ConsentTech set-upThe beef

Friday, May 7, 2010


ConsentTech set-upThe beefDe-installation

Friday, May 7, 2010


ConsentTech set-upThe beefDe-installationThanks & incentive

Friday, May 7, 2010

UNMODERATED must-haves

Friday, May 7, 2010


Expectation setting

Friday, May 7, 2010


Expectation settingInstructions

Friday, May 7, 2010


Expectation settingInstructionsThe beef

Friday, May 7, 2010


Expectation settingInstructionsThe beefThanks & incentive

Friday, May 7, 2010

Recruiting

Friday, May 7, 2010

Recruiting

Same: Good “talkers”Same: Demographic fits

Friday, May 7, 2010

Recruiting

Same: Good “talkers”Same: Demographic fitsDifferent: Relevant current tasksDifferent: Compatible tech

Friday, May 7, 2010

Logistics: What & How

Provide test targetAccessCredentials

Friday, May 7, 2010

Logistics: Who & When

How to connect with the user (we’ll discuss in Tools)

Managing time zonesSession length

Friday, May 7, 2010

Paying Incentives

Collect minimal personal infoDeliver quicklyAppropriate for location

Friday, May 7, 2010

Advanced planning:hybrid studies

Why use only one method?Some methods don’t tell you the why3-5 remote interviews is all you need

Friday, May 7, 2010

Portable Research

Friday, May 7, 2010

BREAK

Friday, May 7, 2010

Recruiting for Remote

Friday, May 7, 2010

We all need people

Relevant tasksCompatible techGood talkersDemographic fitsIn the moment

Friday, May 7, 2010

Gold Standard

Live recruitingGet as close as you can

Page 52

Friday, May 7, 2010

Recruiting Options

EthnioLinks or house ads to online formsRecruiting agenciesCustomer email listsPanelsCraigslist/online ads

Friday, May 7, 2010

use Ethnio when

Website accessSome budgetQuotas are relatively simple

Friday, May 7, 2010

use forms when

You can place a link but not codeLow budgetQuotas are relatively complexOptions:

Google Docs

Wufoo

Friday, May 7, 2010

Google docs

Friday, May 7, 2010

use Agencies when

No website accessHigher budgetOnline recruiting failed

Friday, May 7, 2010

Customer lists

Sub-segmentsOwnership requirementsNo website access

Friday, May 7, 2010

PANELSFriday, May 7, 2010

Craigslist & forums

When other methods don’t workNo doughTakes a whileEmbarrassing to admit

Friday, May 7, 2010

the Gold Standard again

Live recruitingGet as close as you can

Friday, May 7, 2010

Live Recruiting Math(for interviews)

Target: 6 recruits/hourAllow for: 1% response (low average)

You Need: 8-10,000 uniques/day

Friday, May 7, 2010

Friday, May 7, 2010

Screener constraints

Keep it shortDifferent from surveysWho gets paid?

Page 60-63

Friday, May 7, 2010

Remember the Point

Tasks that align with research goalsTechnical compatibilityControl demographics

Friday, May 7, 2010

EXERCISE

Friday, May 7, 2010

Friday, May 7, 2010

Your best questions

Friday, May 7, 2010

GO EAT LUNCH

Famished

Friday, May 7, 2010

exercise

We recommend:Split Pea Seduction, 6th & Minna

Custom Burger, 7th & Natoma

Tony Baloney’s, 7th & Howard

Tin Vietnamese, Howard & 6th

Harvest Market, Natoma & 8th

Sightglass Coffee, 7th & Folsom

Friday, May 7, 2010

ESCAPE THE LAB

Pleasestand by

WILL!RESUME!AT 1:15

Friday, May 7, 2010

welcome back

Friday, May 7, 2010

Tweet why you hate remote research #ETLABWin a copy of Silverback

Time for a

contest!

Friday, May 7, 2010

Remote Research Toolkit

MODERATEDmost of

Chapter 8

Friday, May 7, 2010

5 easy pieces

Talk to the userSee what they’re doingTake notes (optional)

Involve observers (optional)

Record audio & video (optional)

Friday, May 7, 2010

Communication

Reliable

Can be expensive for international

Conference calls needed for observers

phone

Cheap or free

Harder for users

Can combine with screensharing to stress user’s connection

Skype

Friday, May 7, 2010

Adobe Connect

LiveLook

WebEx*

GoToMeeting

Standalone

UserVue (RIP)

FuzeMeeting (new)

Integrated

Screensharing

*WebEx also has mobile versions

Friday, May 7, 2010

Adobe Connect

Friday, May 7, 2010

FuzeMeeting(New)

Friday, May 7, 2010

Livelook

Friday, May 7, 2010

Webex

Friday, May 7, 2010

GoToMeeting

Friday, May 7, 2010

Taking Notes

Second monitor if possibleExcel Hack (for time codes)

Friday, May 7, 2010

How to rig your notes

Leave one column (A) empty for notes.

Set the cells of the next column (B) to record the clock time of each entry in A.

=IF(A2=“”,“”,IF(B2=“”,NOW(),B2))

Set the cells of the next column (C) to record the difference between the first entry and the current entry

=IF(ISERROR(B2-$B$2),“”,B2-$B$2)

Fill down (Ctrl-D). Expect to use up to 120 rows for a session.

Hide column B. You’ll never need to see it

What the formulas look like

Auto timestamp!

Don’t write this

down, it’s on page 104

Friday, May 7, 2010

Observer Involvement

IMWeb Chat RoomsIntegrated ChatPassing Notes

Friday, May 7, 2010

Recording Tools

Phone Patch

Cables

HardwareCamtasia (Windows)

iShowU HD (Mac)

Software

Friday, May 7, 2010

Phone patch set up

Friday, May 7, 2010

Friday, May 7, 2010

Simple Set-up

Friday, May 7, 2010

Intermediate Set-up

Friday, May 7, 2010

Advanced Set-up

Friday, May 7, 2010

Friday, May 7, 2010

exercise

Roll your ownWhat will be your base set-up?

Write it down

Friday, May 7, 2010

Friday, May 7, 2010

Your version?

Friday, May 7, 2010

Winners!!

Friday, May 7, 2010

Moderating for Remote

Friday, May 7, 2010

Is it really any different?

(Short answer: some.)

Page 94-99

Friday, May 7, 2010

Question from the BP inbox yesterday

It would have actually helped to know how many users the authors recommend for a tar* type of test. In a regular user test , many of us have seen how findings repeat a#er 5 -8 users of each user type are observed. However, here we don't always know which task a user will be doing when we interrupt them. So in cases where we really are following users along as they complete what they came to do, how many useres should we target ? Given that the authors have been doing this for a while a recommendation from them on how many users to target and their notes on the topic would have been really interesting.

*Time-Aware Research

Friday, May 7, 2010

Making passionate tasks work

Friday, May 7, 2010

Making passionate tasks work

Recruit for relevant current tasksScript for customizable tasks

Friday, May 7, 2010

Re-script for remote

Friday, May 7, 2010

Re-script for remote

“Ok, I’d like you to start from the homepage, pretend you are shopping for groceries for

a party, and buy some paper plates.”

Friday, May 7, 2010


Also see37-43

Friday, May 7, 2010


ConsentAlso see

37-43

Friday, May 7, 2010


ConsentTech set-up

Also see37-43

Friday, May 7, 2010


ConsentTech set-upThe beef

Also see37-43

Friday, May 7, 2010


ConsentTech set-upThe beefDe-installation

Also see37-43

Friday, May 7, 2010


ConsentTech set-upThe beefDe-installationThanks & incentive

Also see37-43

Friday, May 7, 2010

Chances are they’ve never done this before

Friday, May 7, 2010

What they worry about

Friday, May 7, 2010

Who are you and can I trust you?

Friday, May 7, 2010

If you can see my screen,can you see my face?

Friday, May 7, 2010

Will you be able to takecontrol of my computer?

Friday, May 7, 2010

Will you help me get the plug-inoff my computer at the end?

Friday, May 7, 2010

Will this work if I have afirewall/proxy/strict IT department?

Friday, May 7, 2010

Is this going to end up onYouTube?

Friday, May 7, 2010

Will I really get the incentiveif I do this?

Friday, May 7, 2010

How long will it take?

Friday, May 7, 2010

Is it OK if I have to take a quick callin the middle?

Friday, May 7, 2010

Is it OK if my kids/dog/fratmates are making noise in the background?

Friday, May 7, 2010

OXYMORON?Remote empathy

Friday, May 7, 2010

Phone Interviewing Specifics

Friday, May 7, 2010

Pitfalls

Things sound leading

Friday, May 7, 2010

Body language is hugeActual language is important too

Make sure you’re comfortable

Friday, May 7, 2010

EXERCISE

Practice screensharingModerator, Participant, at least one observerUse the script in your packet as a starting pointGet your participant all set up and ready to start an interview

Friday, May 7, 2010

Friday, May 7, 2010

How did it go?

Friday, May 7, 2010

Advanced TechniquesHigh Emotions

Friday, May 7, 2010

Advanced TechniquesOne to many moderation

Multi-threadingFriday, May 7, 2010

Future Frontiers

Friday, May 7, 2010

BREAK

Friday, May 7, 2010

Remote Research Toolkit

UNMODERATED

Friday, May 7, 2010

3 parts

Provide test targetRecord user behaviorCapture comments (optional)

Friday, May 7, 2010

AUTOMATED Tools

Friday, May 7, 2010

What do you want to do?

Get freeform data cheaplyQuantify user behaviorsConceptual & IA testing

Friday, May 7, 2010

This stuff is pretty cheap

• Usabilla $0-950/yr• Optimal Workshop $559-1350/yr• Loop11 $350/study• OpenHallway $19-199/mo• UserTesting.com $39/user• Feedback Army $10/10 users

Friday, May 7, 2010

UserTesting.com

Friday, May 7, 2010

Using your own recruitsSet up an accountEmail [email protected] with:

the URL where your target livesthe scenario (500 char limit)tasks (2000 char limit)up to 4 questions to be answered in writing at the end

Request a UserTesting URL for your study

Friday, May 7, 2010

mailto:[email protected]


Loop11

Friday, May 7, 2010

Usabilla

Friday, May 7, 2010

Optimal Workshop

Treejack Chalkmark

Friday, May 7, 2010

EXERCISE

Choose one of these toolsSign up for a test accountCreate a 15-minute test for any website you like

Friday, May 7, 2010

Friday, May 7, 2010

RESULTS

Friday, May 7, 2010

Session VideosHeatmaps

NotesCompletion RatesCard Sort Groups

Lions, Tigers, Bears...

Friday, May 7, 2010

Telling a storyIf you have time-coded notes, you can make highlight videosIf you have user comments, you can do tag analysis or make word cloudsIf you have numbers, you can compare to benchmarksIf you have card groups you can do cluster analysis

Friday, May 7, 2010

Case Study: Data Mismatch

Friday, May 7, 2010

Qualitative Results

16 remote interviewsWorst findability BP has ever observedRecommendation to burn it down and start over

Friday, May 7, 2010

Your task is to make an appointment with a pediatric arrhythmia specialist at Children’s Hospital. Once you have found it, please remember or write down the phone number you would call to make an appointment by phone, as you will be asked for that information later on.

Answers distribution N Mean* 1 2 3 4 5 6 7

! The phone number search has been... *[1=Extremely difficult / 7=Extremely easy]

2% (1) 2%

(1) 10% (5) 18%

(9) 18% (9) 24%

(12) 26% (13) 50 5.2

EFFECTIVENESS RATIOS EFFICIENCY Success users Only

Time (mm:ss) Mean time per task 01:36 Standard deviation 00:47

Clicks Mean clicks per task 6

(exact value: 6) Standard deviation 4

(exact value: 4.4)

???

Quantitative Data

Friday, May 7, 2010

What happened?

Friday, May 7, 2010

32 page report, distilled

1 terrible 2 3 4

neutral 5 6 7 excellent UserZoom’s red

flag minimum

Hospital reputation Search ease

of use Navigation ease of use

Friday, May 7, 2010

What’s different IRL?

Friday, May 7, 2010

Recording1 more button to press

Friday, May 7, 2010

Observer inputOne more stream on your screen

(Possibly 2 or 3)

Friday, May 7, 2010

NotesTap, tap, tapping away

Friday, May 7, 2010

So, what could go wrong?

Friday, May 7, 2010

3 points of failure

Technologyyours

participant’s

observer’s

RecruitingTest Design

Friday, May 7, 2010

Control what you can control

Which is, your set up.You wouldn’t go backpacking without checking your gear.Do it 24 hours in advance. Or else.

Page 240-241

Friday, May 7, 2010

Prepare for whatyou can’t control

User’s dog trips over power cordInternet affected by local weatherUser can’t installDoesn’t know how to install

Friday, May 7, 2010

Mid-SessionTech

Problems

Friday, May 7, 2010

Live Recruiting Problems

What if they gave a study and nobody came?

Friday, May 7, 2010

Increase exposure

When you’re not getting enough viewsWhen the wrong type of users are responding

Friday, May 7, 2010

Increase incentives

When it’s cheaperWhen you’re almost getting enoughWhen segments are tough

Friday, May 7, 2010

Switch methodsWhen base traffic level isn’t high enough

Friday, May 7, 2010

Script ProblemsCut the BS.

Friday, May 7, 2010

The bottom lineIf they’re not engaged,

something’s wrong.

Friday, May 7, 2010

Burn the scriptFind out what matters.

Friday, May 7, 2010

QUESTIONS

Friday, May 7, 2010

@[email protected]

[email protected]

Friday, May 7, 2010



kthxbai.

Friday, May 7, 2010

etl may 2010

Technology

remote studies

remote fail

remote usability

remote win

remote interviews

best remote research

real time observers

timeaware tasks