Download - ETL May 2010
ESCAPE THE LAB
Pleasestand by
WILL!BEGIN!AT 9:00
Friday, May 7, 2010
ESCAPE THE LABLET’S!GET!STARTED!MMKAY?
Friday, May 7, 2010
Friday, May 7, 2010
I’m CydHe’s Nate
Friday, May 7, 2010
#ETLAB
Friday, May 7, 2010
remoteUSABILITY
Friday, May 7, 2010
oh
Friday, May 7, 2010
and
Friday, May 7, 2010
also
Friday, May 7, 2010
NO
Friday, May 7, 2010
time
Friday, May 7, 2010
travel
Friday, May 7, 2010
OR
Friday, May 7, 2010
lasers
Friday, May 7, 2010
HOW!TO!BEgraceful
When All Your
TECHNOLOGYBREAKS
Friday, May 7, 2010
PLEASE DON’T TRIPON!THE!CABLES
Friday, May 7, 2010
Follow the buddy system
Friday, May 7, 2010
How about you all?
Best remote research storyMost important thing to get out of today?Biggest fear or skepticism about remote research?
Friday, May 7, 2010
Go TO escapethelab.com NOW
Friday, May 7, 2010
Friday, May 7, 2010
Remote Research
Definition:
Research where the participant is not sharing physical space with the researcher.
In personRemote
% of qualitative research done remotely*
*Source: BP’s best guess
Friday, May 7, 2010
What we’ll cover today
RecruitingModeratedUnmoderatedResultsPitfalls
Friday, May 7, 2010
We shall begin...with a little experiment
Friday, May 7, 2010
EXERCISEJUMP IN THE POOL
Table groups
Pick one moderator and one participant
(they need to be in different locations)
10 minutes of moderating
Friday, May 7, 2010
MODERATOR Instructions
Your participant has a sheet of paper and a linkGoal: find out what’s hardest about presenting physical task instructions onlineHint: ask them to give you a play by play of what they are doingFollow along with your own paper
NO helping, even if you could.
Friday, May 7, 2010
It’s A RACE
First participant to make an origami giraffe is the winner
Friday, May 7, 2010
Friday, May 7, 2010
Surprises?
Friday, May 7, 2010
When does remote win?
Page 10-15
Friday, May 7, 2010
Native Environment
Cheap ethnography, not cheap lab.
Friday, May 7, 2010
Real Time
Allows self-motivated tasks.
Friday, May 7, 2010
Involved Observers
Easier to collaborate in real time.
Friday, May 7, 2010
Geography
Friday, May 7, 2010
When does remote fail?
Friday, May 7, 2010
Bandwidth
Yes, it’s still an issue.
Friday, May 7, 2010
Translation
It’s expensive and difficult to do live.
Friday, May 7, 2010
Minors
Consent is a bear.Especially if they’re under 13.
Friday, May 7, 2010
FacesWe’ve all got webcams today.
Most users don’t.Webcams are invasive.
Friday, May 7, 2010
CASE STUDY: Epic FAIL
Friday, May 7, 2010
What went wrong
Coins were way more valuable than we expectedPeople mis-assessed their English levelsBandwidth was really a problemNobody plays Habbo when their parents are home to consent
Friday, May 7, 2010
making the most
Friday, May 7, 2010
TIME-AWARE TasksTold to find purple pumps.
Doesn’t care.
Has to wear this for sister’s wedding this weekend.
Whole different interaction.
Friday, May 7, 2010
Real Time OBservers
More on page 109-111
Friday, May 7, 2010
Change on the fly
“No one seems engaged in the laptop customizer”
“Let’s add questions about who they get recommendations from”
Friday, May 7, 2010
Designing Remote Studies
Friday, May 7, 2010
Friday, May 7, 2010
Friday, May 7, 2010
plan these things
Friday, May 7, 2010
plan these things
MethodRecruitingLogistics
Friday, May 7, 2010
MODERATED must-haves
Friday, May 7, 2010
MODERATED must-haves
Consent
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-up
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beef
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beefDe-installation
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beefDe-installationThanks & incentive
Friday, May 7, 2010
UNMODERATED must-haves
Friday, May 7, 2010
UNMODERATED must-haves
Expectation setting
Friday, May 7, 2010
UNMODERATED must-haves
Expectation settingInstructions
Friday, May 7, 2010
UNMODERATED must-haves
Expectation settingInstructionsThe beef
Friday, May 7, 2010
UNMODERATED must-haves
Expectation settingInstructionsThe beefThanks & incentive
Friday, May 7, 2010
Recruiting
Friday, May 7, 2010
Recruiting
Same: Good “talkers”Same: Demographic fits
Friday, May 7, 2010
Recruiting
Same: Good “talkers”Same: Demographic fitsDifferent: Relevant current tasksDifferent: Compatible tech
Friday, May 7, 2010
Logistics: What & How
Provide test targetAccessCredentials
Friday, May 7, 2010
Logistics: Who & When
How to connect with the user (we’ll discuss in Tools)
Managing time zonesSession length
Friday, May 7, 2010
Paying Incentives
Collect minimal personal infoDeliver quicklyAppropriate for location
Friday, May 7, 2010
Advanced planning:hybrid studies
Why use only one method?Some methods don’t tell you the why3-5 remote interviews is all you need
Friday, May 7, 2010
Portable Research
Friday, May 7, 2010
BREAK
Friday, May 7, 2010
Recruiting for Remote
Friday, May 7, 2010
We all need people
Relevant tasksCompatible techGood talkersDemographic fitsIn the moment
Friday, May 7, 2010
Gold Standard
Live recruitingGet as close as you can
Page 52
Friday, May 7, 2010
Recruiting Options
EthnioLinks or house ads to online formsRecruiting agenciesCustomer email listsPanelsCraigslist/online ads
Friday, May 7, 2010
use Ethnio when
Website accessSome budgetQuotas are relatively simple
Friday, May 7, 2010
use forms when
You can place a link but not codeLow budgetQuotas are relatively complexOptions:
Google Docs
Wufoo
Friday, May 7, 2010
Google docs
Friday, May 7, 2010
use Agencies when
No website accessHigher budgetOnline recruiting failed
Friday, May 7, 2010
Customer lists
Sub-segmentsOwnership requirementsNo website access
Friday, May 7, 2010
PANELSFriday, May 7, 2010
Craigslist & forums
When other methods don’t workNo doughTakes a whileEmbarrassing to admit
Friday, May 7, 2010
the Gold Standard again
Live recruitingGet as close as you can
Friday, May 7, 2010
Live Recruiting Math(for interviews)
Target: 6 recruits/hourAllow for: 1% response (low average)
You Need: 8-10,000 uniques/day
Friday, May 7, 2010
Page 56
Friday, May 7, 2010
Screener constraints
Keep it shortDifferent from surveysWho gets paid?
Page 60-63
Friday, May 7, 2010
Remember the Point
Tasks that align with research goalsTechnical compatibilityControl demographics
Friday, May 7, 2010
EXERCISE
Friday, May 7, 2010
Friday, May 7, 2010
Your best questions
Friday, May 7, 2010
GO EAT LUNCH
Famished
Friday, May 7, 2010
exercise
We recommend:Split Pea Seduction, 6th & Minna
Custom Burger, 7th & Natoma
Tony Baloney’s, 7th & Howard
Tin Vietnamese, Howard & 6th
Harvest Market, Natoma & 8th
Sightglass Coffee, 7th & Folsom
Friday, May 7, 2010
exercise
We recommend:Split Pea Seduction, 6th & Minna
Custom Burger, 7th & Natoma
Tony Baloney’s, 7th & Howard
Tin Vietnamese, Howard & 6th
Harvest Market, Natoma & 8th
Sightglass Coffee, 7th & Folsom
Friday, May 7, 2010
ESCAPE THE LAB
Pleasestand by
WILL!RESUME!AT 1:15
Friday, May 7, 2010
welcome back
Friday, May 7, 2010
Tweet why you hate remote research #ETLABWin a copy of Silverback
Time for a
contest!
Friday, May 7, 2010
Remote Research Toolkit
MODERATEDmost of
Chapter 8
Friday, May 7, 2010
5 easy pieces
Talk to the userSee what they’re doingTake notes (optional)
Involve observers (optional)
Record audio & video (optional)
Friday, May 7, 2010
Communication
Reliable
Can be expensive for international
Conference calls needed for observers
phone
Cheap or free
Harder for users
Can combine with screensharing to stress user’s connection
Skype
Friday, May 7, 2010
Adobe Connect
LiveLook
WebEx*
GoToMeeting
Standalone
UserVue (RIP)
FuzeMeeting (new)
Integrated
Screensharing
*WebEx also has mobile versions
Friday, May 7, 2010
Adobe Connect
Friday, May 7, 2010
FuzeMeeting(New)
Friday, May 7, 2010
Livelook
Friday, May 7, 2010
Webex
Friday, May 7, 2010
GoToMeeting
Friday, May 7, 2010
Taking Notes
Second monitor if possibleExcel Hack (for time codes)
Friday, May 7, 2010
How to rig your notes
Leave one column (A) empty for notes.
Set the cells of the next column (B) to record the clock time of each entry in A.
=IF(A2=“”,“”,IF(B2=“”,NOW(),B2))
Set the cells of the next column (C) to record the difference between the first entry and the current entry
=IF(ISERROR(B2-$B$2),“”,B2-$B$2)
Fill down (Ctrl-D). Expect to use up to 120 rows for a session.
Hide column B. You’ll never need to see it
What the formulas look like
Auto timestamp!
Don’t write this
down, it’s on page 104
Friday, May 7, 2010
Observer Involvement
IMWeb Chat RoomsIntegrated ChatPassing Notes
Friday, May 7, 2010
Recording Tools
Phone Patch
Cables
HardwareCamtasia (Windows)
iShowU HD (Mac)
Software
Friday, May 7, 2010
Phone patch set up
Friday, May 7, 2010
Friday, May 7, 2010
Simple Set-up
Friday, May 7, 2010
Intermediate Set-up
Friday, May 7, 2010
Advanced Set-up
Friday, May 7, 2010
Friday, May 7, 2010
exercise
Roll your ownWhat will be your base set-up?
Write it down
Friday, May 7, 2010
Friday, May 7, 2010
Your version?
Friday, May 7, 2010
Winners!!
Friday, May 7, 2010
Moderating for Remote
Friday, May 7, 2010
Is it really any different?
(Short answer: some.)
Page 94-99
Friday, May 7, 2010
Question from the BP inbox yesterday
It would have actually helped to know how many users the authors recommend for a tar* type of test. In a regular user test , many of us have seen how findings repeat a#er 5 -8 users of each user type are observed. However, here we don't always know which task a user will be doing when we interrupt them. So in cases where we really are following users along as they complete what they came to do, how many useres should we target ? Given that the authors have been doing this for a while a recommendation from them on how many users to target and their notes on the topic would have been really interesting.
*Time-Aware Research
Friday, May 7, 2010
Making passionate tasks work
Friday, May 7, 2010
Making passionate tasks work
Recruit for relevant current tasksScript for customizable tasks
Friday, May 7, 2010
Re-script for remote
Friday, May 7, 2010
Re-script for remote
“Ok, I’d like you to start from the homepage, pretend you are shopping for groceries for
a party, and buy some paper plates.”
Friday, May 7, 2010
MODERATED must-haves
Also see37-43
Friday, May 7, 2010
MODERATED must-haves
ConsentAlso see
37-43
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-up
Also see37-43
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beef
Also see37-43
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beefDe-installation
Also see37-43
Friday, May 7, 2010
MODERATED must-haves
ConsentTech set-upThe beefDe-installationThanks & incentive
Also see37-43
Friday, May 7, 2010
Chances are they’ve never done this before
Friday, May 7, 2010
What they worry about
Friday, May 7, 2010
Who are you and can I trust you?
Friday, May 7, 2010
If you can see my screen,can you see my face?
Friday, May 7, 2010
Will you be able to takecontrol of my computer?
Friday, May 7, 2010
Will you help me get the plug-inoff my computer at the end?
Friday, May 7, 2010
Will this work if I have afirewall/proxy/strict IT department?
Friday, May 7, 2010
Is this going to end up onYouTube?
Friday, May 7, 2010
Will I really get the incentiveif I do this?
Friday, May 7, 2010
How long will it take?
Friday, May 7, 2010
Is it OK if I have to take a quick callin the middle?
Friday, May 7, 2010
Is it OK if my kids/dog/fratmates are making noise in the background?
Friday, May 7, 2010
OXYMORON?Remote empathy
Friday, May 7, 2010
Phone Interviewing Specifics
Friday, May 7, 2010
Pitfalls
Things sound leading
Friday, May 7, 2010
Body language is hugeActual language is important too
Make sure you’re comfortable
Friday, May 7, 2010
EXERCISE
Practice screensharingModerator, Participant, at least one observerUse the script in your packet as a starting pointGet your participant all set up and ready to start an interview
Friday, May 7, 2010
Friday, May 7, 2010
How did it go?
Friday, May 7, 2010
Advanced TechniquesHigh Emotions
Friday, May 7, 2010
Advanced TechniquesOne to many moderation
Multi-threadingFriday, May 7, 2010
Future Frontiers
Friday, May 7, 2010
BREAK
Friday, May 7, 2010
Remote Research Toolkit
UNMODERATED
Friday, May 7, 2010
3 parts
Provide test targetRecord user behaviorCapture comments (optional)
Friday, May 7, 2010
AUTOMATED Tools
Friday, May 7, 2010
What do you want to do?
Get freeform data cheaplyQuantify user behaviorsConceptual & IA testing
Friday, May 7, 2010
This stuff is pretty cheap
• Usabilla $0-950/yr• Optimal Workshop $559-1350/yr• Loop11 $350/study• OpenHallway $19-199/mo• UserTesting.com $39/user• Feedback Army $10/10 users
Friday, May 7, 2010
UserTesting.com
Friday, May 7, 2010
UserTesting.com
Friday, May 7, 2010
UserTesting.com
Friday, May 7, 2010
UserTesting.com
Friday, May 7, 2010
Using your own recruitsSet up an accountEmail [email protected] with:
the URL where your target livesthe scenario (500 char limit)tasks (2000 char limit)up to 4 questions to be answered in writing at the end
Request a UserTesting URL for your study
Friday, May 7, 2010
Loop11
Friday, May 7, 2010
Loop11
Friday, May 7, 2010
Loop11
Friday, May 7, 2010
Loop11
Friday, May 7, 2010
Loop11
Friday, May 7, 2010
Usabilla
Friday, May 7, 2010
Optimal Workshop
Treejack Chalkmark
Friday, May 7, 2010
EXERCISE
Choose one of these toolsSign up for a test accountCreate a 15-minute test for any website you like
Friday, May 7, 2010
Friday, May 7, 2010
RESULTS
Friday, May 7, 2010
Session VideosHeatmaps
NotesCompletion RatesCard Sort Groups
Lions, Tigers, Bears...
Friday, May 7, 2010
Telling a storyIf you have time-coded notes, you can make highlight videosIf you have user comments, you can do tag analysis or make word cloudsIf you have numbers, you can compare to benchmarksIf you have card groups you can do cluster analysis
Friday, May 7, 2010
Case Study: Data Mismatch
Friday, May 7, 2010
Qualitative Results
16 remote interviewsWorst findability BP has ever observedRecommendation to burn it down and start over
Friday, May 7, 2010
Your task is to make an appointment with a pediatric arrhythmia specialist at Children’s Hospital. Once you have found it, please remember or write down the phone number you would call to make an appointment by phone, as you will be asked for that information later on.
Answers distribution N Mean* 1 2 3 4 5 6 7
! The phone number search has been... *[1=Extremely difficult / 7=Extremely easy]
2% (1) 2%
(1) 10% (5) 18%
(9) 18% (9) 24%
(12) 26% (13) 50 5.2
EFFECTIVENESS RATIOS EFFICIENCY Success users Only
Time (mm:ss) Mean time per task 01:36 Standard deviation 00:47
Clicks Mean clicks per task 6
(exact value: 6) Standard deviation 4
(exact value: 4.4)
???
Quantitative Data
Friday, May 7, 2010
What happened?
Friday, May 7, 2010
32 page report, distilled
1 terrible 2 3 4
neutral 5 6 7 excellent UserZoom’s red
flag minimum
Hospital reputation Search ease
of use Navigation ease of use
Friday, May 7, 2010
What’s different IRL?
Friday, May 7, 2010
Recording1 more button to press
Friday, May 7, 2010
Observer inputOne more stream on your screen
(Possibly 2 or 3)
Friday, May 7, 2010
NotesTap, tap, tapping away
Friday, May 7, 2010
So, what could go wrong?
Friday, May 7, 2010
3 points of failure
Technologyyours
participant’s
observer’s
RecruitingTest Design
Friday, May 7, 2010
Control what you can control
Which is, your set up.You wouldn’t go backpacking without checking your gear.Do it 24 hours in advance. Or else.
Page 240-241
Friday, May 7, 2010
Prepare for whatyou can’t control
User’s dog trips over power cordInternet affected by local weatherUser can’t installDoesn’t know how to install
Friday, May 7, 2010
Mid-SessionTech
Problems
Friday, May 7, 2010
Live Recruiting Problems
What if they gave a study and nobody came?
Friday, May 7, 2010
Increase exposure
When you’re not getting enough viewsWhen the wrong type of users are responding
Friday, May 7, 2010
Increase incentives
When it’s cheaperWhen you’re almost getting enoughWhen segments are tough
Friday, May 7, 2010
Switch methodsWhen base traffic level isn’t high enough
Friday, May 7, 2010
Script ProblemsCut the BS.
Friday, May 7, 2010
The bottom lineIf they’re not engaged,
something’s wrong.
Friday, May 7, 2010
Burn the scriptFind out what matters.
Friday, May 7, 2010
QUESTIONS
Friday, May 7, 2010
Friday, May 7, 2010
kthxbai.
Friday, May 7, 2010