living laboratories': rethinking ecological designs and experimentation in human-computer...

EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter

2009HCICWorkshopatYMCACamp,Colorado

  Asafield,earlyfundamentalcontributionsfrom:–  Computerscientistsinterestedinchangesinwaysweinteract

withinformationsystems–  Psychologistsinterestedintheimplicationsofthesechanges

  Combustible,because:–  Computerscientistswanttocreategreattools,butdidn’tknow

howtomeasureimpact–  Psychologistswanttogobeyondclassicalresearchofthebrain

andhumancognition

  Example:–  anenduringinterestin‘augmentinghumanintellect’:V.Bush,

Licklider,Engelbart,inturninspiringStuCard,AlanNewell,AlanKay,andmanyothers.

2/13/09 HCIC "Living Lab" 2

  TheneedtoestablishHCIasascience–  Adoptmethodsfrompsychology–  Convenientandfitswellwithproblemsathand

  Issuesaroundpersonalcomputing(WIMP)–  Dualpurpose:understandnatureofhuman

behaviorandbuildupascienceofHCItechniques.

–  GoodExamples:Fitts’Law,ModelsofHumanMemory,CognitiveandBehavioralModeling,InformationForaging

–  StuartK.Card,WilliamK.English,andBettyJ.Burr(1978).Evaluationofmouse,rate‐controlledisometricjoystick,stepkeys,andtextkeysfortextselectiononaCRT.Ergonomics,21(8):601–613,1978.

2/13/09 3 HCIC "Living Lab"

  Beyondauserinfrontofcomputer–  Yetevaluationmethodsmostlystayedthesame–  PerceivedCHIpapertemplateforacceptance

  Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore–  Yesterday’sdiscussionaboutLargeDataandHCIwaslargelydrivenby

howHCIevaluationmethodsneedtochangetofitthewild

  Thebestexamples:TrendsinSocialComputingandUbiCompforceustothinkaboutnewcontextofuse


Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with

no time boundries Controllable experimental conditions Uncontrollable experimental conditions


  ThinkaboutresearchthathasbeendoneonUIanimationsorflashingicons.

  Fromvisualperception,weknowmotionintheperipheryismorenoticeablethaninthefovealregion[DaVinci].


  EvaluationssurroundingmanyHCIsystemsforknowledgeworkfocusonproductivityincrease,butwhataboutfactorsforadoption?–  Argument:ifnoproductivityincrease,thenadoptionis

irrelevant–  Buttheoppositeargumentisjustasright:ifnoadoption,no

amountofproductivityincreaseshownisrelevant!

  Academicresearchoftenfocusonproductivityimprovements,increasingtheperceivedgulfbetweentheivorytowerandthetrenches

  AnExample:Colorcopierstudies


  Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations–  Ecologicalconsiderations–  Hardtogeneralizetonewtaskcontexts(withinterruptions,other

tasks,othergoals,unfocusedattention,moredisplays)–  Hardtogeneralizetoothertools,apps–  Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups

  Exampleproblems:–  Adoptionofmobiletechnology

  iPhonesinJapan,single‐handedinput[PARC]  BestsellingphonesinIndonesiacomeswithacompass[Bell]

–  AggregatebehaviorofWikipediaorDelicioususers  Bigdataanalysisofeditlogs


  Wasacomputationalmolecularbiologist  Analogy:Justasbiologistsworkonmodelplantsand

genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.

  Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.


  ObservationalStudies–  Ethnography–  SocialTechnicalDesign–  IterativeDesign–  DiaryStudies–  LongitudinalStudieswithsingleoutcome

  Problems:–  Samplingfromnon‐normaldistributions–  Treatvariationsinsocialcontextsas‘noise’–  Whataboutadoption?

  MixedMethodsisnowpopulartopartiallyameliorate–  Convergentmeasurestellsuswe’regettingclosertothetruth


–  similartoVentureBusinessmentionedbyDavidMillen

  Conductresearchonrealplatformsandservices–  Nottoreplacecontrolledlabstudies–  Expandourarsenaltocovernewsituations

  Someprinciples:–  Embeddedintherealworld–  Ecologicallyvalidsituations–  Embracethecomplexity–  Relyonbig‐data‐sciencetoextractpatterns

  Notfirsttosuggestthis:–  S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On

ecologicalvalidityandubiquitouscomputing.HCIJournal,2008–  EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital

FamilyPortrait[Rowan,Mynatt]


  Twodimensions–  1.Whetherthesystemisunderthecontroloftheresearcher–  2.Whetherthestudyisconductedinthelaborinthewild

System Control System Not in Control

Laboratory (1) Build a system, study in the Lab

(2) Adopt a system, study in the Lab

Wild (Real World)

(4) Build a system, release it, study in the Wild

(3) Adopt a system, study in the Wild


  TraditionalApproach;Numerousexamples  FavoredbyCHIreviewers  Typicalsituationisthestudyofsomeinteractiontechnique

–  Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput

  Typicalmeasuresarequantitativeinnature–  performanceintime,performanceinaccuracy,eyetracking,learning

measures,userpreferences

  Issues:–  Notalwaysecologicallyvalid–  Hardtotakeallinteractionsintoaccount–  Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.


  Hardertofindintheliterature  Oftencomparingagainstanoldersystemasbaseline  Typicalcaseiscomparisonoftwosystems

–  (onewebsitewithanother,onewordprocessorvs.another)–  Whichhighlightingfeatureworksbetter–  Twotextinputtechniqueonacellphone

  Typicalmeasuresaresimilarto(1)  Issues:

–  Somesimilarissuesto(1)becauseit’sinlab–  Systemfeaturenotincontrol,sonotabletocomparefairly,or

isolatethefeature



http://www.flickr.com/photos/crush_images/3245220458/

  Realapplicationsinecologicalvalidsituations  Realfindingscanbeappliedtoarunningsystem  Impactofresearchismoreimmediate,sincesystemisalready

running  Typicalcaseisloganalyticswithlargesubjectpools

–  logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.

  Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis

  Issues:–  Factorsnotincontrol,findingsnotcomparable–  Factorscannotbeisolated–  Reasonsforfailureisoftenjustguesswork


  Hypothesis:ConflictiswhatdrivesWikipediaforward.  Howtostudythis?

–  Tukeyparadigm–  Getalargepaper,andplotthedamndata!

–  DownloadedallofWikipediaandalloftherevisions–  Hadoop/MapReduce,MySQL,etc.


2/13/09 18

60%

65%

70%

75%

80%

85%

90%

95%

100%

2001 2002 2003 2004 2005 2006

Perc

enta

ge o

f tot

al e

dits

Article

User

Article Talk

User Talk

Other

Maintenance

HCIC "Living Lab"

Group A

Group B Group C

Group D

Number of users in user group A B C Total

Users with Korean point of view 10 6 0 16

Users with Japanese point of view 1 8 7 16

Neutral or Unidentified 7 3 6 17 2/13/09 19 HCIC "Living Lab"

Mediators

Sympathetic to parents

Sympathetic to husband

Anonymous (vandals/spammers)


  Hypothesis:SocialTaggingdoesn’tscaleovertime.  Howtostudythis?

–  Crawlasmuchtaggingdataaswecan.–  Studythenoiseinthesystem.

–  40machinesfor3months


Topics

Users Documents

Tags

T1…TnEncodingDecoding

Noise


Concepts


Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)

29 2/13/09 HCIC "Living Lab"

  Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.–  Goodforcasesinwhicheconomicsmakessense[Google]–  Changestosystemispossible;Factorscanbecontrolled.

  TypicalcasemightbeA/Btesting,largesubjectpools  Typicalmeasuresarebeingdeveloped

–  Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?

–  Usabilitymeasuresvs.Usefulnessmeasures

  Issues:–  Effortandresourcerequirementisdroppingbutstillsignificant–  Hardforaresearchlabtotakeon



  CollaborativeFiltering

  TapestrysystematPARC

  GroupLens  6kusers  Ecosystemwith

passionateusers

2/13/09 34

HCIC "Living Lab"

2/13/09 35

HCIC "Living

Lab"

2/13/09 36

HCIC "Living

Lab"

2/13/09 37

Guide

Web

Howto

Tips Help

Tools

Tip

Tricks

Tutorial

Tutorials

Reference

Semantic Similarity Graph

HCIC "Living Lab"

  Interactioncostsdeterminenumberofpeoplewhoparticipate

  Surplusofattention&motivationatsmalltransactioncosts

  Therefore…  Importanttokeep

interactioncostslow

Cost of participation #

Peop

le w

illin

g to

pro

duce

for “

free

”


  In situ tagging while reading –  No new window –  Clicking vs typing

  Tagging + highlighting


  Exitingthecleanroom[Carteretal.]–  SparseData–  CriticalMass–  Dealingwithrealisticenvironments–  Unobtrusivenessofmonitoring–  Ambiguityanderrorofinferencing


  Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.

  Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.


Characteriza7on Models

PrototypesEvalua7ons

  ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.

  LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.

http://asc‐parc.blogspot.comhttp://[email protected]

WikiDashboard MrTaggy SparTag.us

living laboratories': rethinking ecological designs and experimentation in human-computer...

Technology

hcic quotliving labquot2

hcic quotliving labquot3

hcic quotliving labquot9

lab study

lab wild real

long tasks

informationforaging

digit familyportraitrowan