TakingSurveystoPeople’sTechnology:Implica:onsforFederalSta:s:cs

andSocialScienceResearch

Frederick G. Conrad Michael F. Schober

Collaborators: Christopher Antoun, David Carroll, Patrick Ehlen, Stefanie Fail, Andrew L. Hupp, Michael Johnston, Courtney Kellner, Kelly F. Nichols, Leif Percifield, Lucas Vickers, H. Yanna Yan, & Chan Zhang

Support:NSFgrantSES-1025645(Methodology,Measurement,andSta?s?csprogram)

Mobilemul?modalphones(smartphones)

•  Peopleincreasinglycommunicateviasmartphones–  >68%ofUSadultsasof2015–  Con?nuinggrowth

•  Peopleincreasinglyuseandswitchbetweenmul?plemodes(manyna?vetosmartphone)forinterac?ng–  Voice–  Text(SMS)–  Email–  Videochat– Webaccessforcommunica?on(e.g.,blogposts)–  Specializedappsforcommunica?on(e.g.,Facebook)

Peopleincreasinglyexpectcapabilityto:•  communicatewhilemobileand/ormul?tasking•  chooseamodethatfitstheircurrentseWngandneeds

–  e.g.,urgentvs.canwait,publicvs.private,noisyvs.quiet,brightvs.dim

•  switchmodeswhilecommunica?ng•  respondinadifferentmodethancontacted

–  e.g.,respondtovoicemailwithatext

Newpossibili?esandchallengesforsurveydatacollec?on

•  AsR’sexpectmul?plemodesonthesamedevice,mayexpectthatsurveysaremul?modal–  Poten?altointeractviaSMStextwhenconvenient–  Poten?altorespondinmodethatisappropriatetocurrentseWng

(e.g.,textinnoisyenvironment,voicewhenthereisglare,etc.)–  Moregenerally,tobeabletorespondinanymode,any?me,

anywhere

•  Dopeoplerespondtoconven?onalsurveymodes(e.g.,telephoneinterviews)inthesamewayonsmartphonesasonlandlines?

•  Howdopeoplerespondtolessconven?onalsurveymodesthatusesmartphonecapabili?es?

Newop?onsforsurveymodechoice•  Nowpossibletochooseamodeonasingledevice,immediatelyandconveniently

•  Quitedifferentfrompriorimplementa?onsofsurveymodechoice– WhenRinvitedbymailcompleteseitheronpaperorweb,thisrequiresextrastepoftypingURLintobrowser

–  Canreduceresponserates(e.g.,Fulton&Medway,2012)

•  Choiceonsingledevicemayleadtodifferentoutcomes

Modecomparisonstudy(Schoberetal.,2015,PLOSONE)

•  examines–  dataquality(sa?sficing,disclosure)–  comple?onrates–  respondentsa?sfac?on

•  fourexis?ngorplausiblesurveymodesthatworkthroughna?veappsontheiPhone– Asopposedtospeciallydesignedsurveyapps– Asopposedtowebsurveyinphone’sbrowser– UniforminterfaceforallRs

•  Asopposedtomixofplacorms(Android,Windows,etc.)

Experiment:4modesoniPhone

Medium

Voice SMSText

Interviewing

Agent

Human

Humanvoice

(RspeakswithI)

Humantext

(RtextswithI)

Automated

SpeechIVR

(Rspeakswithsystem)

AutomatedText

(Rtextswithsystem)

Surveysviatextmessaging?•  Moreandmorepeopleareembracingtextmessagingforpersonalandprofessionalcommunica?on–  ontheirmobilephones(smartphonesornot)–  onotherdevices(e.g.,tablets,desktops)

•  Tex?ngisbecomingapoten?allyimportantwaytoreachrespondents–  somemayaeendtotextmorethantoemailsorvoicemails

–  respondentsmayexpecttobeabletopar?cipateinasurveyviatext

•  Someorganiza?onsarenowincludingSMStextintheirsuiteofmodesformobilesurveys–  e.g.,GeoPoll,PollEverywhere,iVisionMobile,etc.

Textasamodeofinterac?on•  Turn-by-turn

–  Threaded(onasmartphone)

•  Responsesdon’tneedtobeimmediate–  Allowsmul?tasking

•  Worksevenwithintermieentnetwork/cellservice–  unlikevoice

•  Doesnotrequirewebcapacityondevice–  unlikemobilewebsurvey

Property Voice Text Synchrony Fully synchronous Less or asynchronous

Medium Auditory Visual Language Spoken/heard Written/read

Conversational structure

Turn-by-turn, with potential for simultaneous speech

Turn-by-turn, rarely but possibly out-of-sequence

Persistence of turn No Yes

Persistence of entire conversation

No Yes, threaded

Social presence of partner

Continuous (auditory) presence

Intermittent evidence (when texts arrive)

Character of multitasking

Simultaneous, especially when hands free, unless other task involves talking

Switching required between texting and other tasks

Impact of environmental conditions

Potential interference from ambient noise

Potential interference from visual glare

Impact of nearby others

Others may hear answers; potential audio interference from others’ talk

Others unlikely to see text and answers on screen, though possible

t

Measuresofdataquality•  Conscien/ousresponding(lesssa?sficing)

–  Rsareknowntotakeshortcuts—to“sa?sfice”—differentlyindifferentsurveymodes

•  e.g.,Chang&Krosnick(2009),Heerwegh&Loosveldt(2008)–  Weexamine

•  roundednumericalresponses(e.g.,mul?plesof10)–  Unroundedanswersaremorelikelytoresultfromdeliberate,memory-basedthoughtprocessesthan

es?ma?on(Brown,1995;Conrad,Brown,&Dashen,2003)–  morelikelytobeaccurateinanswerstoobjec?vefactualques?ons(Holbrooketal.,2014)

•  straightlining(nondifferen?a?on)–  givingsameanswertobaeeryofQs

•  Disclosure(moreisbeeer)–  Rsopendisclosemoresensi?veinforma?onwhentheyself-administeraques?onnaire(websurveys,ACASI)

•  e.g.,Kreuteretal.(2009),Tourangeau&Smith(1996)

•  Par/cipa/onandcomple/on

Possibleoutcomes:Conscien?ousresponding

•  TEXTVS.VOICE–  Rsmightbelessconscien>ousintextbecausetheyimport“leasteffortstrategy”fromhowtheyusuallytext

– ORRsmightmoreconscien>ousintextbecausetheyfeelless?mepressuretorespondthaninspokeninterviews

–  andcananswerwhentheyareready•  HUMANVS.AUTOMATED

–  Rsmightbelessconscien?ouswithautomatedinterviewer(selfadministra?on)becausethereisnohumantomo?vatethemtobeconscien?ous

Possibleoutcomes:Disclosure•  TEXTVS.VOICE

–  Rsmightdisclosemoreintextbecauseoffewersocialcuesintheinterac?on

•  lessevidenceofreac?ontoanswers?•  more?metobecomfortablewithanswers?•  nooneelsecanheartheques?onsoranswers?

–  ORRsmightdiscloselessintextbecause•  theyworrythatothersmightseevisuallypersistentanswers?•  theyworrythatanswersarepermanentlystored?•  theycantake?metoanswerinwaysthatgivethebestimpression?

•  HUMANVS.AUTOMATED–  Rsmightdisclosemorewithautomatedinterviewer,asinACASIorwebsurvey

Items•  First,safe-to-talkorsafe-to-textques?on•  32QstakenfrommajorUSsocialsurveysandmethodological

studies–  E.g.,BRFSS,NSDUH,GSS,PewInternet&AmericanLifeProject–  Formost,knowntohaveproduceddifferencesinsa?sficingor

disclosurebetweenconven?onalmodes•  Yes/no,numerical,categorical,baeeryitems(seriesofQs

withsameresponseop?ons)•  Ra?onaleforinclusion

–  Qswithmoreandlesssociallydesirableanswers•  e.g.,sexualhistory,druguse,newspaperreading

–  Qsforwhichfrequencyreportscouldbepreciseores?mated(rounded)

•  e.g.,numberofmoviesseenlastmonth,numberofappsoniPhone–  BaeeryQ’sthatcouldproducestraightlining(non-

differen?a?on)

Implementa?on:Humanvoice•  8interviewers(Is)fromUMichsurveyresearchcenter

•  customdesignedCATIinterfacethatsupportsvoiceandtextinterviews(PAMSS)

Implementa?on:Humantext•  Same8IsfromUMichsurveyresearchcenter•  SamecustomdesignedCATIinterface

–  Iselects,edits,ortypesques?ons/prompts,andclickstosend

•  Textmessagessentthroughthirdparty(Aerialink)•  Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),ornumber

Implementa?on:SpeechIVR

•  Custombuiltspeechdialoguesystem•  UsesATT’sWatsonspeechrecognizer,Asterisktelephonygateway

•  Recordedhumaninterviewer,speechresponses(nottouchtone)

Implementa?on:Auto-text•  Custombuilttextdialoguesystem•  Textmessagessentthroughthirdparty(Aerialink)•  Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),or

number

Respondents:634iPhoneusers•  n=157to160randomlyassignedtoeachmode•  RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk– Webscreenerverifiedage(>21years)andUSareacode–  iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse

•  $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire

•  Age,gender,ethnicity,income,educa?onnotreliablydifferentinfourmodes

•  SomewhatyoungerandlessaffluentthanUSna?onaliPhoneusers

TextRespondent

22

HumanTextInterviewerInterface

23

Datacollec?on

•  InterviewscarriedoutMarch-May2012

•  Resultsbasedonspeech-IVRsystemrecogni?on– 95.6%correctrecogni?onaccuracybasedontranscripts

– Samepaeernofresultsifweusehumanannota?ons(Johnston,etal.,2013)

Percent respondents reporting rounded numbers of… Human Auto Human Auto Estimate SE Odds

ratio Estimate SE Odds ratio Estimate SE Odds

ratio

Movies seen in theaters in past 12 months 24.4% 18.2% 17.1% 12.1% -0.463* 0.211 0.630 -0.383† 0.210 0.682 -0.035 0.425 0.965

Songs on iPhone 66.9% 61.8% 45.2% 51.6% -0.655*** 0.163 0.520 0.026 0.163 1.026 0.478 0.326 1.612

Apps on iPhone 80.6% 78.6% 47.5% 54.1% -1.332*** 0.179 0.264 0.112 0.175 1.119 0.391 0.358 1.479

Text messages sent and received on iPhone in current billing cycle 91.1% 90.1% 73.2% 70.5% -1.331*** 0.233 0.264 -0.125 0.214 0.882 -0.019 0.466 0.981

Times they ate spicy food in last month 46.5% 41.2% 38.6% 52.2% -0.325 0.228 0.640 -0.218 0.229 0.804 0.771* 0.323 2.162

Movies watched in any medium in last month 30.6% 40.9% 32.9% 30.6% -0.179 0.168 0.836 0.179 0.168 1.196 -0.557† 0.338 0.573

Times they shopped in a grocery store in last month 33.8% 41.0% 29.1% 35.0% -0.236 0.168 0.790 0.293† 0.168 1.340 -0.039 0.336 0.961

Times they ate in restaurants in last month 39.4% 36.7% 35.4% 36.9% -0.080 0.165 0.923 -0.025 0.165 0.975 0.178 0.329 1.195

†=p<.10 *p<.05 **p<.01 ***p<.001

Estimates come from the full model when the interaction is statistically significant (p<.05) and from the main effects analysis when the interaction is not statistically significant.

Interaction

VOICE VS. TEXT DIFFERENCES

MORE COMPLEX PATTERNS

NOT RELIABLY DIFFERENT

Text vs. VoiceAutomated vs. Human

InterviewerVoice Text

Conscien?ousresponding:Straightlining

•  Q:supportforvariousdietaryprac?ces(ea?ngredmeat,limi?ngfastfood,etc.)

»  stronglyfavor»  somewhatfavor»  neitherfavornoroppose»  somewhatoppose»  stronglyoppose

•  Wedefineanswersinbaeeryas“straightlining”whenatleast6of7responsesarethesame

•  Significantlylessstraightliningintextthanvoice

37

Table 5. Disclosure effects for each question.

Percent people reporting… Human Auto Human Auto Estimate SEOdds ratio Estimate SE

Odds ratio Estimate SE

Odds ratio

Having smoked at least 100 cigarettes in their entire life 39.2% 34.0% 42.4% 50.3% 0.404* 0.162 1.497 0.054 0.162 1.055 0.547† 0.326 1.727

Exercising less than 1 time per week in a typical week 13.1% 12.6% 21.5% 29.3% 0.838*** 0.212 2.312 0.239 0.206 1.270 0.462 0.425 1.587

Having had 3 or more sexpartners in the last 12 months 7.6% 10.1% 13.6% 14.3% 0.520* 0.257 1.681 0.160 0.254 1.174 -0.261 0.518 0.770

Personally watching television for five or more hours on the average day 10.7% 9.5% 15.9% 15.3% 0.499* 0.243 1.647 -0.083 0.239 0.921 0.084 0.486 1.087

Having had one or more drinks of analcoholic beverage on morethan 15 days of the past 30

10.6% 11.4% 8.2% 19.1% 0.247 0.244 1.280 0.546* 0.248 1.727 0.890† 0.504 2.436

Never attending religious services 32.7% 44.7% 37.6% 44.0% 0.088 0.163 1.092 0.385* 0.164 1.469 -0.243 0.327 0.785

Never reading the newspaper 16.9% 29.6% 14.6% 27.4% -0.134 0.194 0.875 0.759*** 0.198 2.136 0.069 0.397 1.071

Smoking every day 13.8% 13.2% 9.5% 16.6% -0.040 0.235 0.960 0.283 0.236 1.327 0.684 0.477 1.892

Having ever, even once,used marijuana or hashish 58.8% 54.7% 65.0% 61.9% 0.281† 0.163 1.324 -0.148 0.163 0.862 0.034 0.326 1.034

Having had 5 or more drinks on the same occasion on more

than 3 days of the past 3010.6% 12.0% 8.9% 11.5% -0.115 0.257 0.892 0.202 0.258 1.224 0.147 0.517 1.159

Having had more than 30 female partners since their 18th birthday (among straight men

and homosexual or bisexual women)16.1% 11.0% 10.3% 9.3% -0.366 0.346 0.694 -0.299 0.344 0.741 0.334 0.698 1.396

Having had more than 25 male partners since their 18th birthday (among straight women

and homosexual or bisexual and men)9.7% 9.1% 10.5% 12.0% 0.203 0.381 1.224 0.046 0.380 1.047 0.222 0.763 1.248

Having had sex 4 or more times a week during the last 12 months 3.9% 9.7% 9.7% 9.7% 0.391 0.297 1.479 0.391 0.297 1.479 -0.999 0.628 0.368

Describing themselves as homosexual,gay, lesbian, or bisexual 9.5% 10.8% 7.1% 10.9% -0.134 0.272 0.875 0.297 0.274 1.345 0.338 0.551 1.403

†=p<.10 *p<.05 **p<.01 ***p<.001

NOT RELIABLY DIFFERENT

Automated vs. Human Interviewer Interaction

Estimates come from the full model when the interaction is statistically significant (p<.05) and from the main effects analysis when the interaction is not statistically significant.

HUMAN VS. AUTOMATED DIFFERENCES

Voice Text

VOICE VS. TEXT DIFFERENCES

Text vs. Voice

Whataccountsfortextvs.voicedifferencesinprecisionanddisclosure?•  Couldbeanyorallofthemanydifferencesin/mingandbehaviorbetweentextandvoiceinterviews–  aloneorincombina?on

•  Plausiblecontribu?ngfactorsinclude:–  Textreducesimmediate?mepressuretorespond,soRhasmore?metothinkorlookupanswersàCouldexplaingreaterprecision(lessrounding)intext

–  Textreduces“socialpresence”•  ReducedsalienceofI’sabilitytoevaluateorbejudgmental?•  NoimmediateevidenceofI’sreac?on?àCouldexplainmoredisclosureintext

Experimentaldesignhelpsruleinorruleoutaccounts

•  e.g.,maybeR’sroundlessintextbecausetextI’sneverlaugh(noLOL’sorhaha’s)– Maybelaughterinvoiceinterviewssuggeststhatcasualresponsesaresufficient

–  Butthatcan’tbeitbecauseR’sroundjustasmuchinHumanandAutoVoiceinterviews,andautomated“interviewer”neverlaughed

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

Text" Voice"

Human"

Automated"

Examples:Textvs.voiceinterac?ons

HUMANTEXT HUMANVOICE

1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?

1 I: Duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.

2 R: 3 2 R: OH,GOD.U:hman.That’salot.HowmanymoviesIseen?Like30.

3 I: 30.

Totalelapsed>meun>lnextQ:1:21 0:12

Examples:Textvs.voiceinterac?onsHUMANTEXT

1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?

2 R: Medium?

3 I: Here’smoreinforma?on.Pleasecountmoviesyouwatchedintheatersoranydeviceincludingcomputers,tabletssuchasaniPad,smartphonessuchasaniPhone,handheldssuchasiPods,aswellasonTVthroughbroadcast,cable,DVD,orpay-per-view.

4 R: 3

Totalelapsed>meun>lnextQ:2:00

HUMANVOICE

1 I: *Duringthelast*

2 R: Huh?

3 I: Oh,sorry.Um,duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.

4 R: Oh!Let’ssee,whatdidIwatch.Um,shouldIsayhowmanymoviesIwatchedorhowmanymovieswatchedme?[laughs]Allrightlet’s-letmethinkaboutthat.IthinkyesterdayIwatchedu:m,notinitsen?retybutyouknow,comingandgoing.Mykidsarewatchingin.Um,Idon’tknowmaybe2or3?mesaweekmaybe?

Examples:Textvs.voiceinterac?onsHUMANVOICE

5 I: Uh,sowhatwouldbeyourbestes?mateonhowmany,um,yousawinthewholemonth.

6 R: [pause]Um,Idon’tknowI’dsaymaybe3moviesifthatmany.

7 I: 3?

8 R: Isthatgoingtothemoviesorwatchingthemoviesontv.Likeyousaid*anymedium*right?

9 I: That’s*anymovies.*Yep.

10 R: Maybe1or2amonthI’dsay.

11 I: 1or2amonth?[breath]Uh,sowhatwouldbe*closer*

Examples:Textvs.voiceinterac?onsHUMANVOICE

12 R: *Yeah,because*Iuh,um,occasionallyItakethekidsonaTuesdaytoseeamovie,dependingonwhat’splaying.SoI’dmaybeonceortwiceamonth

13 I: Whichwouldbecloser,onceortwice.

14 R: Iwouldsaytwice.

15 I: Twice?

16 R: R:Mhm.Becauseitruns4Tuesdayswhichischeapertogo

17 I: Right

18 R: R:soI’dsaytwice,yah.BecauseIdotakethemtwice.Notlastmonthbutthemonthbefore

Totalelapsed>meun>lnextQ:ß

1:36

ExamplesfromAutomatedVoiceI:Howopendoyoureadthenewspaper?'Everyday','afew?mesaweek','onceaweek','lessthanonceaweek',or'never'?R:EverydayI:Gotit.I:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:NoneI:Ithinkyousaid'9'.Isthatright?YesorNo.R:NoI:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:ZeroI:Thanks

Numerical

FirstHypothesis:“Nine”

LastHypothesis:“Zero”

LastAnnota?on:“Zero”

Categorical

ExplicitConfirma>on

Median?meperques?onandanswer

Mul?tasking(self-reported)

Cannonresponseornoncomple?onexplainthepaeerns?

•  Werepeoplewhoweremoreconscien?ous(lesslikelytoroundtheiranswers)ormorewillingtodisclosesensi?veinforma?onactuallylesslikelytostartorfinishinterviewsinvoicemodesthanintextmodes?

•  Couldourmodeeffectsresultnotfromthecontribu?onofrespondersandcompleters,butinsteadfromthenon-contribu?onofnon-respondersandnon-completers?

Studydesignallowslookingatthisinafocusedway

•  alloursamplemembershadalreadyindicated,byscreeningintothestudy,interestinandatleastsomecommitmenttopar?cipa?nginaninterviewontheiriPhone(inanunspecifiedinterviewmode).

•  Thefactthatourpar?cipantswererandomlyassignedtoaninterviewingmodemeansthattheirini?a?vewasunlikelytohavedifferedacrossthemodes.

Nonresponse?

•  noevidencethatdifferentkindsofpeople(age,gender,ethnicity,race,educa?on,income)fromoursamplewereanymoreorlesslikelytostarttheinterviewsinthedifferentmodes

•  Implausiblethatanotherfactorcouldexplainpaeern:– wouldrequirethattendencyofRstogiveimpreciseanswersandreluctancetoengageinatextinterview(butwillingnesstoengageinavoiceinterview)wouldhavethesameorigin

Noncomple?on?•  Comple?ongreaterinhumanthanautomatedinterviews

•  Nodifferencebetweentextandvoice•  àUnlikelytoaccountforvoicevs.textdifferences•  Fornoncomple?ontoaccountfordisclosure,wouldrequireasystema?creversalofthepaeernofdisclosureobservedforthosewhocompletedandthosewhobrokeoff–  thosewhobrokeoffwithautomatedinterviewerswouldhavetobethosewhohadlesstodisclose

–  Butonewouldthinkthatpeoplewhobreakoffwouldbethosewithmoretodisclose

Prefertext(vs.voice)forfutureiPhoneinterview?

0

10

20

30

40

50

60

70

80

90

100

Text Voice

Percen

t Human

Automated

Othersa?sfac?onmeasures

•  MostRsfoundinterviewveryorsomewhateasy– Morefoundspeech-IVRsomewhathard

•  Futureinterviews:– TextRsoverwhelminglypreferredfutureinterviewintextvs.voice

– VoiceRspreferredvoice,butlesssoifspeech-IVR

Summary:Voicevs.Text•  Textinterviewsproducehigherdataquality:greaterdisclosure,lesssa?sficing,highsa?sfac?on

•  Eventhough(orbecause?)theytakelonger•  Eventhoughdataarelesssecure(morepersistentandtraceable)thanvoice–  Perhapsbecauseofdifferent?mepressurethanvoice?–  PerhapsbecauseofconvenienceofansweringwhenandhowRwants?

–  Perhapsbecauseofgreatersocialdistancewithinterviewer?

•  Caveat:weimplementedtextinterviewsinonepar?cularway,withsingle-characterresponses

Summary:Humanvs.AutomatedInterviewer

•  Automatedinterviewsonasmartphone(inthesemodes)canleadtodataatleastashighinqualityasdatafromhumaninterviewsinsamemodes–  Nomoresa?sficingthanwithhumaninterviewers!–  Moredisclosure

•  Tradeoffs–  Fieldperiodcanbeshorter–  interviewscantakelonger–  Higherbreak-off–  requireaddi?onaldevelopmenteffort,especiallyspeech-IVR

•  Caveat:weimplementedonepar?cularversionofspeech-IVR;otherscoulddiffer

ModeChoiceStudyConradetal.(underreview)

•  Ismessage–  Urgentorcanitwait?–  Sensi?veornot?–  Shortvs.long?

•  WillIbemul?tasking?Ifso,whatelsewillIbedoing?•  Whatmodewillbeeasiestorleastdisrup?veforpartner?

•  IsseWngpublicvs.private,noisyvs.quiet,brightvs.dim?

•  Whatismygenerally(chronically)preferredwayofcommunica?ng?–  e.g.,talkingvs.tex?ng

•  Sopeoplecanusethesamedevice,forexample,torespondto–  avoicecallwithatextmessage–  atextmessagewithaFacebookpost–  emailwithavoicecall

48

Onsmartphones,peoplechooseandswitchmodestofitneeds

Implica?onsforSurveyPrac?ce

•  Nowpossibleformembersofpublictochooseoneofmanysurveymodesonasingledevice–  immediatelyandconveniently

•  Notofferingachoicecoulddeterpar?cipa?onbysmartphoneusers– orreducemo?va?onwhenansweringques?ons

49

Inothertasks,choiceseemstohelpandhurt

•  Choiceenhancesintrinsicmo?va?on(byincreasingautonomy)andperformance–  Patalletal.(2008)meta-analysis:78of91effectsofchoiceonintrinsicmo?va?onareposi?ve

•  Toomanyop?ons(overload)leadstonochoice(paralysis)andreducedsa?sfac?onwithchoices–  IyengarandLepper(2000):par?cipantsmorelikelytopurchasegourmetjams/chocolatesortocompleteop?onalassignmentswhenoffered6vs.24or30choices

•  Howdoeschoiceaffectsurveypar?cipa?on?

50

SurveyModeChoice•  Toincreasepar?cipa?on,researchersofferpoten?alrespondentsachoiceofmodes–  e.g.,mailpaperques?onnaireandgiverandomhalfchoiceofcomple?ngonline;requiresextrastepoftypingURLintobrowser

•  Butthiskindofchoiceseemstoreducepar?cipa?on:–  Fulton&Medway(2012)meta-analysisof19mail/webchoicestudiesfindsthat,comparedtonochoice,modechoicereliablyreducespar?cipa?onby3.8%

–  suggestcouldbeduetoParadoxofChoice(Schwartz,2009)orcostsofswitchingfrominvita?ontointerviewmode

•  Choiceonsingledevicesimplifieschoiceimplementa?on

51

Currentstudy

•  Examineshowmodechoiceonasingledeviceaffects– Par?cipa?on– Dataquality(rounding,straightlininganddisclosure)

– Rsa?sfac?on

•  Same4modes,same32items

52

Possibleoutcomes:Par?cipa?on

•  IfRscanChoose– Mightreducepar?cipa?onbecause

•  Increasedcomplexity(Schwartz,2004;FultonandMedway,2012)•  Breakinresponseprocess(Fulton&Medway,2012)

– Mightincreasepar?cipa?onbecause•  Canchooseamodethatissuitablegiventheircurrentenvironmentandotherdemands(e.g.,whethertheycantalknow)

Possibleoutcomes:Conscien?ousresponding

•  IfRscanChoose– mightprovidefewerconscien?ousanswersbecausetheychooseamodeinwhichit’seasiertotakeshortcuts

•  e.g.,anautomatedmodebecausenohumaninterviewertopressthemtoworkhard

– mightprovidemoreconscien?ousanswersbecausebeingabletochoosemayincreasetheircommitmenttothetask

•  Mayincreasemo?va?on

Possibleoutcomes:Disclosure

•  IfRscanChoose– mightdisclosemorebecausechoosemoreprivatemodewithfewersocialcues

•  e.g.,Automatedtext

– mightdiscloselessbecausechoosemoreconvenient,fastermodewithmoresocialcues

•  E.g.,humanvoice

Possibleoutcomes:Sa?sfac?on

•  IfRscanChoose– Mightreducesa?sfac?onbecause

•  Addingop?onsincreasesR’sexpecta?ons(Schwartz,2004)•  Leadstoregretovernotchoosingimaginedalterna?ve(Schwartz,2004)

– Mightincreasesa?sfac?onbecausepeopleperceivethechosenalterna?veasmoreaerac?ve(Fes?nger,1948;Cooper,2007)

– Orjustbecausemoreconvenientandeasier!

ExperimentalCondi?ons

1.  AssignedMode(NoChoice)•  Rsrandomlyassignedtoamode•  Contactedandinterviewedinsamemode

2.  Choice•  Rsrandomlyassignedtoacontactmode•  Requiredtochooseinterviewmode

–  Couldchoosecontactmodeoranyofotherthree–  Makesexplicittheirmodechoiceinten?on

57

ModeComparisonExperiment

ModeChoiceDesignandImplementa?on(2)

•  ModeChoiceintroduc?on:“Togetstarted,weneedyoutochoosehowyouwanttobeinterviewed--whateverworksbestforyou.Therearefourchoicesandanychoiceisfinewithus.Doyouwantto‘talkwithaperson’,‘talkwithanautomatedinterviewer’,‘textwithaperson’,or‘textwithanautomatedinterviewer’?

•  Withineachcontactmode,orderofinterviewmodeop?onsrotatedacrossRs(16orders)

58

Respondents:1260iPhoneusers•  AssignedMode(NoChoice):n=634

–  n=157to160permode–  InterviewedMarch–May,2012

•  Choice(AbletoChooseInterviewMode):n=626–  n=149to170permodeofcontact–  InterviewedJuly–September,2012

•  RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk–  Webscreenerverifiedage(>21years)andUSareacode–  iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse

•  $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire

•  Age,gender,ethnicity,income,educa?onnotreliablydifferentbetweenAssignedModeandModeChoicegroups

•  13Umich/SRCinterviewers:–  5onlyinAssignedModecondi?on,3onlyinModeChoice,5inbothcondi?ons

59

Par?cipa?on•  Doessimplybeingpresentedwithachoicereducepar?cipa?on?

–  SlightlyfewerRsinModeChoicecondi?on(52.1%)choseamodethanansweredthefirstques?oninAssignedModecondi?on(55.9%).

•  Doeschoosingamodereducecomple?on?–  Overall,46.4%ofRsinChoicecondi?onvs.50.5%inAssignedMode

condi?oncompletedques?onnaire(RR1)

•  Inallcases?–  WhenRschosetostayincontactmode48.3%completedinterview,

notdifferentfrom50.5%AssignedMode–  Modechoiceinautomatedmodeshasnoimpactoncomple?on

(43.4%vs.44.0%)

•  Howdoesmodechoiceaffectbreakoffs?–  MoreRswhochoseaninterviewmodecompletedtheinterview

(94.9%)thanthosewhowereassignedamode(90.4%)

YES

NO

ITREDUCESBREAKOFFS

YES

Par?cipa?on

55.9%

90.4%

48.9%

94.9%

0102030405060708090

100

Startinterview(answerQ1) Completeinterviewoncestart

Percen

t

AssignedMode

ModeChoice

•  Overallcomple?onhigherwithout(50.5%)thanwithchoice(46.4%)•  Noimpactonkindsofpeoplewhopar?cipatesochoiceprobablydoesnot

introducenonresponsebias 61

Breakoffaperchoicebutbeforeinterview

0.7%

11.1%

0

2

4

6

8

10

12

StayinMode(n=301) SwitchMode(n=388)

%don

’tan

swerQ1aW

er

choo

singm

ode

62

Bothgroupsmakechoicesoincreasedbreakoffswhenchoicerequiresswitchingmodesduetotoswitchingcosts,notParadoxofChoice

HumanVoice HumanText

Whatmodeswerechosen?

0

50

100

150

200

250

300

SwitchintoMode

StayinMode

AutomatedText

n=170

n=150 n=157 n=149

OriginalSampleSize(beforemodechoice)

AutomatedVoice

Num

bero

fRs

63

DataQuality:Rounding

•  Wedefineroundinghereasnumericalanswersdivisibleby10– HowmanysongsdoyoucurrentlyhaveonyouriPhone?

•  Exampleroundedanswer:1100•  Exampleunroundedanswer:1126

64

AverageNumberofRoundedNumericalAnswers

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

Effectofchoicenotduetopar?cularchoiceofmode:lessroundingwithchoicethanwithoutapercontrollingformode,p=0.008

2.58

p<0.001

2.28

Num

berRo

unde

dAn

swers

65

Rounding:“Numberof?mesea?nginrestaurants”

0

5

10

15

20

25

AssignedMode ModeChoice

Percen

tRsrep

or>n

groun

ded

answ

er

*During the last month, how many times did you eat in restaurants?

p<0.01

66

Rounding:“NumberofsongsonyouriPhone”

40

42

44

46

48

50

52

AssignedMode ModeChoice

Percen

tRsrep

or>n

groun

ded

answ

er

*How many songs do you currently have on your iPhone?

p=0.02

67

PercentofRsstraightlining

0%

2%

4%

6%

8%

10%

12%

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

Effectofchoicenotduetopar?cularchoiceofmode:marginallylessstraightliningwithchoicethanwithoutapercontrollingformode,p=0.085

6.78%

p=0.029

3.99%

68

AveragenumberofSociallyDesirableAnswers

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

2.71 2.83

Num

berofAnswers

1

Figure4.Disclosure:Averagenumberofresponsesdeemedmostsociallyundesirable:abovethetopdecileforconKnuousnumericalresponsesorthemostextremecategoricalresponseopKoninthesKgmaKzeddirecKon.

Responsesdeemedmostsociallyundesirable:abovetopdecileforcon?nuousnumericalresponsesormostextremecategoricalresponseop?onins?gma?zeddirec?on

Sa?sfac?onhigherwithmodechoice

0

10

20

30

40

50

60

70

80

AssignedMode ModeChoice

%Verysa>sfie

d

p<0.001

Overall,howsa?sfiedwereyouwiththeinterview?

70

Reasonsforchoosingmodes

*Why did you choose this interviewing method?

Mostcommoncategories %ProvidingReason

Ease/simplicity 33.8%Convenience/flexibility 22.8%Quickness(shortestinterview?me) 10.3%Privacy 9.8%Liketex?ng 9.0%Environment--loca?on 8.8%

Threecoders;Agreement=98.1%

•  Codedopen-endedanswersinto29categories

71

Reasonsforchoosingmodes*(examples)

•  Humanvoice:–  “Morecomfortablespeakingwitharealperson”

•  Humantext:–  “IchosetotextbecauseIhadasmallchildwithmeinmyhomeduringthe

interviewandcouldnothaveconcentratedontheques?onsifitwasonthephone.”

–  “Toavoidbackgroundnoiseandtoclearlyunderstandtheques?onandtakemy?metoanswerit.”

•  Automatedtext:–  “Iamatworkandwouldn'talwaysbeabletoanswerques?onsifIspoketo

someoneonthephone.”–  “BecauseIdidn'twanttotalkonthephoenordidIwanttotextaperson

simplybecausIknewsomeofmyresponseswouldhavebeenalielelate”•  AutomatedVoice:

–  “ididn'twanttotalktoanyonebut,IwasdrivingsoIcouldn'tlookatascreen”

–  “Talkingtoanautomatedpersonwaslesspersonal”

*Why did you choose this interviewing method? 72

Summary•  Modechoiceproduced:

–  lessrounding–  lessstraightlining–  fewerbreakoffs–  higherRsa?sfac?on

•  Choicedidnotaffectpaeernsoftextvs.voiceforrounding,straightlining,disclosure

•  Par?cipa?on–  Lowerstartandcomple?onrateswithchoicethannot– Mostlyduetowhetherchoiceinvolvesmodeswitch–  Rswhostartaperchoosingmodemorelikelytocomplete

73

Manyques?onsremain(overall)•  Dodifferentdemographicsubgroups(e.g.,age,income,educa?on)varyindisclosure,effort,preferences?

•  Generalizabilitytoothermobileplacorms?Tolesssmartmobilephones?

•  Generalizabilitytoanon-convenienceornon-incen?vizedsample?

•  Dorespondentswanttobeabletoswitchmodesmid-interviewwhencircumstanceschange(mobile,noisy,private,etc.)?

•  HowmanyQscanbeaskedviatextinterviews?

Implica?ons

•  Tex?ngisworthexploringfurtherasamodeofsurveydatacollec?onforFederalsta?s?csandsocialscienceresearch

•  Asynchronous,less-?me-pressuredrespondingmayreallybebeeerthanusualmodes–  Raisesques?onofwhetherFTFortelephonemodesshoulds?llbeconsideredthegoldstandardinasmartphoneera

– Andwhether“best”modevariesfordifferentresearchques?onsorpar?cipants

Implica?ons(2)

•  Mul?taskingwhileansweringsurveyques?onsdoesnotnecessarilyleadtopoorerdataquality

•  Maywellenhancerespondents’sa?sfac?onandwell-beingbyallowingthemtorespondwhereandwhentheyfinditconvenient

Implica?ons(3)

•  Poten?albenefitsofautoma?onforsocialmeasurementextendtotheuseofapersonalportabledevicesdespitethevaryingcontexts(publicandprivate)inwhichthedeviceisused

Implica?ons(4)

•  Offeringrespondentsamodechoiceonasingledevicemayhaveimportantbenefits

•  Butnotallmodetransi?onsarethesame•  Differentdesignsolu?onswillbeneededfordifferentmodetransi?ons

78

Specula?on:ANewTakeonStandardiza?on

•  Shouldourtradi?onalone-size-fits-allapproachtocollec?ngself-reportdataberethought?

•  Maybedifferentmodesfordifferentpeopleondifferentoccasionscanincreasecomparabilityoftheirresponses

•  Maybewhatisneededisstandardizingpar?cipants’experience–  enhancingeveryone’sabilitytofocusonthetaskinawaythatsuitstheirpreferencesandcircumstances

•  Smartphones–mul?modal,mobiledevices–maybeforcingustothinkthisway

Thankyou!

80