the ux of predictive behavior for the iot (2016: o'reilly designing for the iot)

Good afternoonandthanksforhavingmehere.InthistalkIwanttolookatthedesign

challengesofsystemsthatanticipateusers’needsandthenactonthem.Thatmeansitsitsat

theintersectionoftheinternetofthings,userexperiencedesignandmachinelearning,and

althoughpeoplehavedealtwithoneofthosedisciplinesbefore,Idon’tthinkthey’veever

beencombinedinquitethewaystheyarenow,orwiththecurrententhusiasm.

Thetalkisdividedintoseveralparts:itstartswithanoverviewofhowIthinkInternetof

Thingsdevicesareprimarilycomponentsofservices,ratherthanbeingself-contained

experiences,howpredictivebehaviorenableskeycomponentsofthoseservices,andthenI

finishbytryingtotoidentifyuserexperienceissuesaroundpredictivebehaviorand

suggestionsforpatternstoamelioratethoseissues.

Acoupleofcaveats:

- Mycurrentworkinthisfieldfocusesalmostexclusivelyontheconsumerinternetofthings,

soIseemostthingsthroughthatlens.PredictiveAIhasalonghistoryinindustrial

applications,it’sintheconsumerspacethatwereallythetheUXissues.

- IwanttopointoutthatfewifanyoftheissuesIraisearenew.Thoughtheterms“internet

ofthings”and“machinelearning”arehotrightnow,theideashavebeendiscussedin

researchcirclesfordecades.Searchfor“ubiquitouscomputing,”“ambientintelligence,”and

“pervasivecomputing”andyou’llseealotofgreatthoughtinthespace.Ifyou’rereally

ambitious,youcanreadtheArtificialIntelligenceandCyberneticsworksofthe50sand60s

andyou’llbesurprisedbytheprescienceofthepeopleworkinginthisspacewhentheentire

world’scomputepowerwasaboutasmuchasmykeyfob.

- Therearealotofideashere,andIwillalmostcertainlyunder-explainsomething.ForthatI

apologizeinadvance.Mygoalhereistogiveyouageneralsenseofhowthesethepieces

connect,ratherthananin-depthexplanationofanyoneofthepieces.

- Finally,mostofmyslidesdon’thavewordsonthem,soI’llmakethecompletedeckwitha

transcriptavailableassoonI’mdone.

0

Let me begin by telling you a bit about my background. I�m a user experience designer. I was one of the first professional Web designers. This is the navigation for a hot sauce shopping site I designed in the spring of 1994.

1

I’vealsoworkedontheuserexperiencedesignofalotofconsumerelectronics

productsfromcompaniesyou’veprobablyheardof.

2

Iwroteacoupleofbooksbasedonmyexperienceasadesigner.Oneisacookbookof

userresearchmethods,andtheseconddescribeswhatIthinkaresomeofthecore

concernswhendesigningnetworkedcomputationaldevices.I’malsomarriedtoone

oftheauthorsofthisbook,sothinkingabouttheimpactofthedesignofconnected

devicesonpeopleiskindofafamilybusiness.

3

Ialsostartedacoupleofcompanies.Thefirst,AdaptivePath,wasprimarilyfocused

ontheweb, andwiththesecondone,ThingM,Igotdeepintodevelopinghardware.

4

TodayIworkforPARC,thefamousresearchlabthatinventedthepersonalcomputer,

objectorientedsoftware,thetabletcomputer,andlaserprinter,asaprincipalinits

InnovationServicesgroup.Wehelpcompaniesreducetheriskofadoptingnovel

technologiesusingamixofsocialresearch,designandbusinessstrategy.

5

IwantstartbyfocusingonwhatIfeelisa keyaspectofconsumerIoTthat’soften

missedwhenpeoplefocusonthehardwareoftheIoT,whichisthatconsumerIoT

productshaveaverydifferentbusinessmodelthantraditionalconsumerelectronics.

6

Historically,acompanymadeanelectronicproduct,sayaturntable,theyfound

peopletosellitforthem,theyadvertiseditandpeopleboughtit.Thatwas

traditionallytheendofthecompany’srelationshipwiththecustomeruntilthat

personboughtanotherthing,andallofthevalueoftherelationshipwasinthe

device.WiththeIoT,thesaleofthedeviceisjustthebeginningoftherelationship

andphysicalthingholdsalmostnovalueforeitherthecustomerorthemanufacturer.

7

When you have a multitude of connected devices and apps, value shifts to services and the devices, software applications and websites used to access it—its avatars—become secondary. A camera becomes a really good appliance for taking photos for Instagram, while a TV becomes a nice Instagram display that you don’t have to log into every time, and a phone becomes a convenient way to check your friends’ pictures on the road.

Hardware, physical things, become simultaneously more specialized and devalued as users see “through” each device to the service it represents. The avatars exist to get better value out of the service.

8

Amazon reallygetsthis.Here�satellingolderadfromAmazonfortheKindle. It’s

saying�Look,usewhateverdevice youwant.Wedon�tcare,aslongyoustayloyaltoourservice.Youcanbuyourspecializeddevices,butyoudon�thaveto.�

9

WhenFirewasreleased5yearsago,JeffBezosevencalled itaservice.

10

Mostlarge-scaleIoT productsareserviceavatars.Theyusespecialized sensorsand

actuatorstosupportaservice,buthavelittlevalue—ordon’tworkatall—without

thesupportingservice.SmartThings,whichwasacquiredbySamsung, clearlystates

itsserviceofferingrightupfrontontheirsite.Thefirstthingtheysayabouttheir

productlineisnotwhatthefunctionalityis,butwhateffecttheirservicewillachieve

fortheircustomers.Theirhardwareproducts’functionality,howtheywilltechnically

satisfytheservicepromise,isalmostanafterthought.

11

Compare thattoX10,theirspiritualpredecessorthat’sbeeninthebusinessfor30

years.AllthatX10tellsisyouiswhatthedevicesare,notwhattheservicewill

accomplishforyou.Idon’tevenknowifthereISaservice.WhyshouldIcarethat

theyhave“modules”?Ishouldn’t,andIdon’t.

12

Simplyconnectingexistingstufftotheinternetdoesnotproduce customervalue…

13

Simpleconnectivityhelpswhenyou’retryingtomaximizetheefficiencyofafixed

process,butthat’snotaproblemthatmostpeoplehave.We’vebeenabletosimply

connectvariousdevicestoacomputersinceaTandyColorComputerscouldlightsoff

andonoverX10in1983.TodayyoucanbuyamodulefromParticle,ElectricImpora

dozenothercompaniesandintegrateitinamonthtoconnectanyarbitrarydeviceto

theInternet.Theproblemisthatthatwasn’tveryusefulthen,andit’snotveryuseful

now.IfyoureplacetheTandywithaniPhoneandthelampwithawashingmachine…

14

…oraneggcarton,youstillhavethesameproblem,andit’sauserexperience

problem.

TheUXproblemisthatendusershavetoconnectallthedotstocoordinatebetween

awidevarietyofdevices,andtointerpretthemeaningofallofthesesensorsto

createpersonalvalue.Formanysimplyconnectedproductsthereissolittleefficiency

tobehadrelativetothecognitiveloadthatit’sjustnotworthit.What’sworse,the

extracognitiveloadisexactlyoppositetowhattheproductpromises,andcustomers

feelintenselydisappointed,perhapsevenbetrayed,whentheyrealizehowlittlethey

getoutofsuchaproductThatmakesmostsuchproductseffectivelyWORSEthan

useless.

Thatpromisegapiswhatdistinguishesagadgetfromatool,whythiseggcartonis

funny,andwhyQuirkywhomadeit,filedforbankruptcyafterburningthrough

hundredsofmillionsofdollars.

15

Howdoyoumakemoneyinthisspace ofdematerializeddevicesandcloudservices?

16

Oneapproachistochangefromanownershipmodeltoasubscriptionmodel.Now

thedevicegivesaccesstoadesiredendresult,withouttheburdensofownershipor

maintenance.TheIoT technologyiswhatgivesanefficientwaytotrackandcharge

forassets.Carsharing,bikesharing,Uber andAirBNB followthismodel.Youdon’t

useiteveryday,sowhyownit?High-endclothingisgoingthisway.Doyoureally

needtoownthatPradahandbagsoyoucanuseittwiceayear?

17

Hewlett Packard’sprinterdivisionisreallyaninkcompanythatalsomakesink

consumptiondevices.SimilarlyAmazonistryingtocornerthemarketonall

consumables,whetherthey’redigital…

18

..orphysical.Their Dashreplenishmentservicecanturnanydevicewith

consumables…

19

…intoanautomaticAmazonreorderingmachine.

TheDashbuttonisanetworkedcomputerwhoseonlypurposeistobeanavatarfor

productswhereit’snotyeteconomicallyfeasibletoincludeconnectedelectronics,

likeamacaroniandcheesebox.That’sgoingtochangeastheelectronicsgetcheaper.

Moreover,thebuttonisasensorforpeople’sintent,whichthendovetailsintothe

realbusinessmodel,whichisnotjustshippingyoumintswhenyou’retoolazyto

leavethehouse…buttoidentifyyourbuyingpatterns,yourcravings,yourimpulses,

sothattheycanpredictthemandshipyoumintsnotwhenyouaskforthem,but

whenyouwantthem.

20

Ithink therealvalueconnectedservicesofferistheirabilitytomakesenseofthe

worldonourbehalf,toreducecognitiveloadbyenablingpeopletointeractwith

devicesatahigherlevelthansimpletelemetry,atthelevelofintentionsandgoals,

ratherthandataandcontrol.Humansarenotbuilttocollectandmakesenseofhuge

amountsofdataacrossmanydevices,ortoarticulateourneedsassystemsof

mutuallyinterdependentcomponents.Computersaregreatatit.

21

Theinterestingthingisthatthisnotjusttheory.

Predictionandresponseisattheheartofthevaluepropositionmanyofthemost

compellingIoT services,startingwiththeNest.TheNestsaysthatitknowsyou.How

doesitknowyou?Itpredictswhatyou’regoingtowantbasedonyourpastbehavior.

22

Amazon’sEchospeaker saysit’scontinuallylearning.Howisthat?Predictivemachine

learningbasedonyouractionsandyourwords.

23

The Birdi smartsmokealarmsaysitwilllearnovertime,whichisagainthesame

thing.

24

Jaguar, learning…ANDintelligent.

25

TheEdyn plantwateringsystemadapts toeverychange.Whatisthatadaptation?

Predictivemachinelearning.

26

Canary,ahomesecurity service.

27

Cocoon,anotherhomesecuritysystem knows.Howdoesitknow?Machinelearning.

28

Here’sfoobot,anairqualityservice.

[Ialsolikehowoneof itsimplicitservicepromisesistoidentify whenyourkidsare

smokingpot.]

29

Silk’sSenseadapts

30

Mistbox sprayswaterintoyourairconditionertoreduceyourenergybill.You’dthink

that’saprettysimpleprocess,butno,it’salwayslearning.

31

Anumberofcompaniesaremakingchipsthatmakemachinelearningmuchcheaper

andmorepower-efficient,whichmeansthatit’sgoingtobeveryeasytoinstallitin

everydevice,fromstreetlightstomedicalequipmenttotoys.It’snotjustlikely,it’s

inevitable.Here’sonethatwasannouncedacoupleofweeksago.

32

33

Theydothisthroughprocessesthathavemanynames,butI’lllumpthemallunderMachine

Learning,whichisabigpartofwhatusedtobecalledArtificialIntelligence.Manyofthecore

ideasheregobacktothe1950sandit’sthebasisofeveryemailspamfilter,soifyou’vehad

yourspamautomaticallyfiltered,you’veexperiencedthevalueofmachinelearning.

AbigpartofMachineLearningispatternrecognition.Wehumansevolvedverysophisticated

facultiestorapidlyidentifyvisualimagesinallkindsofdifficultconditions.Youlookata

pictureofanorangeonaredplateandyoucantellinstantlythatit’snotasunset,butuntil

recentlythatwasreally,reallyhardforacomputer.BecauseofacombinationofMoore’s

Lawandsomebreakthroughs,computershavegottenmuchbetteratpatternrecognitionin

thelastcoupleofyears.

Foracomputer,recognizingsomethingstartswithaprocesswheresomebasicattributesof

animageareextracted,suchastheshapeofboundariesbetweenclustersofpixels,orthe

dominantcolorofapatchofanimage.Thesearecalledfeaturesinmachinelearning.By

examininglotsandlotsofexamplesoffeaturesinanimage,amachinelearningsystembuilds

astatisticalmodelofwhatthatclusterrepresents.

Basicformsofthiskindofimagerecognitionhasbeenusedindustriallyfordecade.Legohas

acompletelyautomatedfactorythatinjectionmoldsamillionLegobricksanhour,examines

everysinglepiece,automaticallysorts,bagsandboxesthem,allusingcomputervision.That’s

relativelyold.

Imagesfrom:Region-basedConvolutionalNetworksforAccurateObjectDetectionand

SemanticSegmentation,R.Girshick,J.Donahue,T.Darrell,J.Malik,IEEETransactionson

PatternAnalysisandMachineIntelligence

Real-TimeImageandVideoProcessing:FromResearchtoRealitybyKehtarnavaz and

Gemadia

34

What’snewisaclassofsystemsthatunderstandthecontentofimages.Theydon’tjustlook

atfeatures,butclustersoffeatures,andclustersofclustersoffeatures,andtheycannow

identifyanorangefromthesettingsun,orapersonfromanairplane,orapolarbearfroma

dalmatian.

ThisiswhyFacebookasksyoutosaywhoisinanimage.It’snotjustforyou,it’sfortheirface

recognizer.

Nowhere’stheinterestingpart:we’rebuilttoidentifypatternsinvisualphenomena,but

we’reprettybadatidentifyingtheminotherkindsofsituations.Forexample,ifyou’veever

triedtounderstandsomeone’sfoodsensitivities,it’sreallyhardtoextractwhatthatperson

isreactingto,evenifyoukeepverycarefultrackofwhatthey’veeaten.We’rejustnotbuilt

forit.Itwasneverevolutionarilysufficientlyimportant,sowedidn’tevolveanorganforit.

Computers,ontheotherhand,don’tcare,andnowthatwe’vefoundreallygoodwaystofind

patternsinvisualimages,thesesametechniquescanfindpatternsinanything.

Insteadofamatrixofpixels,whatifyouhadamatrixofmedicalprescriptions,witheachrow

asthehistoryofoneperson’sprescriptionsfromthefirsttimethatpersonwenttothedoctor

foraproblem,throughwhentheywereprescribedcertainthings,towhentheygotbetter,or

theydidn’t.Thesamekindofsystemcouldlearnthetypicalpatternforprescribing,say,a

wheelchair.Itwouldessentiallyseethegeneralshapeofthesequencefortheprescriptionof

achairovertimeandacrossmanypeople.

Thenifyousawawheelchairbeingprescribedthatwasoutsideofthetypicalpattern,you

couldidentifyit.That’scalledanomalydetection.That’sinfactexactlyhowwebuiltasystem

toidentifyMedicarefraud.Peopleareterribleatthatstuff,butcomputersaregreat.

35

Whenoneofthedimensionsistimeandanotheristheoutcomeofaseriesofactions

youcanmakeapatternrecognizerthatassociatesasequenceofactionswithasetof

statisticalprobabilitiesforpossibleoutcomesbasedondatacollectedacrossawide

varietyofsimilarsituations.Inotherwords,becausepeopleandmachinesbehavein

fairlyconsistentways,thesemachinelearningsystemscanincreasinglypredictthe

futureandattempttoadaptthecurrentsituationtocreateamoredesirable

outcome.

36

37

Asinterestingastheseissuesare,Ithinkthat, moreimportantly,whattheyrepresent

isthatwe’reentering intoanewrelationshipwithourdeviceecosystem,asea

changeinourrelationshiptothebuiltworld.

38

Thinkofasewingmachine.It’sverycomplex,butitstillonlyactsinresponsetous.

39

Computersacting autonomously erodethissimpletool/userrelationship.Predictive

IoTismorethanjustrecommendinganewsong,it’sactingonyourbehalfonthe

basisofitsassumptionaboutwhatyouwant,andwhat’sbestforyou.

Atthedawnofcomputinginthelate1940scyberneticistslikeNorbert Wiener

philosophizedabouttheincreasinglycomplexrelationshipbetweenpeopleand

computers,andhowitwasfundamentallydifferentthanthewayweinteractwith

otherkindsofmachines.Developersworkinginsupervisorycontrolofmanufacturing

machinesandroboticshavehadtodealwiththesequestionspragmaticallyforabout

30years,butthankstotheInternetofThings,thisisnowaproblemthateveryone

willhavetograpplewithgoingforward.

Here’sadiagrambythegreatsTomSheridanandBillVerplank from1978,inwhich

theyillustratefourwaysthatsemi-autonomouscomputersandhumanscanwork

togethertosolveaproblem.

40

By2000SheridanexpandedtheseideaswithParasuraman andWickens todefinea

spectrumofresponsibilitybetweenpeopleandcomputers.Itrangesfromhumans

doingallthework(thisisyouwritinganessay)tocomputersdoingallthework

completelyautonomously(thisisyourcar’sfuelinjectioncontroller).Ofcoursethe

goalistogetasystemtolevel9or10.That’sthemaximumreductionincognitive

load.However,forasystemtoqualifyforthat,ithastobeverystable,itseffects

needtobehighlypredictableand,equallyimportantly,it’sroleneedstobe

adequatelyembeddedinsociety.ItneedstobeOKforacomputertotakeonthat

levelofresponsibility.Attheairportwetrustthemonorailcomputerstowork

withouthumanintervention,butwedon’ttrusttheplaneautopilottodothat,even

though-–asIunderstandit—planescanbasicallyflythemselvesthesedays.

PredictiveIoT devicesgenerallyfallbetween5and7onthisscalerightnow.The

problemisthatthisistheexactrangewhereyou’remaximizingsomeone’scognitive

load,butnotnecessarilydoingalltheworkforthem,sotheresultoftheautomation

hadbetterbeworthit.Thisfundamentallyundermineswhatweexpectfromour

tools,andwhenthattoolistryingtoanticipatewhatwe’retryingtodo,it

fundamentallychangesourworkingrelationshipwithit.

41

Theidealscenariothesethingspaintisprettyseductive.Imagineaworldofespresso

machinesthatstart brewingasyou’rethinkingit’sagoodtimeforcoffee;officelights

thatdimwhenit’ssunnytosaveenergy,andmacandcheesethatneverrunsout.The

problemisthatalthoughthevaluepropositionisofabetteruserexperience,it’s

unspecificinthedetails.Previousmachinelearningsystemswereusedinareassuch

aspredictivemaintenance andfinance.Theyweremadebyandforspecialists.Now

thatthesesystemsareforgeneralconsumers,wehavesomesignificantquestions.

Howexactlyhowwillourexperienceoftheworld,ourabilitytouseallthecollecteddata,becomemoreefficientandmorepleasurable?

We’restillearlyinourunderstandingofpredictivedevices,andinthedisciplineof

whatAaronShapiroofHugehasdubbedAnticipatoryDesign,sorightnowthe

problemsareworsethansolutions.IwanttostartbyarticulatingtheissuesI’ve

observedinourwork.

42

We’veneverhad mechanicalthingsthatmakesignificantdecisionsontheirown.As

devicesadapttheirbehavior,howwilltheycommunicatethatthey’redoingso?Do

westickasignonthemthatsays“adapting”,likethelightonavideocamerasays

“recording”?Shouldmychairvibratewhenadjustingtomyposture?Howwillusers,

orjustpassers-by,knowwhichthingsadapt?Icouldendupsittinguncomfortablefor

alongtime,waitingformychairtochange,beforerealizingitdoesn’tadaptonits

own.Howshouldsmartdevicessettheexpectationthattheymaybehavedifferently

inwhatappearstobeidenticalcircumstances?

How doweknowHOWintelligentthesedevicesare?Peoplealreadyoftenproject

moresmartsondevicesthanthosedevicesactuallyhave,soacoupleofaccurate

predictionsmayimplyamuchbettermodelthanactuallyexists.Howdoweknow

we’renotjust homesteadingtheuncannyvalleyhere?

43

Theironyinpredictivesystemsisthat they’reprettyunpredictable,atleastatfirst.

Whenmachinelearningsystemsarenew,they’reofteninaccurateandunpredictable,

whichisnotwhatweexpectfromourdigitaldevices.60%-70% accuracyistypicalfor

afirstpass,buteven90%accuracyisn’tenoughforapredictivesystemtofeelright,

sinceifit’smakingdecisionsallthetime,it’sgoingtobemakingmistakesallthetime,

too.It’sfineifyourhouseisacoupleofdegreescoolerthanyou’dlike,butwhatif

yourwheelchairrefusestogotoadrinkingfountainnexttoadoorbecauseit’sbeen

trainedondoorsanditcan’ttellthat’snotwhatyoumeaninthisoneinstance?For

allthetimesasystemgetsitright,it’sonthemistakesthatwejudgeitandacouple

suchinstancescanshatterpeople’sconfidence.Anxietyisakindofcognitiveload,

andalittledoubtaboutwhetherasystemisgoingtodotherightthingisenoughto

turnaUXthat’srightmostofthetimeintoonethat’smoretroublethanit’sworth.

Whenthathappens,you’vemorethanlikelylostyourcustomer.

Unfortunately,soonerthanwethink, suchinaccuratepredictivebehaviorisn’tgoing

tobeanisolatedincident.Soonwe’regoingtohave100connecteddevices

simultaneouslyactingonpredictionsaboutus.Ifeachis99%accurate,thenoneis

alwayswrong.Sotheproblem is:Howcanyoudesignauserexperiencetomakea

devicestillfunctional,stillvaluable,stillfun,evenwhenit’sspewingjunkbehavior?

Howcanyoudesignforuncertainty?

Photo CCBY2.0photo2011PopCultureGeektakenbyDougKline:

https://www.flickr.com/photos/popculturegeek/6300931073/

44

Thelastissuecomesasaresult oftheprevioustwo:control.Howcanwemaintain

somelevelofcontroloverthese devices,whentheirbehaviorisbydefinition

statisticalandunpredictable?

Ontheonehandyoucanmangleyourdevice’spredictivebehaviorbygivingittoo

muchdata.WhenIvisitedNestoncetheytoldmethatnoneoftheNestsintheir

officeworkedwellbecausethey’reconstantlyfiddlingwiththem.Inmachinelearning

thisiscalledovertraining.Theotherhand,ifIhavenodirectwaytocontrolitother

thanthroughmyownbehavior,howdoIadjustit?AmazonandNetflix’s

recommendationsystems,whichisakindofpredictiveanalyticssystem,giveyou

somecontextaboutwhytheyrecommendedsomething,butwhatdoIdowhenmy

onlyinterfaceisagardenhose?

45

Hereare7 patternsI’veobservedindevelopingpredictivesystemsthatIthinkmapto

theIoT.FormostoftheseI’mgoingtobeusingexamplesfromNestand

recommendersystemslikeAmazon’s,Google’sandNetflix’s.Recommendersystems

havebeenaroundformorethanadecadeandthey’vebeenextensivelystudied.The

moveintopredictivebehaviorisbuiltonacombinationofrecommendersystemsand

supervisorycontrol,soIrecommendnotreinventingthewheel,butlearningfrom

thosedisciplines.

46

To buildaneffectiveanticipatorymachinelearningsystem,youneedtoknowwhatto

anticipate,andtodothatyouneedtomakeamodelofwhatpeopleneed,valueand

desire.Simplyautomatingexistingactivitieswithoutunderstandingwhypeopledo

them,whattheirgoalsareindoingthem,missesthepointofcreatingvalue.

Predictabilityisveryvaluable,evenwhenthepredictabilityisinsomethingthat’s

flawed.Whenweincludeanticipatorybehaviorinanexperience,we’reessentially

tradingawayanincrediblyvaluablecommoditysothattradehadbetterbeworthit.

Toknowwhetherit’sworthit,weneedtohaveamodelofwhatpeoplevaluewhich

we’rereplacingoraugmenting.

47

Whatgoes intothatmentalmodel?

Therearelotsofwaystostructurehowyourepresentpeople’sviewoftheworld.It’s

asignificantfocusofcognitivescience,andIcan’tdoitjustice,buthere’sanicelistI

grabbedfromtheintelligentagentliterature.

Asadesigner,manyoftheseboildowntodecisions.Whatdecisionwillan

anticipatorysystemhelpsomeonemake?Whatdecisionswillitmakeonthat

person’sbehalf?Whataretheparametersofthatdecision?Forexample,ifIhada

real-timebloodglucosemonitorandinsulinpumpthatadjustedmybloodglucosein

realtime,whichofmydecisionswoulditmakeforme?Whichdecisionswouldittell

mehowtomake?Whichdecisionswoulditgivemeadviceabout?

Withoutaclearclearlyarticulatedstoryaboutwhatdecisionsasystemhelps

someonemake,Ibelieveyoudon’thaveaclearstoryaboutwhatvalueitbrings

them.Howdoyoufigureoutwhatthosedecisionsare?Youtalktopeople.User

research.Ethnography.Leavingtheoffice.

48

Oneofthegreatcliches inUXdesignisthesearchfordelight,suchastheseasonally

changingbackgroundsinGoogleCalendar.Mydefinitionfordelightisthatit’s

functionalitythatsubvertspeople’snear-termexpectations,butsupportstheirlong-

termneedsanddesires.Thisisparticularlyimportantindesigningpredictivesystems,

becauseifyousubvertexpectationsWITHOUTsupportingtheirneeds,youget

cognitivedissonanceandyouhaveviolatedtheirmentalmodel.

49

Becausemachinemeansyour toolsadapttoyouandlearnsfromyou,adaptivetoolsare

morelikeapprentices,ratherthanimplementsandouruseofthemismorelikea

conversationratherthanthanlineartooluse.Infact,IheardoneofNest’sUXdesignerssay

thatheconsideredusers’evolvingrelationshipstotheNestasaconversation.

Thisisespeciallyrelevantintheeraofchatbots andvoiceUI.Ifyoulistentoahuman

conversation,it’salmostneveralinear,straightforward,well-structuredprocess.Westop,

werephrase,weaskforcorrections,wetalkpasteachother,weinterrupt.Morelikelythan

not,thisishowapredictivemachinelearningsystemwillinteractwithpeople,fromwhomit

willwantguidance,confirmation,andwhowillaskitforrecommendationsorchangestoits

behavior.

Ethnomethdologists andconversationanalystshavebeenmodelinghowpeopletalktoeach

otherforabout40years,soI’mgoingtoborrowsomeoftheirconcepts.

• Sequenceorganizationisaboutorganizingactionintime.Whathappensfirst,what

happensnext?Howdothetwopartiesexpandonambiguity?Forexample,ifahome

securitysystemdecidesyou’renothome,itcantellyou“Iseeyou’redrivingawayfrom

home.I’mgoingtoturnallthealarmson.”Youcanthensay“Alloftheexceptfortheback

yard.”

• Turn-takingiscritical.Wedon’tjustsimplytaketurnswhentalking,wecontinuously

providefeedbackandcorrect.Wehaveexpectationsforwhoseturnisnextandwhat

they’resupposedtodo.“Ok,chair,I’msittinghere,nowit’syourturn.Confirmyouknow

I’mhere.Warnmeifyou’regoingtoadjust.”

• Repairisbacktracking,clarifying,continuingafteraninterruption,etc.Whathappens

whentheexpectedsequence,eitherfromtheperspectiveofthepersonortheservice,is

brokenandneedstobereconstructed?

50

Inadditiontoteachingapprentices aboutourneeds,wealsolearnfromapprentices

whattheircapabilitiesareandwhytheymadecertaindecisions,ratherthanothers,

whendoingthethingswetaughtthemtodo.Thisisbothapartofhowtheylearn

aboutusandhowwelearntoworkwiththemeffectively.TheBMWiDrive system

wasnotoriousforitsUI,whichdidn’ttellyouwhatitcouldorcouldn’tdo,andhowto

doit.Youhadaknobandthatwasbasicallyit.

HowdoIinterrogateanadaptivesystemtounderstandwhatitcando,andtoaskit

toexplainwhatitjustdid.

HowdoyouknowwhatSiriorGoogleNowhavelearnedtodo?Well,youusethe

app.Butwhataboutservicesforwhichyoudon’thaveadisplay?Chatbots todayare

essentiallycommandlineinterfaces.Theyknowspecificwordsandsequences,but

whatifthosecommandschangeovertime?Whatifthedevicelearnsnewthingsover

time?

51

Thenextpatternisthatyouneedauserstoryforeverystageofthemachinelearning

andpredictionprocess,evenforstepsthatseemsinvisible.Howwillyouincentivize

peopletoaddtheirbehaviordatatothesystematall?WhyshouldIuploadmycar’s

dashcam videotoyourtrafficpredictionsystemEVERYDAY?Howwillyou

communicateyou’reextractingfeatures?IlikethewaythatGooglespeechtotext

showsyoupartialphrasesasyou’respeakingintoit,andhowitcorrectsitself.That

smallbitoffeedbacktellspeopleit’spullinginformationoutandittrainsusershow

tomeetthealgorithmhalfway.Howdomachine-generatedclassificationscompareto

people’sorganizationofthesamephenomena?Howisacontextmodelpresentedto

endusersanddevelopers?Howwillyougetpeopletotrainitandtellyouwhenthe

modeliswrong?Doesthefinalbehavioractuallymatchtheirexpectation?

52

Sincepredictivesystemsareneitherconsistent, norarethereasonsfortheirbehavior

clear,thiscanbereallyconfusing.Thesamethingcanbehavedifferentlyinwhat

appeartobesimilarcircumstances.Ifweunderminepeople’sconfidenceinasystem

byviolatingtheirexpectations,they’relikelytobedisappointedandstopusingit.

Whenwe’redealingwithahuman orananimal,unpredictablebehaviorsare

expectedandtolerated,butthat’snotthecasewithcomputers.ApredictiveUX

needstodoistosetpeople’sexpectationsappropriately.Itneedstoexplain the

natureofthedevice, todescribeitistryingtopredict, thatit’stryingtoadapt,that

it’sgoingtosometimesbewrong,toexplainhowit’slearning,andhowlongit’lltake

beforeitcrossesoverfromcreatingmoretroublethanbenefit.

Recommendersystems,suchasGoogleNow,describewhyacertainkindofcontent

wasselected,andthatsetstheexpectationthatinthefuturethesystemwill

recommendotherthingsbasedonotherkindsofcontentyou’verequested.Nest’s

FAQkindofburiestheinformation,butitdoesexplainthatyoushouldn’texpectyour

thermostattomakeamodelofwhenyou’rehomeornotuntilit’sbeenoperatingfor

aweekorso.

53

Aboutten yearsagoTimo Arnall andhisstudentstriedtoaddressasimilarsetof

questionsaroundinteractionswithRFID-enableddevicesbycreatinganiconography

systemthatcommunicatedtopotentialusersthatthesedeviceshadfunctionality

thatwasinvisiblefromtheoutside.Perhapsweneedsomethinglikethisforbehavior

createdbypredictivebehavior?

54

Predictivebehavior,isallabouttime,aboutsequencesofactivities.Manypredictive

UXissues aroundexpectationsanduncertaintyhavetimeastheirbasis:whatwere

youexpectingtohappenandwhy.Ifitdidn’thappen,why?Ifsomethingelse

happened,orithappenedatanunexpectedtime,whydidthathappen?

Knowing thatadevicehasactedonyourbehalf,andthatit’sgoingtoact—andHOW

it’sgoingtoact—inthefutureisimportanttogivingpeopleamodelofhowit’s

working,settingtheirexpectations,reducingtheuncertainty.Nest,forexample,hasa

calendarofitsexpectedbehavior,anditshowsthatit’sactingonyourbehalfto

changethetemperature,andwhenyoucanexpectthattemperaturewillbereached.

55

Youhavetogivepeopleaclear waytoteachthesystemandtellitwhenitsmodelis

wrong.Statisticalsystems,bydefinition,don’thavesimplerulesthatcanbechanged.

Therearen’tobvious handlestoturnordialstoadjust,becauseeverythingis

probabilistic.Ifthemodelismadefromdatacollectedbyseveraldevices,which

deviceshouldIinteractwithtogetittochangeitsbehavior?GoogleNowasks

whetherIwantmoreinformationfromasiteIvisited,Amazon showsaexplanation

ofwhyitgavemeasuggestion.MappingthistotheconsumerIoT meanswaymore

explanationthanwe’recurrentlygetting,whichiseitherthatathinghashappened,

orithasn’t.

56

Finally,don’tautomate. Thesesystemsshouldn’ttrytoreplacepeople,buttosupport

them,toaugment andextenttheircapabilities,tohelpthembebetteratwhatthey

wanttodo,nottoreplacethem.

Forexample,EmberfromMeshfire, isamachinelearningassistantforsocialmedia

management.Itdoesn’ttrytoreplacethesocialmediamanager.Insteaditmanages

themediamanager’stodo list.Itaddsthingsthatitthinksaregoingtobeinteresting,

deletesoldthings,andreprioritizesthemanager’slistbasedonwhatitthinksis

important.Ithinkthisisagoodmodelforhowsuchsystemscanaddvaluetoa

person’sexperiencewithoutcreatingasituationwhererandom,unexplained

behaviorsconfusepeople,frustratethemandmakethemfeelpowerless.Emberisan

augmentationtothesocialmediamanager,ithelpsthatpersonfocusonwhat’s

importantsothattheycanbesmarterabouttheirdecisions.Itdoesn’ttrytobe

smarterthantheyare.HowcanourdevicesHELPus,ratherthantryingtoreplaceus?

57

Finally,anantipattern:makingpeople doallofthetraining,askingthemtoidentify

whetherabehaviorisappropriateornot,shouldbedoneselectivelyand

infrequently.Yes,itwillreallyhelpyoursupervisedmodel’saccuracytohavepeople

identifythecorrectpositivesfromthefalsepositives,butunlessyou’repayingthese

people,it’sincrediblyannoyingtohavecustomersdoitallthetime.LastFridayone

consumerIoT productwithamachinelearningsystemI’mplayingwithaskedmeto

classifyitsoutputat1:11PM,thenagainat1:26,andagainat1:47andagainand

again.Ithinkitwasonroughlyten-minutesensingcycle,andateverycycleittriedto

makeadecision,andaskedmetoverifyit.I’msureit’sstilldoingit,butIturnedoff

allnotificationsfromit,andnowI’mconsideringturningitoffentirely.Peoplewill

sometimeswillinglyactassensorsandactuatorsforyoursystem,butbecausethey

arenotmachines,theywillnotdoitallthetimeandyou’rejustgoingtohavetofind

abetterwaytotrainyourmodel.

58

Finally,formetheIoT isnotaboutthethings,buttheexperiencecreatedby the

servicesforwhichthethings areavatars.

59

Ultimatelyweareusingthesetoolstoextendourcapabilities,tousethedigitalworld

asanextensionofourminds.Todothatwellwehavetorespectthatasinteresting

andpowerfulasthesetechnologiesare,theyarestillintheirinfancy,andourjobas

entrepreneurs,developersanddesignerswillbetocreatesystems,services,thathelp

people,ratherthanaddingextraworkinthenameofsimplisticautomation.Whatwe

wanttocreateisasymbioticrelationshipwherewe,andourpredictivesystems,work

togethertocreateaworldthatprovidesthemostvalue,fortheleastcost,forthe

mostpeople,forthelongesttime.

Wearecurrentlyshovelingourolddevicesintothisnewmedium.Wehavenotyet

figuredoutwhattheessentialcapabilitiesofthisnewmediumare.

LiteralMcLuhanquotation:"Thecontentofthepressisliterarystatement,asthe

contentofthebookisspeech,andthecontentofthemovieisthenovel."

60

Thankyou.

61