Upload File

uncertain interaction - university of cambridge · 2019-01-30 · computational interaction •...

  • Home
  • Documents
  • Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical
157
Uncertain Interaction Per Ola Kristensson Department of Engineering University of Cambridge Solution principles for

Upload: others

Post on 15-Aug-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

UncertainInteraction

PerOlaKristenssonDepartmentofEngineeringUniversityofCambridge

Solution principles for

Page 2: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

VisionsoftheFuture

Page 3: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

VisionsoftheFuture

•  Ubiquitoussensing–  Smarthome,InternetofThings,etc.

•  Pervasiveagents–  Spokendialogue-basedcommandandquery

interfaces•  Virtualreality–  Portableoffice,training,immersivedataanalytics

•  Phonewithoutaphone– Opticalsee-throughhead-mounteddisplayswithform

factorscomparabletoeverydayglasses

Page 4: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

VisionsoftheFuture

•  Ubiquitoussensing–  Smarthome,InternetofThings,etc.

•  Pervasiveagents–  Spokendialogue-basedcommandandquery

interfaces•  Virtualreality–  Portableoffice,training,immersivedataanalytics

•  Phonewithoutaphone– Opticalsee-throughhead-mounteddisplayswithform

factorscomparabletoeverydayglasses

Allassumefluidinterfacesbasedonfundamentallyuncertaininteraction

Page 5: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Computationalinteraction•  Classichuman-computerinteraction(HCI)method

doesnothandleuserinterfacedesignunderuncertaintyverywell

•  ClassicHCImethodisunderpinnedonelicitinguserneedsusingavarietyofprocessesandthenaniterativeprocessofdesignandevaluation,inwhichdesignisdrivenbydesigningenuityratherthanprinciples

•  Thismeans:–  Noautomateddesignwork–  Noexplicitmodel–  Datainfluenceddesignonlythroughthedesigner

•  ComputationalinteractionisanemergingdisciplineinHCIwhichproposesuserinterfacedevelopmentbyallowingalgorithmstoperformwork,byexplicitmodelling,andbyallowingdatatodirectlyinfluencedesign.

Page 6: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Computationalinteraction•  Computationalinteractionwouldtypicallyinvolve

atleastoneof:I.  anexplicitmathematicalmodelofuser-system

behavior;II.  awayofupdatingthatmodelwithobserveddata

fromusers;III.  analgorithmicelementthat,usingthismodel,can

directlysynthesiseoradaptthedesign;IV.  awayofautomatingandinstrumentingthe

modelinganddesignprocess;V.  theabilitytosimulateorsynthesiseelementsofthe

expecteduser-systembehavior.

Page 7: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical
Page 8: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Intelligenttextentryasanexampleofdesigninginteractionunder

uncertainty

Page 9: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical
Page 10: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 11: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry

1.  Letterssimplifiedtolinemarks

2.  Commonwordstemscompressedintosimplelinemarksordots

3.  Commonwordstemsidentifiedbywordfrequencyanalysisofthebookofpsalms

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 12: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry

1.  Letterssimplifiedtolinemarks

2.  Commonwordstemscompressedintosimplelinemarksordots

3.  Commonwordstemsidentifiedbywordfrequencyanalysisofthebookofpsalms

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 13: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry

1.  Letterssimplifiedtolinemarks

2.  Commonwordstemscompressedintosimplelinemarksordots

3.  Commonwordstemsidentifiedbywordfrequencyanalysisofthebookofpsalms

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 14: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry•  Inotherwords:

1.  Optimisespeedbyminimsingtheamountofinformationusershavetoarticulate

2.  Exploitredundanciesinnaturallanguagesbycreatingalanguagemodel

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 15: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry•  Inotherwords:

1.  Optimisespeedbyminimsingtheamountofinformationusershavetoarticulate

2.  Exploitredundanciesinnaturallanguagesbycreatingalanguagemodel

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 16: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry•  Inotherwords:

1.  Optimisespeedbyminimsingtheamountofinformationusershavetoarticulate

2.  Exploitredundanciesinnaturallanguagesbycreatingalanguagemodel

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 17: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Principlesofintelligenttextentry•  ...whichcanoftenbe

thoughtofasaninferenceproblem:

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

Page 18: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Whydonearlyalltextentrymethodsfail?

Page 19: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Page 20: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Graffiti

Page 21: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Graffiti Multi-tapandpredictivetext

Page 22: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Graffiti Multi-tapandpredictivetext

Touchscreenkeyboards

Page 23: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Graffiti Multi-tapandpredictivetext

Touchscreenkeyboards

Gesturekeyboards

Page 24: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

Graffiti Multi-tapandpredictivetext

Touchscreenkeyboards

Gesturekeyboards Physicalthumb

keyboards

Page 25: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

Page 26: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

Page 27: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

Page 28: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

2006-2008 2014

Page 29: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

40-60wpm(ceilingrates)

2006-2008 2014

Page 30: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

40-60wpm(ceilingrates)

Mainstreamtextentrymethods(touchscreenQWERTY,thumbkeyboard,gesturekeyboard)

2006-2008 2014

Page 31: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

40-60wpm(ceilingrates)

Mainstreamtextentrymethods(touchscreenQWERTY,thumbkeyboard,gesturekeyboard)

2006-2008 2014

Researchtextentrymethods(slowerthancommercialsolutions)

Page 32: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

40-60wpm(ceilingrates)

Mainstreamtextentrymethods(touchscreenQWERTY,thumbkeyboard,gesturekeyboard)

2006-2008 2014

Researchtextentrymethods(slowerthancommercialsolutions) Optimised

keyboards

Page 33: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

Page 34: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Page 35: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Familiarinterface

Page 36: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Unfamiliarinterface

Familiarinterface

Page 37: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Unfamiliarinterface

Familiarinterface

Page 38: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Unfamiliarinterface

Familiarinterface

Cross-overpoint

Page 39: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Unfamiliarinterface

Familiarinterface

Timeinvestment

Cross-overpoint

Page 40: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Unfamiliarinterface

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Page 41: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Page 42: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Page 43: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Page 44: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Objectivebenefit

Nicosia,M.,Oulasvirta,A.andKristensson,P.O.2014.Modelingtheperceptionofuserperformance.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:1747-1756.

Page 45: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Perceivedbenefit

Nicosia,M.,Oulasvirta,A.andKristensson,P.O.2014.Modelingtheperceptionofuserperformance.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:1747-1756.

Page 46: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 47: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 48: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 49: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Interactionstrategies

Page 50: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Efficientencodings

Interactionstrategies

Page 51: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Optimisinglayouts

Efficientencodings

Interactionstrategies

Page 52: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 53: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 54: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thenarrowdesignspace

Page 55: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Solutionprinciples•  Fromclosedtoopen-loop

–  Avoidtheneedforavisualfeedbackloop•  Continuousnovice-to-experttransition

–  Avoidexplicitlearning•  Pathdependency

–  Avoidredesigningtheinteractionlayer•  Flexibility

–  Enableuserstocomposeandeditinavarietyofstyleswithoutexplicitmodeswitching

•  Probabilisticerrorcorrection–  Usethehypothesisspacetodesignoptimalerrorcorrection

strategies•  Fluidregulationofuncertainty

–  Allowuserstoseamlesslyinfluencetheinferenceprocess•  Efficiency

–  Letusers’creativitybethebottle-neck

Page 56: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

FromClosedtoOpenLoop

Page 57: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Reimagingthekeyboard

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 58: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 59: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 60: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 61: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 62: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 63: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 64: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 65: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 66: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 67: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 68: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 69: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Page 70: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Priorprobability

Page 71: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Priorprobability

Likelihood

Page 72: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhiterabbit

Priorprobability

Likelihood

Posteriorprobability

Page 73: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhiterabbit

Priorprobability

Likelihood

Posteriorprobability

Decodingnoisygesturesintotextusingacombinationofgesturerecognitionandlanguagemodelling

Page 74: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Closed-andopen-loop•  Closed-loop:

–  Continuousfeedback-driveninteraction–  Visually-guidedmotion–  Slowandprecise–  Modelledwellbythe“crossinglaw”

•  Averagemovementtime=a+blog2(D/W+1);aandbarelinearregressioncoefficients;DandWarethedistanceandwidthtothecrossinggoalrespectively

•  Open-loop:–  Notfeedback-driven–  Directrecallfrommotormemory–  Fastandimprecise–  Nogoodmodelexits

•  Gesturekeyboardinteractionisamixofclosed-andopen-loopinteraction

Page 75: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

ContinuousNovice-to-ExpertTransition

Page 76: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Continuoustransitionfromnovicetoexpertbehaviour

Skillacquisition

Fallingbackandrelearning

Consistentmovementpattern

Completenovice:TracinglettertoletterClosed-loopSlowandaccurate

Completeexpert:GesturingwordshapesOpen-loopFastandinaccurate

Page 77: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

PathDependency

Page 78: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical
Page 79: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Example:typingonasmartwatch

•  Smallscreensizeisobviouslyaconstraint•  Manynaïvesolutions:– Progressivezoomingtechniques– Reducekeyset(álatheoldtelephonekeypadtechniques)

– Variousmulti-strokestrategies•  Allslow•  Alldemanduserlearning(noimmediateefficacy)

Page 80: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Page 81: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

=40hoursofdedicatedpractice

Page 82: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

=40hoursofdedicatedpractice

Assumetheusertypesforfiveminutesontheirsmartwatcheveryday

Page 83: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

=40hoursofdedicatedpractice

Userperformanceaftermonthsofuse

Assumetheusertypesforfiveminutesontheirsmartwatcheveryday

Page 84: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecross-overpoint

Time

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpointreachedafter480days

=40hoursofdedicatedpractice

Userperformanceaftermonthsofuse

Assumetheusertypesforfiveminutesontheirsmartwatcheveryday

Page 85: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

Page 86: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

•  IttakesaverylongtimetolearnQWERTY(orlearnanewlayout)

Page 87: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

•  IttakesaverylongtimetolearnQWERTY(orlearnanewlayout)

•  UsersarefamiliarwithtouchscreenQWERTY

Page 88: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

•  IttakesaverylongtimetolearnQWERTY(orlearnanewlayout)

•  UsersarefamiliarwithtouchscreenQWERTY

•  KeepQWERTY

Page 89: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Mainstreammobiletextentrymethods

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

entry

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

•  IttakesaverylongtimetolearnQWERTY(orlearnanewlayout)

•  UsersarefamiliarwithtouchscreenQWERTY

•  KeepQWERTY

Page 90: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Touchmodelling

2DGaussianscenteredateachkey.Separatevariancesinthex-andy-dimensions.

Vertanen,K.,Memmi,H.,Emge,J.,Reyal,S.andKristensson,P.O.2015.VelociTap:investigatingfastmobiletextentryusingsentence-baseddecodingoftouchscreenkeyboardinput.InProceedingsofthe33rdACMConferenceonHumanFactorsinComputingSystems(CHI2015).ACMPress:659-668.

Page 91: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Languagemodelling

•  Languagemodels:– 12-gramlettermodel– 4-gramwordmodelwithunknownword– Trainedonbillionsofwordsofdata

§ Twitter,blog,socialmedia,Usenet,andwebdata

– Optimizedforshortemail-likemessages– Letter+wordlanguagemodel=~4GBmemory

Vertanen,K.,Memmi,H.,Emge,J.,Reyal,S.andKristensson,P.O.2015.VelociTap:investigatingfastmobiletextentryusingsentence-baseddecodingoftouchscreenkeyboardinput.InProceedingsofthe33rdACMConferenceonHumanFactorsinComputingSystems(CHI2015).ACMPress:659-668.

Page 92: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Decoding

Observation1 Observation2 Observation3fgc

z

ϵ

abz

Tokenstrack:probability,LMcontext,traceback

o

ϵ

a

z

X X

d

ϵ

o za

X

d

good

god

go

Beamprunetokeeptractable

X XX

X

Page 93: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Entryanderrorrate

Condition

Normal Standardportraitkeyboard,60mmwide

Small Bigsmartwatch,40mmwide

Tiny Smallsmartwatch,25mmwide

Page 94: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Typingonatinykeyboard

Page 95: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Flexibility

Page 96: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”

Page 97: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”•  System:“thebatsat”

Page 98: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”

Page 99: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”•  System:“thebatsatdissectrat”

Page 100: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”•  System:“thebatsatdissectrat”•  (User:“Ihatethis…”)

Page 101: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”

Page 102: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”•  System:“thebatsat”

Page 103: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”•  System:“thebatsat”•  Usergesturestheword:“cat”

Page 104: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”•  System:“thebatsat”•  Usergesturestheword:“cat”•  System:“thecatsat”

Page 105: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”•  System:“thebatsat”•  Usergesturestheword:“cat”•  System:“thecatsat”

•  Thesystemautomaticallyidentifiestheerrorlocationandcorrectstheerror

Kristensson,P.O.andVertanen,K.2011.Asynchronousmultimodaltextentryusingspeechandgesturekeyboards.InProceedingsofthe12thAnnualConferenceoftheInternationalSpeechCommunicationAssociation(Interspeech2011).ISCA:581-584.

Page 106: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

Page 107: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

Timestep1

Page 108: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

Timestep1

Page 109: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

Timestep1 Timestep2

Page 110: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

thee0.3

the0.6

three0.1

Page 111: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

the0.3

ε0.6

thee0.1

Page 112: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Outputfromatextentrymodality

Gesturekeyboard

thee0.3

the0.6

three0.1

Page 113: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

VAT0.2

Outputfromatextentrymodality

Gesturekeyboard

thee0.3

the0.6

three0.1

cat0.6

cart0.2

Page 114: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

at0.28rat0.28

Outputfromtwotextentrymodalities

the0.56

a0.38

cat0.82

at0.06fat0.06

ε0.87

at0.09

sat0.75

nat0.19

the0.94 bat0.57

cat0.09

sat0.47

ε0.21

Page 115: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

at0.28rat0.28

Softeningthewordconfusionnetworks:addingwild-cardtransitions

the0.56

a0.38*0.03

cat0.82

at0.06fat0.06*0.03

ε0.87

at0.09*0.03

sat0.75

nat0.19*0.03

the0.94

*0.03

bat0.57

cat0.09*0.03

sat0.47

ε0.21*0.03

Page 116: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

at0.28rat0.28

Softeningthewordconfusionnetworks:addingepsilontransitions

the0.56

a0.38*0.03ε0.02

cat0.82

at0.06fat0.06*0.03ε0.02

ε0.87

at0.09*0.03

sat0.75

nat0.19*0.03ε0.02

the0.94

*0.03ε0.02

bat0.57

cat0.09*0.03ε0.02

sat0.47

ε0.21*0.03

Page 117: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

at0.28rat0.28

Softeningthewordconfusionnetworks:addingwild-cardself-loops

the0.56

a0.38*0.03ε0.02

0.01*

0.01* cat0.82

at0.06fat0.06*0.03ε0.02

0.01* ε0.87

at0.09*0.03

0.01* sat0.75

nat0.19*0.03ε0.02

0.01*

the0.94

*0.03ε0.02

0.01*

0.01* bat0.57

cat0.09*0.03ε0.02

0.01* sat0.47

ε0.21*0.03

0.01*

Page 118: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

at0.28rat0.28

Searchforthehighestjointpathinbothrecognitionmodalities

the0.56

a0.38*0.03ε0.02

0.01*

0.01* cat0.82

at0.06fat0.06*0.03ε0.02

0.01* ε0.87

at0.09*0.03

0.01* sat0.75

nat0.19*0.03ε0.02

0.01*

the0.94

*0.03ε0.02

0.01*

0.01* bat0.57

cat0.09*0.03ε0.02

0.01* sat0.47

ε0.21*0.03

0.01*

Page 119: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Speech-onlyflexiblerepair

Page 120: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Probabilisticerrorcorrection

Page 121: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Probabilisticerrorcorrection

•  Foranyprobabilistictextentrymethod…–  Capableofassigningposteriorprobability

distributionstowords•  …thereexistsahypothesisspace•  Thebestresultisthemaximumprobabilitypath

inthishypothesisspace– However,itneednotbetheonetheuserintended

•  Byexposingpartofthehypothesisspacetousers,highefficienciescanbegainedwhenuserscorrectwords

Page 122: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Fluidregulationofuncertainty

Page 123: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Theauto-correcttrap•  Auto-correctisgreatwhenitworks•  However,whenauto-correctfailserrorcorrectionactivities

exhibitahighpenalty•  Thesolutionistoprovideuserswithmoreagencyand

allowthemtoregulatetheircertainty

Weir,D.,Pohl,H.,Rogers,S.,Vertanen,K.andKristensson,P.O.2014.Uncertaintextentryonmobiledevices.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:2307-2316.

Page 124: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Pressure-sensitiveauto-correct

•  LikelihoodofaGaussianwithstandarddeviationregulatedbypressure

•  StandarddeviationcomputedasC/ωT,whereCisaconstantandωTisthepressurefortouchT

•  TunedCsothatthepressureofatypicaltouchhadastandarddeviationofhalfakeywidth

Weir,D.,Pohl,H.,Rogers,S.,Vertanen,K.andKristensson,P.O.2014.Uncertaintextentryonmobiledevices.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:2307-2316.

Page 125: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Results•  Enablinguserstoregulatetheircertaintybyforce

resultedina10%percentagedropinactivecorrections(fixingawordbybackspacingorretyping)

•  Thisimprovedentryrateby20%

Page 126: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Efficiency

Page 127: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 128: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 129: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

125ms

Page 130: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

250ms

Page 131: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

375ms

Page 132: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

500ms

Page 133: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

625ms

Page 134: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

750ms

Page 135: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

875ms

Page 136: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

1000ms

Page 137: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Recordspeedsachievedwhenwritingbygaze

•  Eye-typing– 5–10wpm(MajarantaandRäihä2002;Roughetal.2014)

•  Eye-typingwithadjustable-dwell– 7-20wpm(Majarantaetal.2009;RäihäandOvaska2012;Roughetal.2014)

•  Dasher– 12–26wpm(Tuiskuetal.2008;WardandMacKay2002;Roughetal.2014)

Page 138: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Kristensson,P.O.andVertanen,K.2012.Thepotentialofdwell-freeeye-typingforfastassistivegazecommunication.InProceedingsofthe7thACMSymposiumonEye-TrackingResearch&Applications(ETRA2012).ACMPress:241-244.

Page 139: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 140: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 141: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 142: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 143: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 144: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 145: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 146: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Thecat

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Page 147: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Humanperformanceestimateofdwell-freeeye-typing

•  Recorded400minutesofeye-tracedata•  Participantsenteredatotalof2026phrases•  Participantswerepromptedphrasesandaskedtocopy

themasquicklyandasaccuratelyaspossible•  Oursystemknewwhattheuserwassupposedtowriteand

verifiedthattheuserisgazingatthelettersequencecorrespondingtothestimulus

Page 148: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Entryrate

Page 149: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Humanperformancemodel

Page 150: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Humanperformancemodel

Eye-typingusingadjustabledwell,finalentryrate(mean=20wpm)

Page 151: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Humanperformancemodel

Eye-typingusingadjustabledwell,finalentryrate(mean=20wpm)

Page 152: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Humanperformancemodel

Eye-typingusingadjustabledwell,finalentryrate(mean=20wpm)

230%

Page 153: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Entryrate,first10-15minutes

Page 154: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Entryrate,first10-15minutes

Eye-typingusingadjustabledwell,entryrateinthefirstsession(mean=6.9wpm)

Page 155: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Entryrate,first10-15minutes

Eye-typingusingadjustabledwell,entryrateinthefirstsession(mean=6.9wpm)

520%

Page 156: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Astep-changeingazecommunication

•  Existinggazecommunicationsolutions– Limitedtocirca20wpm

•  Dwell-freeeye-typing– Empiricallymeasuredhumanperformancepotential:46wpmaverage

•  Releasedasaproduct:Tobii-DynavoxI-Series+

Page 157: Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction • Computational interaction would typically involve at least one of: I. an explicit mathematical

Conclusions•  Atextentrymethodlikelytobeadoptedbyusersis

probablysimilartoexistingsolutionsandatleastasfast•  Itisstillpossibletomakeprogressbyusingafewsolution

principles:–  Fromclosedtoopen-loop–  Continuousnovice-to-experttransition–  Pathdependency–  Flexibility–  Probabilisticerrorcorrection–  Fluidregulationofuncertainty–  Efficiency

•  Ingeneral,thesecanbeviewedassolutionprinciplesforuncertaininteraction

Kristensson,P.O.2015.Next-generationtextentry.IEEEComputer48(7):84-87.


Towards A Dual Process Approach to Computational Explanation in Human-Robot Social Interaction

Computational Modeling of Protein-Protein Interaction · 2014. 10. 8. · Computational Modeling of Protein-Protein Interaction YinghaoWu Department of Systems and Computational Biology

Interactive Visualization for Computational Linguisticsgpenn/esslli09/day3.pdf · 2009. 7. 30. · Interactive Visualization for Computational Linguistics ESSLII 2009 2 Interaction

COMPUTATIONAL MODELING OF FLUID–STRUCTURE INTERACTION … · COMPUTATIONAL MODELING OF FLUID–STRUCTURE INTERACTION IN BIOLOGICAL FLYING AND SWIMMING By Hu Dai Dissertation

Neural Modeling and Computational Neurosciencedidawiki.cli.di.unipi.it/.../computational-neuroscience/neuralmodeling_3.pdf · computational brain, modeling neuron-astrocyte interaction,

Protein interaction Computational (inferred) Experimental (observed)

Interaction of Computational Tools for Multiscale ... · Interaction of Computational Tools for Multiscale Multiphysics Simulation of Generation-IV Reactors I.A. Bolotnov1, F. Behafarid1,

Autonomy, Interaction & Complexity in Computational Systems. Preliminary Notes

ELECTRONIC SUPPLEMENTARY INFORMATION (ESI) Computational ... · ELECTRONIC SUPPLEMENTARY INFORMATION (ESI) Computational insight into the interaction of oxaliplatin with insulin Giuseppe

FROM UNCERTAIN PROTEIN INTERACTION NETWORKS TO SIGNALING PATHWAYS

A one-dimensional computational model for the interaction

Computational Benchmark of Commercial Fluid-Structure ...web.aeromech.usyd.edu.au/~gareth/Articles/giannelis2015... · Computational Benchmark of Commercial Fluid-Structure Interaction

Comparative Analysis of Molecular Interaction Networks · Comparative Analysis of Molecular Interaction Networks ... Challenges in Computational Analysis: Thesis ... Frequency is

Complexity & Interaction: Blurring Borders between Physical, Computational, and Social Systems. Preliminary Notes

1 Protein-Protein Interaction Networks MSC Seminar in Computational Biology 19.1.2006

A computational method for dislocation–precipitate interaction · interaction problems based on linear elasticity theory, which enables us to investigate the interaction between

Webinar: Discovering Small Molecule Protein-Protein Interaction Inhibitors through Computational Drug Design

A Computational Investigation of Wheel and …publications.lib.chalmers.se/records/fulltext/184130/184130.pdfA Computational Investigation of Wheel and Underbody Flow Interaction Investigating

Computational Prediction and Analysis of Interaction of Silver Nitrate with Chitinase Enzyme

Validation of Computational Fluid-Structure Interaction

Computational Benchmark of Commercial Fluid-Structure Interaction Software for Aeroelastic Applications

Supporting Vulnerable Children in an Uncertain World · Supporting Vulnerable Children in an Uncertain World ... See separate booklet for information about your interaction ... ISPA

Computational simulation of the interaction between a

Computational modelling of full interaction ... - Amazon S3

Complexity & Interaction: Blurring Borders between Physical, Computational, and Social Systems. A Coordination Perspective

Dynamic Network Flow with Uncertain Arc … › ~ctovey › publications › papers › master...Dynamic Network Flow with Uncertain Arc Capacities: Decomposition Algorithm and Computational

Human-Robot interaction with low computational-power humanoids

Computational Study of Amino Mediated Molecular ...lv2117/SIs/BalducciJPCC2015SI.pdf · Supporting Information Computational Study of Amino Mediated Molecular Interaction evidenced

Structuralism: Patterns of Interaction Computational

A unique computational algorithm to simulate probabilistic ...€¦ · A unique computational algorithm to simulate probabilistic multi-factor interaction model complex material point

Effectofwavelengthandbeamwidthonpenetrationinlight-tissue ... · ORIGINAL ARTICLE Effectofwavelengthandbeamwidthonpenetrationinlight-tissue interaction using computational methods

Protein-Protein Interaction Networks - · PDF fileProtein-Protein Interaction Networks William Andreopoulos Center for Computational Biology and Bioinformatics Department of Electrical

COMPUTATIONAL MODEL OF TISSUE …ut27/Computational Modelling Of...School of Biomedical Engineering, Science and Health Systems, Drexel University COMPUTATIONAL MODEL OF TISSUE INTERACTION

Towards a computational representation of host-pathogen interaction networks

Experimental and Computational Investigation on ... · Experimental and Computational Investigation on Interaction between Nano Rotor and Aerodynamic Rudder ... and minimum induced