enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile...

Enhancing the interaction spaceof a tabletop computing system

to design paper prototypesfor mobile applications

Master ThesisFrancesco Bonadiman

2016

Introduction

1

Blended Prototyping

● tabletop system based on hand-drawn paper sketches

2

Blended Prototyping


○ converted into digital versions→ projected on the table→ expanded further

2



○ transformed into testable applications→ on a mobile device

Blended Prototyping

2



○ transformed into testable applications→ on a mobile device

● enhances development of mobile applications○ accelerates early design phases

Blended Prototyping

2

Hardware

● video projector

○ projects mobile frames & prototypes

3

Hardware

● video projector


● webcam

○ recognizes barcodes of screens

3

Hardware

● video projector


● webcam


● DSLR camera

○ shoots HQ pics of the sketches

3

Hardware

● video projector


● webcam


● DSLR camera

○ shoots HQ pics of the sketches

● tablet

○ allows to perform actions on the prototypes

3

Software

● Java application

○ controls behavior of projector & cameras

4

Software



○ identifies barcode markers on paper sheets

4

Software




○ digitize sketches & perform actions on them

4

Software




○ digitize sketches & perform actions on them

● Barcode (similar to QR-code)

○ wider → optimized for webcam

○ gives sheet’s position & rotation on table

4

Context

5

Advantages of Paper Prototyping

● potential of Paper-Prototyping & mobile devices = combined

6



● first impression of not-yet-developed product

6




● paper → cheap, fast & intuitive (Snyder)

○ no unimportant details → iterative refinement (Nielsen)

6






○ quickly create multiple design alternatives (Landay)

6






○ quickly create multiple design alternatives (Landay)

● same usability issues as Hi-Fi discovered

○ benefits of early usability data = 10+ times bigger (Snyder)

6

● encourages collaboration & critiques →

interdisciplinary & creative (no “gaps”)

Advantages of the System

7



● testing within real-life scenarios (de Sá / Carriço)

→ in thousand of different usage conditions


7



● testing within real-life scenarios (de Sá / Carriço)

→ in thousand of different usage conditions

● add code & functionalities

○ define dynamic interface behavior

○ smooth transition to development


7

Core features

Via TABLET → Determine widgets & semantics

8

Core features


● manually define “hotspots” on prototype

● turn these into different design patterns

8

Core features




● create links between the prototypes

● remove components & connections

8

Core features




● create links between the prototypes

● remove components & connections

● convert into working code

8

Problem

● several devices + media → different degrees of fidelity

○ paper & office supplies → computers & smart objects

9

Problem



BUT

9

Problem



BUTChanging fidelity of tool & modality of interaction

→ disrupts creative design process

9

Problem



BUTChanging fidelity of tool & modality of interaction

→ disrupts creative design process

→ shifting continuously confuses users

9

Fidelity Clash

● combination of low- & high-tech solutions = puzzling

10

Fidelity Clash


○ one single user stops ideation process

→ to perform any action

10

Fidelity Clash




○ breaking of collaborative & creative moment (Snyder)

10

Fidelity Clash





● users distracted → do not interact anymore

○ get unfocused → lose the “flow”

10

Fidelity Clash





● users distracted → do not interact anymore

○ get unfocused → lose the “flow”

→ perceived as too technical, isolating & distracting

10

Objectives

● replace high-tech interactions with low-tech approaches

11

Objectives


● new interaction techniques → not to shift fidelity of media

11

Objectives



THEN

11

Objectives



THEN● find & implement alternative solutions to tablet

11

Objectives



THEN● find & implement alternative solutions to tablet

● low-tech approaches to define “hotspots” & perform actions

○ keep interaction techniques learnable & usable

11

Enhanced System

12

Tasks

1. digitize a screen by taking a picture

13

Tasks


2. duplicate a whole screen

13

Tasks



3. detect a component (button, image or textbox)

13

Tasks




4. connect two screens (link a button to next screen)

13

Tasks





5. remove a component from a screen

13

Tasks






6. remove a connection between two screens

13

Tasks






6. remove a connection between two screens

7. remove a whole screen

13

Alternatives considered

● Voice recognition

● Object recognition

● Gesture recognition

● Special pen

● Special button

● Colored objects

● Barcode recognition

● Transparent layers

14

Alternatives considered

● Voice recognition

● Object recognition

● Gesture recognition

● Special pen

● Special button

● Colored objects

● Barcode recognition

● Transparent layers

● most approaches = unfeasible

○ noisy / confusing environment

○ need of not-yet-existing hardware

○ time constraints

14

Color Detection

A

BarcodeRecognition

B15

A. Color Detection

● no specific hardware

● dynamic & “playful” interaction

16

A. Color Detection



Concept: different color = different component

16

A. Color Detection




→ colored markers + semi-transparent BBPapier

→ users can see through without “ruining” sketches

16

A. Color Detection




→ colored markers + semi-transparent BBPapier

→ users can see through without “ruining” sketches

→ JavaCV & OpenCV

16

→ Procedure

● control card = “toolbox” → paint with markers● same color → fill up the chosen component

17

→ Procedure


● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values

→ avoid mistakes & colors overlapping

17

→ Procedure


● digitize the sketch → creating threshold○ average color in toolbox → range (+30/-30)○ exterior pixels discarded → central values

→ avoid mistakes & colors overlapping

● threshold = calculate component’s RGB value● any pixel within → associated to specific UI component

17

→ Approximation

Once pixels & colors detected

● algorithm recognizes contours of shape

● approximates it to a rectangle

● component created & projected

18

→ Approximation

Once pixels & colors detected

● algorithm recognizes contours of shape

● approximates it to a rectangle

● component created & projected

Need for filling component

● algorithm not efficient only for outline

● rough drawings typical of sketching

18

B. Barcode Recognition

Barcode marker = tool for specific operations

19



● take pictures of device

● connect screens

● recognize & delete components

● copy screens

19



● take pictures of device

● connect screens

● recognize & delete components

● copy screens

Algorithm continuously running

→ checks if new barcodes are detected

19

→ Tools

20

Further improvements

Sidebar → alternative approach for some functionalities

21


Sidebar → alternative approach for some functionalities● rectangular transparent area + at top of table● if barcode placed inside → operation performed

21



Clock (progress indicator) → added onto center-top area

21



Clock (progress indicator) → added onto center-top area● immediate visual feedback to users● if appears → users realize they are performing an action

21



Clock (progress indicator) → added onto center-top area● immediate visual feedback to users● if appears → users realize they are performing an action

● 3 seconds = time-frame users might change their mind→ effective to avoid accidental mistakes

21

Evaluation

22

Modality A: Sidebar

● digitize a screen → taking a picture

○ by placing single barcode inside

23

Modality A: Sidebar

● digitize a screen → taking a picture

○ by placing single barcode inside

● duplicate a screen

○ by placing two barcodes inside

○ one is already digitized (source)

○ one is empty (destination)

23

Modality A: Colors

● toolbox projected

→ boxes for button, image, textbox…

24

Modality A: Colors



● BBPapier over toolbox → paint with color

● same color → BBPapier over component

24

Modality A: Colors



● BBPapier over toolbox → paint with color

● same color → BBPapier over component

● digitize screen with toolbox visible on table

→ colored square displayed to represent component

24

Modality A: Colors

To connect two screens

● same + use FROM → TO box

To delete component → digitize again

● physically remove colored BBPapier

● or cover it with white paper

To remove connection

● same with button-connector

25

Modality B

● camera tool to digitize

● copy tool to duplicate

● handles to detect component

● arrow tool to connect screens

● rubber tool to delete○ component○ connection○ screen (only way)

26

Dependent Variables

● measure overall success of new interaction techniques

○ if quick & easy-to-use without distracting user

○ which of the two modalities = most effective

27

Dependent Variables




● prove if efficient & offers satisfactory user-experience

27

Dependent Variables




● prove if efficient & offers satisfactory user-experience

4 parameters:

→ Quickness → Ease-of-use → Distraction → User-Experience

27

Quickness & Ease-of-Use

Quickness

● time for task completion

● evaluation is videotaped

28


Quickness



● Effective Time

○ duration of task

→ without system errors

→ avoids malfunctionings

28


Quickness



● Effective Time




Ease-of-use

● % successfully completed tasks

● ratio calculated by checking interactions executed perfectly

28


Quickness



● Effective Time




Ease-of-use

● % successfully completed tasks

● ratio calculated by checking interactions executed perfectly

○ users make no mistakes

○ no significant problems

○ not need any help / hint

28

Distraction

Two factors: quickness & workload index (RTLX)

● unweighted (Raw) version of NASA Task Load Index (TLX)

● 6 subscales:

○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration

29

Distraction

Two factors: quickness & workload index (RTLX)

● unweighted (Raw) version of NASA Task Load Index (TLX)

● 6 subscales:

○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration

● own survey → filled in by users after every task

● 7-points linear scales → then averaged = RTLX

● the lower → the less demanding & distracting task is

29

User-Experience

● dimensions of AttrakDiff Survey → studies by Hassenzahl

○ “how users rate the usability and design of your interactive product”

30

User-Experience

● dimensions of AttrakDiff Survey → studies by Hassenzahl

○ “how users rate the usability and design of your interactive product”

● own custom version → filled in after finishing whole study

● 28 7-points semantic differential scales

○ opposite adjectives at both poles ("good - bad" / "human - technical")

○ implicitly divided into 4 dimensions (Pragmatic Quality, Hedonic Quality - Identity, Hedonic Quality - Stimulation, Attractiveness)

30

Experiment Design

● two different design solutions → A/B Testing

● “within-subject” design → every user tests both versions

○ no interactions in common → no influence

32

Experiment Design

● two different design solutions → A/B Testing

● “within-subject” design → every user tests both versions

○ no interactions in common → no influence

● to avoid possible bias = AB & BA groups

32

User Demographics

● one user at a time → focus on interactions

○ easier to observe & more difficult to get biased

33

User Demographics

● one user at a time → focus on interactions

○ easier to observe & more difficult to get biased

● 24 participants → 11 females & 13 males

● half aged 18-24, 10 aged 25-34, two older

● 7 background in HCI, 8 IT, rest other expertise

● via Facebook groups & personal connections

● reward = tasty gift or house utensils

33

Analysis of the Results

34

Results

● reviewed videos from GoPro

● investigate how long took users to complete tasks

● detect when they had troubles & needed help

35

Results

● reviewed videos from GoPro

● investigate how long took users to complete tasks

● detect when they had troubles & needed help

● SPSS → ONE-WAY ANOVA test

○ ideal for type of Dependent Variables (continuous)

○ valid for any group of users of study

○ robust to violations in underlying assumptions

35

Quickness

● Modality B faster than Modality A

○ time for A = 40.1s, B only 23.25s

○ statistically significant (p < .05)

36

Quickness




● Effective Time (without system errors)



36

Quickness




● Effective Time (without system errors)



→ color detection = long process

36

Ease-of-use

● Modality A easier to use than B

37

Ease-of-use

● Modality A easier to use than B

● perfectly accomplished tasks:

○ ratio: A = 84%, B = 60%


37

Distraction

● A almost as demanding as B

○ A RTLX index = 1.90

○ B RTLX index = 1.84

38

Distraction

● A almost as demanding as B

○ A RTLX index = 1.90

○ B RTLX index = 1.84

● On a 7-points scale →

not statistically significant

(F1,286 = 0.289, p = 0.591)

38

User-Experience

● users feel “assisted by the product”

○ usability can be improved

39

User-Experience



● users are “stimulated by the product”

○ users can identify with it

○ room for hedonic improvement

39

User-Experience



● users are “stimulated by the product”

○ users can identify with it

○ room for hedonic improvement

● system considered as “rather desired”

○ attractiveness above average

39

Single Tasks Analysis

40

Task 1 Analysis

● sidebar (A) better

○ faster

○ easier to use

○ less demanding

41

Task 1 Analysis

● sidebar (A) better

○ faster

○ easier to use

○ less demanding

● camera tool slower

○ no clue on how /

where to place it

41

Task 2 Analysis

● sidebar (A)

○ easier to use

42

Task 2 Analysis

● sidebar (A)

○ easier to use

○ faster (only E.T.)

○ causes several

system errors →

see later

42

Task 2 Analysis

● sidebar (A)

○ easier to use


○ causes several

system errors →

see later

● RTLX == copy tool

42

Task 3 Analysis

● handles (B)

○ faster

○ less demanding

○ easier to use

43

Task 3 Analysis

● handles (B)

○ faster

○ less demanding

○ easier to use

● color detection

○ 3 tasks in one

○ digitize + create

43

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

44

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

○ where?

44

Task 4 Analysis

● arrow (B)

○ faster

○ less demanding

○ where?

● color detection

○ easier to use

○ create + connect

44

Task 5 Analysis

● rubber (B)


45

Task 5 Analysis

● rubber (B)


○ bug found!

45

Task 5 Analysis

● rubber (B)


○ bug found!

● color detection (A)

○ easier to use

○ less demanding

○ more accurate

45

Task 6 Analysis

● rubber (B)

○ faster

○ less demanding

46

Task 6 Analysis

● rubber (B)

○ faster

○ less demanding


○ easier to use

○ experience?

46

Interesting facts

● people with background in HCI

→ performed more easily & faster

● good sketching abilities

→ same + found system more attractive

47

Interesting facts





● people familiar with mobile devices

→ faster + system less distracting

47

Interesting facts





● people familiar with mobile devices

→ faster + system less distracting

● during day-time → better performance & lower RTLX

47

Considerations

● accuracy = ratio of task’s precision

○ global accuracy = 86%

○ color detection = 67%

→ more delicate + error-prone

48

Considerations





● barcodes + hands into digitized image → distraction + inexperience

48

Considerations





● barcodes + hands into digitized image → distraction + inexperience

● if BBPapier wavy → grey area projected

○ projector’s brightness → shadow (“ghost shape”)

48

Projector’s beam

● when toolbox at sides of projection → outer areas appear darker

○ either no color is detected

○ “fake” grey color recognized → multiple components created

49

Projector’s beam




● most system errors

○ excessively intense light causing reflections on table

49

Projector’s beam





○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing

49

Projector’s beam





○ excessively intense light causing reflections on table→ closest barcodes could not be detected→ devices continuously refreshing

○ “effective time” → task completed if interaction is correct

49

Conclusions

50

Which better?

51

Which better?


○ easier to use

○ takes longer time

51

Which better?


○ easier to use


● barcode recognition (B)

○ definitely faster

○ harder to understand

51

Which better?


○ easier to use


● barcode recognition (B)

○ definitely faster

○ harder to understand

● both have similar workload

○ around 1.9 out of 7

○ positively low →

not too distracting51

Ideal implementation

1-2. sidebar → digitize & duplicate screens

52



2. copy tool too → duplicate without moving papers

52




3-4. both approaches to detect components & create connections

a. color detection = from scratch + multiple tasks at same time

52






b. handles & arrow = fast & easy-to-use

52






b. handles & arrow = fast & easy-to-use

5-6-7. rubber tool to remove components, connections & screens

→ implement separate ones to avoid possible mistakes

52

Future Work

● visual feedback when projections disappear from table

53

Future Work


● icon near clock → tells which action is being performed

53

Future Work



● exploring color detection further

○ better algorithms to calculate the threshold

○ better ways to detect shapes through contours

53

Future Work






● barcodes = built with wood / opaque plastic → reduce reflections

53

Future Work







● compare interaction techniques with original system

53

Future Work







● compare interaction techniques with original system

● test system in real-life scenario → inside company or startup53

Thanks!Any questions?

images from pixabay.com & papers by Bähr

enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile...

Technology