gesture interaction with children for education, fitness, games

Lisa Anthony, PhD

[email protected]

Pictures from flickr.com by Creative Commons License

X equals 4 liters…

Turbo boost!

@drlanthony

Research Approach What are “natural user interactions”?

Why NUIs for Children

Touchscreen Interaction Projects

Future Research

Understanding, Designing, and Developing Natural User Interactions for Children / Lisa Anthony, PhD 3

@drlanthony




Future Research


@drlanthonyUnderstanding, Designing, and Developing Natural User Interactions for Children / Lisa Anthony, PhD 5

VIDEO LINK: https://www.youtube.com/watch?v=6ECVHsUVuJg


speech

intent

vision

touch

gesture

motion


natural interaction doesn’t interfere with the user’s task

@drlanthony

Understanding user needs and

capabilities

Developing new intelligent

algorithms


Design the algorithms informed by user behavior

Design the interfaces informed by tech strengths

@drlanthony

Understanding user needs and

capabilities

Developing new intelligent

algorithms


Design the algorithms informed by user behavior

Design the interfaces informed by tech strengths

PsychologySocial scienceErgonomics…

Computer scienceMachine learningStatistics…


speech

intent

vision

touch

gesture

motion

@drlanthony




Future Research



kids are hands-on and learn by doing


kids apply real-world metaphors to technology


VIDEO LINK: https://www.youtube.com/watch?v=aXV-yaFmQNk

@drlanthony

LETTERS SHAPES


A

E

Q

Δ

O

+drawn to scale

@drlanthony

LETTERS SHAPES


A

E

Q

Δ

O

+drawn to scale

Adult Child Adult Child


X equals 4 liters…

@drlanthony




Future Research


@drlanthony

1. Understanding how kids use touchscreens now

2. Understanding patterns in how kids make touch gestures

3. Understanding when and why current touch gesture recognition approaches don’t work

4. Designing and developing new algorithms for touch gestures

5. Designing and developing tools to help developers analyze input behavior patterns and choose the best gestures / recognizers for their specific context


@drlanthony

“Mobile Touch and Gesture Interaction for Children”

https://mtagic.wordpress.com/

Funding support: NSF HCC Small NSF CISE #IIS-1218395/IIS-1433228 and IIS-1218664

Collaborators: Quincy Brown (Bowie State University)

Students from UMBC, Bowie State, and UF


Gesture

Touch cloud

@drlanthony

4 studies with kids and adults (S1^, S2*, S3*, S4-ongoing) 49 kids (ages 7-17), 36 adults (ages 18+)


Study 2 setting

^ Brown, Q. and Anthony, L. 2012. Toward Comparing the Touchscreen Interaction Patterns of Kids and Adults. Proc. ACM SIGCHI Workshop on Educational Software, Interfaces and Technology (EIST’2012), Austin, TX, 05-06 May 2012.* Anthony, L., Brown, Q., Nias, J., Tate, B., and Mohan, S. 2012. Interaction and Recognition Challenges in Interpreting Children’s Touch and Gesture Input on Mobile Devices. Proc. ACM Conf. on Interactive Tabletops and Surfaces (ITS’2012), Cambridge, MA, 14 Nov 2012, p.225-234.


Touch TaskStudy 2

Gesture TaskStudy 2


Blue triangles: adultsRed squares: childrenBlack outline: target

Phone screen outline

@drlanthony

1. Children miss more targets than adults**

2. Smallest targets most challenging*


** significant at the p<0.01 level * significant at the p<0.05 level

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Very Small (0.125") Small (0.25") Medium (0.375") Large (0.5")

Av

era

ge

Pro

port

ion

of

Mis

ses

Target Size (inches)

Adults

Children

Average proportion of misses by target size for adults and children over all targets. Error bars show 95% confidence interval.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Very Small (0.125") Small (0.25") Medium (0.375") Large (0.5")

Av

era

ge

Pro

port

ion

of

Mis

ses

Target Size (inches)

7 to 10

11 to 13

14 to 17

Adults 18+

Study 2a (ITS 2012) Study 2b (JPUC 2014)


Adults

Children

Study 1 (EIST 2012)


No Visual Feedback(children)

Visual Feedback(children)

Study 2 (IDC 2013)

@drlanthony

1. Kids make gestures differently than adults S1: kids make bigger gestures**

S1: kids make gestures with more strokes**

2. Kids’ gestures are recognized^less accurately than adults’**

S2: correlated to age**


** significant at the p<0.01 level

^ Anthony, L. and Wobbrock, J.O. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. Proc. Graphics Interface (GI’2010), Ottawa, Canada, 02 Jun 2010, p.245-252.

0

1

2

3

4

5

6

<60% 60-65% 65-70% 70-75% 75-80% 80-85% 85-90% 90-95% 95-100%

Nu

mb

er

of P

art

icip

ants

Recognition Accuracy

Adults

Children

Histogram of recognition accuracy distribution for gestures drawn by children and by adults.

Study 2a (ITS 2012)

@drlanthony

Demonstrate design recommendations in an open-source app as a tool for developers

Kids make gestures and touch targets differently than do adults: we can designand develop for this

Next steps: Understanding younger kids’ touch and

gesture patterns

Understanding and designing contextualizedinteractions

Developing child-specific recognizers


Design guidelines app

@drlanthony

Kids’ gestures are recognized less accurately than adults’ by modern recognizers designed for adults Even when trained on kids gestures!


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

7 to 10 11 to 13 14 to 17 Adults 18+

Re

cog

nit

ion

Acc

ura

cy

Participant Age GroupsUser-Independent TabletPC User-Independent $P

User-Dependent $N-Protractor User-Dependent $P

Recognition performance by age group for each of the four recognizer configurations tested.

Error bars indicate the 95% confidence interval.

Study 2b (JPUC 2014)Best performer on kids aged 7 to 10 is still only at 84%

@drlanthony




Future Research


@drlanthony

1. Understanding how kids use whole-body interaction now

2. Understanding patterns in how kids move and make hand and body gestures

3. Understanding when and why current hand and body gesture recognition approaches don’t work

4. Designing and developing new algorithms for hand and body gestures


@drlanthony

Education

Fitness

Games


Multimodal Algebra Equation Solving(PhD thesis, Carnegie Mellon, 2008)

@drlanthony

Education

Fitness

Games


Image from PufferFish Ltd.

Investigating Public Engagement with Science(new project)

@drlanthony

Education

Fitness

Games


Kinect Games for Exercisehttp://kinectforexercise.wordpress.com

@drlanthony

Lisa Anthony, PhD [email protected] @drlanthony

http://lisa-anthony.com and http://init.cise.ufl.edu


Acknowledgments: National Science Foundation, Pittsburgh Science of Learning Center, University of Florida, University of Maryland Baltimore County

speech

intent

vision

touch

gesture

motion

gesture interaction with children for education, fitness, games

Education

natural user interactions

children lisa anthony

user behaviordesign

natural interaction

drlanthonyresearch approach

video link

litersturbo boost

creative commons licensex