an introduction to nui

An Introduction to NUI

NUI programming seminar

Jun Iio, Ph.D.Chuo University

An introduction to NUI Copyright © Jun Iio 2

Self-introduction● Prof. Jun Iio, Ph.D.

Dept. of Socio-Informatics, Faculty ofLetters, Chuo University

● 1994-2013, Working for Mitsubishi ResearchInstitute

● 2013-2014, Assoc. Prof. of Chuo University● 2014-Current, Prof. of Chuo University● Also, 2009-, Guest Assoc. Prof. of Tokyo University

of Agriculture and Technology● My interests are

– Software Engineering, Image Processing,Open-Source Software, User Interface, etc.

An introduction to NUI Copyright © Jun Iio 3

What is NUI?● NUI stands for Natural User Interface

● Improvement of the User Interface?– CUI -> GUI -> NUI？

● Let's look back on the history of UI...

An introduction to NUI Copyright © Jun Iio 4

The first step of data input methodsfor computers

source: Wikimedia

An introduction to NUI Copyright © Jun Iio 5

Emerging personal computers

source: Wikimedia (CC BY-SA 2.0 fr)

Commodore PET 2001 computer. On display at the Musée Bolo, EPFL, Lausanne. The making of this document was supported by Wikimedia CH.

An introduction to NUI Copyright © Jun Iio 6

CUI（Character User Interface）

An introduction to NUI Copyright © Jun Iio 7

GUI（Graphical User Interface）

An introduction to NUI Copyright © Jun Iio 8

WIMP

Ｗ … WindowsＩ … IconsＭ … MenusＰ … Pointers

An introduction to NUI Copyright © Jun Iio 9

Physical peripherals● Keyboard● Pointing devices

– Mouse– Track-point– Track-ball– Touch pad– Touch panel

An introduction to NUI Copyright © Jun Iio 10

Novel operations● Tap

– Double tap– Tap and hold

● Flick, drag● Pinch

– Pinch in– Pinch out

● Shake, bump(!)– (Grab) source: http://bump.softonic.jp/iphone

An introduction to NUI Copyright © Jun Iio 11

Elemental technologies● Contact-type

– Touch interface● Multi touch, touch display, pressure-sensitive input● (Image input, acceleration sensor)

● Contactless-type– Space devices

● Image input, three dimensional measurement– Gesture interface

● Human body modeling, inverse kinematics● Acceleration sensor (motion, direction, inclination)● Muscle potential measurement

An introduction to NUI Copyright © Jun Iio 12

An impact of Windows 8

source: http://fesoku.net/archives/6029501.html

An introduction to NUI Copyright © Jun Iio 13

Multi-touch● http://kivy.org/

An introduction to NUI Copyright © Jun Iio 14

TUIO● Tangible User Interface Objects

An introduction to NUI Copyright © Jun Iio 15

CCV（Community Core Vision）● Poorman's multi-touch device? Video

An introduction to NUI Copyright © Jun Iio 16

3D mouse

An introduction to NUI Copyright © Jun Iio 17

Kinect

An introduction to NUI Copyright © Jun Iio 18

OpenNI

An introduction to NUI Copyright © Jun Iio 19

Leap MotionVideo

An introduction to NUI Copyright © Jun Iio 20

MyoVideo

An introduction to NUI Copyright © Jun Iio 21

RingVideo

An introduction to NUI Copyright © Jun Iio 22

Gesture interface● Maximum benefit

– “Direct manipulation”– It enables to control the target directly

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 23

Note for the NUI designs● Natural User Interface development is still

in progress● Considerable features:

1. How to realize input methods which can notmake users tired?

2. “Mode” problem of gesture input3. Haptic information (feedbacks of user's

operation)● Learning from SciFi movies

– What are applications using NUI efficiently?

An introduction to NUI Copyright © Jun Iio 24

References“Make It So” (Japanese Translation)

An introduction to NUI Copyright © Jun Iio 25

“Minority Report”● A “prototype meter” for the gesture

interface● Lessons learned:

– A grate demo hide many flaws– A gesture interface should understand intent

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 26

Enthusiasts for keyboards

source: http://www.pfu.fujitsu.com/hhkeyboard/hhkbpro/black.htmlsource: http://www.kmecs.com/products/detail_267.html

An introduction to NUI Copyright © Jun Iio 27

Haptic devices● An input was completed? or not?● Why does this keyboard make user feel

unpleasant?

source: http://www.itmedia.co.jp/pcuser/articles/1204/06/news033.html

An introduction to NUI Copyright © Jun Iio 28

Haptic devices● It enables the

feedbacks of user'sinput by touchinformation such asvibration, resistantforce, and/ormotion.

● Cf. “Myo uses apattern of vibrationas the notice foruser's input”

source: http://lab.sekaimon.com/index/index/c/58058/ym/201201/p/2/ajax/1

An introduction to NUI Copyright © Jun Iio 29

Grammar for the gesture input

source: Leap Motion SDK API document

An introduction to NUI Copyright © Jun Iio 30

Grammar for the gesture input

source: https://support.getmyo.com/hc/en-us/articles/201135025-Controlling-media-with-your-Myo-armband

An introduction to NUI Copyright © Jun Iio 31

Hollywood's pidgin● Wave to activate● Push to move● Turn to rotate● Swipe to dismiss● Point or touch to select● Extend the hand to shoot● Pinch and spread to scale

An introduction to NUI Copyright © Jun Iio 32

Wave to activate

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 33

Push to move

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 34

Turn to rotate

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 35

Swipe to dismiss

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 36

Point or touch to select

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 37

Extend the hand to shoot

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 38

Pinch and spread to scale

source: ”Make It So”

An introduction to NUI Copyright © Jun Iio 39

Be careful if you propose a novelgesture out of the seven patterns

● Another patter canlead usersmisunderstandingor confusing

● Considering globalrules and local rule– Gestures in

Japanese culture– Gestures in another

country's culture

source: Fuji television

An introduction to NUI Copyright © Jun Iio 40

Dependency of local cultures

source: http://matome.naver.jp/odai/2133536386030621501

An introduction to NUI Copyright © Jun Iio 41

Dependency of local cultures

source: http://matome.naver.jp/odai/2133536386030621501

An introduction to NUI Copyright © Jun Iio 42

An application of Kinect● Y. Nakano, T.

Tohyama, and S.Kanemune,“Comparison ofJapanese InputMethods for Kinect” Inproc. of IPSJ-SSS2013, 173-175.– https://www.faceboo

k.com/media/set/?set=a.183589891823257.1073741846.172555706260009

An introduction to NUI Copyright © Jun Iio 43

Applications using NUI● It should be considered what scene

requires NUI fundamentally?

● In the case of promotion videos for Myo:– Controlling drone, sphero, and UMV...

“It would be enough to use a controller!”

● It is suitable for the situations where auser cannot use his/her own hands– Controlling recipe video when cooking– tele-surgery

An introduction to NUI Copyright © Jun Iio 44

Future of interfaces● NUI does not replace {CUI, GUI}

– Programming using NUI? -> a distant idea

Practical

Abstract

Professionaluse

Generalpurpose

CUI

GUI

NUI

An introduction to NUI Copyright © Jun Iio 45

Conclusions● Progress of interacting user and computer:

CUI -> GUI -> NUI– NUI study is the blue ocean and in progress

● Elemental technologies for NUI– multi-touch, 3D measurement, acceleration

sensors, muscle potential measurement● How to solve some problems in gesture UI

– Tiredness, mode-problem, haptic interface● How to make a grammar for gesture UI

– Not only hardware but also cultural constraintsshould be considered!