simple expressions at play

SimpleExpressions@Play

ShashankTiwariwww.treasuryofideas.com|www.shanky.org

shashank.>[email protected]|[email protected]|twi?er:tshanky

What’scoming?

•  Gestureseverywhere!•  GesturesinFlashtoday•  What’sbrewing?

•  WheredoIgofromhere?

Whatexactlyisagesture?

•  Touch,mo>on,voice,thought?•  Sta>c–performandhold– Example–thumbsup!,touch

•  Dynamic–movementover>me– Example–wave,flick/pinch

•  Welldefineddeviceconstrained|freeform

GesturesEverywhere!

JeffHan

1Of15

GesturesEverywhere!

Wii

2Of15

GesturesEverywhere!

TheClapper

3Of15

GesturesEverywhere!

GoogleNexusOne

4Of15

GesturesEverywhere!

ATM

5Of15

GesturesEverywhere!

AirportCheck‐inkiosk

6Of15

GesturesEverywhere!

GlowingPlaces

7Of15

GesturesEverywhere!

MicrosoYSurface

8Of15

GesturesEverywhere!

Fukuda’sautoma>cdoor

9Of15

GesturesEverywhere!

enMo>on’sPapertoweldispenser

10Of15

GesturesEverywhere!

DysonAirBlade

11Of15

GesturesEverywhere!

DanceDanceRevolu>on

12Of15

GesturesEverywhere!

Cubit

13Of15

GesturesEverywhere!

Intel’stouchscreenwall

14Of15

GesturesEverywhere!

PhilipsImageocandlelight

15Of15

GesturesinFlash

•  It’snotnewontheFlashpla\orm!•  Sohowdowegetgesturesworking–  Ifyouarenotblindtoinputs,youcanpossibleactonit.

•  OldStyle–FP‐>Socket‐>deviceinputs•  NewStyle–FP10.1‐>driverbakedintotheplayer

1Of45

GesturesinFlash

•  OldStyle– Touchlib‐‐h?p://code.google.com/p/touchlib/

– Gestureworks‐‐h?p://gestureworks.com/–  Ideo‐‐h?p://code.google.com/p/ideo‐mul>touch/

– Touche‐‐h?p://code.google.com/p/touche/–  (op>calmul>‐touch)

2Of45

GesturesinFlash

•  NewStyle– Build‐insupportinFlashPlayer10.1andAIR2•  Formul>‐touchgestures•  Promisingtobecrosspla\orminfuture

–  (providedthatispossible!)

3Of45

GesturesinFlash

•  Beforewelearnaboutthemul>‐touchsupport,letsseewhat’sinfluencingit…

4Of45

GesturesinFlash

•  Typesoftouchdevices:•  Resis>ve‐‐(example:Kiosks)– 2thinelectricallyconduc>velayersseparatedbyagap

– Ontouchpanelsgetconnected– Panelsactasvoltagedividerpairwithconnectedoutputs

– Changeinelectriccurrentregisteredasatouchevent

5Of45

GesturesinFlash

•  Capaci>ve––  InsulatorlikeglasscoveredwithaconductorlikeIndiumTinOxide(ITO)

– Humanbodyisagoodcondutor

– Ontouchelectrosta>cfieldisdistorted,whichisrecordedasachangeincapacitance(abilityofabodytoholdcharge)

6Of45

GesturesinFlash

•  Op>calImaging–– 2or>imagesensorsplacedaroundedges

– Ontheotherside,inthecamera’sfieldofviewinfraredlightisplaced

–  touchtranslatestoshadow– Pairofcamerastriangulatedtolocateatouch

7Of45

GesturesinFlash

•  Surfacewaveacous>cs–– Ultrasonicwavespassoverthesurface– Touchabsorbsultrasonicwaves

•  Dispersivesignal– Sensorstouchmechanicalenergyintheglassduetotouch

•  Straingauge– Springmountedon4corners– Gaugesdeterminedeflec>onontouch

8Of45

GesturesinFlash

•  Numberoftouchpoints– 2seemstoomany!

9Of45

GesturesinFlash

•  Mul>‐touchdevices–whathavewegot?–  3M—M2256PWwithten‐fingersupport–  Acer—AcerAspire1820PT&5738PG–  Apple—iPhone,iPad,iPodTouch,MacBook,MacBookAir,MacBookPro,MagicMouse

–  Asus—EEEPCT91MT&T101MT–  CircleTwelve—DiamondTouch–  Dell—La>tudeXT&XT2,Mini5,Studio17–  Google—NexusOne–  Hewle?‐Packard—HPTouchsmart–  HTC—HTCHero,HTCHD2–  Ideum—MT‐50Mul>touchTable

10Of45

GesturesinFlash–  Lenovo—X200&T400–  LGElectronics—Arena,BL40NewChocolate–  MicrosoY—Surface,ZuneHD–  Motorola—Droid–  MULTIVISION—Mul>‐TouchLCD—upto32fingers–  Nortd—TouchKit–  Palm—Pre,Pixi–  Percep>vePixel—Mul>‐TouchCollabora>onWall–  Shu?leInc.—Mul>‐touchLCDX50v2–  Sony—VAIOLSeriesAll‐in‐onedesktops–  TycoElectronics/EloTouchSystems‐IntelliTouchPlusSurface

Acous>cWaveMul>‐touchTouchTechnology–  Wacom—Bambootablets

11Of45

GesturesinFlash

•  ForstartershowdoIfindifFP10.1|AIR2knowsofthetouchevents?

12Of45

GesturesinFlash

Mul>touch.supportsGestureEvents– Doesit?

Mul>touch.supportsTouchEvents– Doesit?

13Of45

GesturesinFlash

•  Isitsingletouchorgestures?

– SetthecorrectinputMode

– Whichissetflash.ui.Mul>touch.inputMode

14Of45

GesturesinFlash

•  flash.ui.Mul>touchInputMode– GESTURE–gesturesrecognized,simpletouchevent=mouseevent

– NONE–everythingistreatedasamouseevent

– TOUCH_POINT–onlythebasictouchevents,nogestures

15Of45

GesturesinFlash

•  Howmanytouchpointsaresupported?

– Mul>touch.maxTouchPoints 16Of45

GesturesinFlash

•  ToucheventorGestureevent?•  flash.events.TouchEvent

•  flash.events.GestureEvent•  flash.events.TransformGestureEvent

•  flash.events.PressAndTapGestureEvent

17Of45

GesturesinFlash

•  TouchEvent– TOUCH_BEGIN– TOUCH_END– TOUCH_MOVE– TOUCH_OUT

18Of45

GesturesinFlash

•  TouchEvent(…cont)– TOUCH_OVER– TOUCH_ROLL_OUT– TOUCH_ROLL_OVER– TOUCH_TAP

19Of45

GesturesinFlash

•  Proper>estheTouchEventobjecthas––  touchPointID–uniqueIDa?achedtothetouchpoint

–  localX–xcoordinaterela>vetothesprite–  localY–ycordinaterela>vetothesprite– sizeX–widthofthecontactarea– sizeY–heightofthecontactarea

20Of45

GesturesinFlash

•  Proper>estheTouchEventobjecthas(…cont)–– controlKey– commandKey– altKey– shiYKey

21Of45

GesturesinFlash

•  Rawtoucheventscanbehandledtounderstandgestures,buttherearesomealreadythere. 22

Of45

GesturesinFlash

•  TransformGestureEvent– GESTURE_PAN– GESTURE_ROTATE– GESTURE_SWIPE– GESTURE_ZOOM

23Of45

GesturesinFlash

•  GESTURE_PAN

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/lock_two_pan.png

•  (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐us,VS.85%29.png)

24Of45

GesturesinFlash

•  GESTURE_ROTATE

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_finger_rotate.png


25Of45

GesturesinFlash

•  Whataboutthis?

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_hand_rotate.png

26Of45

GesturesinFlash

•  GESTURE_SWIPE

•  h?p://regmedia.co.uk/2008/12/29/swipe_patent1.jpg•  (h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐us,VS.85%29.png)

27Of45

GesturesinFlash


•  Onefingerflick‐‐h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/one_finger_swipe.png

28Of45

GesturesinFlash

•  Andthis?

•  Twofingerflick‐‐h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_finger_swipe.png

29Of45

GesturesinFlash

•  GESTURE_ZOOM

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_finger_zoom_in.png


30Of45

GesturesinFlash


•  Mul>pointzoom‐‐h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_hand_zoom_in.png

31Of45

GesturesinFlash

•  Andthis?

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/pinch_zoom.png

32Of45

GesturesinFlash

•  GestureEvent– GESTURE_TWO_FINGER_TAP

33Of45

GesturesinFlash

•  GESTURE_TWO_FINGER_TAP

•  h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_finger_tap.png•  h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐us,VS.85%29.png

34Of45

GesturesinFlash


•  Twofingerdoubletap‐‐h?p://gestureworks.com/wp‐content/themes/gestureWorks/images/gesturePngs/two_finger_double_tap.png

35Of45

GesturesinFlash

•  PressAndTapGestureEvent– GESTURE_PRESS_AND_TAP

36Of45

GesturesinFlash

•  GESTURE_PRESS_AND_TAP

•  h?p://i.msdn.microsoY.com/Dd940543.gestures%28en‐us,VS.85%29.png

37Of45

GesturesinFlash

•  Letsseeasimpleexample(amediasorter)– FP10.1– AIR2 38

Of45

GesturesinFlash

•  Sowhatworks?•  Win7–allgood

•  Mac–AIR2|FP10.1

•  Linux–some>mesworks!

39Of45

GesturesinFlash

•  Anotherexample(asoYkeyboard)– OlderFP(earlierthantheonethatofficiallysupportstouchandgestures)

– FP10.1– AIR2

40Of45

GesturesinFlash

•  Yetanotherexample(anelementaryrota>ngglobe)

41Of45

GesturesinFlash

•  Whataboutcustomgestures?– Combina>on|Aggrega>on

– Extension 42Of45

GesturesinFlash

•  Aggrega>on•  GESTURE_ZOOM+TAP– <>.addEventListener(TransformGestureEvent.GESTURE_ZOOM,onZoom);

–  func>ononZoom(evt:TransformGestureEvent):void{if(evt.phase==GesturePhase.BEGIN){addeventlistenerfortap–whichistranslatedtoamouseclick}

43Of45

GesturesinFlash

•  Aggrega>on•  GESTURE_ZOOM+GESTURE_ROTATE

•  ? 44Of45

GesturesinFlash

•  Extension– Listentorawtoucheventsanduseyourbrilliance!

45Of45

What’sbrewing?

•  Extendgesturesupportbeyondthebasics– Gesturekit‐‐h?p://code.google.com/p/gesturekit/ 1

Of5

What’sbrewing?

•  Helpthosewhodon’thaveaccesstoamul>‐touchdevice– Startwithmul>touchup.com– Relyonyourmouse– Mactrackpadisn’tbad(withAIR2)

2Of5

What’sbrewing?

•  TouchdragsupportwithintheFPapi•  Supportonallpla\orms– Mobileseemsapriority 3

Of5

What’sbrewing?

•  Newerdevices–smartphones,slatesandnetbooks

•  Newertouchaccesspoints–kiosks,surfacesandwalls

4Of5

What’sbrewing?

•  Integra>onwithgesturesbeyondtouch– Mo>ondetec>on

– Soundrecogni>on– Freeformexpressions– Simultaneoustrackingofeachofthese

5Of5

WheredoIgofromhere?

•  GetFP10.1&AIR2•  Getholdofatouchscreen|device•  Readthedocumenta>on– h?p://help.adobe.com/en_US/as3/dev/WSb2ba3b1aad8a27b0‐6�37601221e58cc29‐8000.html

– h?p://help.adobe.com/en_US/FlashPla\orm/beta/reference/ac>onscript/3/flash/events/TouchEvent.html

1Of4

WheredoIgofromhere?

•  Writeanapportwo•  GiveupyoursleeptrackingtheTouchEvent!

2Of4

WheredoIgofromhere?

•  Explore– Unistrokerecognizer‐‐h?p://depts.washington.edu/aimgroup/proj/dollar/

– h?p://www.betriebsraum.de/projects/gestures/

•  Gestureworks–Opensourcegesturelibrary– h?p://gestureworks.com/about/open‐source‐gesture‐library/

3Of4

WheredoIgofromhere?

•  Gesturerecogni>on‐‐h?p://github.com/sqrtof5/GestureRecogni>on

•  Touchlibsource–– h?p://code.google.com/p/touchlib/source/browse/trunk/AS3/int/app/core/ac>on/gestures/GestureNormalizer.as

4Of4

Thispresenta>on&demos?

•  h?p://shanky.org/publish‐and‐present/flex•  h?p://www.treasuryofideas.com/gestures/touch/MediaSorter/MediaSorter.html

–  Iwilltweettheexactlinksandwhentheupdatedversionsareready–twi?er:tshanky

–  Imagesforslidesweredownloadedfromtheweb.Thecopyrightsiswiththeirrespec>veowners

–  Referencestomaterialusedbetweenslides4and18isavailableath?p://www.shanky.org/references/simple_expressions_at_play_references.pdf.

Thanksforcoming!

ShashankTiwariwww.treasuryofideas.com|www.shanky.org

[email protected]|[email protected]|

twi?er:tshanky