Kinect Interactions: Get a Grip in C# or HTML5/JavascriptBen LowerProgram Manager – Kinect for Windows3-705

Periodic TableInteractionStreamSneak Peek of 1.8 SDKNew Kinect?Q&A

Agenda

Periodic Table

CHUCK NORRISHAD TO DESTROYTHE PERIODIC TABLE…

HE ONLY RECOGNIZES THEELEMENT OF SURPRISE!

AAudio

Waveform

Stream Symbol

Stream Name

Data Type

The Periodic Table of Kinect for Windows

DDepth

DepthImagePixel[]

AAudio

Waveform

IRInfrared

byte[]

CColorbyte[]

ACAcceleromete

rVector4

Data Streams (Eyes & Ears)

IInteractionsInteractionFrame

(UserInfo[])

KFKinect Fusion

Point Cloud/Mesh

FTFace Tracking

FaceTrackFrame

SSpeech

Recognition Events

STSkeleton TrackingSkeleton[]

Recognition Streams (Brains)

Demo: Controls Basics +Interaction Gallery

InteractionStream

Why InteractionStream?Moving to world that is multi-modal

Have experiences feel natural

Consistency of interactions

Interaction StreamTranslates hand position to screen coordinates• Physical Interaction Zone (PhIZ) is shaped to be ergonomic for an

individual independent of position relative to the sensorTrack and recognize press action• Press action based on arm extension• Press distance dependent on starting arm position• Press adjusted to improve accuracy over buttons

Recognize hand grip and release gesture• Computer vision and machine learning algorithm recognize grip gesture• Grip recognized for a wide variety of hand poses and orientations

The PhIZPhysical Interaction Zone (PhIZ)The area in front of the user where there is a spatial mapping relationship between the user’s hand movements in physical space and the cursor movement on the screen.

Features• Ergonomics and Human Movement• Scaling to fit user• Separate PhIZ for each hand• Consistent experience

Targeting & Selection

Cursor Default (L)

Cursor Default (R)

New WayTarget button with hand cursorPress toward button, then releaseCenter “pulls hand in” during press

Old WayTarget button with hand cursorHover, hold, wait (for action)

Press Cursors

Roll Over Press Half Full Press Press Complete

Research showed people like visual cues & feedbackHand “fills up” to indicate press progress

Scrolling & Direct Manipulation

Cursor Default

Grip Over

New WayTarget anywhere w/hand cursorGrip to manipulate

Old WayTarget scroll button w/hand cursorHold and wait for scroll

InteractionStream APIsNative COM based InteractionStream API• Core implementation• Usable with any UI frameworks that can call C functions and COM

objectsManaged .NET Wrapper• Wraps the InteractionStream API for .NET languages, e.g. C#, VB etc.

WPF Adapter – KinectRegion, Cursor and Controls

• Implements IInteractionClient for hit-testing WPF controls• Translates Interaction Frame to WPF events• Implements Cursor to provide feedback for: handedness, press progress

and grip state

Interactions Architecture

NUI_INTERACTION STREAM

KinectInteraction170_*.dll (32 or 64)

DirectXKinect

Support InteractionStream

Microsoft.Kinect.Toolkit.Interaction.dll

.NET/WPF

Kinect-Enabled App

WPF Kinect Controls

Microsoft.Kinect.Toolkit.Controls.dll

XNA Kinect

SupportC++/Win32 Kinect

Controls

Windows Forms Kinect

Controls

C++/DirectX Kinect-Enabled

App

C++/Win32 Kinect-Enabled

App

XNAKinect-Enabled

App

Windows FormsKinect-Enabled

App

Working with InteractionStream

Application

InteractionStream

UI Framework Adapter:IInteractionClient

ProcessSkeleton(skeleton, accelerometer, timestamp)

ProcessDepth(depthData, timestamp)

GetInteractionInfoAtLocation(id, handType, x, y)

new (sensor, interactionClient)

return InteractionInfo{ isGripTarget, isPressTraget,PressAttractionPointX, PressAttractionPointY,PressControlId}

InteractionFrameReady

Integration ConsiderationsPrimary User and Primary Hand• InteractionStream tracks 2 users and 4 hands• Primary User and Primary Hand implemented by KinectRegion• UX Guidance: Avoid 2 handed actions – difficult to learn & master

Cursor – press and grip visualization• Continuous press progress feedback critical to user experience• Press initiates action on release – similar to button press with mouse

Pressable button affordance• Button provides visual feedback when cursor is over pressable button

Sneak Peek: v1.8 SDK

Demo: Background Removal

Demo: Photobombr

Interactions Architecture (1.8)

NUI_INTERACTION STREAM

KinectInteraction170_*.dll (32 or 64)

DirectXKinect

Support InteractionStream

Microsoft.Kinect.Toolkit.Interaction.dll

HTML5/Javascript

Kinect-Enabled AppHTML5/JS Controls

Microsoft.Samples.

Kinect.Webserver.dll

XNA Kinect

SupportC++/Win32 Kinect

Controls

Windows Forms Kinect

Controls

C++/DirectX Kinect-Enabled

App

C++/Win32 Kinect-Enabled

App

XNAKinect-Enabled

App

Windows FormsKinect-Enabled

App

.NET/WPF

Kinect-Enabled AppWPF Kinect

Controls

Microsoft.Kinect.Toolkit.Controls.dl

l

Announcing: v1.8 SDK (September 2013)Background Removal API (green screening w/out the screen)

HTML5/Javascript support for Interactions

Kinect Fusion Color

Plus more…stay tuned

What’s up with the new Kinect?

New Kinect for Windows Sensor• Wider field of view• HD resolution• Track six skeletons• Better ST, audio, infrared• Coming next year

Announcing: Developer Kit ProgramDirect access to our engineering teamExclusive access to pre-release SDKsPrivate access to all API & sample docsPre-release/alpha sensorNew RTM sensor next year

$399 (coming November 2013)Apply now: http://

aka.ms/k4wdevkitapplication

Who should apply?

Already working today to build & deploy NUI experiencesYou are interested in or experimenting with NUIYou want to help push the boundaries of NUIExperienced developer who wants to jump inYou have an awesome idea for a touch-free app

Thank you!Get 1.7 SDK now

Follow our blog to know when 1.8 SDK is released

Apply today for new dev kit program

Kinect for Windows @ Build 2013All Week

• Kinect for Windows Booth (demos, 3D printing with Fusion)

Wednesday, June 26th• 2:00 PM Real-world Machine Learning: How Kinect Gesture Recognition Works, 3-704

Alisson Sol, Bing (formerly Kinect for Windows), North Hall: 123

Thursday, June 27th• 2:00 PM Kinect Interactions: Get a Grip in C# or HTML5/Javascript, 3-705

Ben Lower, Kinect for Windows, North Hall: 131

• 5:00 PM Scanning & Printing the Real-World using Kinect for Windows and Preview of vNext, 3-702

Chris White, Kinect for Windows, North Hall: 131

Kinect for Windows ResourcesDeveloper Kit Program Application

http://aka.ms/k4wdevkitapplication

Developer Center (SDKs, Samples, Videos)http://aka.ms/k4wdev

Product Infohttp://aka.ms/k4w

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.