chuck norris had to destroy the periodic table… he only recognizes the element of surprise!
TRANSCRIPT
Kinect Interactions: Get a Grip in C# or HTML5/JavascriptBen LowerProgram Manager – Kinect for Windows3-705
Periodic TableInteractionStreamSneak Peek of 1.8 SDKNew Kinect?Q&A
Agenda
Periodic Table
CHUCK NORRISHAD TO DESTROYTHE PERIODIC TABLE…
HE ONLY RECOGNIZES THEELEMENT OF SURPRISE!
AAudio
Waveform
Stream Symbol
Stream Name
Data Type
The Periodic Table of Kinect for Windows
DDepth
DepthImagePixel[]
AAudio
Waveform
IRInfrared
byte[]
CColor
byte[]
ACAcceleromete
rVector4
Data Streams (Eyes & Ears)
IInteractionsInteractionFrame
(UserInfo[])
KFKinect Fusion
Point Cloud/Mesh
FTFace Tracking
FaceTrackFrame
SSpeech
Recognition Events
STSkeleton TrackingSkeleton[]
Recognition Streams (Brains)
Demo: Controls Basics +Interaction Gallery
InteractionStream
Why InteractionStream?
Moving to world that is multi-modal
Have experiences feel natural
Consistency of interactions
Interaction Stream
Translates hand position to screen coordinates• Physical Interaction Zone (PhIZ) is shaped to be ergonomic for an
individual independent of position relative to the sensor
Track and recognize press action• Press action based on arm extension• Press distance dependent on starting arm position• Press adjusted to improve accuracy over buttons
Recognize hand grip and release gesture• Computer vision and machine learning algorithm recognize grip gesture• Grip recognized for a wide variety of hand poses and orientations
The PhIZ
Physical Interaction Zone (PhIZ)The area in front of the user where there is a spatial mapping relationship between the user’s hand movements in physical space and the cursor movement on the screen.
Features• Ergonomics and Human Movement• Scaling to fit user• Separate PhIZ for each hand• Consistent experience
Targeting & Selection
Cursor Default (L)
Cursor Default (R)
New WayTarget button with hand cursorPress toward button, then releaseCenter “pulls hand in” during press
Old WayTarget button with hand cursorHover, hold, wait (for action)
Press Cursors
Roll Over Press Half Full Press Press Complete
Research showed people like visual cues &
feedback
Hand “fills up” to indicate press progress
Scrolling & Direct Manipulation
Cursor Default
Grip Over
New WayTarget anywhere w/hand cursorGrip to manipulate
Old WayTarget scroll button w/hand cursorHold and wait for scroll
InteractionStream APIs
Native COM based InteractionStream API• Core implementation• Usable with any UI frameworks that can call C functions and COM
objects
Managed .NET Wrapper• Wraps the InteractionStream API for .NET languages, e.g. C#, VB etc.
WPF Adapter – KinectRegion, Cursor and Controls
• Implements IInteractionClient for hit-testing WPF controls• Translates Interaction Frame to WPF events• Implements Cursor to provide feedback for: handedness, press progress
and grip state
Interactions Architecture
NUI_INTERACTION STREAM
KinectInteraction170_*.dll (32 or 64)
DirectXKinect
Support InteractionStream
Microsoft.Kinect.Toolkit.Interaction.dll
.NET/WPF
Kinect-Enabled App
WPF Kinect Controls
Microsoft.Kinect.Toolkit.Controls.dll
XNA Kinect
SupportC++/Win32 Kinect
Controls
Windows Forms Kinect
Controls
C++/DirectX Kinect-Enabled
App
C++/Win32 Kinect-Enabled
App
XNAKinect-Enabled
App
Windows FormsKinect-Enabled
App
Working with InteractionStream
Application
InteractionStream
UI Framework Adapter:IInteractionClient
ProcessSkeleton(skeleton, accelerometer, timestamp)
ProcessDepth(depthData, timestamp)
GetInteractionInfoAtLocation(id, handType, x, y)
new (sensor, interactionClient)
return InteractionInfo{ isGripTarget, isPressTraget,PressAttractionPointX, PressAttractionPointY,PressControlId}
InteractionFrameReady
Integration Considerations
Primary User and Primary Hand• InteractionStream tracks 2 users and 4 hands• Primary User and Primary Hand implemented by KinectRegion• UX Guidance: Avoid 2 handed actions – difficult to learn & master
Cursor – press and grip visualization• Continuous press progress feedback critical to user experience• Press initiates action on release – similar to button press with mouse
Pressable button affordance• Button provides visual feedback when cursor is over pressable button
Sneak Peek: v1.8 SDK
Demo: Background Removal
Demo: Photobombr
Interactions Architecture (1.8)
NUI_INTERACTION STREAM
KinectInteraction170_*.dll (32 or 64)
DirectXKinect
Support InteractionStream
Microsoft.Kinect.Toolkit.Interaction.dll
HTML5/Javascript
Kinect-Enabled AppHTML5/JS Controls
Microsoft.Samples.
Kinect.Webserver.dll
XNA Kinect
SupportC++/Win32 Kinect
Controls
Windows Forms Kinect
Controls
C++/DirectX Kinect-Enabled
App
C++/Win32 Kinect-Enabled
App
XNAKinect-Enabled
App
Windows FormsKinect-Enabled
App
.NET/WPF
Kinect-Enabled App
WPF Kinect Controls
Microsoft.Kinect.Toolkit.Controls.dl
l
Announcing: v1.8 SDK (September 2013)Background Removal API (green screening w/out the screen)
HTML5/Javascript support for Interactions
Kinect Fusion Color
Plus more…stay tuned
What’s up with the new Kinect?
New Kinect for Windows Sensor
• Wider field of view• HD resolution• Track six skeletons• Better ST, audio, infrared• Coming next year
Announcing: Developer Kit Program
Direct access to our engineering teamExclusive access to pre-release SDKsPrivate access to all API & sample docsPre-release/alpha sensorNew RTM sensor next year
$399 (coming November 2013)
Apply now: http://
aka.ms/k4wdevkitapplication
Who should apply?
Already working today to build & deploy NUI
experiences
You are interested in or experimenting with NUI
You want to help push the boundaries of NUI
Experienced developer who wants to jump in
You have an awesome idea for a touch-free app
Thank you!
Get 1.7 SDK now
Follow our blog to know when 1.8 SDK is released
Apply today for new dev kit program
Kinect for Windows @ Build 2013
All Week• Kinect for Windows Booth (demos, 3D printing with Fusion)
Wednesday, June 26th
• 2:00 PM Real-world Machine Learning: How Kinect Gesture Recognition Works, 3-704Alisson Sol, Bing (formerly Kinect for Windows), North Hall: 123
Thursday, June 27th
• 2:00 PM Kinect Interactions: Get a Grip in C# or HTML5/Javascript, 3-705Ben Lower, Kinect for Windows, North Hall: 131
• 5:00 PM Scanning & Printing the Real-World using Kinect for Windows and Preview of vNext, 3-702
Chris White, Kinect for Windows, North Hall: 131
Kinect for Windows Resources
Developer Kit Program Applicationhttp://aka.ms/k4wdevkitapplication
Developer Center (SDKs, Samples, Videos)http://aka.ms/k4wdev
Product Infohttp://aka.ms/k4w
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.