Kinect basics: Systems for Smart Home Automation

Nilanjan Banerjee

AgendaOverviewCamera BasicsDepth DataSkeletal TrackingGestures & InteractionFace TrackingKinect Socket ServiceKinect FusionNext Version of Kinect

What’s new with Kinect 1.6 & 1.7Kinect 1.6Improved Depth Data APICoordinate MapperMore Sensor data – IR, Accelerometer, Camera Settings

Kinect 1.7Kinect FusionKinect Interactions – Interaction Stream, Click, Grip, ScrollInteraction Controls – KinectRegion, KinectCursor, KinectItems Control, KinectTileButton, KinectCircleButton, KinectScrollViewer, UserViewer

Getting Started

Kinect - Xbox 360 or Kinect for Windows Kinect for Windows SDK 1.7Kinect for Windows Developer Toolkit 1.7

Coding for fun Kinect WPF

Kinect API BasicsSetup Kinect1. KinectSensorChooser – KinectChangedEvent or KinectSensor.Connected2. Enable Color, Depth, Skeleton, Audio3. Start Kinect

Get Data4. Interaction Controls5. Events – AllFramesReady, InteractionFrameReady6. Polling – OpenNextFrame

Camera Basics

Image Processing 101BGR32 Format – Every Pixel (0,0 | 0,1 | 0,2) has blue, green, red, empty

B G R Empty255 255 255 0

Kinect ImageSizes: 80x60, 320x240, 640x480DPI: 96Stride: # of bytes per single line:

Width (320 or 640) x 4 bytes (B,G,R,E)

B G R Empty255 255 255 0

320x240

640x480

80x60

First order task: Retrieve a RGB image from the KinectWe will be using WPF1. Next generation windows presentation foundation2. Helps you build pretty user interface in Windows3. Much more flexible than a windows form framework

1. Allow dynamic and flexible layouts --- sizes of views like textbox etc. changes dynamically

2. Separates the design of the UI and the logic: design of the UI is in a xaml file and the logic is in C# similar to web programming

Important data structure: WriteableBitmap4. Stores the bitmap of an image but unlike the ordinary bitmap, WriteableBitmap is

mutable5. Read and write to bitmap

Camera BasicsDemoName

Depth Data

Depth Data

Distance and player index for every pixelDistance is in millimeters (2,000 mm = 6.56 feet)Player index 1-6 or 0 for no player

DepthDataDemoName

Skeletal Tracking

Skeletal Tracking

Max of two skeletons per Kinect6 Player Proposals

HandLeftWristLeft

ShoulderLeft

Head

ElbowLeft

ShoulderRight

HandRightWristRight

ElbowRight

ShoulderCenter

FootLeftAnkleLeft

KneeLeft

HipLeftHipCenter

HipRight

FootRightAnkleRight

KneeRight

Spine

Seated (10)

Default (20)

Coordinates

Z Axis X Axis

Y AxisJoints in meters from camera

Negative

Positive

Positive Negative

Positive

Joint Tracking & SmoothingEach joint has associated tracking state

Tracked, Not tracked, or InferredInferred - Occluded, clipped, or low confidence joints

Use TransformSmoothParameters to smooth joint data to reduce jitter

Kinect transform smoothing uses Holt Double Exponential Smoothing - http://en.wikipedia.org/wiki/Holt-Winters

Skeleton.ClippedEdges = FrameEdges.Top;

Physical Interaction Zone

Fatigue Kills Gestures Use Coding4Fun ScaleTo extension Interaction Controls support

Skeletal TrackingDemoName

Gestures & Interaction

Gesture ControlsDefine Kinect controls in KinectRegion KinectItems Control, KinectTileButton, KinectCircleButton, KinectScrollViewerAutomatically get cursor, grip, and scroll support

GesturesDemoName

Face Tracking

Face Tracking

2D/3D Mesh and Points3D Head PoseAnimation Units

Z Axis

3D Head Pose

Pitch

+20

-20

Yaw

-45 +45

Roll

-90 +90

Upper Lip Raiser

-1 Max pushed down lip+1 Showing teeth fully

Jaw Lowerer

-1 Closed+1 Fully open

Lip Stretcher

-1 Fully rounded (kissing mouth)+1 Fully stretched (joker’s smile)

Brow Lowerer

-1 Raised almost all the way+1 Fully Lowered

+1 Very sad frown

Lip Corner Depressor

-1 Very happy smile

+1 raised in expression of deep surprise

Outer Brow Raiser

-1 fully lowered, very sad face

Animation Units

FaceTrackingDemoName

Kinect Service

Kinect Socket ServiceKinect data over TCP SocketsColor, Depth, Skeleton, and AudioClient libraries for WPF, Windows Phone & WinRTConsole or Windows Service host

ScenariosWindows Store, Windows Phone, WPF1 Kinect to many PCs, Many Kinects to 1 PC

KinectServiceDemoName