kinect v2 introduction and tutorial

Introduction to Kinect v2

Tsukasa Sugiura

@UnaNancyOwen

Self-Introduction

Tsukasa SugiuraMicrosoft MVP for Kinect for Windows

@UnaNancyOwen

http://UnaNancyOwen.com

[email protected]

(July 2014 - June 2015)

https://twitter.com/UnaNancyOwen

http://unanancyowen.com/

mailto:[email protected]

Agenda

What is Kinect v2

Specifications

New Features

Demo

Tutorial

Disclaimer

“This is preliminary software and/or hardware and APIs

are preliminary and subject to change.”（Kinect for Windows v2は暫定的なものであり、ソフトウェア、ハードウェア、およびAPIは製品版で変更される可能性があります。）

Kinect for Windows v1 Sensor

MULTI-ARRAY MIC MOTORIZED TILT

3D DEPTH SENSORS

RGB CAMERA


MULTI-ARRAY MIC

3D DEPTH SENSOR

( IR Camera + IR Emitters )

RGB CAMERA


Image by iFixit

IR EMITTERS

IR CAMERA

Specifications

Kinect for Windows v1 Kinect for Windows v2

Color 640×480 @ 30fps 1920×1080 @ 30fps

Depth 320×240 @ 30fps 512×424 @ 30fps

Sensor Structured Light(PrimeSense Light Coding)

Time of Flight(ToF)

Range 0.8~4.0 m 0.5~4.5 m

Angle of ViewHorizontal / Vertical

57 / 43 degree 70 / 60 degree

Microphone Array ◯ ◯

Specifications


BodyIndex 6 people 6 people

Body 2 people 6 people

Joint 20 joint/people 25 joint/people

Hand State Open / Closed Open / Closed / Lasso

Gesture ☓ ◯

Face ◯

Speech / Beamforming ◯ ◯

Basic Features

Color

1920×1080@30fps / 15fps (Lighting Condition)

RGBA, YUV, BGRA, Bayer, YUY2

Basic Features

Depth

512×424@30fps

500~4500[mm]

ToF (Time of Flight)

Basic Features

Infrared / LongExposureInfrared

512×424@30fps

16bit (higher 8 bits)

Basic Features

BodyIndex

512×424@30fps

6 people

Body Area : 0~5, Other Area : 255 (5 < Index)

255

0 1

Basic Features

Body

6 people

25 joint / people (Add Tip, Thumb, Neck)

Orientation (Quaternion)

Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)

Basic Features

Audio

Beamforming (+/-50 degree)

Speaker Estimation

Speech Recognition

Application Features

Gesture

Gesture Recognition using Machine Learning

Discrete (detected true/false), Continuous (progress 0.0f~1.0f)

Learning Classifier Tool “Visual Gesture Builder”

Video by http://youtu.be/-XYoblrnDpg

http://youtu.be/-XYoblrnDpg


Face

Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)

Activity, Appearance, Expression Activity … Eye Closed, Mouth Open / Moved, Looking Away

Appearance … Wearing Glasses

Expression … Happy


HDFace

For Creating 3D Face Model

Points (1347), Triangles (2340), Hair Color, Skin Color

Fitting Face Model


Other

Kinect Fusion (3D Shape Reconstruction)

Controls (Assist in implementation of NUI)

System / Software Requirements

OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)

CPU Intel Core i7 3.1GHz (or higher)

RAM 4GB (or more)

GPU * DirectX 11 supported

USB * USB 3.0 (Intel or Renesas Host Controller)

Compiler * Visual Studio 2012, 2013 (Supported Express)

Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)

Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

Connection


PCPC

Example Multiple Connection

PCPC PC

Hub

Server

System

Kinect

Driver

Kinect SDK

Application

Kinect

Driver

Kinect SDK

Application

Kinect

Service

Kinect SDK

Application

Kinect SDK

Application


Tutorial

Basic Flow of Programming (C++)

Sensor Stream Frame Data

Sensor Source Reader Frame Data

Kinect for Windows SDK v1

Kinect for Windows SDK v2

Source independent to each Data(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)

Doesn’t depend on each other Source(e.g. Doesn't need to Depth Source when retrieve Body Data)

Tutorial

Sensor


// Sensor

IKinectSensor* pSensor;

HRESULT hResult = S_OK;

hResult = GetDefaultKinectSensor( &pSensor );

if( FAILED( hResult ) ){

std::cerr << "Error : GetDefaultKinectSensor" << std::endl;

return -1;

}

hResult = pSensor->Open();


std::cerr << "Error : IKinectSensor::Open()" << std::endl;

return -1;

}

IKinectSensor

Tutorial

Source


// Source

I***FrameSource* pSource;

hResult = pSensor->get_***FrameSource( &pSource );


std::cerr << "Error : IKinectSensor::get_FrameSource()" << std::endl;

return -1;

}

IColorFrameSource

IDepthFrameSource

IInfraredFrameSource

IBodyIndexFrameSource

IBodyFrameSource

Tutorial

Reader


// Reader

I***FrameReader* pReader;

hResult = pSource->OpenReader( &pReader );


std::cerr << "Error : IFrameSource::OpenReader()" << std::endl;

return -1;

}

IColorFrameReader

IDepthFrameReader

IInfraredFrameReader

IBodyIndexFrameReader

IBodyFrameReader

Tutorial

Frame


while( 1 ){

// Frame

I***Frame* pFrame = nullptr;

hResult = pReader->AcquireLatestFrame( &pFrame );

if( SUCCEEDED( hResult ) ){

/* Data */

}

SafeRelease( pFrame );

}

IColorFrame

IDepthFrame

IInfraredFrame

IBodyIndexFrame

IBodyFrame

Tutorial

Data (Depth, Infrared, BodyIndex)


while( 1 ){

// Frame

IDepthFrame* pDepthFrame = nullptr;

hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame );


unsigned int bufferSize = 0;

unsigned short* pBuffer = nullptr;

hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );


/* Processing*/

}

}

SafeRelease( pDepthFrame );

}

Depth, Infrared … unsigned short

BodyIndex … unsigned char

Tutorial

Data (Color)


while( 1 ){

// Frame

IColorFrame* pColorFrame = nullptr;

hResult = pColorReader->AcquireLatestFrame( &pColorFrame );


unsigned int bufferSize = 0;

unsigned char* pBuffer = nullptr;

hResult = pColorFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );


/* Processing*/

}

}

SafeRelease( pColorFrame );

}

Color … unsigned char

Default Color Image Format … YUY2

Tutorial

Data (Color with Converted Format)


unsigned int buffeerSize = width * height * channel * sizeof(unsigned char);

unsigned char* pBuffer = new unsigned char[bufferSize];

while( 1 ){

// Frame

IColorFrame* pColorFrame = nullptr;

hResult = pReader->AcquireLatestFrame( &pColorFrame );


hResult = pColorFrame->CopyConvertedFrameDataToArray( bufferSize, pBuffer,

ColorImageFormat::ColorImageFormat_Bgra );


/* Processing */

}

}

SafeRelease( pColorFrame );

}

delete[] pBuffer;

Tutorial


while( 1 ){

IBodyFrame* pFrame = nullptr;

hResult = pBodyReader->AcquireLatestFrame( &pBodyFrame );


IBody* pBody[BODY_COUNT] = { 0 };

hResult = pBodyFrame->GetAndRefreshBodyData( BODY_COUNT, pBody );


/* Processing */

}

for( int count = 0; count < BODY_COUNT; count++ ){

SafeRelease( pBody[count] );

}

}

SafeRelease( pBodyFrame );

}

Data (Body)

Tutorial


for( int count = 0; count < BODY_COUNT; count++ ){

BOOLEAN bTracked = false;

hResult = pBody[count]->get_IsTracked( &bTracked );

if( SUCCEEDED( hResult ) && bTracked ){

Joint joint[JointType::JointType_Count];

hResult = pBody[count]->GetJoints( JointType::JointType_Count, joint );


for( int type = 0; type < JointType::JointType_Count; type++ ){

if( joint[type].TrackingState != TrackingState::TrackingState_NotTracked ){

CameraSpacePoint cameraSpacePoint = joint[type].Position;

cameraSpacePoint.x; // x (+/- 1.0f)

cameraSpacePoint.y; // y (+/- 1.0f)

cameraSpacePoint.z; // z (500~4500[mm])

}

}

}

}

}

Data (Joint)

Tutorial

Coordinate System

ColorSpace (Coordinate System of the Color Image)

… Color

DepthSpace (Coordinate System of the Depth Data)

… Depth, Infrared, BodyIndex

CameraSpace (Coordinate System with the origin located the Depth Sensor)

… Body (Joint)

Tutorial

Coordinate Mapper// Coordinate Mapper

ICoordinateMapper* pCoordinateMapper;

hResult = pSensor->get_CoordinateMapper( &pCoordinateMapper );


std::cerr << "Error : IKinectSensor::get_CoordinateMapper()" << std::endl;

return -1;

}

ICoordinateMapper::Map○○○FrameTo△△△Space()… Mapping Coordinate System in the Frame

ICoordinateMapper::Map○○○PointsTo△△△Space()… Mapping Coordinate System in the Array of Points

ICoordinateMapper::Map○○○PointTo△△△Space()… Mapping Coordinate System in the Point

Reference

Sample Program

Kinect2Sample | GitHubhttps://github.com/UnaNancyOwen/Kinect2Sample

Article Series

Introduction to Kinect for Windows v2 | Build Insider http://www.buildinsider.net/small/kinectv2cpp

Blog

Kinect | Summary?Bloghttp://unanancyowen.com/?cat=3

https://github.com/UnaNancyOwen/Kinect2Sample

http://www.buildinsider.net/small/kinectv2cpp

http://unanancyowen.com/?cat=3

kinect v2 introduction and tutorial

Technology