programming with realsense using .net

Template designed by

Programming with RealSense

Matteo Valorianimvaloriani@gmail.com@matteovaloriani

Matteo ValorianiPhD at Politecnico of MilanoCEO of Fifth ElementSpeaker and Consultant

Microsoft MVP

Intel Software Innovator

email: mvaloriani@gmail.comtwitter: @MatteoValorianilinkedin: https://it.linkedin.com/in/matteovaloriani

Nice to Meet You

2

RealSense Introduction

Capture Hands Data

Detect HandsGestures

Understands Hardwsare Limits

RealSense Cameras

Agenda

The Intel® RealSense™ 3D Camera

4* Other names and brands may be claimed as the property of others

5Intel Confidential - for use under NDA only

Dell* Inspiron 23 7000

* Other Names and Brands may be claimed as the property of others

Available

Enthusiast-grade performance- Intel 4th-gen Core i7/i5 processors- NV GTX850M/860M

Silent cooling system- Efficient heat removal with minimum fan noise

Specially-designed Game Keys- Specially marked WASD keys- Chiclet Keyboard with Red backlighting

Majestic Sound- SonicMaster Premium- ROG Audio Wizard- Headphone amp to

enhance gaming headset experience

Gaming-centric design- Matte black finish with

red diamond cut- Full HD IPS display

with 178-degree wide-view

- Aluminum Design with illuminated ROG logo

Intel® RealSense3D Camera- Enhance experience with natural

interaction.- Immersive Collaboration/ Creation- Capture faces/ objects in full 3D

Asus* G771

Intel Confidential - for use under NDA only

* Other Names and Brands may be claimed as the property of others

What Kinds of Applications can you develop?

7

Capture and Share

Immersive Collaboration/Creation

Interact Naturally

Gamingand Play

Learning andEdutainment

dem

o

8

When we roll out the public beta SDK later in 2014, we’re exploring a developer kit to help you get startedKit would include:

• The Intel® RealSense™ SDK for Windows Beta

• Peripheral depth camera

• Full VGA depth resolution

• 1080p RGB camera

• 20cm – 120cm range

• USB 3.0 interface

• Requires 4th generation Intel® Core™ processor (or later) & Windows* 8.1

The Developer Kit

9

Competitive technologies focus on a living-room experience or a sub-set of Intel RealSense technology features

Designed for close-range interactions

10

Intel® RealSense™ Hands-On Lab - Milan

120 cm

Intel®RealSense™ 3D camera

56°(v) x 72° (v)

20 cm

Leap, RealSense, Kinect

11

Intel® RealSense™ Hands-On Lab - Milan

2,5 cm

60 cm 2 m 4 m

12

What OS and hardware works with RealSense technologies?

1Specific algorithms may have different range and accuracy.**Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.*Other names and brands may be claimed as the property of others.

Intel® RealSense™ SDKfor Windows*, Version 2014 supports Windows 8.x (64-bit), Windows 10

Download the SDK intel.com/realsense/sdk

Intel® RealSense™ Developer Kit 3D cameraGet started developing quickly :

• Full VGA depth resolution• 1080p RGB camera• 0.2 – 1.2 meter range1

• USB 3.0 interface• Requires Intel® Core™ processor

Reserve Yours Now

13

System Requirements

Required Hardware • A dedicated USB 3 port in the client system (for peripheral dev kit camera), do not use a hub

• 4th generation (or later) Intel® Core™ i5 or i7 processor with Intel® Iris™ Graphics and Intel® HD Graphics

• 8GB free hard disk space• Intel® RealSense™ 3D Camera (front-facing, F200 – integrated or peripheral)

Required OS • Microsoft Windows* 8.1 64-bit Desktop Mode, 10• Microsoft Windows 8.1 New UI (Metro) Mode (coming soon)**• Support for Android* coming soon**

Supported Programming Languages

• C++• C# (Microsoft .Net* 4.0 Framework is required)• Java* (JDK 1.7.0_11 or later)• JavaScript*

Supported IDE • Microsoft* Visual Studio 2010-2013 with service pack 1 or newer• Unity* PRO 4. 1. 0 or later for Unity game development

Supported Development Tools

• Microsoft* .NET 4.0 Framework for C# development• Processing* 2.1.2 or higher for Processing framework development• Java* JDK 1.7.0_11 or higher for Java development

Supported Browsers for JavaScript Development

• Microsoft Internet Explorer* 10.0.13• Google Chrome* 33.0.1750.146• Mozilla Firefox* 27.0.1

Understands 4 basic types of input

14

Categories of Input

Capabilities Features

Hands • Hand and Finger Tracking

• Gesture Recognition

• 22-point Hand and Finger Tracking• 9 static and dynamic mid-air gestures

Face • Face Detection and Tracking

• Multiple Face Detection and tracking• 78-point Landmark Detection (facial features)• Emotion Recognition (7 emotions, coming post-Beta)• Pulse Estimation• Face Recognition (Coming post-beta)

Speech • Speech Recognition • Command and Control• Dictation• Text to Speech

Environment • Segmentation• 3D Scanning• Augmented Reality

• Background Removal• 3D Object / Face / Room Scanning (Coming post-beta)• 2D/3D Object Tracking• Scene Perception (coming post-beta)

Coordinate Systems

15

World coordinates Image coordinates

Mass Center: Image and world coordinates of the calculated hand image mass center.

Extremity Points: Special tracking points such as the closest, left-most, top-most, right-most, bottom-most and center points which form the boundaries of the mask of the hand silhouette.

Body Side: Whether it is a left or right hand.

Palm Orientation: Estimation of where the hand is facing.

Tracked Joints: Positions and rotations of the user’s hand in world and image coordinates.

Hands Data

16

Normalized Joints: Posture of user’s hand without changing dimensions of the hand, i.e., the distances between each joint (bone-length) are always the same.

Finger Data: Degree of foldedness and the radius of a particular fingertip.

Segmentation Image: A mask of the hand silhouette allowing to separate a hand from the background image.

Hand Openness: (0-100 value) indicating if all fingers are completely folded to all fingers fully spread.

Hands Data

17

/* Initialize a PXCMSenseManager instance */

manager = PXCMSenseManager.CreateInstance();

if (manager == null){

Debug.Writeline("SenseManager Initialization Failed");

}

Creating a Session for Hand Tracking

18

PXCMSenseManager: organizes a pipeline by starting, stopping, and pausing the operations of its various modalities.

1. Initialize a PXCMSenseManager instance

2. Enable hand tracking using EnableHand.

3. Retrieve an instance of hand module using QueryHand on the instance of PXCMSenseManager.

4. Initialize the pipeline using Init

Initializing the Pipeline

19

/* Enable the hand tracking module*/ sts = manager.EnableHand();if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR) Debug.WriteLine("PXCSenseManager.EnableHand: " + sts);

/* Retrieve an instance of hand to configure */handAnalyzer = manager.QueryHand();if (handAnalyzer == null)

Debug.WriteLine("PXCSenseManager.QueryHand"); /* Initialize the execution pipeline */sts = manager.Init();if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR){ Debug.WriteLine("PXCMSenseManager.Init Failed");}

handData = handAnalyzer.CreateOutput();

Initializing the Pipeline

20

EnableHand(): Enable the hand tracking module.

QueryHand(): Retrieve an instance of hand to configure

Init(): Initialize the execution pipeline

config = handAnalyzer.CreateActiveConfiguration();

config.EnableAllGestures();

//config.EnableGesture("v_sign");

config.EnableAllAlerts();

config.ApplyChanges();

Enable Gestures

21

Enable All gestures or specific gestures to recognize

private CancellationTokenSource streamCancellationTokenSource;

private CancellationToken streamCancellationToken;

Task streamTask = new Task(new Action(AcquireFrames),

streamCancellationToken);

streamTask.Start();

Acquire Data using Task

22

private void AcquireFrames() {

while (manager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR) {

this.outputData.Update();

PXCMHandData.IHand handData;

PXCMHandData.GestureData gestureData;

//AcquireGestures

// AcquireHands

manager.ReleaseFrame();

}

}

AcquireFrames

23

Acquire new data end refresh the elaborated output

Release the frame or you can not elaborate the next !!!

Gesture Tracking

24

//Gestures

for (int i = 0; i < outputData.QueryFiredGesturesNumber(); i++){

if (outputData.QueryFiredGestureData(i, out gestureData) == pxcmStatus.PXCM_STATUS_NO_ERROR){

outputData.QueryHandDataById(gestureData.handId, out handData);

var bodySide = handData.QueryBodySide();

//Do somethings with data

sb.Clear();

sb.AppendLine("Timestamp: " + gestureData.timeStamp);

sb.AppendLine("Gestrue identified: " + gestureData.name);

sb.ppendLine("Gestrue state: " + gestureData.state);

sb.AppendLine("Gestrue handId: " + gestureData.handId);

sb.AppendLine("Gestrue bodyside: " + bodySide.GetType());

sb.AppendLine();

}

}

Analyze Gestures

25

Hand Tracking Modes

26

Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>[] hands = new Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>[4];

PXCMHandData.IHand[] totalHands = new PXCMHandData.IHand[4];

int numberOfHands = outputData.QueryNumberOfHands();

if (numberOfHands > 0) {

for (int i = 0; i < numberOfHands; i++) {

Dictionary<PXCMHandData.JointType, PXCMHandData.JointData> handDictionary =

new Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>();

PXCMHandData.JointData temp;

if (outputData.QueryHandData(PXCMHandData.AccessOrderType.ACCESS_ORDER_BY_ID, i, out handData) ==

pxcmStatus.PXCM_STATUS_NO_ERROR) {

// iterate through Joints

for (int j = 0; j < PXCMHandData.NUMBER_OF_JOINTS; j++) {

handData.QueryTrackedJoint((PXCMHandData.JointType)j, out temp);

handDictionary.Add((PXCMHandData.JointType)j, temp);

}

hands[i] = handDictionary;

totalHands[i] = handData;

// Do somethings with data

}}

Analyze Hands Joints

27

dem

o

28

Capture Volumes

29

Intel® RealSense™ Hands-On Lab - Milan

The user is performing a hand gesture outside of the capture volume. The camera will not see this gesture

Understands Hardware Limits

30

Vertical Rages

31

Intel® RealSense™ Hands-On Lab - Milan

60cm

58 c

m

120cm

56°

20cm

17 c

m

70cm 35cm73 c

m

Effective Range

Gestures Range

Effective Range

3D Facial Traking Range

2D Facial Traking Range

115c

m

Vertical Misalignment

32

Intel® RealSense™ Hands-On Lab - Milan

120c

m

56°

115c

m

Horizontal Rages

33

Intel® RealSense™ Hands-On Lab - Milan

60cm

87cm

120cm

72°

20cm

24cm

Effective Range

Gestures Range

Effective Range

3D Facial Traking Range

2D Facial Traking Range

170c

m

70cm 35cm108

cm

50cm

RealSense Cameras

FRONT-FACING

Intel RealSense 3D Camera F200Availability starting Q4’14

REAR-FACING

Intel RealSense 3D Camera R200In-Market Target 2H’15

Intel® RealSense™ Cameras

Primary: 8 MP, 3264 x 2448 pixels, autofocus

Secondary: 2 MP

Intel RealSense Snapshot, 2 x 720p cameras

RealSense SnapShot

36

Display: 8.4" OLED infinity (2560 x 1600)

Dimensions: 215.9 x 124.2 x 6.1 mm

Weight: 305 g

OS: Android OS, v4.4.2 (KitKat),

CPU: Intel Atom Z3580,Quad-core 2.3 GHz

Memory: 16 GB (+microSD), 2 GB RAM

Dell Venue 8 7000

37

Target Platform:Tablets, 2-in-1s, and moreThe camera typically faces away from the user. It was designed to sense the environment around it in 3 dimensions.

Specifications- Longer range** (up to 3-4 meters indoors, longer range outdoors)- Depth/IR: 640x480 resolution at 60fps- RGB: 1080p at 30fps- USB 3.0 required- Developer Kit Dimensions: 130mm x 20mm x 7mm- Supports Microsoft Windows* 8.1 (64-bit mode), Android* support coming soon.

Targeted Usages- 3D Capture: faces, people, and environments- Depth-enhanced Augmented Reality- Depth-enhanced photography and video- Measurement- Face detection and tracking

RealSense R200

Free Intel® RealSense™ SDK for Windows Beta in 2nd Half of 2014

Intel® RealSense™ Developer Kit being explored

Huge opportunity to reach customers with integrated 3D camera

Works with languages/frameworks/engines you already use

High-level APIs for NUI beginners. Low-level APIs for NUI experts

Getting Started

39 All products, software, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.

software.intel.com/RealSense

Q&A

Grazie a tutti per la partecipazione

Riceverete il link per il download a slide e demo via email nei prossimi giorni

Per contattarmi

mvaloriani@gmail.com

Grazie