Gesture and Facial Tracking for HTML5 using Intel® RealSense Technology

Shwetha DossSr. Application Engineer

Software and Services Group

Intel Corporation

Agenda

• Intel® RealSense™ Technology

• Exploring the Intel® RealSense SDK

• Integration of Intel® RealSense SDK with

HTML5

• Summary

Intel® RealSense Technology

4

Interfaces of computing devices…

What comes next?

Adding Human-like Perceptual

abilities to Computing Devices

5

Add “Senses” to the “Brain”

The Intel® RealSense™ SDK

6

Free Tools and APIs for building natural user interfaces.

Public Beta for Windows available

Accessible for beginners. Open and extensible for experts.

Support for Windows 8.1 today. Android support coming.

Reach a broad-base of customers by writing apps for the Intel RealSense 3D Camera.

Stand out in a crowded market. Delight your customers.

Works with languages, frameworks, and game engines you already use

Intel® RealSense™ 3D cameras

8

The Intel® RealSense™ 3D Camera

9

Devices with Intel® RealSense™

camera

Dell Venue 8 7000

tablet

INTEL CONFIDENTIALSoftware and Services GroupSoftware and Services GroupSoftware and Services GroupSoftware and Services Group

Front-Facing Camera Usage

Learning and Edutainment

Immersive Collaboration,Creation

Capture and Share

Interact NaturallyGaming and Play

Broad Enabling

INTEL CONFIDENTIALSoftware and Services GroupSoftware and Services GroupSoftware and Services GroupSoftware and Services Group

Rear-Facing Camera Usage

Learning and Edutainment

Capture the world in 3D

Immersive Gaming

Enhanced Photo and Video

Collaboration

INTEL CONFIDENTIALSoftware and Services GroupSoftware and Services GroupSoftware and Services GroupSoftware and Services Group12

Snapshot Camera Usage

13

SDK Supported FeaturesRequired Hardware A system with a minimum of a 4th generation Intel® Core™

processor

(codenamed Haswell), either IA-32 or Intel® 64, with

integrated depth camera

Required OS Microsoft* Windows 8.1 OS (32- and 64-bit)

Microsoft* Windows 8.1 Metro (coming soon)

Supported Programming

Languages

C++, C#, JavaScript, Java (JDK 1.7.0_11 or later for face,

voice, and gesture)

Supported IDE Microsoft* Visual Studio C++ 2010-2013 with service pack

1 or newer

Eclipse.org IDE

Supported Development Tools -Microsoft* .NET 4.0 Framework for C# development

-Unity* PRO 4.0.0 or later for Unity game development

-Processing* 2.0.7 or later for Processing framework

development

-OpenFrameworks* v0071 or later for OpenFrameworks

development

-Havok* Vision SDK 2012.2.1 or later for Havok Vision

SDK development

-Intel® Media SDK 2014

Exploring the Intel® RealSense

SDK

Hands

Face

Speech

Environment

15

The Intel® RealSense™ SDK

Modules

Use standalone or in combination

Categories of

Input

Capabilities Features

Hands • Hand and Finger

Tracking

• Gesture Recognition

• 22-point Hand and Finger Tracking

• 9 static and dynamic mid-air gestures

Face • Face Detection and

Tracking

• Multiple Face Detection and tracking

• 78-point Landmark Detection (facial features)

• Emotion Recognition (7 emotions, coming post-Beta)

• Pulse Estimation

• Face Recognition (Coming post-beta)

Speech • Speech Recognition • Command and Control

• Dictation

• Text to Speech

Environment • Segmentation

• 3D Scanning

• Augmented Reality

• Background Removal

• 3D Object / Face / Room Scanning (Coming post-beta)

• 2D/3D Object Tracking

• Scene Perception (coming post-beta)

SDK Runtime

Hand

Tracking

Microsoft® USB Video Class Camera Driver

Background

Segmentatio

n

Core SpeechFace

Analytics

Aug-

mented

Reality

Application

Intel® RealSense SDK Architecture

RealSense SDK Package

Developer Tools

Libs

Docs & Samples

…

Power Mgmt FW UpdateIOModule

Camera Module Camera

Firmware

Depth Camera

Management

17

Intel® RealSense Programming

18

Integration with HTML5

19

19

Gestures and Facial Analysis

Unity* Web Player games

HTML5 / JavaScript (interactive websites)

Unity Web Player App

HTML5 / JavaScript

SDK Local

Web-Socket Server

Intel® RealSense™

SDK runtime

Operating System

Web Socket Communication

127.0.0.1:4181

* Other names and brands may be claimed as the property of others

Web/ HTML5

Accessing Intel® RealSense modules

20

21

Accessing Intel® RealSense

modules

22

Hand Tracking

Output

Accessing tracked hands

Hand data

Joint data

Finger data

Extremity Points

23

Skeleton Tracking

24

Alerts

– HandDetected:

– a hand was identified

– HandTracked:

– full skeleton available for detected hand

– HandCalibrated:

– hand measurements ready

– HandOutOfLeftBorder:

– hand touching the tracking borders

– Same for: HandOutOfRightBorder, HandOutOfTopBorder,

HandOutOfBottomBorder

– HandTooClose / HandTooFar: hand reaching closest/furthest range

25

Predefined Gestures

26

Hand Tracking

27

Hand Tracking

28

29

Demo : Hand Tracking

30

Face Tracking

Output

Face Detection

Bounding rectangle

Average face depth

Face landmark

Points

Face Pose

Facial Expressions

31

Alerts

- Face in scene

- New face detected

- Face lost

- FOV

- Face out of FOV

- Face Back to FOV

- Occlusion

- Face Occluded

- Face not occluded

- Face attached object

32

Face Tracking

33

Face Detection

34

Face Landmark

35

Face Pose & Expression

36

37

Demo : Face Tracking

38

Speech Recognition

Speech Recognition

39

40

Demo : Speech Recognition

Create A New World of

Interactivity• Intel RealSense offers an opportunity to

enhance how users interact with compute

devices

• The ecosystem is gaining momentum and Intel

is making a significant investment

• Depth camera integration in 2014 paves the

way to volume

41

More @ https://software.intel.com/realsense

Connect with us

• Visit us at the Intel booth

• Attend the Intel sessions at MODS

• Building your First Internet Of Things Solution - a Software Developer Perspective:

Oct 9, Hall B,11.10 – 12.10

• Accelerating Your Software Development for Android* on Intel® Platforms : Oct 9,

SD Hall,15.20 – 16.20

• Gesture and Facial Tracking for HTML5 apps using Intel® RealSense™ Technology

Oct 10, Hall B,12.20 – 13.20

• Join us at Intel® Developer Zone: software.intel.com

• Look for us on:

facebook.com/inteldeveloperzone

twitter.com/intelsoftwareinFacebook#IntelSoftwa

re

Thank You