gesture processing using 3d depth data on mobile platforms...
TRANSCRIPT
©2013 SoftKinetic. All Rights Reserved.
S3498 – Gesture Processing Using 3D Depth Data On Mobile Platforms
Tim Droz SoftKinetic, Inc.
VP, US Operations
SoftKinetic DepthSenseTM Cameras
Patented Time-of-Flight sensors
Most advanced Depth cameras available
iisuTM middleware (“the interface is you”)
Whole body recognition
Pioneer in close interaction
– 2 Hand, 10 finger tracking
Powerful SDK available for developers
Content Internal team developing applications & UX frameworks
3rd party developer network
Hardware | Middleware | Content
SoftKinetic , Founded in 2007 with offices in Belgium, China, USA, and Korea, is the only independent end-to-end provider for natural gesture solutions
Body Recognition natural gesture solution How Does It Work?
Refine to user-level primitives iisu™ User API
iisu™ Volume Body parts
iisu™ Parts API
Depth map processing iisu™ World API
3D avatar Upper body & hands
iisu™ Remote
Standardized 3D depth map; Sensor calibration iisu™ Source API
iisu Gesture Recognition Middleware
Layer 0
Layer 1
Layer 2
Layer 3
Tegra 3 Solution TOF + Driver + iisu stack results in low latency
Your App Game, App, UI
Mobile Platform
DepthSense Camera Physical Layer
Events
Commands
iisu Gesture Recognition Frame Grabbing
SIMD in general Perform same instruction on multiple data simultaneously. Benefit from optimized performance, power, and
area for our embedded and media intensive gesture recognition software.
SoftKinetic Internal Library Platform independent SIMD code (SSE2, SSE3, SSE4.1, SSE4.2, AVX) for DepthSense™ camera drivers and
IISU™ middleware.
NEON-specific – DepthSense camera drivers Perform computations on 8 pixels (NEON) at a time in the camera driver to compute the Confidence, Depth,
Vertices and other 3D data and have an improvement of over 4 times the scalar implementation.
NEON-specific – iisu middleware We use SIMD instructions in iisu™ layers such as 3D Points Generation, Scene Segmentation, Signal Filtering,
Gaussian filtering, Depth Image Post Processing, Source 3D and have an improvement of over 2.5 times the scalar implementation.
Result iisu™ runs with a depth stream at 60 fps on most of ARMv7 platforms running Android or Linux.
SIMD Optimizations Tegra NEON boosts performance
Platform Acer Iconia A700 tablet, 1.3GHz Tegra 3, Android 4.1.1
SoftKinetic DS311 Camera – USB attached — QQVGA (160x120) depth, VGA color, stereo mic
— 15cm-4m operation distance
DepthSense SDK, iisu SDK
2 apps: Skeleton & Stuntman
SoftKinetic Android build targeted for: ARMv7 based CPU architecture
— Cortex A9, Cortex A15
Android OS at least v4.0.3
Demo Skeleton & Stuntman
More Pixels More x-y resolution in depth
More x-y resolution in color (RGB)
Wide FOV Best for small spaces (don’t need to rearrange the
furniture)
Full body from 1.5m
Higher Performance Silicon Better (finer) depth uncertainty
Higher fps (60+) = even lower lag
Smaller form factors Higher levels of integration (TI partnership)
Modules for PCs, Tablets, Smartphones, and more
What’s Next DepthSenseTM Sensors & Cameras
Large play-space, sees players closer to the Screen
Kinect sweet spot from 2m to 2.5m for single player!
Detection 4 to 6 player tracking
Hands & fingers from 10cm
Open/closed detection
3D Scanning
Object detection | tracking
Head tracking
Biometrics ID Face and volume identification
Robustness Increased flexibility with less false positives
Auto calibration
What’s Next iisuTM Gesture Recognition Middleware
Its already here Gaming IS mobile Allows people to have their content with them at all times
Multimodal Gameplay Close Interaction gesture & touch on mobile Plug in to TV’s & AIO’s for full body tracking It’s not about one or the other, it’s all of the above
Tegra 4+ (Logan) Platform GPGPU, OpenCL, CUDA –
— RGB+Z UV mapping on a scene, augmented reality
— Better graphics rendering in games/applications
— 4K Output
What’s Next Play Is The Mother Of Invention
Contact: Tim Droz [email protected] www.softkinetic.com www.youtube.com/SoftKinetic