Skeltrack - Open Source Skeleton Tracking

Joaquim Rocha, IgaliaGUADEC 2012 - A Coruña

Igalia's Interactivity Team was on to investigateimmersive applications using FOSS

These applications representnew/different ways of interaction for users

Those can go from entertainment to more serious applications

The Kinect

Microsoft's Kinect was the first camerawith a price affordable to the public

The USB connection is open and thus hackable

This originated Open Source projects like the libfreenect,a library to control the Kinect device and get its information

We created a GLib wrapper for libfreenect called GFreenect

GFreenect offers asynchronous functions (and some synchronous aswell) and makes it easy to use with other GNOME technologies

GObject Introspection opens these technologies for users ofPython, Javascript, Vala, etc.

The Kinect has a structured light camera which gives depth information

But that's raw information... values from 0-2048

libfreenect/GFreenect can give those values in mm

Still...

It does NOT tell you there is a person in the picture

Or a cow

Or a monkey

Let alone a skeleton and where its joints are!

For this you need a skeleton tracking solution

Skeleton Tracking

Three proprietary/closed solutions exist:

Microsoft Kinect SDK: non-commercial only

OpenNI: commercial compatible

Kinect for Windows: commercial use allowedbut incompatible with the XBox's Kinect

Conclusion: There were no Free solutions toperform skeleton tracking... :(

So Igalia built one!

Enter Skeltrack

What we wanted:✩ A shared library, no fancy SDK✩ Modular (do one thing)✩ Device independent✩ No pattern matching, no databases✩ Easy to use (everybody wants that!)

Not as easy as it sounds!

The current version tracks up to 7 joints:✩ Head;✩ Shoulders;✩ Elbows;✩ Hands.

Smoothing is implemented usingHolt's Double Exponential formulas

Smoothing and heuristics usedcan be tweaked using GObject properties

There are still some things missing...

Future work

Hands from elbows: If one of the extremas is an elbow, weneed to infer where the hand is

Robustness: Use restrictions to ignore objects that are notthe user

Multi-user: Track more than one person at a time

And of course, get the rest of the joints: hips, knees, etc.

How to use it?

First reduce the depth buffer's dimension

Reducing the buffer's dimension will make things faster

While at it, it's wise to also apply a depth threshold

Asynchronous API

SkeltrackSkeleton *skeleton = SKELTRACK_SKELETON (skeltrack_skeleton_new ());skeltrack_skeleton_track_joints (skeleton, depth_buffer, buffer_width, buffer_height, NULL, on_track_joints, NULL);

Synchronous API

SkeltrackJointList list;list = skeltrack_skeleton_track_joints_sync (skeleton, depth_buffer, buffer_width, buffer_height, NULL, NULL);

Skeleton Joint:

ID: HEAD, LEFT_ELBOW, RIGHT_HAND, ...x: X coordinate in real world (in mm)y: Y coordinate in real world (in mm)screen_x: X coordinate in the screen (in pixels)screen_y: Y coordinate in the screen (in pixels)

Join us!

https://github.com/joaquimrocha/Skeltrack

Nifty Tools for Development:

GFreenect: https://github.com/elima/GFreenect

GFreenect Utils: https://github.com/joaquimrocha/gfreenect-utils

GFreenect Python Example

Tool: depth-file-viewer

Tool: record-depth-file

Demo?

Warning: Might go wrong!

Questions?

Creative Commons pictures from flickr:Kinect: Auxo.co.krMonkey: nothingtosayKid Playing: Rob WelshSkeleton: Dark Botxy

Other non CC images:Doctor: http://research.microsoft.com/en-us/projects/touchless/Minority Report:http://www.ovelf.com/2010/04/14/oblong-tan-g-speak-arttirilmis-gerceklikte-hareketlerin-3-boyutta-yorumlanmasi/