JugEm: Software to help the world juggle

Level 4 Project 2013 PresentationDavid McBrierty

Juggling

The act of continuously tossing into the air and catching (a number of objects) so as to keep at least one in the air while handling the others.

The idea…

• Build a piece of software that can be used to teach people how to juggle.

Existing Software

• Lots of existing tutorial help out there already….

Existing Software

More Existing Software

Theres something missing….

• Existing tutorials are not interactive!• Learn by doing!

What is JugEm?

• JugEm is an interactive application to help teach people how to juggle.

Initial ideas….

• Use mobile devices to get sensor data• Make users juggle actual devices• Strap devices to users hands

• Wii Controllers• Existing project has done this!

Kinect

• Kinect Sensor provides an interesting interface

How good is a good juggle?

• Lots of lovely physics behind what happens• Claude Shannon’s Uniform Juggle Theorem

Third Party Libraries

• Kinect for Windows SDK• Gives access to raw Kinect Data

• EmguCV Image Processing• .NET Wrapper around OpenCV library

• Microsoft Speech Platform 11• Allows Kinect Sensor to use voice commands

• Built around Microsoft XNA Framework• Framework for making Xbox and Windows software

System RequirementsMUST HAVE SHOULD HAVE

COULD HAVE WOULD LIKE TO HAVE

Detect user throwsDetect user catchesDetect peaksTrack users hand positionsProvide report on techniqueDetect 1,2 and 3 Ball patterns

Calculate Shannon ScoreSuggestions on better techniqueDetect dropsDetect 4 Ball patterns

Tasks to improve timingTasks to improve throwing

Define patterns using Siteswap

The good stuff….

• The system goes through 5 stages:

Kinect Data Input

• Color Image represented as byte[]

• byte[] can easily be converted to an image.

Kinect Data Input

• Depth Data provided as DepthImagePixel[]

• Each pixel from Color Image has DepthImagePixel for its distance from sensor

• Depth data has to be processed before it is useful!

The good stuff….

• The system goes through 5 stages:

Depth Image Filtering

• Raw depth data is very noisy, has to be filtered• 76800 pixels filtered (30 times a second)• Image is converted to gray scale (0-255)• All non player pixels are removed• EmguCV Binary threshold applied to results

• Anything behind players wrists is removed!

• Kinect thinks the balls are part of the player

The good stuff….

• The system goes through 5 stages:

Ball Detection

• EmguCV runs Canny Edge Detection on image• Finds edges with specific radius• Need to detect circles and capped ellipses

Screenshot

Ball Color Detection

• Uses Kinect Color Image• RGB Color Detection proved to be inaccurate

• Hue, Saturation, Value Color Detection instead!

• Extra work, but necessary

Frames

• A Frame is used to store all the data collected from processing

• All the Juggling Balls seen with color, direction and position• Positions of Users hands• Distance from each ball to each hand• Time Frame was seen

• Each new Frame is compared to the previous one to find out directions and detect Events.

The good stuff….

• The system goes through 5 stages:

Detecting Events

• Two areas around users hands• Hand• Throw

Detecting Throws and Catches

• THROW: Ball seen inside Throw Zone and then outside

• CATCH: Ball seen outside Throw Zone and then inside

The good stuff….

• The system goes through 5 stages:

• System collects the Events generated and processes them once the user is finished

• Each type of Juggle has a pattern

• System must be able to cope with missed events

Processing Events

Missed Events

• Inferring missed Events can be tricky…

• Pattern = P, RT, RC, P, LT, LC.

Missed Events

• Inferring missed Events can be tricky…

• Pattern = P, RT, RC, P, LT, LC.

• Inferring missed Events can be tricky…

• Pattern = P, RT, RC, P, LT, LC.

Missed Events

• Inferring missed Events can be tricky…

• Pattern = P, RT, RC, P, LT, LC.

Missed Events

• Inferring missed Events can be tricky…• The times of some inferred Events are slightly

out• Just sometimes a problem…

• Overall effect on time is small.

Missed Events

Juggle Report

• After the Events have been analysed and processed

• System has all the information it needs for Shannon Score

• Display report to the user

Personal Reflection

• Learned how to juggle!• Got to work in a new language• Working with Kinect• Working with all the frameworks• Image processing• Easily the biggest bit of work completed to

date….

Personal Reflection

fin