PowerPoint Presentation

Exergamingxbox fitness technical povBy Kaushik Dashttp://twitter.com/theKaushik

Xbox Fitness is a service exclusively for the Xbox One console, which features fitness and exercise programs from top-tier and popular trainers Jillian Michaels, Tracy Anderson, Tony Horton of P90X and Shaun T of Insanity

List of sensorsXbox Fitness uses a Kinect sensor which internally uses the following sensorsRGB Camera 1080p wide angle cameraDepth Sensor Find distance between objectsMulti Microphones Array Noise cancellation and voice commandActive Infrared Camera See in the dark

RGB Camera

Functionalities The RGB camera is a very important part of the Kinect. Its the eye of the Kinect to view the world.Beside its basic functionality to capture the world and the player movement two other major functionalities of the camera for Xbox One Kinect which are used by Exergaming are as follows: Heart Rate monitorUsed to calculate calories burnedMakes sure that the player is putting right amount of effortEmotion DetectionDetect player emotion and change game play based on it.

Heart RateThe camera can measure heart rate without any hardware attached to a human body.It uses a framework called Eulerian Video Modification which when used with right RGB camera can reveal temporal variations in a human face video which are impossible to see with the naked eye. The variations it detects is the pulse variation which is in sync with our heart beat, similar to pulse beat.

An example of heart rate detection in the game

Emotion DetectionThe camera can do facial recognition and capture small expressions on the face.An emotion detection can understand if the player is distressed or happy. The sensor can detect up to 2 players emotions.An internal algorithm calculates the emotion based on facial expressions.The game AI can change the game play based on the expression detected.

An example of facial expression detection matrix

DEPTH SENSOR

Math behind depth

Line of sightSmall SizeLarge SizeFor the camera in 2D both looks the sameDistance to objectIf there is one small object nearer to a camera and a larger object which is further away from the camera then both the object may look similar from the prospective of the camera.

Please play the animation. Use Slide Show

Distance XUsing the angles of the triangle we can calculate how far the box is from the cameraA way to solve this problem would be to have 2 camera, and draw a triangle to join the object with the camera. Using the angle of the triangle and some trigonometry we should be able to find the distance. Illustration as shown below. Distance Y

Kinetic Depth SensorKinect uses the depth sensor to figure out how far away things are. This is very important when playing games using gestures like detecting hand movement and speed of movement etc.It uses a combination of RGB camera and an Infrared Projector to find the depth.Referring to the previous slide illustration, in place of 2 cameras Kinect uses Infrared Projector along with a camera to virtually paint identification numbers on objects to be recognized by the sensor.

Please play the animation. Use Slide ShowIF the video is not playable use thisURL: https://youtu.be/uq9SEJxZiUg?t=54s

Example of how the Infrared projector projects infrared light viewed with night vision camera.

IF the video is not playable use thisURL https://youtu.be/CIkscN6nXcE?t=1m10s

MULTI-MICROPHONE ARRAY

Why a multi microphone array?A single microphone can help in detecting sound but multiple microphones can help in achievingNoise cancellationIn a multi microphone scenario some of the mics can detect external sounds like sound from streets, speaker sounds, etc. , and the algorithm build in can cancel them while sound is captured from other microphones at the same time.Sound localizationSound localizationrefers to a listener's ability to identify the location or origin of a detectedsound in direction and distanceThis helps the game understand the position of the user. An interesting fact is that our ears are like multi microphone array of two microphones and thats why we are able to do sound localization.

A diagram representation of human ears for sound localization

Kinect multi microphone arrayKinect uses 4 microphones in an arrayIt uses it for Voice recognitionNoise cancellationSound localization

ACTIVE INFRARED CAMERA

Why is a infrared camera needed?Kinect already has a RGB camera, but why is an infrared camera needed?Simple answer, to be independent of the environmental light which might effect an RGB camera.An example: Right side: RGB camera only detect partial information of the player. But Active IR has captured full image without been affected by environmental light.

KINECT SENSOR COMPUTER VISION

What is a Kinect SensorKinect Sensor is a line of motion sensing input devices by Microsoft for Xbox 360 and Xbox One video game consoles and Windows PCs.So Kinect is a combination of RGB Camera 1080p wide angle cameraDepth Sensor Find distance between objectsMulti Microphones Array Noise cancellation and voice commandActive Infrared Camera See in the darkWhat Kinect does is takes input of various sensors and intelligently maps together to create an highly precise 3D depth map of a human (in this case the player of Exergame) aka Computer Vision for the Xbox.

How does the Kinect Sensor detects movement?Kinect Sensor has the intelligence to create a 3D depth map of a user. Using these the algorithm can figure out various basic movement parameters for the player.Skeleton Detects basic skeleton of the player, with joints and hinges in the body including finger joints.Orientation Detects how body part moves using a concept called Box Man. Muscle + Force When the 3D depth map is added to a physics model then the sensors can output forces and muscle strength used using algorithms.

IF the video is not playable use thisURL https://youtu.be/Hi5kMNfgDS4?t=3m

AI AlgorithmsUsing the Muscle + Force map the game can detect if you are able to do an exercise properly. Will suggest alternative level when you fail or not able to do something right. Thus the AI in a way motivates and helps the user. If you workout better then expected there is a multiplier added to the score to motivate.

3rd Party Service IntegrationXbox Fitness can sync with Microsoft Health the cloud-based health integrationplatform

Social Network integrationsRecord play and upload using integrated Youtube upload feature.

Thank You.

ReferenceYoutube: https://www.youtube.com/watch?v=uq9SEJxZiUg, https://www.youtube.com/watch?v=CIkscN6nXcE, https://www.youtube.com/watch?v=Hi5kMNfgDS4 Microsoft: http://www.xbox.com/