senior design group 16 client / adviser: dr. tom daniels brenton hankins, rick hanton, harsh goel,...
TRANSCRIPT
Senior Design Group 16Client / Adviser: Dr. Tom Daniels
Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer
Cpre 185 Wii/Kinect Lab Software
Freshmen computer engineering college students and high school students need an interesting way to be introduced to programming and problem solving.
In the past, Iowa State students have been taught to create simple programs in the C programming language that use Nintendo Wiimotes as input devices.
Software needs to be developed similar to Wii Wrap, but uses the Microsoft Kinect™ sensor as input to student-written programs.
Problem Statement
RGB Webcam Camera 640x480 @ 30 Hz (center) IR Projector Projects Grid for Depth Mapping IR Camera at 640x480 @ 30 Hz collects depth data
(right) Microphones give room audio
Hardware - Kinect
Small Panning motor turns sensor bar from side to side
Onboard image preprocessing
Projector power draw requires extra AC power
Wii Remote / Wiimote 11 buttons + Power Roll/Pitch/Yaw Gyro Infrared Sensor Speaker (unsupported) Optional Nunchuck
Hardware - Wii
Schedule / Cost
Three Modules:1. Hardware
interface to read in raw data (provided)
2. Interface simplification / data processing
3. Graphical interface to display output
Architecture
Flow Diagram
Issue: Decrease delay in printf output Solution: Added a fflush(stdout) statement
immediately after printf
Issue: IR data becomes erratic if sensor loses source for one frame of data
Solution: Save last good frame of IR data (x,y) and loop the last good data until new data arrives
WiiMote Solutions
OpenKinect◦ Tricky to get started with in windows◦ Natively supports Unix◦ Body tracking not built-in◦ Needs a calibration pose for body tracking◦ No event when new Video or Depth frame is available
Microsoft Kinect SDK◦ Sample Projects◦ Documentation◦ Simple installer◦ Natively supports Windows◦ Full body tracking built-in
Kinect Design Approaches
Data Available◦ RGB Video Frames (640x480 max resolution)◦ Depth (mm) Data Frames (320x240 max
resolution)◦ Skeletal tracking points (20 points per player)
How we get the data◦ Pass an event object to alert us when new data is
available◦ Pass a video/depth/skeleton frame pointer to read
in next frame◦ Run event loop and process each frame as
RGB/Depth/Skeleton data
Microsoft Kinect SDK
Windows File Mapping is an inter-process communication tool that maps a block of memory by name
Shared Memory in Kinect Wrap◦ File Mapping is set up by Kinect Wrap backend
program with a specific name for each block◦ Backend gets data from Kinect SDK and dumps it into
shared memory space when each new frame becomes available
◦ Our API Library copies frame from shared memory into local memory space for student program to use.
Shared Memory Use
Provides an easy-to-use interface for students while maximizing functionality
Functions:◦ getVideoFrame(rgb** pFrame)◦ getDepthFrame(short** pFrame)◦ getSkeleton(Point3D* pSkel)◦ setOutputDisplay(rgb** pFrame)◦ setOutputText(char* sText)◦ setPixel(rgb** pFrame, int x, int y, KCOLOR color)
Kinect Wrap API
Kinect Wrap Library is a static Windows library that students must include their C projects
Library handles backend processing of API Abstracts all of the technical workings of the
Kinect SDK, File Mapping API, and GUI design
Kinect Wrap Library
Abstracts away the details of drawing Kinect data via a graphical API
Kinect GUI receives data via shared memory Data is drawn as a grid of vertices, each
with a custom RGB value via DirectX Intend to port GUI code to OpenGL and work
on constructing a more intuitive GUI in future
Kinect GUI
Field Testing◦ Compared time delay of data transmission
between the current and previous iterations of the Wii Wrap code.
◦ Found the apparent delay seen in the Wii Wrap GUI to be significantly decreased.
◦ Test results corroborated by CPRE 185 lab TAs during classroom use.
◦ No previous functionality was affected by these changes.
Testing – Wiimotes
Assumed that the Kinect SDK functions deliver correct data from Kinect sensor
Shared Memory◦ Wrote known data into shared memory space with
one program and had second program read it out and verify that the data has not been altered
Student API Interface◦ Verified that buffers of known data sent to the
student program are not altered before being read◦ Verified that a known function run by student code
produces the expected output frame data given a known input frame
Kinect Black Box Testing
GUI Output◦ Tested whether frames were being updated
consistently by writing specific colors to known pixels in a given output frame
Kinect Black Box Testing
setPixel()◦ Tested for boundary values.◦ Attempted to set pixels outside the frame (640x480)◦ Attempted to set pixel RGB values outside of 0-255
getVideoFrame() & setOutputDisplay()◦ Check that output frame RGB data matches input data◦ Found that with simple student processing code, any
reduction in output fps from device frame rate is minimal getDepthFrame()
◦ Used a plane at a known distance from the Kinect to find how accurate the depth values in mm are vs. reality.
Kinect Integration Testing
int main()
{
KinectLib_Init(); // Initialize the library
rgb* frame[VIDEO_HEIGHT]; // Create 2D frame of rgb data
short* dframe[DEPTH_HEIGHT]; // Create 2D frame of short depth data
Point3D* skeleton; // Create array of skeleton data
while(1){
getVideoFrame(frame); // Map video frame to pointers
getDepthFrame(dframe); // Map depth frame to pointers
if(skeletonTracked()) {
skeleton = getSkeleton(); // Map Skeleton data
}
setPixel(frame, 23, 18, BLUE); // Arbitrarily set pixel (23, 18) to Blue
displayVideoFrame(frame); // Display the modified video frame in the GUI
}
KinectLib_Destroy(); // Destroy Kinect library instance
return 1;
}
Kinect Lab Example
Multi-Kinect Support More comprehensive GUI program Additional Library Functionality Create code to share Kinect data over
Ethernet
Future Work
Questions?seniord.ece.iastate.edu/dec1116/