![Page 1: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/1.jpg)
Kinect in A.I.
Akihiro Eguchi
![Page 2: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/2.jpg)
What is Kinect
a motion sensing input device by Microsoft for the Xbox 360 video game console, which has
◦ RGB Camera
◦ 3D Depth Sensors
◦ Motorized Tilt
◦ Multi-Array Microphone
![Page 3: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/3.jpg)
Kinect Hacks
![Page 4: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/4.jpg)
3D Video Capture with Kinect
http://www.youtube.com/watch?v=7QrnwoO1-8A
![Page 5: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/5.jpg)
KinectFusion - 3D reconstruction
and Augmented Reality
http://www.youtube.com/watch?v=quGhaggn3cQ
![Page 6: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/6.jpg)
Kinect Enabled Autonomous
Mini Robot Car Navigation
http://www.youtube.com/watch?v=NmnepqAQIlk
![Page 7: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/7.jpg)
Some More Links
Top ten Kinect hacks
◦ http://spectrum.ieee.org/automaton/robotics/diy/top-10-robotic-kinect-hacks
Robots use Kinect to understand our world (object recognition)
◦ http://www.newscientist.com/article/dn20723-robots-use-kinect-to-understand-our-world.html
3d models from a Kinect on a car
◦ http://blog.decoratorpattern.com/2011/01/23/real-world-mapping-with-the-kinect/
Kinect RGB Demo v0.3.1
◦ http://nicolas.burrus.name/index.php/Research/KinectRgbDemoV3
Drawing in the air with Kinect
◦ http://vimeo.com/16818988
Teaching Kinect to recognize objects
◦ http://www.youtube.com/watch?v=fQ59dXOo63o
![Page 8: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/8.jpg)
Microsoft Official SDK
Microsoft has released of a non-
commercial Kinect software development
kit for Windows on June 16, 2011.
C++/CLI, C#, or Visual Basic .NET.
Tutorial of how to use the SDK
◦ Kinect for Windows SDK Quickstarts
![Page 9: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/9.jpg)
Object learning
![Page 10: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/10.jpg)
How human learn name of objects
Human children use mainly two biases
when learning name of objects:
◦ Shape bias (Landau, Smith, and Jones (1988))
Generalize name of the object if the shape is similar
◦ Function bias (Nelson, Russsel, Duke, and Jones (2000))
Generalize name of the object if the function of the
use seem to be the same
![Page 11: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/11.jpg)
Previous work in object name
learning Simulation of biases (Grabner1, Gall, Gool (2011))
◦ Shape bias:
Grab 3D models that are labeled with a same name (e.g., chair) and run a machine learning algorithm to train the classifier.
◦ Function bias:
First define the use of the object; e.g., chair is a object to sit on.
Then, let the program learn the posture of sitting.
If the object is sittable, the object is more likely to be a chair.
![Page 12: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/12.jpg)
Shape Bias Implementation with
Kinect Take the depth info. of the environment.
Use RANSAC algorithm to remove the ground the object is placed.
Name the objects and run machine learning to train the classifiers.
Google 3D warehouse should enhance the learning (Lai and Fox (2010))
![Page 13: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/13.jpg)
E.g., of machine learning techniques
K-NN algorithm
◦ Simplest machine learning technique
◦ Store all the input associated with a name.
◦ To identify a new object, take k closest match and take a majority vote to find the most probable name of the object.
Problem:
◦ Memory consumption and speed limitation
One solution:
◦ Use SVM, which is a lot better in the way it does not store every exact data of the input.
![Page 14: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/14.jpg)
Function Bias implementation with
Kinect Kinect has a feature to track 20 different
joints of human body.
◦ Easily be able to draw skeletal body image
So, program can learn a posture of human (like sitting) easily.
Like Grabner1, Gall, Gool (2011), we can add a simple function bias with this way.
![Page 15: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/15.jpg)
Activity learning
![Page 16: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/16.jpg)
How human learn name of action
Children distinguish actions based on
◦ Manner bias
Based on the manner the action includes
Shrink-expand, movement of the hands, and etc.
◦ Path bias:
Based on the path the action takes
E.g., left to right, up to down, and etc.
Pulverman, Hirsh-Pasek, Golinkoff, Pruden and Salkind (2006)
![Page 17: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/17.jpg)
Previous work in action learning
Laptev, and Caputo (2004)
◦ used video recorded action data like walking,
jogging, and running, which is four seconds in
average, and run SVM to train the classifier to
recognize those actions.
![Page 18: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/18.jpg)
Implementation of the action
learning with Kinect records 10 seconds of each action and name
it.
motion data can be represented as a relative distances between all of 20 joints tracked by Kinect.
By running SVM on the cumulated data, program can recognize what kind of action humans are performing in front of the Kinect sensor based on a manner bias.
Path bias can be implemented easily as well.
![Page 19: Kinect in A.I. - University of Arkansascsce.uark.edu/~cwt/COURSES/2011-08--CSCE-4613--AI/DOCS/Kinect--Akihiro... · What is Kinect a motion sensing input device by Microsoft for the](https://reader030.vdocuments.net/reader030/viewer/2022041216/5e0499cda8c6680527096962/html5/thumbnails/19.jpg)
Activity learning demo on youtube
http://www.youtube.com/watch?v=AxCn0eKWkiQ