cinemacraft - virginia tech...kinect sensing + operacraft = cinemacraft pd-l2ork kinect (facial...
TRANSCRIPT
![Page 1: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/1.jpg)
OPERAcraft + Kinect =
Cinemacraft
Members: Brittany Barnes, Elsi Godolja, Marina Kiseleva
Client: Ivica Ico Bukvic, Institute for Creativity, Arts & Technology (ICAT)
CS 4624, Virginia Tech, Blacksburg VA.
Spring 2016
![Page 2: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/2.jpg)
❖ OPERAcraft: Combines video games and opera to produce live productions
❖ Opera + Minecraft = OPERAcraft
![Page 3: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/3.jpg)
OPERAcraft Controls● Limited to the keyboard and predefined controls
*avatar not pictured
![Page 4: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/4.jpg)
Kinect Sensing + OPERAcraft = CINEMAcraft
Pd-L2Ork
Kinect (Facial detection)
Kinect (Body/skeletal detection) OPERAcraft Visual Display
❖ Allows for production creation in real-time from unique perspectives❖ Goal: Get Minecraft avatar to ‘mirror’ user’s facial and skeletal movements
via Kinect sensors
![Page 5: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/5.jpg)
Software❖ Kinect code
➢ Language: C#➢ Environment: Visual Basic (64 or 32 bit)
❖ Pd-L2Ork
❖ Minecraft code➢ Language: Java➢ Environment: Eclipse Luna/Mars (64 or 32 bit)
![Page 6: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/6.jpg)
What’s Done?
Integral Feature Base Level of Functionality
Does it need to be expanded?
Could it be expanded?
Skeletal: arm movement complete no no
leg movement complete no no
torso movement complete no no
head movement complete no yes
Facial: eyebrow movement in progress yes yes
mouth movement in progress yes yes
Minecraft: avatar art complete no yes
world art complete no yes
![Page 7: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/7.jpg)
Kinect → Pd-L2Ork → Minecraft
![Page 8: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/8.jpg)
Skeletal Recognition
![Page 9: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/9.jpg)
Kinect Skeletal Tracking Infer points
Calculate angles of rotation
Translating Kinect data -> OPERAcraft movement
Send angles to OPERAcraft
@karm ICAT01 24 36 …
UDP Packet:
![Page 10: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/10.jpg)
Angle Calculations
❖ Compute angles of rotation in each plane
❖ Points -> 3D Vectors -> compute angle between
❖ Around Y axis: rotation in X plane
❖ Around Z axis: rotation in Y plane
❖ Around X axis: rotation in Z plane
![Page 11: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/11.jpg)
Facial Recognition
Eyebrow and Mouth Combination Matrix. The 20 possibilities for avatar helmets, based on the user’s facial expressions.
EYEBROW
MO
UTH
Tracked Animation Units (AUs). The areas of 2D points that would affect various AUs to yield the delta values from a face’s neutral position (4).
![Page 12: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/12.jpg)
Kinect → Pd-L2Ork → Minecraft
![Page 13: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/13.jpg)
UDP Packet Constraints
Delimiter(s) Line Ending(s)
Avatar Movement Keywords Movement Position Value(s) Movement
Position Types
whitespace ;\n
eyes mouthhead
shoulderarmleg
torso
{0-3}{0-4}
{(-90)-(90)} {(-180)-(180)}{(y)}
{(-180)-(180)}{(-180)-(180)}{(-90)-(90)} {(-90)-(90)}
{(x)} {(y)} {(-180)-(180)}
integerinteger
float (xrot, yrot)float (y)
float (xrot, zrot)float (xrot, zrot)float(x, y, yrot)
![Page 14: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/14.jpg)
Pd-L2Ork: Averages/Normalization
● Feature position data gets sent from Kinect for each frame
● Pd-L2Ork averages the last five frames to reduce jumpiness in expressions/gestures
● Sends the “patched” data to Minecraft
![Page 15: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/15.jpg)
Kinect → Pd-L2Ork → Minecraft
![Page 16: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/16.jpg)
Minecraft Code: Dependencies
and Flow
![Page 17: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/17.jpg)
Minecraft Art: Facial Expression Helmets
Sad Anxious brows
+ frown
SurprisedSurprised brows
+open lips
AngryFurrowed brows
+ frown
Kiss FaceNeutral brows
+puckered lips
● Microsoft Paint ● 16x16 pixels ● Female and male
![Page 18: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/18.jpg)
![Page 19: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/19.jpg)
Minecraft Art: Avatar Skin
● Microsoft Paint● 32x64 pixels● Female and male
100% zoom
800% zoom
![Page 20: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/20.jpg)
![Page 21: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/21.jpg)
Minecraft Art: Background World● In-game with various block types
![Page 22: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/22.jpg)
Prototype
1. Primary display monitor
2. Computer to process skeletal movements
3. Computer to process facial movements
4. Kinect sensor for skeletal tracking
5. Kinect sensor for facial tracking
![Page 23: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/23.jpg)
SXSW Prototype Demo
Video: https://drive.google.com/file/d/0B12jba-1Ut5jT2pPYTZoOExNWDg/view
![Page 24: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/24.jpg)
QUESTIONS?
![Page 25: Cinemacraft - Virginia Tech...Kinect Sensing + OPERAcraft = CINEMAcraft Pd-L2Ork Kinect (Facial detection) Kinect (Body/skeletal detection) OPERAcraft Visual Display Allows for production](https://reader033.vdocuments.net/reader033/viewer/2022042402/5f135e08d567a3405776625d/html5/thumbnails/25.jpg)
Resources[Image] https://hidale.com/shop/dp-kinect/
[Image] http://vector.me/search/walking-stick-figure
[Image] https://www.icat.vt.edu/funding/operacraft