Play with Kinect

키넥트 가지고 놀아보기2013/5/11

최윤규 (Yunkyu Choi)

강사 소개• Kincet 관련 작업–키넥트 펀치백 테스트–키넥트 기능성 게임 달걀 받기 , 두더쥐 잡기–키넥트 농사 체험 ( 중앙 박물관 어린이 박물관 )–키넥트 울트라 빔 , 포스 , 날아다니기등 예제– 3D 프레젠테이션 데모

Contents

• Overview• Device• Driver, Library and Software• Kinect Application Basic• Application

OVERVIEW키넥트란 무엇인가 ?

Kinect

• Natural User Interface– Motion sensing– Speech Recognition

적외선 발사 반사된 적외선 수신

Natural User Interface?

• NUI http://en.wikipedia.org/wiki/Natural_user_interface

How does it work?

Labeled Depth Image

Image

이미지 (Image) 깊이 이미지 (Depth Image)

LabelingSkeleton TrackingEtc..

Demo

• NI MATE–키넥트로 이미지 , 깊이 이미지등을 받아 “뼈대

(Skeleton)” 등을 구성한 후 다른 프로그램에 전달 해줌

– http://www.ni-mate.com/• Skanect–키넥트를 통해 3 차원 스캔 가능– http://skanect.manctl.com/

키넥트가 인식하는 뼈대 구조• 20 parts

Kinect 응용 분야• Kinect Games– http://www.youtube.com/watch?v=7jEJS9rkc48– http://starwars.com/watch/vg_kinect_trailer.html

• Kinect for Education– https://www.youtube.com/watch?v=

V0nv1A9nFy8– http://marketplace.xbox.com/en-US/Product/Kin

ect-Sesame-Street-TV/66acd000-77fe-1000-9115-d8024d5309da

Kinect 응용 분야• 3D Scan

– http://skanect.manctl.com/– http://reconstructme.net/

• Motion Capture– http://www.ni-mate.com/– http://ipisoft.com/download/

• You can do crazy things!!– http://code.google.com/p/kinect-ultra/– http://www.youtube.com/watch?v=jhoITmb4518

Other NUIs

• Microsoft PixelSense– http://www.youtube.com/user/mspixelsense

• 3D Immersive Touch (e.g. Apple’s Devices)– Multi User– Multi Touch– Interact with 3D Virtual Environments

• Kinect like Devices• LeapMotion• MYO

DEVICES기넥트 기기에 대해 자세히 알아보자

Kinect or similar devices• Kinect for Xbox 360

– The original(old) one– Developed for Xbox360– Can be used in other env (Mac, PC)

• Kinect for Windows– Include Developer License – Additional Charge– Support Near Mode

• Xtion (ASUS)– Smaller, Lighter– Less quality, No Motor

• Xtion Live (ASUS)

http://www.microsoft.com/en-us/kinectforwindows/news/faq.aspx

Coming Soon!!

• Prime Sense Capri

Assembly

• http://www.ifixit.com/Teardown/Microsoft+Kinect+Teardown/4066/3

DRIVER, LIBRARY AND SOFTWARE필요한 라이브러리 및 소프트웨어들의 구성을 알아보자

Development Environments

• MS Kinect SDK– Only Support Kinect for Windows and Xbox360 (?)– Support all Kinect functions

• Motor, Speech, Near Mode and etc.

– Good Examples (Face, eye tracking etc)• OpenNI– Open Source– Supported by PrimeSense– Support all Kinect Like Devices– PrimeSense

• Israeli company provides 3D sensing technology for Kinect

OpenNI

• Architecture

OpenNI API

• OpenNI– Voice– Voice Command– Hand Gesture– Body Motion Tracking– Not full function support for Kinect (?)

Middleware for OpenNI

• NITE (PrimeSense)

Physical Device

Voice, Hand, Motion APIs

MS Kinect SDK

Summarize @_@

• Complex

Kinect SDK OpenNI

Windows MacOS Linux

설치– 기본 드라이버 및 라이브러리• MS Kinect SDK 또는 OpenNI 설치 ( 우리는

OpenNI 만 ) – 설치시 기기는 분리– MS Kinect Runtime 1.6 :• http://www.microsoft.com/en-us/download/details.asp

x?id=34811

– OpenNI v2.1 : • http://www.openni.org/openni-sdk/

설치할 것들 - 미들웨어들• 적당히 필요한 것들 있으면 설치– http

://www.openni.org/software/?pg=1&cat_slug=file-cat1#.UY6bLCuYFa8

KINECT APPLICATION BASICS

키넥트 가지고 놀때 필요한 기술들• AI (Artificial Intelligence)– Neural Net– Support Vector Machine (SVM)– …

• Computer Vision• Image Processing• Skeleton Animation (with skin)• FSM (Finite State Machine)• 3D Graphics

Basic Process

• 1. Recognition Setting / Start– Initial Pose or Etc

• 2. Input Data (Depth, Image, Skeleton, Hand Point, …)

• 3. Process Data–이 부분을 이전 장의 기술들로 처리

Recognition Setting

• Hand point or Skeleton Mode– Hand point

• Start with gesture (wave hand)• Unstable

– Skeleton• Initial Pose• Stable when all bones are visible from Kinect• Mirror Mode Support

• MS Kinect only– Seated Mode (Unstable)– Kinect for windows only

• Near Mode

주요 Input Data

• Depth Image• Image• Hand points• Skeleton– Joint position– Joint rotation– Root position

• Voice and voice position

Output

• Screen 상에 물체의 위치 , 색 , 기타속성등을 변경

• Motor Control

APPLICATIONPython Programming, 3D Scan & Motion Capture

응용 해보기 예제 소개1. 간단한 조작 해보기– Blender + Python– Blender Game Engine + Python– OpenNI -> NI Mate -> Blender Game Engine Python 코딩

2. Motion Capture– OpenNI -> NI Mate -> Blender -> .bvh 파일 생성

3. 3D Scan– OpenNI/MS SDK -> SKNECT -> .ply 등 모델 파일 생성

플러그인들 및 프로그램• NI Mate : 모션 데이터등 응용프로그램에 전달– http://www.ni-mate.com/download/– 설치 후 Trial 버전으로 인증하는 과정이 필요함– Blender Plugin 설치

• http://www.ni-mate.com/use/blender/

• 3D 파일 및 애니메이션 편집 툴– Blender 2.66 버전으로 설치 (2.67 은 문제 있음 )

• http://download.blender.org/release/Blender2.66/

• SKANECT: 3D Scan– http://skanect.manctl.com/download/

Blender Python

• Blender Python Tutorial– http://cgcookie.com/blender/2011/08/26/introduction

-to-scripting-with-python-in-blender/

• Basics1. 새로운 창 생성2. Text Editor, Syntax Highlighting3. New Text4. 코딩 하기5. Run Script6. Save File / Open File

Blender Python Ex 1

1. Open Python Console2. bpy.context.active_object• ctrl + space : 자동 완성

3. bpy.context.active_object.location4. bpy.context.active_object.location.x = 25. Obj = bpy.data.objects[“Cube”]6. Obj.location.z = 37. 기타 물체 속성 테스트

Blender Python Ex 2

Blender Python Ex 3

Blender Python & Game Engine

1. Open Logic Editor2. Sensors -> Add Sensor -> Always– Activate TRUE level triggering

3. Controllers -> Add Controller -> Python Script– Set Python Script

4. Property -> Physics tab– Physics Test

• Tutorials– http://cgcookie.com/blender/2011/09/22/introduction-to-scrip

ting-in-the-blender-game-engine-part-01/– http://cgcookie.com/blender/2011/09/26/introduction-to-scrip

ting-in-the-blender-game-engine-part-02/

Blender Python Game Engine Ex 1

• 물체를 위쪽 화살표키로 움직이기

Blender Python Game Engine Ex 2

• 부닫치면 “ Hit” 메세지 표시하기

NI Mate & Blender Tutorial

• Blender, NI Mate Tutorial– Basic

• http://www.youtube.com/watch?feature=player_embedded&v=bQhS0SFQ8Gg

– Motion Capture• http://www.youtube.com/watch?v=bQhS0SFQ8Gg• http://www.youtube.com/watch?v=-nN0neRp444• http://www.youtube.com/watch?v=EogegvTG3Po

– Model Download: http://www.blendswap.com/blends/view/7093

– Bone, Skinning• http://www.youtube.com/watch?v=8aFbTy5nDeo

NI Mate & Blender Game Engine

• 아바타 조종해서 물방울 터트리기–스스로 해봅시다

NI Mate + Blender Game Engine + SKNECT(3D Scan)

• Don’t Touch Me! 프로그램 만들어 보기–키넥트를 사용해 모델을 건드리면 뒤로 물러나게

하기• 준비– SKNECT• 3D 모델 스캔• 텍스쳐 맵핑기능은 아직 지원 안함• 버텍스 칼라 지원

Other Helpful Programs

• 3D Scan– ReconstructMe

• http://reconstructme.net/

– SCENECT• http://3d-app-center.faro.com/index.php/stand-alone-apps-faro-

scenect

– 123D Catch: 아이폰 만으로 3 차원 모델 생성가능• http://www.123dapp.com/catch

• Motion Capture– iPi Motion Capture: 여러 개의 키넥트 / 카메라로 정교한 모션

캡쳐 가능• http://ipisoft.com/