Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections

Hui-Shan Kao

2

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

3

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

4

Introduction Regular interactive surface only support

Multi-Touch Tangible input

Extend more possibility on interactive surface Add another display Enable intuitive 3D manipulation

Introduction5

Beyond-Surface Interactions Base on regular interactive surface Embedded IR markers

Invisible 6DOF of IR camera

pico projector Tablet PC

IR Projecto

r

Location & Orientation

6

Outline Introduction Related Work

Beyond-surface Interaction Localization of Device Invisible Projection

System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

7

Beyond-surface Interaction Second Light

Use an electronically switchable diffuser Turn any translucent sheet above the

surface into a mobile display via the second projection

With the camera that sees through the surface, it can localize a mobile panel in six-degrees.

Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08

8

Localization of Device Enable 3D interaction on tabletop display

Need to recognize the 6DOF of device The way to know 6DOF

Magnetic tracker Penlight

Vision based tracker Handheld projector JanusVF

Marker based trackerVisible

9

Invisible projection Invisible projection

Spectrum IR/Color polarization

Time high frequency

Synchronization of camera and projector Encoding in content

Embedded code in color channel

10

Invisible projection Hybrid Infrared and Visible Light Projection for

Location Tracking A projector capable of projecting visible images and infrared images Using gray-coded pattern to locate the sensors.

Dynamic adaptation of projected imperceptible codes Using high frequent temporal image modulation to

project an invisible pattern

Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07 A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07

11

Outline Introduction Related Work System Design

Hardware Configuration Markers System

System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

12

System Design Goal:

Support multi-touch and multi-user DI based touch detection

Add beyond surface interaction Using invisible marker

13

 Hardware Configuration

IR ProjectionColor Projection

IR Projector

IR Camera IR

Camera

Color Projector

Mirror

Pico projector + IR Camera

Tablet PC + IR Camera

14

 Hardware Design The order of glass layer and diffuse layer

diffuse layer should on top Not to degrade the luminance of pico projector The reflection of pico projector may offending

the user

15

Hardware Configuration Problem:

IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views

Use two cameras to reduce the IR spot

16

Markers System ARToolkitPlus

Fiducial marker Self-identify

Enable error correct bit Localization

Camera and projector calibration Camera pose estimation

17

Outline Introduction Related Work System Design System Calibration

Tabletop System Calibration Projector and Camera System Calibration

Beyond Surface Implementation Working with multi-touch Application Conclusion

18

Tabletop System Calibration Original tabletop system calibration

Finding the homographs between table, camera, and projector

The corners are manually specified by the users Time-consuming Human intervention Pixel-level accuracy

Tabletop System Calibration19

Only need four points to be manually specified Adding an additional IR-Color camera Project predefined markers for calibration

IR Camera

IR Projector

Color Projector

IR-Color Camera

20

Pico projector and camera calibration

The projector as an inverse camera Mapping pixel from a 2D image into 3D rays Using standard camera calibration procedure

Find the 3D points of the projected pattern and the 2D points of the image projected

Rc , Tc

Rp , Tp

Rcp , Tcp

21

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation

3D posture estimation Multi-Resolution Markers

Working with multi-touch Application Conclusion

22

Beyond Surface Implementation

3D posture?

23

3D  Posture Estimation ARToolKitPlus for 3D estimation

Camera looks within image for markers Encode identity Allow recovery of camera pose relative to

marker

24

Multi-Resolution Markers Uni-resolution marker

Camera could observe the markers too small or big The marker with unfit size will not be

recognized

Multi-resolution marker System resizes the IR markers according to

the camera’s posture

25

Marker Split and Merge Marker Split

Not enough : split the markers into smaller size Marker Merge

Too much : merge the markers for higher accuracy

How to re-arrange the layout of IR makers? Ensure that camera will see at least 4 markers Only re-arrange the layout in camera’s view field The nearest camera will have high priority

27

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch

Foreground detection Background Simulation Software Synchronization

Application Conclusion

28

Working with multi-touch Traditional DI process

Take few frames for building background subtract the background obtain the foreground

IR markers projection will interrupt the traditional detection of multi-touch Foreground can not be recognized

3131

IR Projecto

r

IR Cameras

IR Camera of Mobile Device

Simulated Background

Observed Image

Foregrounds

Layout Manager

Color Project

or

ROI

Real scene

ROI Generation

34

IR Projecto

r

IR Cameras

IR Camera of Mobile Device

Smoothing

Simulated Background

Observed Image

Foregrounds

Tangible Objects

Finger Touches

Layout Manager

Prediction

Color Project

or

ROI

Applications

KalmanFiltering

Real scene

Foreground Detection

35

Capture Image

Simulate BG

Real BG

Software Synchronization The camera and projectors are two

independent systems The simulated background will not

synchronize to the capture image Some of the markers will be treat as

foreground

36

Software Synchronization Keep the simulated backgrounds in a

buffer by time Find the most similar background by

subtraction

Background Candidate queue

Real BG

Capture Image

BG Candidate

Real BG

37

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

38

 Application 3 types of the application provide

intuitive and natural manipulation.

iLamp

iFlash

iView

iLamp

Goal: Project high-resolution content, bring

more detailed and fine-grained information

Combine a Pico projector and an IR camera

iFlashlight A mobile version of iLamp, can be

moved easily.

iView Tablet PC + IR Camera An intuitive tool to see 3D content or

augmented information of the 2D map from different perspectives.

42

iView Problem in iView

The user will lose the connection with the surface.

Adding the boundary of surface Instruct user to manipulate the surface for

farther information.

43

Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion

44

Conclusion A new interactive surface based on the

programmable invisible markers. Supporting both on-surface and above-

surface interaction for any device outfitted with an IR camera.

Bring another level of information on interactive surface.

45