Ramesh RaskarMedia Lab, MIT

Cambridge, MA

Second Skin

Second Skin and RFIG

• Second Skin– Sensing and Actuation

• Every mm at every ms– Impercetible, natural environment

• Overcoming the ‘Dark side of VR’

• RFIG– Marker technology– Marker= RFID + photosensor (not barcode)– Locator= Projector (not camera)– Durable (10+ yrs), long range, imperceptible (zero area)

• Think inverse

Vicon Motion Capture

High-speed IR Camera

Body-worn markers

Medical Rehabilitation Athlete Analysis

Performance Capture Biomechanical Analysis

‘Motion’ Capture ?• Building a real time human model

– Dense sampling over surface– Geometry with Id at every millisecond at every milimeter– Bio parameters

• Getting intimate– Cameras ..– Wearables– Second Skin (Sensor suit)– Tapping inside

• Close the loop in bio-I/O– Remote monitoring: Elderly care, training– Robot observation:learning, worker safety– Feedback for biomech/neuro interfaces

Tagged Books in a Library Id : List of books in RF range

No Precise Location Data Are books in sorted order ?

RF Tag + Photosensor

Conventional RF tag

Photo-sensing RF tag

READER

Micro Controller

RF Data

Light

Memory

Photosensor

Computer

READER

Micro Controller

RF Data

Memory

Conventional RFID

Computer

READER

Projector Micro

Controller RF Data

Light

Memory

Photosensor

Computer

Projector + Photo-sensing RF tag

Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

00 11 11 00 00 X=12

X=12

Visual feedback of 2D position

a. Receive via RF {(Id1,x1,y1), (Id2,x2,y2), …}

b. Illuminate those positions

a. Receive via RF {(Id1,x1,y1), (Id2,x2,y2), …}

b. Illuminate those positions

Siggraph 2004

Where are products about

to expire ?

Where are products about

to expire ?

Find tag location using

handheld Projector

Find tag location using

handheld Projector

Photosensing

Wireless Tags

Photosensing

Wireless Tags

Many geometric ops Many geometric ops

R F I

R F I D

Interactive stabilized projection

Interactive stabilized projection

(Radio Frequency Id & Geometry)

G

Siggraph 2004Siggraph 2004

Mitsubishi Pocket Projector

AR with Photosensing RFID and Handheld Projector

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,

SIGGRAPH 2003

AR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display device

• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

AR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display device

• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

RFID ?

AR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display device

• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

RFID

Photosensing RFID

Projector for visual feedback

Inside of Projector

The Gray code pattern

Focusing Optics

Gray code Slide

Condensing Optics Light Source

Tag

2D Location 3D Location

Y data

X data

Y data

X dataX2 data

Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

00 11 11 00 00 X=12

X=12

Imperceptible Tags under clothing, tracked under ambient light

Towards Second SkinCoded Illumination Motion Capture Clothing

• 500 Hz with Id for each Marker Tag• Capture in Natural Environment

– Visually imperceptible tags– Photosensing Tag can be hidden under clothes– Ambient lighting is ok

• Unlimited Number of Tags– Light sensitive fabric for dense sampling

• Non-imaging, complete privacy• Base station and tags only a few 10’s $

• Full body scan + actions– Elderly, patients, athletes, performers– Breathing, small twists, multiple segments or people– Animation Analysis

Second Skin and RFIG

• Second Skin– Sensing and Actuation

• Every mm at every ms– Impercetible, natural environment

• Overcoming the ‘Dark side of VR’

• RFIG– Marker technology– Marker= RFID + photosensor (not barcode)– Locator= Projector (not camera)– Durable (10+ yrs), long range, imperceptible (zero area)

• Think inversehttp://raskar.info

Acknowledgements• MERL

• Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Joe Marks, Darren Leigh

• Office of the Future group at UNC Chapel Hill• Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder, Michael Brown,

Ruigang Yang• Henry Fuchs, Herman Towles• Wei-chao Chen

END

END

END

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006

Ramesh RaskarMitsubishi Electric Research Labs (MERL)

Cambridge, MA

The Poor Man’s Palace:Special Effects in the Real World

Special Effects and Virtual Worlds

– Photorealism around us ? Stays on screens

– Does it affect daily life in real time ? Unlike other fields

Changing Appearance

Changing Virtual Illumination

Special Effects and Virtual Worlds

– Photorealism around us ? Stays on screens

– Does it affect daily life in real time ? Unlike other fields

– Fusion: real world with graphics• Next big challenge in CG/Second Life ?• Believable, seamless co-existence

Changing Appearance

Projector Projector

Virtual light source

Changing Virtual Illumination

Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

– Preprocessing• Scan 3D object and create model• Roughly align projector(s)• Calibrate by finding pose

– Run-time• Render images of 3D model• Warp/Correct

Virtual Motion

Virtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

Dynamic Augmentation

Projecting on Tracked Objects

d ( x )2

k ( x ) cos( p )

Radiance Adjustment

Ip (x, p) =

I

d

, k ( x ) > 0L ( x, )

Virtual Real

Intensity correction

Desired radiance

Pixelintensity

Reflectance

Virtual Reflectance Virtual Illumination

InteractionVirtual Motion

ShaderLamps

www.ShaderLamps.com

Poor Man’s Palace

Poor Man’s Palace

Maya: World is an Illusion

• Fake World– We all live in one

• Social Issues– Real-life Fakes

• Not just photos and videos but surroundings– Privacy

• X-reality/AR/Virtual Worlds – Delivers years of CG/Sensors/Robotics research into the real world– Time and Space shifting with non-linear distortions

• Complex Reflectance – Specular or arbitrary BRDF surfaces– View-dependent appearance

• Participating Media – Simulating or in presence of smoke, fog

• Complex Geometry– Spaghetti

• Motion – Animation of real surfaces

• NPR, Distortions, Perceptual factors– Great thesis topics ..

Beyond Gouraud Shading of White Objects

Pieces of the PuzzleActuated Surfaces

Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]

Displays Contenders

OrganicLED

LightEmittingPolymers

E-Ink

Recap

• Special Effects in Real World– Photorealism yet to impact daily life– Poor Man’s Palace

• Spatial Augmented Reality– Untethered solution for fusion– Geometry, Photometry, Id– Sense, Control, Compensate– Projectors, RFID, Sensors

• Open Problems– All senses: haptic, olfactory, proprioception– Natural phenomenon, complex BRDF, other displays

• Next Challenge: Photorealistic AR around us