raskar graphics interface may05 web
TRANSCRIPT
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Ramesh RaskarMitsubishi Electric Research Labs (MERL)
Cambridge, MA
Special Effects in the Real WorldSpecial Effects in the Real World
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Special EffectsSpecial Effects
– Photorealism •Stays on screens•Yet to affect daily life in real time
– Entertainment and Visualization– Unlike other fields of CS
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Changing AppearanceChanging Appearance
Virtual light source
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Changing Virtual IlluminationChanging Virtual Illumination
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Special Effects in the Real Special Effects in the Real WorldWorld
– Photorealism •Limited to display screens•Yet to affect daily life in real time
– Entertainment and Visualization– Unlike other fields of CS
– Fusion: real world with graphics•Next big challenge in CG ?•Believable co-existence
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Changing AppearanceChanging Appearance
Projector Projector
Virtual light source
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Changing Virtual IlluminationChanging Virtual Illumination
Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Virtual MotionVirtual Motion
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Virtual MotionVirtual Motion
Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Dynamic AugmentationDynamic Augmentation
Projecting on Tracked Objects
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLampsShaderLamps
www.ShaderLamps.com
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Projector-based AugmentationProjector-based Augmentation
www.ShaderLamps.com
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Poor Man’s PalacePoor Man’s Palace
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Maya: World is an IllusionMaya: World is an Illusion
• Fake World– We all live in one
• Real-life Fakes– Not just photos and videos but surroundings
• Privacy
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Augmented Reality (AR)Augmented Reality (AR)
With See-through Head-mounted Display
© Andrei State
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
hand-helddisplay
spatial see-throughdisplay
head-mounted display
retinal display
Head-attached Body-attached spatial
projector
projector
projector
Spatial Augmented Reality
Classification of Augmented RealityClassification of Augmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Being ThereBeing There
• Recreate Large Environments– ‘BeingThere’, walk-around– Museums, Exhibitions, Training
Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Desired Virtual Model
© Andrei State
Projected Guidance for Placement
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Head-mounted Display Projectors
© Andrei State
Augmented RealityAugmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
OpportunitiesOpportunities
• Information Augmentation- Office, factory or assembly Training
• Increasing value of real products - Smart illumination to enhance appearance of object
• Entertainment- Animating objects, Museums, Special Effects, Laser Shows
• Advertising- Car showrooms, Furniture store
• Design and Ergonomics- Protyping products, visualize material changes
Projector + Camera
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
• Why– Special Effects in the Real World– Photorealism for everyday
• Tools for Augmentation– Projectors, Sensors, RFID
• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud
Spatially Augmented RealitySpatially Augmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Spatially Augmented Reality Spatially Augmented Reality (SAR)(SAR)
HMD-VR Spatially Immersive-VR
AR using HMDSpatially Augmented
Reality
e.g. CAVE
Video or Optical see-through
SAR, Shaderlamps
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
AR IssuesAR Issues
• Preprocessing:– Authoring
• Runtime:– Identification: Recognition of objects
• Using markers and visual tags
– Registration: Finding relative pose of display device• Dynamic estimate of translation and rotation• Render/Warp images
– Interaction:• Widgets, Gesture recognition, Visual feedback
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Spatially Augmented RealitySpatially Augmented Reality
Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,
SIGGRAPH 2003
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
AR IssuesAR Issues
• Preprocessing:– Authoring
• Runtime:– Identification: Recognition of objects
• Using markers and visual tags
– Registration: Finding relative pose of display device• Dynamic estimate of translation and rotation• Render/Warp images
– Interaction:• Widgets, Gesture recognition, Visual feedback
– Photorealism ?
RFID
Cameras
Trackers
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
StepsSteps
– Preprocessing• Scan 3D object• Roughly align projectors• Compute correction
– Run time• Render images of 3D model• Correct
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
StepsSteps
– Preprocessing• Scan 3D object and create virtual model
Faro arm
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
StepsSteps
– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
StepsSteps
– Preprocessing• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose
– Run-time• Inverse Rendering of 3D Models
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
WarehousingWarehousing
RoutingRouting
Library Library
Baggage
handling
Baggage
handling
CurrencyCurrency
Livestock trackingLivestock tracking
RFID
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Micro Controller
Memory Computer
READER
Micro Controller
Memory
Conventional Passive RFID
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Tagged Books in a LibraryTagged Books in a Library Id : List of books in RF range
No Precise Location Data Are books in sorted order ?
Which book is upside down ?
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Handheld ProjectorHandheld Projector
iLamps 2002 RFIG Lamps 2003-04
Pocket Projector 2004-05
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Mitsubishi Pocket ProjectorMitsubishi Pocket Projector
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Prototype TagPrototype Tag
RF tag + photosensor
RF tag + photosensor
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
READER
Micro Controller
RF Data
Memory
Conventional RFID
Computer
READER
Micro Controller
RF Data
Light
Memory
Photosensor
Computer
Conventional RF tag
Photo-sensing RF tag
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
READER
Projector Micro
Controller RF Data
Light
Memory
Photosensor
Computer
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Find tag location using
handheld Projector
Find tag location using
handheld Projector
Photosensing
Wireless Tags
Photosensing
Wireless Tags
Many geometric ops Many geometric ops
R F I R F I
R F I DR F I D
Interactive stabilized projection
Interactive stabilized projection
(Radio Frequency Id & Geometry)(Radio Frequency Id & Geometry)
GG
Siggraph 2004Siggraph 2004
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
AR with Photosensing RFID AR with Photosensing RFID and Handheld Projectorand Handheld Projector
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Visual feedback of 2D positionVisual feedback of 2D position
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
RFID
(Radio Frequency Identification)
RFIG
(Radio Frequency Id and Geometry)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
CurvedPlanar Non-planar
SingleProjector
MultipleProjectors
Projector
jUser : T
?
Pocket-ProjObjects
Projectors and SurfacesProjectors and Surfaces
2000
2000 2002
2001
1999
2002
20031999
1998
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Pieces of the PuzzlePieces of the PuzzleActuated SurfacesActuated Surfaces
Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
• Why– Special Effects in the Real World– Photorealism for everyday
• Tools for Augmentation– Projectors, Sensors, RFID
• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud
Spatially Augmented RealitySpatially Augmented Reality
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Believable Co-existenceBelievable Co-existence
• Other Senses – Haptic, Smell, Taste, Locomotion
• Beyond Gouraud Effects– Complex BRDF
• Display Choices– Polymers, e-Ink
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Augmentation of “five senses”Augmentation of “five senses”
• Visual Displays• Auditory Displays• Haptic Displays• Locomotion Interfaces• Olfactory Displays• Taste Displays ?• Sixth Sense ..
Extendingsensorymodalities
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Sense of BalanceSense of Balance
Galvanic Vestibular Stimulation
Masahiko Inami (UEC)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Electric Cutaneous DisplayElectric Cutaneous Display
Skin
Nerve
Electrode
Dr. H. Kajimoto, U TokyoMasahiko Inami (UEC)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Smart TouchSmart Touch
Masahiko Inami (UEC)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Scent DisplayScent Display
Tracking nose direction
Aiming atthe nose
Air cannon
Deliveringscented air byvortex rings Camera
Platform
Yasuyuki Yanagida, ATR
Spatio-Temporal Controlled Delivery of Scent
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Air CannonAir Cannon
Generating a doughnut-shaped vortex ring
Aperture
Push
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Air CannonAir Cannon
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Prototype SystemPrototype System
• Nose tracking• Multiple scents• Scent switching• Improved delivery range
Yasuyuki Yanagida, ATR
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Beyond White ObjectsBeyond White Objects
Grossberg, Nayar et alGrossberg, Nayar et al
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Grossberg, Nayar et alGrossberg, Nayar et al
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
The Virtual ShowcaseThe Virtual Showcase
Oliver BimberOliver BimberTracking and MirrorsTracking and Mirrors
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
DisplaysDisplays ContendersContenders
OrganicLED
LightEmittingPolymers
E-Ink
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
HolodustHolodustPerlin et alPerlin et al
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Fake WorldFake World
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
•Participating Media –simulating or in presence of smoke, fog
•Complex Geometry–Spaghetti
•Complex Reflectance –Arbitrary appearance on arbitrary BRDF surfaces
•Shape and Motion –Transformation and Animation of real surfaces
•NPR, Distortions, Perceptual factors–Great thesis topics ..
Beyond Gouraud Shading of White ObjectsBeyond Gouraud Shading of White Objects
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
RecapRecap
• Special Effects in Real World– Photorealism yet to impact daily life– Poor Man’s Palace
• Spatial Augmented Reality– Solution for fusion– Geometry, Photometry, Id– Sense, Control, Compensate– Needs hardware components: Projectors, RFID, Sensors
• Open Problems– Natural phenomenon, complex BRDF, other displays– All senses: haptic, olfactory, preprioception
• Next Challenge: Photorealism
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Spatial Augmented RealitySpatial Augmented Reality
• Graphics– Photorealism yet to impact daily life– Special effects in real world
• Spatial Augmentation– Solution for fusion– Needs hardware components– Projectors, RFID, Sensors
• Next Challenge: Photorealism
raskar.com
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
MoreMore
• Networked fake world• Poor man’s palace• Intelligence moving from PC to Proj to
Surf• Electronic items eventually become
cheaper than those scavenged from nature
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Special EffectsSpecial Effects
– Photorealism • Limited to display screens• Does not affect daily life in real
time– Entertainment and Visualization– Unlike other fields of CS
• Provoke not offend• The primary advantage AR has over VR is
that AR uses the parts of the body that aren't just the eyes and ears: proprioception, vestibular perception, and othe cues that old-fashioned VR just can't handle. The disjunct between vestibular information and visual information that you get in VR is the source of the motion-sickness that often accompanies it.
• VR, like a lot of early 'cyberspace' mythology, was built on an unrealistic rejection of the body, and a fantasy of "pure mind."
• Bud Fake Cup of Coffee 3000http://www.visit4info.com/details.cfm?
adid=21038
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
ApplicationsApplications
– Static setup• Color everyday objects• Engineering drilling locations• Stage backdrops, seasons or aging
– Laser shows on buildings
– Simulate motion• Car shows, wheels, dirt marks• Virtual moving lights
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
ApplicationsApplicationsIndoors, under controlled lightingIndoors, under controlled lighting
– Architectural models• Augment walk-around scaled model of buildings• Project and ‘paint’ surface colors, textures• Lighting, sunlight, seasons• Internal structure, pipes, wiring
– Assembly line• Instructional text, images and procedures
– Entertainment• Live shows, exhibits, demonstrations
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005
Tools for AugmentationTools for Augmentation
• Environment– Sense, Compensate and Control– Id, Geometry, Photometry(reflectance/Illumination), Id
• Display
• Rendering