rfig sig04 presentation
TRANSCRIPT
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang,
Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher
Mitsubishi Electric Research Labs (MERL), Cambridge, MA
R F I R F I GG Lamps Lamps : : Interacting with a Self-describing World Interacting with a Self-describing World
via Photosensing Wireless Tags and via Photosensing Wireless Tags and ProjectorsProjectors
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Radio Frequency Identification Tags Radio Frequency Identification Tags (RFID)(RFID)
microchip
Antenna
No batteries,
Small size,
Cost few cents
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
WarehousingWarehousing
RoutingRouting
Library Library
Baggage
handling
Baggage
handling
CurrencyCurrency
Livestock trackingLivestock tracking
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Micro Controller
Memory Computer
READER
Micro Controller
Memory
Conventional Passive RFID
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Tagged Books in a LibraryTagged Books in a Library
Id List of books in RF range
No Precise Location Data Are books in sorted order ?Which book is upside down ?
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Where are boxes with Where are boxes with Products close to Expiry Date ?Products close to Expiry Date ?
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
READER
Micro Controller
RF Data
Memory
Conventional RFID
Computer
READER
Micro Controller
RF Data
Light
Memory
Photosensor
Computer
Conventional RF tag
Photo-sensing RF tag
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
READER
Projector Micro
Controller RF Data
Light
Memory
Photosensor
Computer
Photosensor ?
Compatible with RFID size and power needs
Projector ?
Directional transfer,AR with Image overlay
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
b. Tags respond with date and precise (x,y) pixel location, Projector beams ‘Ok’ at that location
b. Tags respond with date and precise (x,y) pixel location, Projector beams ‘Ok’ at that location
a. Photosensing RFID tagsare queried via RF
a. Photosensing RFID tagsare queried via RF
c. Multiple users can simultaneously work from a distance without RF collision
c. Multiple users can simultaneously work from a distance without RF collision
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
RFID
(Radio Frequency Identification)
RFIG
(Radio Frequency Id and Geometry)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Prototype TagPrototype Tag
RF tag + photosensor
RF tag + photosensor
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
OutlineOutline
• Photosensing RF tags– Location sensing
• Geometric Operations– Multi-view analysis– Deformation
• Interactive Projection– Mouse-like interaction– Image stabilization
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
OutlineOutline
• Photosensing RF tags– Location sensing
• Geometric Operations– Multi-view analysis– Deformation
• Interactive Projection– Mouse-like interaction– Image stabilization
• Applications
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
Projected Sequential Frames
•Handheld Projector beams binary coded stripes
•Tags decode temporal code
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
00 11 11 00 00 X=12
X=12
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Visual feedback of 2D Visual feedback of 2D positionposition
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
a. Receive via RF {(x1,y1), (x2,y2), …} pixels
b. Illuminate those positions
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
LimitationsLimitations
• Line of sight~Surface patches~Multiple tags per object
• Ambient light~Modulated infrared
• Surface reflectance and shadows
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
OutlineOutline
• Photosensing RF tags– Location sensing
• Geometric Operations– Multi-view analysis– Deformation
• Interactive Projection– Mouse-like interaction– Image stabilization
• Applications
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Find 3D coordinatesFind 3D coordinates
a. Observe structured patterns via Camera
b. Triangulate cam/proj view
a. Observe structured patterns via Camera
b. Triangulate cam/proj view
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
3D from 2 Projector Views3D from 2 Projector Views(Structure from Motion)(Structure from Motion)
• Two+ unknown projector views• Correspondence is trivial• Applications
– Detect 3D deformations– Trajectory grouping
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Change Detection Change Detection without fixed camera, in any lighting conditionwithout fixed camera, in any lighting condition
Compare with new coordinates from a different
view
Record coordinates of tags from one view
Before After
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Texture AdaptationTexture Adaptation
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
OutlineOutline
• Photosensing RF tags– Location sensing
• Geometric Operations– Multi-view analysis– Deformation
• Interactive Projection– Desktop-like interaction– Image stabilization
• Applications
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Desktop-like InteractionDesktop-like Interaction
Selecting Selecting tagstags
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Support for handheld Support for handheld projectionprojection
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Mouse SimulationMouse Simulation
• Cursor follows handheld projector motion
• Pre-warped image remains stable
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Image Quasi-StabilizationImage Quasi-Stabilization
Eliminate hand jitter using inertial Eliminate hand jitter using inertial sensors+camerasensors+camera
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Absolute StabilizationAbsolute Stabilization
Image stays registered with world Image stays registered with world featuresfeatures
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Image StabilizationImage Stabilization
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Interactive ProjectionInteractive Projection
(Also in Emerging Technologies Booth)(Also in Emerging Technologies Booth)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Adaptive ProjectionAdaptive Projection
‘‘Paste’ : Geometric and Photometric Paste’ : Geometric and Photometric compensationcompensation
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Prototype Handheld Prototype Handheld ProjectorProjector
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Related WorkRelated Work• Smart objects
– Smart-its– FindIT Flashlight (Ma and Paradiso 2002)
• Location sensing – Multiple readers, large antennas– Olivetti Active Badge, Xerox PARCtab (Want 1995)
• Interaction and Display– Augmented Reality– Projector-based AR, Shaderlamps
(Raskar 1998, Underkoffler 1999, Pinhanez 2001)
– Handheld projector (Raskar 2003)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
ApplicationsApplications
• Single Tags– Authoring for AR, Store augmentation: display per
object
• Multiple tags on objects– 3D Pose– Packing and placement strategy– Robot navigation
• Interaction between tagged objects– History of geometric interaction– Orientation mismatch
• Distributed tags– Deformation– Interpolated values for temperature sensors
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
AcknowledgementsAcknowledgements
• MERL– Cliff Forlines– Joe Marks, – Dick Waters, Kent Wittenburg– Vlad Branzoi– Rebecca Xiong, Debbi van Baar
• Mitsubishi Electric, Japan– Mamuro Kato, – Keiichi Shiotani
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Interacting with a Self-Describing WorldInteracting with a Self-Describing World
• Hybrid optical and RF communication– Photosensing Wireless Tags + Projector illumination
• Geometric Analysis– Location, Selection,
Augmentation At a distance
• Interactive Projection– Desktop like interaction– Stabilized images
• RFIG Applications beyond logistics– Identity, Geometry, History, Annotation
www.MERL.com
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Find tag location using
handheld Projector
Find tag location using
handheld Projector
Photosensing
Wireless Tags
Photosensing
Wireless Tags
Many geometric ops Many geometric ops
R F I R F I R F I DR F I D
Interactive stabilized projection
Interactive stabilized projection
(Radio Frequency Id & (Radio Frequency Id & GeometryGeometry))GG
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
(End of Presentation)(End of Presentation)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Towards Passive RFIDTowards Passive RFID• Photosensor
– Most compatible with passive batteryless RFID– Power: near zero power for sensing– Size: Each photocell is tens of micrometer (smaller than RFID
which are about 500 micrometer)– Very low cost, can be built with silicon of RFID microchip– RFID+Photosensor can be size of grain of rice– RFIG tag is ‘visible’ to projector but not to humans– Current prototype uses battery because passive tags are difficult
to program
• Other possibilities– Adding a light emitting diode (LED)
• Requires on board power• Size has to be considerably large• Size and power requirements are high
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Future DirectionsFuture Directions
• Sophisticated tags• Light modulation
• Handheld projector
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Comparison with Bar-codesComparison with Bar-codes
• RFID– Barcodes take up more space– Long bit sequences– Multiple tags can be simultaneously read– Read distance is high
• RFIG– Multiple barcodes per object is cumbersome– Difficult to attach sensor (e.g. temperature)
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Locate Tags Precisely using a Locate Tags Precisely using a handheld scanner/projectorhandheld scanner/projector
•Handheld device
•Eliminates multiple RF readers for triangulation or large antennas
•Projector indicates location of tag
Illuminate tags with binary coded stripes with projector
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Photosensor added to RFID tags
Hybrid communication via RF and via light
Solve 3 main issues with conventional RFIDs
1. Locate tags precisely
2. Select a subset of tags for operation
3. Long distance operation without collision
Applications beyond inventory management
1. Warehouse: locate objects for specific queries
2. Training and maintenance: interactive augmented reality
3. Surveillance: keep trail of object locations
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
2. Select subset of tags for operation2. Select subset of tags for operation
Advantages
•Only tags which are illuminated respond to queries
•No RF collision between multiple tags
•Projector also use as a display to indicate result of operation
Technique
•Illuminate subset of tags with flashlight or projector
3. Long distance operation without RF collision3. Long distance operation without RF collision
Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,
Lee, Leigh, WillwacherR F I G Lamps
Interactive ProjectionInteractive Projection
• Desktop like interaction– Cursor follows use motion– Center pixel is the pointer– Rest of the image is stabilized– Selection
• Image stabilization– Absolute: camera and inertial sensor– Relative: world markers
• Copy-paste– Camera captures texture during ‘copy’– Projection after geometric/photometric adaptation