image based rendering(ibr) jiao-ying shi state key laboratory of computer aided design and graphics...
TRANSCRIPT
Image Based Rendering(IBR)
Jiao-ying ShiState Key laboratory of Computer Aided Design and
Graphics
Zhejiang University, Hangzhou, China
http://www.cad.zju.edu.cn/home/jyshi
Survey on Image Based Rendering (IBR)
PART I
Traditional Computer Graphics
Use Geometry and lighting model to simulate the imaging process and generate realistic scene
– No Guarantees for the rightness of the models– A lot of computation time needed
Use of images In Computer Graphics
Texture Mapping Environment map
• How about more images?
Computer vision
Extract Geometry model from real scene(photos)
Combined with Computer Graphics:
Image based renderingBypass the “model”,driectly from real image to s
ynthesized image
Image Based Rendering
Images
Geometry
Images
Computer
vision
analyze
Computer
Graphics:
simulate
Image based
rendering
A Framework of Image Based Rendering
Real
Scene
Sampling System
Data
Storage
System
Data representation
System
Rendering
System
Synthesized view
The Key Part of IBR
The data representation system is the key part of IBR, It determines the other three subsystems.
-A taxonomy based on the data representation system
A Taxonomy of IBR The Geometry based data representation The Image based data representation The plenoptic function based data representation
The Geometry based data representation Geometry elements used as data
representation in IBR:– polyhedra(Debevec, et. al 1996)– layers (Baker, Szeliski and Anandan 1998)– points(Shade et al. 1998)
Similar to Traditional Computer Graphics, except the geometry model comes from images
General working processImage User input
stereo
Geometry
Interactive
modeling
Ranger
3D Warping
Rendering
Image based data representation
data are treated as a series of images with correspondence relations
“optical flow” ”morphing map” “Trifocal/Trilinear tensor” are used to control the generation of novel image
forward/ reverse mapping;morphing
Examples: View interpolation (Chen and William,1993)
View Morphing(Seitz and Dyer 1998)
General working process
Images User input
Correspondence relations
Existing geometry model
(Synthetic images only)
Stereo
2D image warping
rendering
Plenoptic function based data representation Plenoptic function (Adelson and Bergen,1991)
),,,,,,( tVVVPlenoptic zyx
General working process
images
image
processingstereo
resampling
rendering
plenoptic function
User Input
Representative IBR methods based on Plenoptic Functions
Plenoptic Modeling: 5D Light field/Lumigraph: 4D Concentric Mosaics : 3D Panorama: 2D
IBR
The Geometry based data
representation
The Geometry based data
representation
The Plenoptic function based
data representation
IBR data are composed of
geometry elements
IBR data are composed of a
set of images with
correspondence relations
IBR data are composed of
a set of light rays
polyhedra
layers
points
View interpolation[CW93]
View morphing[SD96]]
Transfer mode [LF94]
Plenoptic
modeling[MB95]
Lightgield [LH96]/
Lumigraph[GGSC96]
Concentric
Mosaics[SH99]
Panorama[Chen95],[SS97]
MCOP images[RB98],
LDI[SGHS98]
Depth based [BSA98]、Motion based [LS97]、TIP [HAA97] etc.
Hybrid approach of
geometry and image
[DTM96]
5D plenoptic
function
4D plenoptic
function
3D plenoptic
function
2D plenoptic
function
Conclusion
The progress of IBR technique is also the progress of new data representation method, We treat an image:
– as texture in geometry texture mapping– as images with correspondence relation
view interpolation /morphing– as light beams light field– as slit image concentric mosaics
...
The study on slit images in image based rendering
PART II
The concept of slit images
The slit image is a kind of 1-D image with width only 1 pixel.
An example of slit image
Previous work based on slit images
MCOP images concentric mosaics
The advantage of using slit images
Most computer graphics technology is used to simulate human motion and observing usually only in 3 DOF:
The walk through task in virtual reality applications requires human motion only in 3 DOF:
Left/right, forward/backward and look around.
The representation of slit images
A slit image is identified by the camera 2D position and orientation (azimuth angle)
in polar coordinates
in Cartesian coordinates
S(x,y,θ)
φ
ρ θ
),,( S
(x,y)
θ
Slit image sets(I)
A scene view at position (ρv, φv), with azimuth θv and horizontal FOV ω:
Sv=
Panorama at position (ρp, φp)
Sp =
2/2/,,|),,( vvvvS
ppS ,|),,(
Slit image sets(II) Concentric mosaic with its center at origin :
Sc ={S(ρ, φ,θ)|θ=-π/2 orθ=π/2 , ρ≤R}
– (camera alone normal direction)Scn ={S(ρ, φ,θ)|-ω /2<θ< ω/2 , ρ=R}
– (camera alone tangential direction)Sct ={S(ρ, φ,θ)|-ω /2 + π/2 <θ< ω/2 + π/2 , ρ=R}
moving straight forward from origin, with horizontal FOV ω
{S(x , y,θ)|y=0, x>0, -ω /2<θ< ω/2 }
Slit image field
Slit images that captured at any position and any azimuth inside a 2D region.– Inside a circle:
{S(ρ, φ , θ)|ρ≤R}– Inside a rectangle:
{S(x, y,θ)| x 1≤x≤ x 2 , y1≤y≤ y2}
From the slit image field we can generate the walk-through scenes inside th region just by resampling
Analogical Slit Images
f
hdhn
dddn
H
Cn Cd
f
M
O
md
mn
on od
Object
in scene
hr
dr
Cr
for
mr
Relations between analogical slit images
rcr
crr
dcd
cdd
d
Hf
Z
fYy
d
Hf
Z
fYy
Let hd=| yd| , hr=| yr|
or
d
r
r
d
d
d
y
y
d
r
r
d
d
d
h
h
r
rd
d
dr
d
dd
h
hh
and
let and
– analogical slit images are highly coherent– slit images can be synthesized by their analogical slit image
Relations between analogical Slit Images
)1)1/(( rd
nddn offset
offsetkhh
d
r
r
dd
d
h
h
d
n
n
dd
d
h
h
dnnd ddoffset drrd ddoffset rd hhk /
Analogical relation of slit images
LR
S1
S2
Analogical relation of slit images is
– reflexive
S1 ~S1
– symmetric
if S1 ~S2 , there will be S2 ~S1
– transitive
if S1 ~S2 、 S2 ~S3 , there will be S1 ~S3
Written as S1 ~S2
So analogical relation of slit image is an equivalence relation
Analogical slit image set
Slit images that are analogical each other are consisted to be a analogical slit image set.
Analogical relation is a kind of equivalence relation
an analogical slit image set is a partition of slit image field
A slit image field can be obtained approximately by limited sampling Each analogical slit image set can be approximated by one or a
few its member silt images The set of slit image sets can cover the slit image field. A slit image field can be approximated by limited sampling
Depth correction for Concentric Mosaics
-A slit image segments based approach
Application of analogical slit images
Motivation
In concentric mosaics, only one slit image is captured for every analogical slit image set. And this slit image is simply used as substitution for all its analogical slit images. Distortion caused
find the pixel relations between analogical slit images and correct the distortion of images
Slit image segments
Definition: a slit image segment consists of a series of adjacent pixels in one slit image which have either similar color or similar depth. Segment is used as primitive of image.
Applications: use segment mapping instead of pixel mapping between analogical slit images
Advantages:
– Reduce big amount of data.– Segment is used as basic block in VQ compress
ion
How to segment slit images
Analyze 2 analogical slit images Initial segment
– Find edge point of slit image
Warp slit image segment of one slit image to its analogical slit image, find the best segmentation and correspondence relations between two slit images.
Data slit image and reference slit image
In the 2 slit images:– one is used to synthesize novel view, called
data slit image.– The other is used to find the best segmentation
and define the segment mapping of data slit image, called reference slit image.
Implementation
Capturing Slit Images using normal camera Calibration between concentric mosaics Slit Image Segments Matching Synthesizing novel view
Capturing slit images using normal camera
'
ω /2
R’
Rω /2
R’
R
inward Setup outward Setup
Capturing two set of Concentric Mosaics
θ d
θ r
O
Pr(Rr,φ r)Pd(Rd,φ d)
φ r=φ d=0
θ r
θ d
Pd(R,φ d)
Pr(R,φ r)
φ r=φ d=0
a)Same direction setup b)Opposite direction setup
Possible errors
Δ φ
O φ r=φ d=0
Δ θ
eRd
O
Lead to wrong “analogical” slit images
Calibration between concentric mosaics
– Estimate the errors parameters so that we can find the correct analogical slit images.
smallΔθ is treated asΔφ for simplification. Only consider the relative error e of R 。
Calibration between concentric mosaics Method:– analogical slit images should be alike– select a set of slit images in one CM, calculate their analogical slit images in another CM with the
consideration of introduced error parameters.
Sadjis the set of slit images in one CM for calibration use
Conform() is a likelihood measurement between data and reference slit images.
S
RSReRReRRSconformadjdS
ddddrdddrrdddrrre
)),,()),,,,,,(),,,,,,(,((max,
Calibration in the Same Direction Setup
Two error parameters notice when |θd| is small, e has only small effect to θr andφr .
De-coupling: Select the slit images with small |θd|, estimate Δφ, then estimate e.
))1(
)(arcsin(sinr
ddr
drdr
R
Re
Calibration in the Opposite Direction Setup
dr
drdr
Only need to estimate 1 error parameter
Preprocessing
Edge detection inside slit image: find the initial segment
warp the initial segments to its analogical slit images, find the best segmentation and correspondence relations between two analogical slit image.
Generate corrected image
θ n
θ d
O
Pd(Rd,φ d)Pn(ρ n,φ n)
φ d=0
2/2/,,|),,( vvvvS
||
)sin()sin(sin
nd
dn
d
n
n
d
nddn
PPR
Desired Slit image Set: )1)1/(( rd
nddn offset
offsetkhh
Generate images from known slit image segment relations
between analogical slit images
Result
Panoramic mosaics of slit images with depth
Panorama Method (Chen, 1995)
Only several picture captured at a viewpoint needed, small data size and easy to sampling.
The only off-the-shelf IBR technological for large scene althoughalthough
Fixed viewpoint, can only look around and zoom in / zoom out, or hop between viewpoints
Data size Vs. Motion range in IBR
Small data size very limited DOF of virtual
camera more DOF huge data size
of virtual camera
Slit images with depth
Assume a uniform depth value is
used for every slit image
Panoramaic mosaics of Slit image with depth
recover or assign depth
recover depth from correspondence relations between analogical slit images– search correspondence points– interactive assign correspondence points
recover depth
Depth may be got from a known map
10001
100 ddh
hh
ddd
Interactive Rendering: Finding Slit Images
φ s
φ n
λ
ρ n
ρ sθ
Novel
viewpoint
Slit images with
depth
dn
O
2/2/,,|),,( vvvvS
sn
n
sn
ns d
sin)sin()sin(
Interactive Rendering: Adjusting
Looming effects simulation– scale slit images uniformly
fill holes– fill holes using nearby slit images
s
n
n
s
d
d
l
l
Sample multiple panoramic mosaics of slit image with depth
Join multiple mosaics together
Join multiple mosaics together to achieve a wider motion range of virtual point
Specify reference points
reference circle
reference point
Map slit images to reference point
reference circle reference point
slit images with united depth
Generate novel view
reference
point
virtual camera
postion
Implementation
Sampling– capture slit images– recover or assign depth
Preprocessing– mapping slit images to reference circle
Interactive Rendering
Synthesized view
Move forward and
backward
Move left and right
Advantages and Disadvantages
Advantages– 3 DOF (move left and right, forward and
backward, look around)for the virtual camera with small data size
– multiple mosaics can be joined up smoothly
Disadvantages– only fit for those scene depth variation is small
along the vertical direction scene or for open scene
Conclusion
For 3 DOF motion, slit image is a good data representation for scene
We studied the slit image proprieties and introduced the following 3 concepts:– analogical slit images– analogical slit image set– slit image field
Conclusion
Applications of slit image concepts– Use of slit image segments to correct vertical
distortion of concentric mosaics– A new IBR method: panoramic mosaics of slit
image with depth