Image-Based Rendering using Hardware Accelerated

Dynamic Textures

Keith Yerex

Dana Cobzas Martin Jagersand

Motivation

Rendering

Traditional geometry based techniques:

detailed 3D model + texture

hard to achieve photorealism

Image-based models

non-geometric model from images

practically hard to apply

Challenges

Hard to generate detailed 3D models Texturing from images require very precise

alignment with the model

Rendering arbitrary views using IBM requires dense sample of the plenoptic function

IBR techniques don’t deal with dynamic scenes

Overview I1 It

…

= =

+ +

Warped texture

Motion params

(R1 a1 b1) …(Rt at bt)

Structure P

Texture basis

y1 … yt

Texture coeff

New pose

(R a b)

Training Model New view

uv

ô õ= RP + a

b

ô õ

I wt = Byt+ IöI wqt = I qt(W(ut;vt))

I wt =P

q=1Q I wqt

I t =P

q=1Q I qt

Structure from motion

Input – Structure from set of corresponding points tracked in a set of images

Assumptions Static scene Camera model

injective perspective weak perspective orthographic

Estimated modelprojective affine metric euclidean

Structure from motionTracked features

Structure from motion

algorithm

poses

structure

SFM algorithms

Few images, perspective camera, precise calibration

epipolar constraint

trilinear tensor

Long motion, affine or perspective structure

factorization methods

Metric structure

Weak perspective camera

Extension of Tomasi Kanade factorisation

algorithm

Extract affine structure Relation between the sffine structure and camera

coordinate frame Transform the structure into metric (unit pixel size)

Weak perspective projection

N points

Normalized with respect to centroid

Rank theorem

Factorization

uti = stiTtP i +atvti = stjTtP i +bt

W= RP + t1

Wê = RP where Wê =Wà t1

Wê = O1Î O2

Rê = O01

Pê= Î 0O02

Metric constraints

Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles

Model P ReprojectionPose x = (r,s,a,b)

R =RêQP =Qà 1Pê

iêTtQQ

Tiêt =jêTtQQ

Tjêt ( = s2t)

iêTtQQ

Tjêt =0 t 2 f1::Mg

[u;v]= sR(r)P + ab

ô õ

Dynamic Textures

Purpose

Model image intensity variations due to

1. Small geometric errors due to tracking

2. Non planarity of real surface

3. Non-rigidity of real object

4. Pose varying lighting effects

Non-geometric, mixing of spatial basis

Spatial Basis Intro

1. Moving sine wave can be modeled:

2. Small image motion

I = sin(u+at) = sin(u) cos(at) + cos(u) sin(at) = sin(u)y1+cos(u)y2

I = I 0+@u@IÉ u+@v

@IÉ v

Spatially fixed basis

2 basis vectors 6 basis vectors

Image Variability

Formally consider residual variation in an image stabilization problem

Optic flow type constraint

É I = I q(wê) à I (t)

É I = I (w+É w) à I w = I (f ) +@w@I (f )É wà I w = @w

@I (w)É w

Structural Image Variability

Affine warp function

Corresponding image variability

Discretized for images

uwvw

ô õ

= W(p;a) =a3 a4a5 a6

ô õ

p +a1a2

ô õ

É I a = @u@I ;@v

@Iâ ã 1 0 ãu 0 ãv 00 1 0 ãu 0 ãv

ô õ y1...y6

2

4

3

5

= [B 1. . .B 6][y1; . . .;y6]T = Baya

É I a =P

i=16

@ai@I wÉ ai = @u

@I ;@v@Iâ ã @a1

@u ááá @a6@u

@a1@v ááá @a6

@v

" # É a1...É a6

2

4

3

5

Composite Image variability

Similarily can show that composite image variability

Can be modeled as sum of basis

É I = É I s+É I d+É I n+É I l +É I e

É I = B sys+B dyd+B nyn+B lyl +É I e = By +É I e

Struct Depth Non-plan Light Res Err

Example Lighting variation

Statistical Image Variability

In practice image variability hard to compute from one image

Instead we use PCA to estimate image variability from a large sequence of images

This yields a transformed basis

Can estimate linear model J

In practice Delaunay triang & bi-linear model

É I = Bêyê

É yê = J É x

Image variability comparison

Derivatives from one picture

Statistically estimated variability

Implementation

Matlab for geometric modeling and prototyping

mexVision for tracking (30Hz frame rate)Hardware accelerated OpenGL for rendering

(2.8Hz in SW, 18Hz on GeForce 2)pthreads and pvm for parallel processing

MATLAB

meXVision OpenGL

Hardware rendering

Unsigned basis

Scaling to 8 bit

Where

I w(t) = B+y(t) à Bày(t) + Iö

Bà+ =255B+ðà 1

Bàà =255Bàðà 1

yà =255à 1ðy

ð= diag(maxjBj)

OpenGL==draw the meanBi ndTexture(Iö);DrawTr i angl es();==add basi s t exturesf or (each i)fSetBl endCoef f i ci ent (jyài(t)j);Bi ndTexture(Bà+i );i f (yài(t) > 0)SetBl endEquat i on(ADD);el se SetBl endEquat i on(SUBTRACT);DrawTr i angl es();Bi ndTexture(Bàài );i f (yài(t) > 0)SetBl endEquat i on(SUBTRACT);el se SetBl endEquat i on(ADD);DrawTr i angl es();

g

Example Renderings

Kinematic arm

Geometric errors

static dynamic

Geometric errors

static dynamic

Geometric errors

DynamicStaticStatic texturing Dynamic

Pixel error

Vertical jitter Horizontal jitter

Static texture 1.15 0.98

Dynamic texture 0.52 0.71

Conclusions

Coarse geometry tractable to estimateErrors from small geometric misalignments

compensated for using dynamic textureSystem runs on consumer PC with web cam

and game graphics cardApplications

Insert characters/objects into gamesVideo phone