special effects techniques - nus computingcs4340/documents/tricks.pdf · 20 cs5245: vision &...

1

Special Effects TechniquesView Morphing,

Inpainting,Texture Synthesis,

Day for Night

2

CS5245: Vision & Graphics for Special Effects

Agenda

View MorphingOther Special Effects Techniques

InpaintingTexture SynthesisSimulating Night Scenes

3


(ref: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

View morphing is not image morphing. Image morphing encounters distortion when:

• cameras are not parallel

• scene has depth (i.e. 3D variations)

On the other hand, view morphing does not cause distortion

View Morphing

4


Source: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)

Image Morphing Example

5


Source: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)

Example 2Top: image morphing resultBottom: view morphing result

6


Let

I0 be input image from camera 1I1 be input image from camera 2

Is be the new image to be synthesized, with viewpointlying along the straight line joining the optical centres of cameras 1 and 2

Π0 be the projection matrix of camera 1 Π1 be the projection matrix of camera 2Πs be the projection matrix of camera 3 (virtual camera)

Theory: Parallel Views

7


8


Π, a projection matrix, can always be written as

[ ]HCH −=Π

where C is the optical centre of the camera, and H is 3x3 matrix containing the position & orientation info

Express a 3D scene point [x y z] as homogeneous coordinates

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

1zyx

P

Express the corresponding image point [u v] as homogeneous coordinates

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

1vu

m

9


In other words, Pm Π=

For parallel views, we can w.l.o.g. assume that the 1st camera is at the origin with focal length f0, and 2nd camera is at (CX, CY, 0) with focal length f1. Hence:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=Π

000

1000000

0

0

0 ff

and

Interpolating image coordinates gives the new image point

PZ

PZ

sPZ

smsms

sΠ=

Π+Π−=+−

1

11)1()1( 1021

where10)1( Π+Π−=Π sss

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−−

=Π0100

0000

1

1

1

1

1 Y

X

CfCf

ff

10


Therefore, when camera image planes are parallel, view morphing means a linear interpolation of camera projection matrices. No distortion in this case.

In fact, represents a camera at position Cs with focal length fsgiven by:

sΠ

10)1()0,,(fsfsf

sCsCC

s

YXs

+−==

11


What happens if camera image planes are not parallel?

[ ]1111 CHH −=Π [ ]2222 CHH −=Π

In this case, additional steps are required.

12


Non-parallel Views

13


Step 1: Pre-warp

Note that H0 and H1 are obtained by solving

where F is the fundamental matrix between I0 and I1

and

Step 2: Linearly interpolate positions and intensities of corresponding pixels in and to form

Step 3: Post-warp

11

11 IHI −=01

00 IHI −=

0I 1I sI

sss IHI ˆ=

FHFH T ˆ)( 10

11 =−−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−=010100

000F

14


Some view morphing examples:

15


Dynamic View Morphing

Given two views of a dynamic scene, taken by different cameras in different locations at different times, create a sequence of views showing a smooth transition between the two camera views that also shows the dynamic scene changing smoothly.

http://www.cs.wisc.edu/~rmanning/homepage/research/view.interpolation.html

16


Often you need to:

Special effects:Remove objectsFill holesReplicate patternTransfer textureShoot night scenes

TechniquesInpaintingTexture synthesisDay for Night

17


InpaintingObject Removal by Exemplar-Based InpaintingCriminisi, Perez & Toyamahttp://www.research.microsoft.com/%7Eantcrim/papers/Criminisi_cvpr2003.pdf

18


Idea

Fill target region starting from its boundary and working inwards.Fill order is important: try to preserve strong edges (isophotes)

19


3 steps

Which patch to fill first?Fill patch with what?Update confidence values

20


Which patch to fill first?

Given a patch Ψp centred at the point p for some p ∈ δΩits priority P(p) is defined as the product of two terms:

P(p) = C(p)D(p).

We call C(p) the confidence term and D(p) the data term,where |Ψp| is the area of Ψp, α is a normalization factor(e.g., α = 255 for a typical grey-level image), and np is aunit vector orthogonal to the front δΩ in the point p.

21


Which patch to fill first?

C(p) measures amount of reliable info surrounding p.

Intention is to fill first those patches which have more of their pixels already filled

D(p) measures strength of isophotes hitting boundary δΩ

Encourages propagation of linear structures (e.g. edges)

22


Fill with what?

After computing priorities, patch with highest priority is chosen.Then select patch in source region Φ that is most similar to chosen patch:

d( , ) is sum-squared-difference of pixels

Now fill empty region in with corresponding pixels in source patch

23


Update confidence

After filling patch , update confidence in the area by:

Repeat 3 steps until target region Ω shrinks to nothing

24


Results

(a) Original image(b) Target region manually selected(c) Result of filling by concentric layers(d) Result of their algorithm

25


Results

26


More results

27


28


Other inpainting papers

www.inf.ufrgs.br/~oliveira/pubs_files/inpainting.pdf

www.iua.upf.es/~mbertalmio/restoration.html

29


Texture Synthesis

How to get from this

To this?

30


Texture Synthesis

Image Quilting for Texture Synthesis and Transfer A. Efros and W. Freemanhttp://graphics.cs.cmu.edu/people/efros/research/quilting.html

Basic Idea:Cut up input image into smaller blocksCleverly choose blocks and join themTake care of the boundaryhttp://graphics.cs.cmu.edu/people/efros/research/quilting/efros-siggraph01.ppt

31


The Goal of Texture Synthesis

Given a finite sample of some texture, the goal is to synthesize other samples from that same texture

The sample needs to be "large enough“

True (infinite) texture

SYNTHESIS

generated image

input image

32


The Challenge

Need to model the whole spectrum: from repeated to stochastic texture

repeated

stochastic

Both?

33


Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

34

CS5245: Vision & Graphics for Special Effects min. error boundary

Minimal error boundaryoverlapping blocks vertical boundary

__ ==22

overlap error

Error surface e

35


Min. Error Cut Algorithm

For row i = 2 … NFor column j = 1 … M

(Efficiently solved using Dynamic programming)

In last row, search for minTrace this back up to 1st row for best path

i

j

36


Our Philosophy

The “Corrupt Professor’s Algorithm”:Plagiarize as much of the source image as you canThen try to cover up the evidence

Rationale: Texture blocks are by definition correct samples of texture so problem only connecting them together

37


AlgorithmPick size of block and size of overlapSynthesize blocks in raster order

Search input texture for block that satisfies overlap constraints (above and left)

Easy to optimize using NN search [Liang et.al., ’01]Paste new block into resulting texture

use dynamic programming to compute minimal error boundary cut

38


39


40


41


Portilla & Simoncelli

Wei & Levoy Image Quilting

input image

Xu, Guo & Shum

42


Failures(ChernobylHarvest)

43


Texture TransferTake the texture from one object and “paint” it onto another object

This requires separating texture and shapeThat’s HARD, but we can cheat Assume we can capture shape by boundary and rough shading

Then, just add another constraint when sampling: Then, just add another constraint when sampling: similarity to underlying image at that spotsimilarity to underlying image at that spot

44


Modified error

Weighted sum of overlap error and correspondence map error (at current location (x,y))α controls the contribution of each termCorrespondence map could be intensity (pixel value) or anything else.

2),(),(

221 ][)1(][ target

yxsource

yxovov CCBBe −−+−= αα

45


++ ==

++ ==

parmesan

rice

46

+ =

47

+ =

48


Sourcetexture

Target image

Sourcecorrespondence

image

Targetcorrespondence image

Blurred intensities

49


++ ==

50


Other papershttp://graphics.cs.cmu.edu/people/efros/research/EfrosLeung.htmlhttp://www.debonet.com/Research/TextureSynthesis/http://research.microsoft.com/research/pubs/view.aspx?msr_tr_id=MSR-TR-2000-32http://www.comp.nus.edu.sg/~cs4243/demo/textsyn/textsyn.html

Benadryl Landscapers Commericalhttp://www.theorphanage.com/ocp/portfolio/recent/1005

51

OBJECTIVE: Transform a ‘day image’ into a ‘night’ version of it including the loss of acuity at low luminances.

+ desired luminance level =

Day for nightVisual Acuity in Day for Night, by G. Haro, M. Bertalmio, V. Caselles2nd IEEE Workshop on Variational, Geometric and Level Set Methods in Computer Vision http://www.tecn.upf.es/~gharo/Download/upc.ppt

52


Night vs. Day

Traditionally done using blue filters.Not realistic

Night scene is not simply a darkened version of day scene

Colors are different (rods vs. cones)Contrast lowerLoss of details (visual acuity)

53


TRANSFORMATION IN 5 STEPS

1. Estimation of reflectance values and modification of illuminant.

2. Modification of chromaticity.

3. Modification of luminance.

4. Modification of contrast.

5. Loss of acuity: Diffusion.

Day for night algorithm

54


Emulating human vision at night.

Emulating a photographic film (γn=3, γ=1).

Simulated scene at 0.3 log cd/m2

Results

55


Without changing the variance, a=1

Increasing the variance, a=0.1

Simulated scene at 0.1 log cd/m2

Results

56


Results

Day scene

Simulated night scene

special effects techniques - nus computingcs4340/documents/tricks.pdf · 20 cs5245: vision &...

Documents