an interactive background blurring mechanism and its applications

An Interactive Background Blurring Mechanism and Its Applications

NTU CSIE

Yan Chih-Yu

Advisor: Wu Ja-Ling, Ph.D.

2

3

Outline IntroductionRelated WorkMethod

◦Object Segmentation◦Depth Map Generation◦Image Defocus

Experimental ResultApplicationsConclusion

Introduction

Real world

Photo plane

Depth of field◦Circle of confusion

5

Photo plane

Introduction

Real light

Circle of Confusion

Out of focus

Depth of field◦Circle of confusion

Introduction Depth of field

Real light

Circle of Confusion- Readability range

Focus plane Photo planeOut of focus – blur range

Depth of field

Introduction

Real world

Photo plane

Shallow focus

8

IntroductionShallow focus

◦Highlight the subject by softening background diffusion

Deep focus by DC

NIKON E4300 (2003)

Shallow focus by DSLR

NIKON D90(2008)

Method

Lazy SnappingLazy Snapping

Alpha MattingAlpha Matting

Face DetectionFace Detection

Perspective BoxPerspective Box

Pop-up CardPop-up Card

Camera SettingsCamera Settings

Defocus BlurDefocus Blur

Privacy PreservingPrivacy Preserving

Photo BrowserPhoto Browser

Object Segmentation

Depth Map Generation

Applications

Image Defocus

10

Method

Segment◦Lazy snapping◦Mean shift◦Alpha matting◦Face detection

Resize to 20%

Do hard graph cut by lazy snapping

User Stroke

Mean shift

Do hard graph cut by lazy snapping

User Stroke

Resize to 20%

Mean shift

Segmentation tri-Map

Segmentation alpha Map

Lazy Snapping. ACM Trans. On Graphics 2004.Mean Shift: A Robust Approach Toward Feature Space Analysis. PAMI 2002.A Bayesian Approach to Digital Matting. CVPR 2001.

11

MethodDepth map

◦ Perspective Box Vanish point Rear wall

Depth Map

Perspective Box

Normal vector

Pop-up card

Depth Map

Perspective Box

Normal vector

Pop-up card

Tour Into the Picture: Using a spidery mesh user interface to make animation from a single image. SIGGRAPH 1997

12

MethodDepth map

◦ Pop-up card

Depth Map

Perspective Box

Normal vector

Pop-up card

Depth Map

Normal vector

Pop-up card

13

MethodDepth map

◦ Normal vector

Depth Map

Perspective Box

Normal vector

Pop-up card

Depth Map

Normal vector

14

Method

Image defocus◦Blur circle diameter

Blur circle diameter

Segment Map

Depth Map

Camera setting

Aperture sizeFocal length of the

lensDistance of focus

u v

bd

vbub

u v

bd

vbub

(a) (b)𝑏= 𝑓𝑁ȁ�(𝑢𝑏 − 𝑢)𝑓ȁ�𝑢𝑏(𝑢− 𝑓) 1𝑢+ 1𝑣 = 1𝑓

15

MethodDefocus blur

◦bokeh

𝐼𝑑𝑒𝑓𝑜𝑐𝑢𝑠 ሺ𝑖,𝑗ሻ= σ𝐸𝑛ሺ𝑖,𝑗ሻσ𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗) + 𝛼𝑛(𝑖,𝑗)

𝐸𝑛ሺ𝑖,𝑗ሻ= 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ

𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1 𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛,𝑠,𝑏)Τ

𝛼𝑛(𝑖,𝑗) =൜𝑤𝑒𝑖𝑔ℎ𝑡𝑛(𝑖,𝑗), 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ< 𝑇ℎ𝑏𝑜𝑘𝑒ℎ0, 𝐼ሺ𝑥𝑛,𝑦𝑛ሻ≥ 𝑇ℎ𝑏𝑜𝑘𝑒ℎ

𝑤𝑒𝑖𝑔ℎ𝑡𝑛ሺ𝑖,𝑗ሻ= 1𝑎𝑟𝑒𝑎 𝑜𝑓 𝑅𝑒𝑔𝑖𝑜𝑛ሺ𝑥𝑛,𝑦𝑛,𝑠,𝑏ሻ× 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛(𝑥𝑛,𝑦𝑛)𝑙𝑢𝑚𝑖𝑛𝑜𝑢𝑠(𝑥𝑛,𝑦𝑛)

……

……E1(i,j)

I(x1,y1) I(x2,y2) I(x3,y3)

I(xn,yn) I(xn+1,yn+1)

E2(i,j) E3(i,j)

En(i,j) En+1(i,j)

En+2(i,j)

I(xn+2,yn+2)

16

MethodDefocus blur

……

……E1(i,j)

I(x1,y1) I(x2,y2) I(x3,y3)

I(xn,yn) I(xn+1,yn+1)

E2(i,j) E3(i,j)

En(i,j) En+1(i,j)

En+2(i,j)

I(xn+2,yn+2)

17

Method

◦Near-by object case

Focus on the flower

Focus on the flower

18

◦Near-by object case

Method Blur circle diameter

Objects in front of the

focus

Objects behind

the focus

Defocus Blur

Text synthesis inpainting

Blur shape as alpha map

Defocus Blur

Interpolation by alpha map

Shallow focus image

19

Experimental Result 1Defocus blur method proposed in the system

comparing with other blur filter results

Deep focus Shallow focus

20

Experimental Result 1Defocus blur method proposed in the system

comparing with other blur filter results

Gaussian blur Defocus blur Defocus blur + Bokeh

21

Experimental Result 1

22Defocus blur

Experimental Result 1

Gaussian blur Defocus blur + Bokeh

Original photograph

• Defocus blur method proposed in the system comparing with other blur filter results

23

Experimental Result 2with / without depth variation in the

background

Real photograph taken by DSLR

24

Experimental Result 2with / without depth

variation in the background

Original photograph

Result after post-processing

Depth variation

Depth fixed

25

Experimental Result 3

◦Near-by object case

Focus on the flower

Focus on the flower

26

◦Near-by object case

Without inpainting

Interpolation result

Experimental Result 3

27

Experimental Result 4Comparing with Photoshop

Photoshop (1 Hour) Our system (3~5min)

Original image

28

Applications - Privacy Preserving

29

Applications - Partial Viewing

30

Applications - Partial Viewing

31

Applications - Image Transition at Photo Browser

32

Applications - Image Transition at Photo Browser

33

Movie with defocus blur

effect

Applications - Image Transition at Photo Browser

34

Applications - Image Transition at Photo Browser

Concatenate two unrelated

images

35

ConclusionWe proposed an interactive refocusing

tool for background blurring◦ Simple user hint◦ Defocus blur kernel ◦ Concatenate two related picture

Future work◦ concatenate two or more unrelated

pictures Color based image retrieval technique

36

Q&A

37

Experimental Result 1Defocus blur method proposed in the system

comparing with other blur filter results

Deep focus Shallow focus Difference

38

Experimental Result 1Defocus blur method proposed in the system

comparing with other blur filter results

Shallow focus

Difference

Defocus blurPSNR:22.79

Gaussian blurPSNR:20.34

39

END

THANK YOU