School of Electrical Engineering and Computer Science

Kyungpook National Univ.

Edge-Preserving Decomposition for

Multi-Scale Tone and Detail Manipulation

ACM Transactions on Graphics,

Vol. 27, No. 3, 2008

Zeev Farbman, Raanan Fattal, Dani Lischinski and Richard Szeliski

Presented by Bong Seok Choi

Abstract

Proposed method

– Construction of edge-preserving multi-scale image

decomposition

• Base detail decomposition

– Based on bilateral filter

• Using edge-preserving smoothing operator

– Based on weighted least squares optimization frame work

2 / 47

Introduction

Application of edge-preserving image smoothing

– Decomposing image into base layer and detail layer

• HDR tone mapping

• Flash/no-flash image fusion

• Transfer of photographic look

• Image editing

– Spatial scale of details captured by detail layer

– Operating detail at variety of scales

3 / 47

4 / 47

Fig. 1. Multi-scale tone manipulation. Left: input image. Middle: results of (exaggerated)

detail boosting at three different spatial scales. Bottom: final result, combining a somewhat

milder detail enhancement at all three scales

– Operating on images at multi scales

• Using multi-scale decomposition

– Laplacian Pyramid

» Using linear fililter

» Producing halo artifact near edge

– Application to tone mapping

• Multi-scale decomposition for reducing halo artifact

– Using non-linear edge-preserving smoothing filter

» Anisotropic diffusion

» Weighted least squares

» Bilateral filter

5 / 47

proposed method

– Using edge-preserving operator

• Based on weighted least squares framework

– Used to reduction ringing in deblurring images in noise

– Using smoothing propagation of sparse constraints

• Well-suited for coarsening of image

• Extraction of detail at various spatial scales

– Application to edge-preserving operator

• Tone mapping

• Detail enhancement

• Contrast manipulation

6 / 47

Background

Multi-scale image decomposition

– Using base layer and detail layers

• Base layer

– Larger scale variations in intensity

– Applying edge-preserving smoothing operator in image

• Detail layer

– Difference between original image and base layer

7 / 47

– Application of multi-scale image decomposition

• Reducing dynamic range of HDR

– Capturing Base layer by non-linear compressive mapping

– Recombination with detail layer

– Process for shape and detail enhancement

• Image and video stylization and abstraction

– Discarding details

» Retaining detail region of interest

» Abstraction in background

– Achieving stylized look for base layer

8 / 47

Image coarsening process with base layer

– Purpose of coarsening

• Avoiding artifact from base and detail layers manipulation

– Causing artifact by base and detail component

• Demonstrating blurring and sharpening of edges in

coarsening image

– Causing ringing in detail layer

» Manifesting halo and gradient reversal

– Unfitness Linear filtering and segmentation for computing

base-detail decomposition

9 / 47

10 / 47

Fig. 2. Artifacts resulting from edge blurring (left) and edge sharpening (right). The original

signal is shown in gray, and the coarsened signal in blue. Boosting the details (red) and

recombining with the base results in halos and gradient reversals (black).

Application of Edge-preserving smoothing operator

– Use in tone mapping

• Introducing LCIS

• Using variant of anisotropic diffusion

– Smoothing and preserving crisp edges between smooth region

– Application to multi scale image and edge detection

– Drawback of anisotropic diffusion

» Producing over sharpen edge

» Slowly converging non-linear iterative process

11 / 47

Improvements to anisotropic diffusion

– Using bilateral filter in image processing

• Effecting bilateral filter

– Smoothing small changes in intensity

– Preserving strong edge

• Non-linear filter

• Using pair of gaussian kernel function

• Weights decreasing both with spatial distance and different

in value

12 / 47

1( )

s rp p q q

qp

BLF g G p q G g g gk

(1)

s rp p q

q

k G p q G g g (2)

where is an image, subscripts and indicate spatial locations of pixels

kernel function and are typically gaussians

determines spatial support, controls sesitivity to edges

g p q

sG r

G

sr

– Limitation of bilateral filter

• BLF trade off edge preservation and smoothing abilities

– As scale of extracted details increase

» BLF tend to blur over more edges

» Producing halo artifact

– Demonstrating limitation of bilateral filter

• Input image

– Contains Several step edges of different magnitude

– Contains noise at two different scales

• Visualization of image intensity by color map

13 / 47

Fig. 3. Filtering a set of noisy step edges (constant regions) with a variety of coarsening

filters. Left : input image , Right : visualization

• Application of linear Gaussian filter

– Using small spatial kernel and large spatial kernel

– Result of filtering image

» Removing fine scale noise

» Blurring step edge

14 / 47

Fig. 3. Filtering a set of noisy step edges (constant regions) with a variety of coarsening

filters.

: 4sGaussian : 12sGaussian

• Application of Bilateral filter

15 / 47

: 4, 0.15s rBLF

: 12, 0.45s rBLF: 12, 3s rBLF

: 12, 0.15s rBLF

Fig. 3. Filtering a set of noisy step edges (constant regions) with a variety of coarsening

filters.

• Application of WLS method

– Preserving step edge

– Without introducing artifact

16 / 47

Fig. 3. Filtering a set of noisy step edges (constant regions) with a variety of coarsening

filters.

: 1.2, 0.25WLS : 1.8, 0.25WLS

Researching to Shortcoming of bilateral filter

– Survey of bilateral filter

• Elad ; Buades et al.

– Handling BLF use to piecewise linear function

• Choundhury and Tumblin et al.

– Using trilateral filter

– Introducing artifact in sharp features

• Durand and Dorsey et al.

– Describing variant designed specifically to avoid halos in thin,

high contrast feature

• Bae et al.

– Manipulating detail layer

» Detecting and fix reversed gradient

– Drawback of previous research

• Representation of wide halos

17 / 47

Edge-Preserving Smoothing via WLS

WLS optimization framework

– Goal of edge-preserving smoothing operator

• As close as possible input image

• As Smooth as possible everywhere

– Excepting across significant gradient in input image

– Expressing goal of optimization

• Data term Minimize distance between and

• Achieving Regularization term for smoothness

18 / 47

222

, ,p p x p y p

p p p

u uu g a g a g

x y(3)

g u

where, input image , new image , subscript denotes spatial location of

pixel. Smoothing weights and , increasing value of result in

progressively smoother image u

pg u

xaya

• Using matrix notation

• Minimizing vector in eq.(4)

– Solution of linear system

– Implementing and for forward difference operators

– Implementing and for backward difference operators

19 / 47

T T T T T

x x x y y yu g u g u D A D u u D A D u (4)

where and are diagonal matrices containing smoothness weights

and , and are discrete differentiation operator xA

yA xa g

ya gxD yD

u

gI L u g (5)

where , is five point spatially inhomogeneous

Laplacian matrix

T T

g x x x y y yL D A D D A D gL

xD yDT

xDT

yD

• Definition of smoothness term

– Exposition of complete WLS-based operator

• Considering relationship between value of parameter and

degree of smoothing

– Using linear spatially invariant smoothing filter

– Doubling spatial support of kernel

» Making filter roughly twice narrower in frequency domain

20 / 47

11

, ,,x p y pa g p a g px y

(6)

where is log-luminance channel of input image ,exponent (typically

between 1.2 and 2.0) determines sensitivity to gradients of .

is small constant (typically 0.0001) that prevent division by zero

gg

• Spatial invariant operator

– New image obtain form input image by non-linear operator

» hard to analyze frequency domain

» Not contain significant edges

– Roughly constant region and smoothness weights

• Frequency response of

21 / 47

1

gu F g I L g

F

(7)

1F g I aL g (8)

g

T T

x x y yL D D D Dwhere is ordinary(homogeneous) Laplacian matrix

F21 1 aF (9)

Multi-scale edge-preserving decomposition

– Construction of multi-scale edge preserving

decomposition

• Consisting decomposition

– Coarse

– Piecewise smooth

– Capturing detail at progerssively finer scales

• Construction of -level decomposition

22 / 47

1k

1i i id u u (11)

where denote input image, denote progressively coarser version

of . will serve as base layer with detail layers.

1,..., ku ugg ku b k

0u g

• Recovering original image from decomposition

– Adding up base and detail layers

23 / 47

1

ki

i

g b d (12)

– Computing progressive coarsening sequence

• First method

– Solving linear system (5) times

– Each time increasing value of parameter

• Second method

– Applying iterative at operator

» Smoothing image repeatedly

» Similarly to mean shift filtering and multi-scale bilateral

transform

24 / 47

1,..., ku u

k

1i

i

cu F g (13)

1i

i i

cu F u (14)

25 / 47

Fig. 4. WLS-based multi-scale decompositions. Left column: three levels computed

using eq. (13). The left half of each image shows the coarsening, while the right half

visualizes the corresponding detail layer. The spatial scale of the details increases from

one level to the next

Input image

1.2, 0.1 1.2, 0.8 1.2, 6.4

26 / 47

Fig. 4. WLS-based multi-scale decompositions. levels computed using eq. (14). The left

half of each image shows the coarsening, while the right half visualizes the corresponding

detail layer. The spatial scale of the details increases from one level to the next

1.8, 0.2 1.8, 0.8 1.8, 3.2

Comparison

Comparison of WLS and previous schmes

– Chen et al.

• Computing bilateral pyramid for video abstraction

• Producing smoothing region

• Blurring of strong edges

• Comparison of WLS

– Chen`s method

» Smoothing edges in large and small feature

» Generating ringing in detail signal

– WLS method

» Eroding same edges

» Eroding faster in small feature than large feature

» Without noticeable ringing 27 / 47

– Fattal et al.

• Applying bilateral filter to previous image

• Reducing range parameter at each iteration

– Ensuring preserve edge in previous level

• Drawback of Fattal`s method

– Manipulation of detail layer

» Unable to remove, suppress, and emphasize detail

– Over-sharpening of edges

» Producing thin gradient reversal artifact

28 / 47

– Comparison of previous methods

• Comparison by 1D section of image

– Containing large feature in left half of image

– Containing narrower feature in right half of image

29 / 47

Fig. 5. Progressively coarsening a signal using different edge-preserving schemes. The

coarsened versions are shown superimposed on the signal (using different shades of blue:

lighter is coarser). The corresponding detail signals are plotted in shades of red below.

• Comparison by real image

30 / 47

Fig. 6. Coarsened images and their corresponding detail layers for several different edge-

preserving filtering schemes. Coarsening progresses from top to bottom. The bilateral filter, LCIS,

and the trilateral filter exhibit ringing in the detail layer (easiest to see in the bottom row). [Fattal

et al. 2007] retains many small features even in the coarsest image, which never make their way

into the detail layer.

Connections with Other Operation

Analyzing mathematical relationship between

various edge-preserving operator

– Expressing Edge-preserving smoothing operator

• Smoothing process as spatially-variant filter

– Applying operator to input image vector

– Each row of thought as kernel

» Affect to proximity edge by kernel`s weights

• Spatial-variant filter

31 / 47

1

gF I L g

F

Application and Results

Multi-scale tone manipulation

– Manipulating tone and contrast of detail at different scales

• Implementation of multi-scale tone manipulation

– Construction three-level decomposition

» Decomposition for CIELAB lightness channel

– Using detail layer and base layer for controlling image

» Exposure of base layer

» for medium and fine detail layer

» Avoiding hard clipping by sigmoid curve S

» Term controls exposure and contrast of base layer

» Remaining terms control medium and fine scale details

32 / 47

1 2

0 1 2ˆ , , ,p p p pg S b S d S d

1 2,

(16)

where is mean of lightness range, and S is sigmoid curve, 1 1 exp ax

0 , pS b

33 / 47

Fig. 9. Multi-scale tone manipulation with our tool. The boosting of the individual scales

is intentionally exaggerated.

Input image Coarse scale boosting Medium scale boosting

Fine scale boosting Combine result

Detail exaggeration

– Enhancing shape and detail from multi-light image

• removing objectionable artifact

• Appearing edges much clear

34 / 47

Fig. 10. Multi-scale detail enhancement of Fattal et al. [2007] (left) compared to results

produced with our decomposition (right). We are able to achieve more aggressive detail

enhancement and exaggeration, while avoiding artifacts.

– HDR tone mapping

35 / 47

Fig. 8. Boosting BLF-based detail layers (top) results in artifacts along the high-contrast

edges, which are absent when the decomposition is WLS-based (bottom). In the right

part of each image medium scale details have been boosted, also resulting in halos

when done using BLF. (Input image courtesy of Norman Koren.)

36 / 47

Fig. 11. HDR tone mapping with our tool. Saturation and exposure were manually

adjusted in the WLS results in order to match the overall appearance of the other two

images. (HDR imagec Industrial Light & Magic. All rights reserved.)

37 / 47

Fig. 7. Top: a tone-mapped image, taken directly from [Durand and Dorsey 2002], with

some halos visible around the picture frames and the light fixture. Bottom: a halo-free

result with a similar amount of local contrast may be produced using the same

tone mapping algorithm, simply by replacing BLF with WLS-based smoothing (a = 1:2; l

= 2).

– Progressive image abstraction

38 / 47

Fig. 12. HDR tone mapping with our tool. Saturation and exposure were manually

adjusted in the WLS results in order to match the overall appearance of the other two

images. (HDR imagec Industrial Light & Magic. All rights reserved.)

Conclusions

39 / 47

Proposed method

– Construction of edge-preserving multi-scale image

decomposition

• Using edge-preserving smoothing operator

– Based on weighted least squares optimization frame work

– Features to gracefully fade in magnitude without introducing

significant blurring

• Except for some of drawback of bilateral filter and other

approaches