digital photography with flash and no-flash image pairs

21
Digital Photography with Flash and No- Flash Image Pairs By: Georg Petschnigg Maneesh Agrawala Hugues Hoppe Richard Szeliski Michael CohenKentaro Toyama, Microsoft Corporation Presented by: Yael Amsterdamer Advanced Topics in Visual Computing, Spring 2012

Upload: darby

Post on 26-Feb-2016

153 views

Category:

Documents


0 download

DESCRIPTION

Digital Photography with Flash and No-Flash Image Pairs. By: Georg PetschniggManeesh AgrawalaHugues HoppeRichard Szeliski Michael CohenKentaro Toyama, Microsoft Corporation Presented by : Yael Amsterdamer Advanced Topics in Visual Computing, Spring 2012. Introduction. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Digital Photography with Flash and No-Flash Image Pairs

Digital Photography with Flash and No-Flash Image Pairs

By: Georg Petschnigg Maneesh AgrawalaHugues Hoppe Richard Szeliski

Michael Cohen Kentaro Toyama, Microsoft Corporation

Presented by: Yael AmsterdamerAdvanced Topics in Visual Computing, Spring 2012

Page 2: Digital Photography with Flash and No-Flash Image Pairs

2Digital Photography with Flash and No-Flash Image Pairs

The Dilemma: to Flash or not to flash?

Introduction

• Natural lighting• Low signal-to-noise ratio

(SNR)• Loss of details• Longer exposure –

motion blur

• Harsh, unnatural lighting• High SNR• More details• May cause unwanted

artifacts (red eye, shadows, specularities)

Page 3: Digital Photography with Flash and No-Flash Image Pairs

3Digital Photography with Flash and No-Flash Image Pairs

Why not both?

• The idea: use the good features of each photo to create a better image

Introduction

Page 4: Digital Photography with Flash and No-Flash Image Pairs

4Digital Photography with Flash and No-Flash Image Pairs

Other works on combining images

Introduction

Different exposures

Panoramic view

Relighting

Current work • handles 2 images with specific

characteristics,• Envisioned for camera firmware

Page 5: Digital Photography with Flash and No-Flash Image Pairs

5Digital Photography with Flash and No-Flash Image Pairs

Different Applications

• Denoising• Detail transfer– Side requirement: handling flash induced

shadows and specularities• White Balancing• Continuous Flash Adjustment• Red Eye Removal

Introduction

Page 6: Digital Photography with Flash and No-Flash Image Pairs

6Digital Photography with Flash and No-Flash Image Pairs

Camera Adjustment to light• A certain amount of light is needed for a photo

• Taking pictures of a dark scene

– Increasing exposure time

– Opening the aperture

– Increasing ISO (sensor sensitivity)

• The problem – high ISO noise affects high-frequency details

Background

Page 7: Digital Photography with Flash and No-Flash Image Pairs

7Digital Photography with Flash and No-Flash Image Pairs

The Input

• F – the photo with flash• A – the no-flash (ambient) photo– Composed of 3 RGB intensity values for each pixel– Ap – one pixel in A

• Same camera position, focus, aperture– Otherwise, registration is required

• Adjusted ISO and exposure time• Undergo non-linear tone-mapping operations

Background

Page 8: Digital Photography with Flash and No-Flash Image Pairs

8Digital Photography with Flash and No-Flash Image Pairs

1st Application: Denoising

• A classic problem in image processing– The target: analytically reduce SNR

• One type of solution: smoothing filters• Bilateral Filter [Tomasi and Manduchi 1998]

– Fast, non-iterative– Averaging close-by pixels– Low-pass filter, combined with– Edge-stopping function

Denoising

Page 9: Digital Photography with Flash and No-Flash Image Pairs

9Digital Photography with Flash and No-Flash Image Pairs

Bilateral filter

• However, results still have noise or blur (or both)

Denoising

ambient

flashBilateral

filter

Page 10: Digital Photography with Flash and No-Flash Image Pairs

10Digital Photography with Flash and No-Flash Image Pairs

Improvement: Joint Bilateral Filter

• In the flash image there are much more details• Why not use F to find edges?

Denoising

Bilateral filter

Joint Bilateral filter

The difference

Page 11: Digital Photography with Flash and No-Flash Image Pairs

11Digital Photography with Flash and No-Flash Image Pairs

Not all edges are real …

• May cause over- or under-blur in joint bilateral filter• We need to eliminate their effect

Denoising

Page 12: Digital Photography with Flash and No-Flash Image Pairs

12Digital Photography with Flash and No-Flash Image Pairs

Detecting shadows• Observation: the pixels in the flash shadow should be similar to

the ambient image

• Not identical:

– Noise

– Inter-reflected flash

• Compute a shadow mask

• Take pixel p if

• is manually adjusted

• Mask is smoothed and dilated

Denoising

Pixels for which we will not use F

Page 13: Digital Photography with Flash and No-Flash Image Pairs

13Digital Photography with Flash and No-Flash Image Pairs

Detecting specularities• Take pixels where sensor input is close to maximum

(very bright)

– Over fixed threshold

• Create a specularity mask

• Also smoothed and dilated

• M – the combination of shadow and specularity masks

– Where Mp=1, we use ABase. For other pixels we use ANR.

Denoising

Page 14: Digital Photography with Flash and No-Flash Image Pairs

14Digital Photography with Flash and No-Flash Image Pairs

There is so much denoising can do• It cannot add details missing in the ambient image• Exist in flash image because of high SNR• We use a quotient image:

• Multiply with ANR to add the details• Masked in the same way

Detail Transfer

Bilateral filtered

Reduces the effect of

noise in F

Page 15: Digital Photography with Flash and No-Flash Image Pairs

15Digital Photography with Flash and No-Flash Image Pairs

Unified process• Denoising + detail transfer + masking shadows and

specularities

Detail Transfer

Page 16: Digital Photography with Flash and No-Flash Image Pairs

16Digital Photography with Flash and No-Flash Image Pairs

More examples

Detail Transfer

Page 17: Digital Photography with Flash and No-Flash Image Pairs

17Digital Photography with Flash and No-Flash Image Pairs

The opposite of preserving the atmosphere• The goal: simulate white light in the scene

– What color is the light in the scene?

• Flash image can help us:The difference between F and A is only the flash light

• Assume it is proportional to the surface albedo– Inherent property of the surface

– Assume the surface reflects light in the same color (will not work for plastic, glass…)

White Balancing

Page 18: Digital Photography with Flash and No-Flash Image Pairs

18Digital Photography with Flash and No-Flash Image Pairs

Computing the ambient light• We cancel the color of the surface

• The average c of Cp(col) for pixels with certain properties is the ambient light color

• Then we scale A to cancel the ambient light

White Balancing

Page 19: Digital Photography with Flash and No-Flash Image Pairs

19Digital Photography with Flash and No-Flash Image Pairs

Example

White Balancing

Estimated ambient light

Computed color

Ambient original

White-balanced

Page 20: Digital Photography with Flash and No-Flash Image Pairs

20Digital Photography with Flash and No-Flash Image Pairs

Combining flash and no-flash images• Providing complementary details about the scene

• Use for denoising and detail transfer

• Use for white balancing

• Future work

– More robustness

– Automatization of configuration

Summary

Page 21: Digital Photography with Flash and No-Flash Image Pairs

Thank you!