EE 7700

Demosaicking Problem in Digital Cameras

Bahadir K. Gunturk 2

Multi-Chip Digital Camera

Lens

Scene

Spectral

filters

Beam-splitter

s

Sensors

To produce a color image, at least three spectral components are needed at each pixel.

One approach is to use beam-splitters and multiple chips.

Bahadir K. Gunturk 3

Single-Chip Digital Camera Multi-chip approach is expensive. Precise chip alignment is

required. The alternative is to use a color filter array.

Lens

Scene

Color filter array

Sensors

Bahadir K. Gunturk 4

Single-Chip Digital Camera The missing color samples must be estimated to produce

the full color image. Since a mosaic of samples are available, this estimation

(interpolation) process is called demosaicking.

Bahadir K. Gunturk 5

Single-Chip Digital Camera Images suffer from color artifacts when the samples are not

estimated correctly.

Original image Bilinearly interpolated from CFA-filtered

samples

Bahadir K. Gunturk 6

Demosaicking Approaches Non-Adaptive Single-Channel Interpolation: Interpolate

each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc.

Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges.

1 2

3

4

x

Edge-directed interpolation 1.Calculate horizontal gradient ΔH = |G1 – G2| 2.Calculate vertical gradient ΔV = |G3 – G4|3.If ΔH > ΔV,Gx = (G3 + G4)/2 Else if ΔH < ΔV, Gx = (G1 + G2)/2 Else Gx = (G1 + G2 + G3 + G4)/4

Bahadir K. Gunturk 7

Demosaicking Approaches Edge-Directed Interpolation: Based on the assumption that

color channels have similar texture, various edge detectors can be used.

Edge-directed interpolation 1. Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 | 2. Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 | 3. If ΔH > ΔV,

G5 = (G2 + G8)/2 Else if ΔH < ΔV, G5 = (G4 + G6)/2 Else G5 = (G2 + G8 + G4 + G6)/4

1

2

4 5 6 7

8

9

3

Bahadir K. Gunturk 8

Demosaicking Approaches Constant-Hue-Based Interpolation: Hue does not change

abruptly within a small neighborhood. Interpolate green channel first. Interpolate hue (defined as either color differences or color

ratios). Estimate the missing (red/blue) from the interpolated hue.

Red Interpolated Red

InterpolateGreen

Interpolate

Bahadir K. Gunturk 9

Demosaicking Approaches Edge-Directed Interpolation of Hue: It is a combination of

edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm).

Bahadir K. Gunturk 10

Demosaicking Approaches Using Laplacian For Enhancement: Use the second-order

gradients of red/blue channels to enhance green channel.

1

2

4 5 6 7

8

9

3

1. Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7| 2. Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9| 3. If ΔH > ΔV,

G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4Else if ΔH < ΔV,

G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4 Else

G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8

Bahadir K. Gunturk 11

Aliasing

mf 1f

2f

2f

1f

2f

1f

Green channel Red/Blue channel

Frequency spectrum of an image:

After CFA sampling:

Bahadir K. Gunturk 12

Demosaicking Approach Alias Cancelling: Based on the assumption that red, green,

and blue channels have similar frequency components, the high-frequency components of red and blue channels are replaced by the high-frequency components of green channel.

2f

1f

Red/Blue channel

Bahadir K. Gunturk 13

Experiment

Full Red/Green/Blue

channels

Subband decomposition

CFA Sampling

Subband decomposition

Interpolate

LL

HL

HH

LH

LLLL

HLHL

LHLH

LL

HL

HH

LH

LLLL

HLHL

LHLH

Bahadir K. Gunturk 14

Constraint Sets

Detail Constraint Set: Detail subbands of the red and blue channels must be similar to the detail subbands of the green channel.

HL

HH

LH

HL

LH

HH

HLRHLG

1 2 1 2 1 2 1 2( , ) : ( , ) ( , ) ( , ),

, ,k k

d

R n n R n n G n n T n nC

for k HL LH HH

Bahadir K. Gunturk 15

Constraint Sets

Observation Constraint Set: Interpolated channels must be consistent with the observed data.

CFASensors

1 2( , )O n n

R

1 2 1 2 1 2 1 2( , ) : ( , ) ( , ), ( , )o RC R n n R n n O n n n n

Bahadir K. Gunturk 16

HL

LH

HH

Projection Operations

Projection onto the Detail Constraint Set:

Decompose the color channels.

Update the detail subbands of red and blue channels.

1 2( , )HLG n n

1 2 1 2( , ) ( , )HLG n n T n n

Apply synthesis filters to reconstruct back the channels.

1 2( , )HLR n n

Bahadir K. Gunturk 17

Projection Operations

Projection onto the Observation Constraint Set:

Insert the observed data to their corresponding positions.

CFASensors

1 2( , )O n n

Bahadir K. Gunturk 18

Alternating Projections Algorithm

Samples of color channels

Initial interpolation

0h

1h

0g

1gUpdate

Insert the observed data

Projection onto the detail constraint set

Projection onto the observation constraint set

Iteration

Bahadir K. Gunturk 19

Results

Original Hibbard 1995 Laroche and Prescott 1994

Hamilton and Adams 1997 Kimmel 1999 Gunturk 2002

Bahadir K. Gunturk 20

Results

OriginalHibbard

1995

Laroche and

Prescott 1994

Hamilton and

Adams 1997

Kimmel 1999

Gunturk 2002

Bahadir K. Gunturk 21

Previous Methods

Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital Cameras,” to appear in IEEE Signal Processing Magazine.

[Gunturk02]

Bahadir K. Gunturk 22

References

[Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating Projections,” IEEE Trans. Image Processing, 2002.

[Hibbard 1995] R. H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5,382,976, January, 1995.

[Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5,373,322, December, 1994.

[Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent 5,629,734, May, 1997.

[Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp. 1221-1228, 1999.