hardware implementation of color image demosaicing for...

Project:

Hardware Implementation of Color Image Demosaicing for

Camera Cellphones

Under Guidance of: By:Prof M Balakrishnan S Vijaya RaghavanAss. Prof. Prem Kalra 2004JVL2439

Associated with: Cypress Semiconductors Pvt. Ltd., Bangalore

What is Demosaicing

Nearest Neighbor Replication

•Each interpolated output pixel is assigned the value of the nearest pixel in the input image. •The nearest neighbor can be any one of the upper, lower, left and right pixels. •An example is illustrated below in Figure 2 for a 3x3 block in green plane. Here we assume the left neighboring pixel value is used to fill the missing ones.

Bilinear Interpolation

•For convenience, Bayer pattern is duplicated here

•Interpolation of green pixels :• Average of the upper, lower, left and right pixel values is assigned as the G value of the interpolated pixel

G8 = (G3+G7+G9+G13) / 4

•Interpolation of red/blue pixels :• Interpolation of a red/blue pixel at a green position :

• the average of two adjacent pixel values in corresponding color is assigned to the interpolated pixel. •Example : B7 = (B6+B8) / 2 ; R7 = (R2+R12) / 2

• Interpolation of a red/blue pixel at a blue/red position : • the average of four adjacent diagonal pixel values is assigned to the interpolated pixel. •Example : R8 = (R2+R4+R12+R14) / 4 ; B12 = (B6+B8+B16+B18) / 4

Smooth Hue Transition

•Interpolation of green pixels : •same as in bilinear interpolation•interpolation of green pixels-- done before red/blue pixels

• Interpolation of red/blue pixels : •try to impose a smooth transition in hue from pixel to pixel. •define blue "hue value" as : B/G. And red "hue value" as R/G •three different cases of blue pixel value interpolations

• blue pixel value at green position + adjacent blue pixels are on left and right : e.g. B7 = G7 / 2 * (B6 / G6 + B8 / G8) • blue pixel value at green position + adjacent blue pixels are on top and bottom : e.g. B13 = G13 / 2 * (B8 / G8 + B18 / G18) • blue pixel value at the red postion :

•e.g. B12 = G12 / 4 * (B6 / G6 + B8 / G8 + B16 / G16 + B18 / G18)

Smooth Hue Transition (log space)

•Interpolation of green pixels : •same as in bilinear interpolation•interpolation of green pixels-- done before red/blue pixels

• Interpolation of red/blue pixels in log space :•define blue "hue value": logB- logG , red "hue value" : logB- logG•three different cases of blue pixel value interpolations

• blue pixel value at green position + adjacent blue pixels are on left and right : e.g. logB7 = logG7 + (logB6 - logG6 + logB8 - logG8) / 2• blue pixel value at green position + adjacent blue pixels are on top and bottom : e.g. logB13 = logG13 + (logB8 - logG8 + logB18 - logG18) / 2 • blue pixel value at the red position :

•e.g. logB12 = logG12 + (logB6 - logG6 + logB8 - logG8 + logB16 - logG16 + logB18 - logG18) / 4

•Finally, transfer all pixel values back into linear exposure space

Edge Sensing Algorithm I

•Human visual systems are sensitive to edges in images•Non-adaptive interpolation algorithms often fail around edges

•since they are not able to detect "edges“

•Interpolation of green pixels : •First, define two gradients, one in horizontal direction, the other in vertical direction, for each blue/red position. For instance, consider B8 : define two gradients as , where | . | denotes absolute value •Define some threshold value T •The algorithm then can be described as follows :

Edge Sensing Algorithm I

• Choice of T depends on the images•T-- different optimum values from different neighborhoods•Common choice of T is

•Algorithm now becomes :

• Interpolation of red/blue pixels : •Same as in Smooth hue transition interpolation in logarithmic space• Other methods may be used as well

Edge Sensing Algorithm II

•A slightly different edge sensing interpolation algorithm

•Interpolation of green pixels : •Same as in edge sensing algorithm I except that horzitonal and vertical gradients are defined differently. •Refer to figure below, to estimate G5 at B5, we define gradients as :

•Interpolation of red/blue pixels : •similar to bilinear interpolation algorithm except that color difference is interpolated instead of color itself.

Linear Interpolation with Laplacian second-order correction terms

•Interpolation of green pixels : •Consider estimating the green value at a blue pixel•Find out G5 at B5

•Define horizontal and vertical gradients as :

•The algorithm is :


•Interpolation of red/blue pixels : • Depending on the positions, we have three cases

1. find red/blue values at a green pixel where nearest neighbors are in the same column.

•estimate R4 at G4, •R4 = (R1 + R7) / 2 + (G4 - G1 + G4 - G7) / 4

2. find red/blue values at a green pixel where nearest neightbors are in the same row

•estimate R2 at G2, •R2 = (R1 + R3) / 2 + (G2 - G1 + G2 - G3) / 4


•Interpolation of red/blue pixels : 3. find red/blue values at a blue/red pixel

• estimate R5 at B5 •Define two diagonal (negative and positive) directions

•Now the interpolation is described as :

High Quality Linear Interpolation•Edges have much stronger luminance than chrominance components•Example: Compare red value to its estimate for a bilinear interpolation

•If different there’s a sharp luminance change at that pixel•Correct a bilinearly interpolated green value by adding a portion of this estimated change

•To interpolate G value at R location: ¿ ¿¿

subscript B means bilinear interpolation

•To interpolate R value at G location: ¿ ¿¿•To interpolate R value at B location: ¿ ¿¿

where gradient of R at that location is RαΔ i,j

•Gradient correction Gains•Determine gain parameters using Weiner approach

•Compute values that lead to minimum mean-square error interpolation

=α 1 /2, =β 5 /8, =γ 3 /4

High Quality Linear Interpolation

Filter coefficients

•Simple•Linear•Better than some nonlinear algorithms•Computational complexity is more

Discriminated Color Correlation

•Use color signal correlation•Select image signal with higher correlation (horizontal/ vertical)

•HIGH HOR. FREQ. => HIGH COR. VER. DIR.•HIGH VER. FREQ. => HIGH COR. HOR. DIR.

•Improve resolution and reduce aliasing•In local region, variations in three signals are similar to one another

G3:R3=GLPF:RLPFG3=R3∗GLPF /RLPF


•Interpolate G at R or B pixel•Calculate gradients

•Obtain G values using below formula

•Interpolate R and B signals at G position


•Interpolate B at R pixel

•Similarly for R at B pixel

Freeman Demosaicing

• G sensor class is sampled at a higher rate than the R and B sensor classes

• ALGORITHM4. Apply bilinear interpolation separately to each sensor class5. The G image plane is now used directly as the output G image plane6. For R and B planes, result of bilinear interpolation is modified before

output1. Two difference planes R-G and B-G are created2. Median filtering of R-G and B-G planes done to eliminate artifacts3. R and B planes are created by adding the G plane to the median

filtered R-G and B-G planes4. In final output image, R and B values estimated from the median

filtered R-G and B-G planes are used only at pixels where there is no R or B sensor value directly available

Freeman Demosaicing

• PROBLEM• Straightaway implementation of this leads to bad performance in general• Generally, R, G and B sensor classes have very different mean responses in

images• This algorithm is sensitive to differential scaling

• SOLUTION• Scaling R, G and B sensor planes to have the same mean before applying this

algorithm and undoing this scaling afterwards produces much better results

Fourier Domain Demosaicing

•Spatial multiplexed imageR x,y =∑

i

Cix,y m ix,y

•wheremR x,y =1cosπx 1cosπy /4mGx,y =1−cosπx cosπy /2mBx,y =1−cosπx 1−cosπy /4

•Color demosaicing by estimating luminance and opponent chromatic signals in Fourier Domain

{Ci x,y },i∈[R,G,B ] , x,y ∈N2

•One-color per pixel image can be written as sum of luminance and chrominance•Luminance =scalar Chrominance= 3-D vector•Luminance and chrominance are well-localized in spatial frequency domain


R x,y =∑i

piCix,y ∑i

x,y ¿

•First part is Luminance => doesn’t depend on modulation function at each pixel•Second is Chrominance => has color information that is spatially sub-sampled

•Thus mi(x,y)= pi + mi~(x,y)

•Hence we have:

ALGORITHM


Luminance Filter

•Chrominance Filter •Orthogonal filter to Luminance

Vector Color Filter Array Demosaicing

•1. Solving for a color vector instead of plane-by-plane pixel interpolation•2. Replacing original color value at a pixel with value that matches better with other color dimensions

•Pseudo-pixels: group neighboring Red, Green and Blue values together•Use column vector to represent Pseudo-pixels•Replace unknown color value with average of surrounding pseudo-pixel values•NOT arithmetic average – smearing•To Preserve edges, take vector median of pseudo-pixels to be missing color vector

R 21G21B21

righ

R15:R15:R15:R15:R27:R27:R27:R27:G21:G21:G16:G14:G21:G21:G28:G26B22:B20:B22:B20:B22:B20:B22:B20

righ

[ ][ ]

Vector Color Filter Array Demosaicing

•Vector Median is a vector A that minimizes the sum of L2 distances (Euclidean) to each point in the set.•Let {x1,x2, … ,xn} be set of K-dimensional vectors, then Vector Median is vector A that solves:

argmin A∑i=1

N ∑k=1

K

Ak−xik 21/2

here Ak and xik are k-th element of vector A and xi, respectively

•Vector median has to be found by iterative means and can be approximated in a number of ways•“Exact” vector median is one that is accurate to some , where is smaller ε εthan the quantization error of imaging system

Variance Interpolation

•Maintains sharpness of edges•Compute amount of variance ( difference between this pixel and its neighboring pixels)•Assume variance is similar in all three color bands•Requires temporary storage to hold 5 rows

Algorithm overview

•For each pixel, determine the actual color value it stores (Refer Bayer pattern)•Now, compute the average value of the same color for this pixel from its corresponding neighboring pixels•Compute the “difference” between the actual and average color value•Compute the average value of the missing color values for this pixel from the neighboring corresponding pixel values

-can use bilinear interpolation here•To each estimated missing color value add the “difference” value computed

Image Enhancement by Efficient Demosaicing

•Two main stages•Adaptive initial interpolation•Correction of visual artifacts that remain in image

•In whole algorithm has 4 steps as shown in figure below:

•Initial estimation (Initial interpolation)•Green Channel

•Classify missing pixels of green channel into regions•Apply different interpolation schemes according to the classification•Have an estimate now of the location of highly problematic regions•Classify these regions further into:

•Horizontal region: interpolate in horizontal direction•Vertical region: interpolate in vertical direction

•Right direction is determined in the correction stage•Red/ Blue Channel

•Interpolate these channels twice ( for both estimates of green channel)•Resulting two color images are passed to correction stage, color artifacts are reduced

•Main key detecting artifacts:•Relative smoothness of chrominance component of each region (YIQ space)


Image Enhancement by Efficient Demosaicing•Green Estimation

•Class A pixels (very smooth)•Average value of neighbors

•Class B pixels•Interpolate using larger neighborhood•Neighbors of missing pixel+pixels from other two channels

•Class C pixels (problematic pixels)•Neither neighbor pixels nor inter-channel pixels help in estimating unknown pixel•Caused due to aliasing and sub-sampling•Unknown value = average of left & right or average of upper & lower neighbors

•Red/ Blue Estimation•Red-Green and Blue-Green values interpolated•Performed twice for each channel (Horizontal and Vertical regions)

•YIQ Transformation•Considering I component, for each aliased region

•Select smoother region based on relative smoothness of horizontal and vertical versions


•RED/ BLUE PIXELG2R1−G1 ifR1<R3G2R3−G3 otherwise

R2=¿{}

•Performed twice--- for horizontal and vertical Green image

•Class A pixels¿ ¿ ¿¿

•Class B pixels

¿ ¿¿

•GREEN PIXEL

•Class C pixels•Horizontal or Vertical average


•Correction Method•For Class C pixels•In YIQ domain, cluster pixels into 3*3 neighborhood•Each cluster has two versions --- horizontal and vertical•If Ch and Cv are two versions

•Average value of I components are AvCh and AvCv

•Average of horizontal and vertical environments are AvEh and AvEv

•Selected cluster Cs is determined by

Chif AvCh /AvEh AvCv /AvEv Cvotherwise

Cs=¿{}¿ ¿¿

Using Natural Image Properties

• Generic Demosaicing Algorithm3. Store 2 copies of image mosaic, H and V4. Calculate “Difficult missing samples” mask5. Estimate the “simple” missing green samples according to Algorithm_A (for

both H and V)6. Estimate the difficult missing green sample twice, once according to its

horizontal neighbors (store in H) and once according to its vertical ones (store in V)

7. Partition “Difficult” missing samples mask into regions (according to spatial adjacency)

8. Calculate color variation and corner scores for each difficult region in H and V

9. Combine H and V by taking the region that scores better according to our criteria in each difficult regions

• Difficult Regions– mask containing regions where traditional gradient based demosaicing methods tend to fail

• Estimated horizontal gradient similar to estimated vertical gradient, yet estimated values in both cases differ significantly

Using Natural Image Properties

•Divide Difficult Regions mask into components simple by slicing it into 8*8 regions•For each such component, calculate a score according to the demosaicing hints•Choose the one that matches best

•Demosaicing Hints•Least Color Variation Criterion

•Assumption: Intensity variation in images is larger than color variation•In high frequency regions, restore missing samples of green channel such that sum of variance of R/G ratio and B/G ratio is reduced

•Minimal corner value criterion•In images edges are sparse and corners are much sparser•Grade demosaicing results according to response to a corner detection filter•Zippering, very often leads to many false corners (erroneous demosaicing)•Use Harris Corner Detection filter and sum it over entire region•Lower the score, better the demosaic

Pixel Grouping

• Natural scene images usually composed of numerous groups of pixels• Pixels in same group are reasonably connected to each other• They have approximately same brightness and colour• Identifying which pixels are in same group, we can obtain better estimates of

missing colors• Only green (luminance) channel is interpolated through identification of

pixel groups• Red and blue channels are generated by a hybrid color correction and

modified smooth-hue-transition algorithm

• Interpolation of green values at blue or red pixels G13 at R13• Calculate gradients in four directions

N=Δ ∣R3−R13∣∗2∣G8−G18∣E=Δ ∣R 13−R15∣∗2∣G12−G14∣W=Δ ∣R 11−R 13∣∗2∣G12−G14∣S=Δ ∣R 13−R23∣∗2∣G8−G18∣

R25G24R23G22R21

G20B19G18B17G16

R15G14R13G12R11

G10B9G8B7G6

R5G4R3G2R1

Pixel Grouping

• Determine direction of smallest gradient and estimate G13 as follows

Δ N is minimum: G13=G8*3R13G18-R3 /4EΔ is minimum: G13=G14*3R13G12-R15 /4WΔ is minimum: G13=G12*3R13G14-R11 /4SΔ is minimum: G13=G18*3R13G8-R23 /4

• Interpolation of blue and red values at green pixels (B8 and R8 at G8)• B8=hue_transit (G7, G8, G9, B7, B9) • R8=hue_transit (G3, G8, G13, R3, R13)

• Function hue_transit (l1, l2, l3, v1, v3) =• If (l1<l2<l3 or l1>l2>l3)• Then return v1+(v3-v1)*(l2-l1)/(l3-l1)• Else return (v1+v3)/2+(l2*2-l1-l3)/4

Pixel Grouping

• Interpolation of blue or red values at red or blue pixels (B13 at R13)

ne = |B9 - B17|+|R5 - R13|+|R13 – R21|+|G9 - G13|+|G13 - G17|Δnw = |B7 - B19|+|R1 - R13|+|R13 – R25|+|G7 - G13|+|G13 - G19|Δ

if ( ne <= nw)Δ Δthen B13 = hue_transit (G9, G13, G17, B9, B17)else B13 = hue_transit (G7, G13, G19, B7, B19)

• PROBLEMS14. Turns out images that are blocky if image has lots of varying gradients15. Optimized for artificial scenes with uniform colors and clear boundaries

between regions like in printed matter

SVM BasedDemosaicing using Optimal Recovery

• Selection and Shape of Training Vectors -- learning local quadratic signal class

• Estimates missing sample of local image patch y from known samples of y

• Local patch: local patch belongs to a known quadratic signal class K, where K is defined as

K= {x Rn : xTQx <= }ε

Also called as Training vectors

• Inverse of Q matrix is covariance matrix of local image patches

• Minimizes maximum error over all possible vectors in K that have same known samples as y

• Algorithm::

• Interpolate green plane by optimal recovery

• Interpolate differences red-green and blue-green using optimal recovery

•Training set is formed under below criteria

•1. Closeness in space

–Size of local window centered around y

–Vectors in this window are closer to pixel interpolated

–This is initial training set S

•2. Closeness in value

•Generally, boundary regions are split in half by the local window

•Pruning S to contain only half the vectors that are closest to y in value

•Taking half of the closest vectors is enough to guarantee good boundary interpolation

•Shape of training vectors

•Quincunx sampling of green plane forces certain structure on shape of training vectors

•Coarse scale vectors Vs Fine Scale vectors

SVM Based

SVM Based

•Interpolation of green plane

•Determine local Quadratic class: •each local region use coarse scale vectors to generate set S

•Use optimal recovery: •estimate the missing green pixel values

•Second pass: •rerun optimal recovery interpolation from the training set S using fine-scale vectors. Necessary for high frequency patterns

•Interpolation of red and blue plane

•Apply above algorithm to differences of red-green and blue-green•Difference image doesn’t contain high frequencies as it is low-passed•Can use simpler interpolation here to increase speed

PRELIMINARY RESULTS AND

PROBLEMS

Bayer Pattern from 24-bit image

Original Image Mosaiced Bayer Image

Nearest Neighbor Replication

Original Image Interpolated Image

Bilinear Interpolation


Smooth Hue Interpolation


Problem: Situations of divide-by-zero occuring in certain regionsSo the results are not correct. This may not be actual interpolation.

Log-Smooth Hue Interpolation


Problem: Situations of log of zero occuring in certain regionsSo the results are not correct. This may not be actual interpolation.

Nearest Neighbor Bilinear

Smooth hue Log-Smooth hue

Original

To be completed shortly

• Edge Sensing 1 Results

• Edge Sensing 2 Results

hardware implementation of color image demosaicing for...

Documents