59

CHAPTER 4

PRINCIPAL COMPONENT ANALYSIS-BASED FUSION

4.1 INTRODUCTION

Weighted average-based fusion algorithms are one of the widely

used fusion methods for multi-sensor data integration. These methods involve

the selection of appropriate weights to combine the images, so as to reduce

the effects of distortion and also give a satisfactory visual image quality. The

method of selecting weights based on the energy of the coefficients of the

decomposed wavelet coefficients was described in Chapter 3. This chapter

outlines a fusion scheme that uses weights based on the statistical nature of

the data to be combined. The statistical measure used is the Principal

Component Analysis (PCA).

Two approaches of using PCA in image fusion have been listed by

Genderen and Pohl (1998):

1. PCA of multi-channel images where the first principal

component is replaced by different images. This method,

known as the Principal Component Substitution (PCS), has

been used for remote sensing data fusion by Chavez et al.

(1991).

2. PCA of all multi-image data channels as outlined by Yesou

et al. (1993) and whose results were reported by Richards

(1984).

60

In addition to these, a third approach can be included in which the

PCA transform has been used to calculate the weights of a linear combination

of the input images as shown in the algorithms developed by Das et al.

(2000), Haq et al. (2005) and Zheng et al. (2007). Haq et al. and Zheng et al.

describe the PCA fusion rule along with multi-resolution image

decomposition using the DWT. This research work focuses on the third

method detailed above and offers alternative algorithms to determine the

weights of fusion.

The PCA-based weighted fusion involves separately fusing the high

frequency (HF) and the low frequency (LF) parts of an image. The two

frequency components are obtained by a filtering mechanism and finally the

fused components are added together to get the resultant fused output.

This chapter begins with an introduction to the PCA transform from

a statistical standpoint; followed by the principal component analysis of

images in section 4.3. The newly designed fusion algorithms that make use of

the PCA for calculating the weights are described in the next section. This

includes a discussion of the PCA – Gaussian weighted average fusion method

(5) and the efficient PCA – Max Fusion scheme (8) in sections 4.4.2 and 4.4.3

respectively. The performance of the fusion schemes detailed in the previous

section is analysed in section 4.5. This chapter concludes with a summary of

the proposed techniques.

4.2 PRINCIPAL COMPONENT ANALYSIS (PCA) TRANSFORM

4.2.1 Introduction

Principal Component Analysis is a quantitatively rigorous method

for achieving simplification. Often, its operation can be thought of as

revealing the internal structure of the data in a way which best explains the

variance in the data. The method generates a new set of variables called

61

Principal Components (PC). Each principal component is a linear

combination of the original variables and all the PCs are orthogonal to each

other, and as a whole form an orthogonal basis for the space of the data;

thereby removing redundant information, if any.

The first principal component is a single axis in space. When each

of the observations in the data set is projected on this axis, the resulting values

form a new variable and the variance of this variable is the maximum among

all possible choices of the first axis. The second principal component is

another axis in space, perpendicular to the first. Projecting the observations on

this axis generates another new variable, such that the variance of this

variable is the maximum among all possible choices of this second axis. The

full set of principal components is as large as the original set of variables. In

general, the sum of the variances of the first few principal components

exceeds 80% of the total variance of the original data. The original data can

be recovered from the first few PCs themselves hence the Principal

Component Analysis is a method that enables a decrease in the number of

channels (or bands) by reducing the inter-channel dependencies.

The multidimensional space is mapped into a space of fewer

dimensions by transforming the original space using a linear transformation

via a principal component analysis. The steps involved in the PCA transform,

also called the Karhunen - Loeve (KL) transform, are:

1. Calculate the covariance matrix or the correlation matrix of

the data sets to be transformed. The covariance matrix is used

in the case of the unstandardised PCA, while the standardised

PCA uses the correlation matrix.

2. Calculate the eigenvalues and the eigenvectors from the

correlation / covariance matrix.

62

3. Principal Components of the given data set are the

eigenvectors of the covariance matrix of the input.

If the correlation matrix of the data is constructed and the

eigenvectors found and listed in eigenvalue order, then just the first few

eigenvectors can be used to reconstruct a large fraction of the variance of the

original data. The first few eigenvectors can often be interpreted in terms of

the large-scale physical behaviour of the system. The original space has been

reduced to the space spanned by a few eigenvectors, with data loss, but

retaining the most important variance. A lossless dimensionality reduction is

possible if the data in question falls exactly on a smooth, locally flat subspace;

however the noisy data prevents such an exact mapping, introducing some

loss of information.

4.2.2 Mathematical Analysis of the Principal Component Analysis

Transform

Consider n-dimensional data in the form of M-vectors

Mx....x,x,x 321 defined by X as,

1 2 3 MX [x ,x ,x ....x ]= (4.1)

The first principal component is that the n dimensional vector along

whose direction, the variance is maximized. The principal component can be

easily computed as the eigenvector of the correlation matrix having the largest

eigenvalue.

The next step is the computation of mean vector of the population,

which is defined by the equation,

å=÷øöçè

æ= M

1k

kx xM

1m (4.2)

63

where,

xm is the mean vector associated with the population of x vectors.

For M vector samples from a random population, the covariance

matrix can be approximated from the samples by,

[ ]å= -÷øöçè

æ= M

1k

T

xx

T

kkx mmxxM

1C (4.3)

where,

‘T’ indicates vector transposition.

xC is covariance matrix of order n x n.

Element cii of xC is the variance of xi; the ith

component of the x

vectors in the population, and element cij of Cx is the covariance between

elements xi and xj of these vectors. The matrix xC is real and symmetric. If

elements xi and xj are uncorrelated, their covariance is zero and, therefore,

cij=cji=0.

Since Cx is real and symmetric, finding a set of n orthogonal

eigenvectors always is possible. Let ei and λi, i = 1, 2 … n, be the

eigenvectors and corresponding eigenvalues of Cx, arranged in descending

order so that

1+³ jj ll for j = 1,2, ... n-1 (4.4)

Let A be a matrix whose rows are formed from the eigenvectors of

Cx ordered so that the first row of A is the eigenvector corresponding to the

largest eigenvalue and the last row is the eigenvector corresponding to the

64

smallest eigenvalue. The matrix A is called the transformation matrix that

maps the vectors x into vectors denoted by y as follows:

( )xmxAY -= (4.5)

This equation is called the Principal components transform (also

called the Hotelling transform). The mean of the y vectors resulting from this

transformation is zero;

0m y = (4.6)

the covariance matrix of the y can be obtained in terms of A and Cx by

T

Xy ACAC ××= (4.7)

Furthermore, Cy is the diagonal matrix whose elements along the

main diagonal are Eigen values of Cx; that is,

úúúû

ùêêêë

é=

3

2

1

00

00

00

ll

lyC (4.8)

The off-diagonal elements of the covariance matrix are 0, and

hence the elements of the y vectors are uncorrelated. The rows of matrix A

are the normalized eigenvectors of Cx. Because Cx is real and symmetric,

these vectors form an orthonormal set, and it follows that the elements along

the main diagonal of Cy are the eigenvalues of Cx. The main diagonal element

in the ith row of Cy is the variance of vector element yi.

65

Because the rows of A are orthonormal, its inverse equals its

transpose. Thus, one can recover the vector x by performing the inverse

transformation,

x

T myAx +×= (4.9)

To conclude this section, the principal component analysis is a

mathematical way of determining the linear transformation of a sample of

points in an N-dimensional space which exhibits the properties of the sample

most clearly along the coordinate axes and in this process reduces the inter-

channel dependencies. The eigenvalues for each principal component

correspond to the amount of total variance in the data described by this

component.

4.3 PRINCIPAL COMPONENTS ANALYSIS FOR IMAGES

4.3.1 Basic Steps Involved (Gonzales and Woods 2002)

PCA finds applications in image processing, where it has been used

for identifying the patterns in data, and expressing the data in such a way as to

highlight their similarities and differences.

The steps involved in finding the PCA of the given images are the

same as followed in the previous section for statistical data sets. However, as

the first stage, the given 2 – dimensional image is represented as a single

vector. In general, an M x N image is represented as an MN x 1 column

vector. Then, the covariances of the given images are calculated, followed by

the eigenvector and eigenvalue for covariance matrix and finally transforming

the images using the eigenvector. This transformed image consists of the

principal components of the given images. The principal components of the

images are the original data that are represented solely in terms of the

eigenvectors.

66

This process can be illustrated using an example. Consider two

images of the same scene (a road) captured using two different image sensing

systems. The first image shown in Figure 4.1 is taken using an infrared

camera, that responds to changes in heat intensity of the source being imaged

and the second image is obtained using the conventional CCD camera, shown

in Figure 4.2.

Figure 4.1 IR Image Figure 4.2 CCD Image

The pixels, which are of two dimensional matrix form M x N, from

both the images are vector populated, similar to equation (4.1), and shown in

Figure 4.5:

67

1x1 1x2 1xm 1x1 1x2 1xm

2x1 2x2 2xm 2x1 2x2 2xm

nx1 nx2 nxm nx1 nx2 Nxm

Figure 4.3 Matrix representation

of Image1 – X

Figure 4.4 Matrix representation

of Image2 – Y

1st image pixels - X 2

st image pixels - Y

1x1 1x1

2x1 2x1

... ...

1xm 1xm

2x1 2x1

... ...

... ...

Nxm Nxm

Figure 4.5 Vector populated matrix

The next step is to find the covariance of the vector populated

matrix. The covariance matrix gives the relation between the given images. If

one is given ‘n’ images, the covariance matrix will be of n x n dimensions.

68

Similar to equation (4.3), for the 2–dimensional case, the covariance can be

represented by,

( )( )( )n

mXmY

Y)(X,C

n

1i

XiYiå=

--= (4.10)

where,

mX is the mean of the samples of one image represented by vector X

mY is the mean of the samples of one image represented by vector Y

In case of more than 2 source images, more than one covariance

measurement is required. For example, from a 3 image (X, Y, Z) vector

populated matrix of dimension (No of pixels in a single image x 3), a

covariance matrix of dimension 3x3 is obtained,

÷÷÷ø

öçççè

æ=

Z)cov(Z,Y)cov(Z,X)cov(Z,

Z)cov(Y,Y)cov(Y,X)cov(Y,

Z)cov(X,Y)cov(X,X)cov(X,

Z)Y,C(X, (4.11)

The value of the matrix elements (1,1), (2,2) and (3,3) gives the

variance of images x, y and z, respectively. The element (1,2) is the relation

between image x and y and similarly for elements (1,3) and (2,3). The values

of (1,2) and (2,1) are same. Each entry in the matrix is the result of calculating

the covariance between two separate dimensions.

The following conclusions can be drawn from the covariance

matrix: if the value of covariance is positive, then it indicates that the gray

values in both the images increase together; if the value is negative, then it

implies a negative correlation, that is, as the gray level in one image increases,

69

the value in the other image decreases and finally, the last case of zero

covariance indicates that the two images are independent of each other.

4.3.2 Determination of Eigen Values of the Covariance Matrix

Since the covariance matrix is a square matrix, the eigenvectors and

eigenvalues for this matrix can be computed, which give useful information

about the images. The eigenvector and eigenvalue is in the form as shown

below for the images under consideration,

3

3

4.4306 10Eigenvalues

1.6518 10

æ ö´= ç ÷´è ø (4.12)

0.8444 0.5355Eigenvectors

0.5355 0.8444

- -æ ö= ç ÷-è ø (4.13)

The graphical representation of the eigenvectors of the images is

given in Figure 4.6.

Figure 4.6 Plot of the images with the eigenvectors of the covariance

matrix overlaid on top

Grey Levels of IR Image

Gre

y L

evel

s of

CC

D I

mag

e

70

When plotted, both the eigenvectors appear as diagonal dotted lines

on the plot and are perpendicular to each other. More importantly, they

provide with information about the patterns in the data. It can be seen from

the plot that one of the eigenvectors goes through the middle of the points,

similar to plotting a line of best fit, and shows how these two data sets are

related along that line. The second eigenvector gives the other, less important,

pattern in the data; signified by the amount by which all the points following

the main line are offset from it. This process of taking the eigenvectors of the

covariance matrix enables extraction of the lines that characterise the data. In

the above example, the eigenvector with the largest eigenvalue is the one that

points down the middle of the data, giving the most significant relationship

between the images.

The next step in finding the principal components of the given

images is to transform the image using the eigenvectors using the

equation (4.5), which is expressed below in words;

( )matrixpopulated vectorsubtractedMeanrEigenvectoComponentPrincipal ´=

The eigenvector used above is either one or both of the

eigenvectors (if two images are taken). In order that the matrix dimensions

agree, either the eigenvector or vector populated matrix has to be transposed.

The principal component using the eigenvector with the largest

eigenvalue will have more information than that obtained using the

eigenvector with the smallest eigenvalue. This can be seen from Figures 4.8

and 4.9 which show the images constructed using the largest eigenvector and

the smallest eigenvector, respectively.

71

Figure 4.7 Principal components using eigenvector corresponding to

the largest eigenvalue

Figure 4.8 Principal components using eigenvector corresponding to

the smallest eigenvalue

From the discussion in this section, it can be summarized that the

PCA technique has three effects: it orthogonalizes the components of the

input vectors (so that they are uncorrelated with each other); it orders the

resulting orthogonal components (principal components) so that those with

the largest variation come first; and it eliminates those components that

contribute the least to the variation in the data set.

72

Figure 4.9 Plot of the principal components obtained using both the

eigenvectors

The PCA process has transformed the data so that it is expressed in

terms of the patterns between the images, these patterns being the

eigenvectors that most closely describe the relationships between the data.

This is helpful because it enables the classification of all the data points

(pixels) as a combination of the contributions from each of the eigenvectors.

Initially, as in Figure 4.6, the plot has a simple grey level axes, each axis

representing grey levels of one image, which does not convey any information

on the relationship of the data points with each other. However, after

processing the images using the PCA transform, the values of the data points

specify exactly where they are present with respect to the trend lines, as seen

in Figure 4.7. In the case of the transformation using both the eigenvectors,

the data has been altered so that it is in terms of the eigenvectors instead of

the usual axes.

Gray Levels of IR Image

Gra

y L

evel

s of

CC

D I

mag

e

73

4.4 PCA-BASED WEIGHTED FUSION

4.4.1 Introduction

In this section, two new methods of computing weights for additive

fusion of two images are presented. The fusion schemes involve separating

the low frequency (LF) and the high frequency (HF) components of the

images and detail rules for fusing the frequency components. Consider the

multi-sensor input images obtained using an infrared sensor and a visible

spectrum CCD sensor as Iir and Ivis. The algorithm assumes that the source

images are registered with each other and processed to be of the same size.

The frequency component separation is achieved by the use of a Gaussian low

pass filter used for smoothing the images. A typical smoothing convolution

filter is essentially a matrix having an integer value for each row and column,

the value chosen depending on the type of filter being used. For the Gaussian

low pass filter, the 2-dimensional kernel is given by,

0 0625 0 1250 0 0625

0 1250 0 2500 0 1250

0 0625 0 1250 0 0625

. . .

h . . .

. . .

= (4.14)

When an image is convolved with this type of filter, the gray value

of each pixel is replaced by the average intensity of its eight nearest neighbors

and itself. If the gray value of any pixel overlaid by the convolution kernel is

dramatically different than that of its neighbors, the averaging effect of the

filter will tend to reduce the effect of the noise by distributing it among all of

the neighboring pixels. The smoothed images are

hIS irir *= (4.15)

hIS visvis *= (4.16)

74

where * is the convolution operator

The images Sir and Svis are the low frequency components

representing the visible portion of the source images.

The high frequency components are obtained by finding the

deviations from the smoothed images. This is given by

visvisvis SID -= (4.17)

irirr SID i -= (4.18)

The result of the filtering is shown in Figure 4.10 for the image of a

house on a hill captured using a CCD camera and that of the infrared image is

given in Figure 4.11.

Figure 4.10 Effect of low pass filtering to separate frequency

components of CCD image

Input CCD ImageLow Frequency Component

Image – Result of

Low Pass Filtering

High Frequency

Component Image

75

Figure 4.11 Effect of low pass filtering to separate frequency

components of IR image

Then these two frequency components are then fused separately

using different fusion rules. The fused components are then added together to

get the fused output image. The fusion rules proposed in this thesis are

discussed in the following sub-sections.

4.4.2 PCA Gaussian Fusion Algorithm

This fusion rule involves combining the low frequency (LF)

components, Sir and Svis, using simple averaging. The high frequency (HF)

components, Dir and Dvis are combined by weighted addition.

4.4.2.1 High Frequency Component Fusion Rule

The weights for fusion are calculated from the principal

components of the high frequency part of the source images. It involves the

computation of principal component of the deviation components as discussed

in section 4.3.1. Let these components be denoted as PC1 and PC2,

corresponding to the largest eigenvalue and the smallest eigenvalue,

respectively.

Input IR ImageLow Frequency Component

Image – Result of

Low Pass Filtering

High Frequency

Component Image

76

These principal components are used to define the ratios P1, P2;

which are in turn used to calculate the weights for the fusion. The ratios P1

and P2 are obtained as follows:

21

11

PCPC

PCP += (4.19)

21

22

PCPC

PCP += (4.20)

where PC1 is principal components corresponding to largest eigen value, and

PC2 is principal components corresponding to smallest eigen value.

The weights are obtained by smoothing the ratio of principal

components using a similar low pass filter, as in section 4.4.1 with a kernel

defined by equations (4.15) and (4.16).

The weights are given by,

hPw 11 *= (4.21)

hPw 22 *= (4.22)

where * is the convolution operator and h is the Gaussian kernel defined in

equation (4.14).

Image fusion of the high frequency components is achieved by the

weighted, normalized sum of the deviations defined by the fusion rule;

( ) ( ){ }]w[w

wDwDD

21

2vis1irfuse +

×+×= (4.23)

The fused output is shown in Figure 4.12

77

Figure 4.12 Fusion of HF components

4.4.2.2 Low Frequency Component Fusion Rule

The low frequency components are combined by averaging the

intensity levels of the two images. This LF image contributes to the

background information in the picture and is used to identify the presence of

objects in the image. Mathematically,

( )2

visirfuse

SSS

+= (4.24)

78

Figure 4.13 Fusion of LF components

The final fused output is obtained by adding the weighted

deviations, equations (4.17) and (4.18), to the low frequency background

image and is shown below.

fusefusefuse DSI += (4.25)

Figure 4.14 Fused image output using additive weighted fusion

79

4.4.3 Fusion Scheme Using the PCA – Max Rule

This newly designed scheme overcomes the performance limitation

of the weight-based fusion algorithm proposed in section 4.4.2. It uses the

same technique of separating the image into the low and high frequency

components. The low frequency components are combined using the principle

of choosing maximum intensity pixels from each LF image. The HF

components are fused using the weighted additive fusion rule as used in

section 4.4.2.1.

4.4.3.1 Low Frequency Component Fusion Rule

The low frequency images Sir and Svis are fused using the ‘Select

Max’ principle as discussed by Zheng et al. (2007). Since the visible

information is contained in the low frequency components, fusing the images

by selecting the pixel values with the highest intensity gives an output image

that has a very high quality as perceived by the human observer. This rule

involves choosing the maximum intensity levels of corresponding pixels from

each LF image to represent the pixel value in the fused image. However, to

enable comparison of the images on a pixel by pixel basis a histogram

matching of the two images is performed. This is achieved by matching the

histogram of the visible image to that of the IIR image (Gonzales and Woods

2002, Jain 1989).

The decision map of the low frequency image fusion rule is:

îíì

<>=

n)(m,Sn)(m,Sifn)(m,S

n)(m,Sn)(m,Sifn)(m,Sn)(m,S

visirvis

visirir

fuse (4.26)

where , m = 1, 2, 3, …. Sm

80

n = 1, 2, 3, … Sn

and Sm x Sn are the dimensions of the LF component image

The fused output is shown in Figure 4.15

Figure 4.15 LF component fusion using choose max rule

4.4.3.2 High Frequency Component Fusion Rule

The weights for fusion are calculated from the principal

components of the high frequency part of the source images. It involves the

computation of principal component of the deviation components as discussed

in section 4.3.1. Let these components be denoted as PC1 and PC2,

corresponding to the largest eigenvalue and the smallest eigenvalue,

respectively.

These principal components are used to define the ratios P1, P2,

which are in turn used to calculate the weights for the fusion. The ratios P1

and P2 are obtained as follows:

81

21

11

PCPC

PCP += (4.27)

21

22

PCPC

PCP += (4.28)

where PC1 is principal components corresponding to largest eigen value, and

PC2 is principal components corresponding to smallest eigen value.

The weights are obtained by smoothing the ratio of principal

components using a similar low pass filter as in section 4.4.1 with a kernel

defined by equations (4.15) and (4.16).

The weights are given by,

hPw 11 *= (4.29)

hPw 22 *= (4.30)

where * is the convolution operator and h is the Gaussian kernel defined in

equation (4.14).

Image fusion of the high frequency components is achieved by the

weighted, normalized sum of the deviations defined by the fusion rule;

( ) ( ){ }]w[w

wDwDD

21

2vis1irfuse +

×+×= (4.31)

The fused output is shown in Figure 4.16,

82

Figure 4.16 HF component fusion using weighted average method

The final fused output is given by

fusefusefuse DSI += (4.32)

as shown in Figure 4.17.

Figure 4.17 Fused output using the PCA – max method

83

4.5 RESULTS

Experiments were carried out with the newly developed PCA

fusion algorithms on different sets of images pertaining to surveillance and

night vision applications. The results of the experiments are tabulated in

Table 4.1. A more detailed comparison of the results is given in Chapter 6. In

addition, the average results obtained by conducting experiments on 20 sets of

images are shown in Table 4.2. Here the results are compared with the

existing DWT-PCA-Max algorithm (Zheng et al. 2007) and the PCA

weighted superposition method (Rockinger 1999).

Based on the experimental work performed using the various

quality metrics for the newly designed PCA-based fusion, the results obtained

are given in Table 4.1.

Table 4.1 Performance metrics for the newly designed PCA-based

fusion algorithms

Image Fusion Scheme En SSIM MI SD CE

Boat

PCA Max 7.58349 0.664093 4.46357 72.6022 0.397248

PCA Gaussian 7.20551 0.733626 3.92757 38.6822 0.750742

Road

Scene

PCA Max 7.62854 0.654167 4.99468 75.8738 0.171729

PCA Gaussian 7.67004 0.800508 3.54768 40.0106 0.135513

84

(a) PCA Max (b)PCA Gaussian

Figure 4.18 Fusion output for boat image

(c) PCA Max (d) PCA Gaussian

Figure 4.19 Fusion output for road scene image

Table 4.2 Average value of different set of images

Image Fusion

Scheme

PCA

Max

PCA

Gaussian

DWT

PCA Max

PCA

fusion

40 Sets

of images

CE 0.389034 0.561836 0.696009 0.69634

EN 7.123141 7.014801 6.992489 7.06783

SSIM 0.695554 0.72741 0.628094 0.577012

MI 4.292513 2.825529 3.298267 4.426463

SD 37.75803 27.95105 343.3215 68.03573

85

4.6 CONCLUDING REMARKS

This chapter gave a brief introduction to the Principal Component

Analysis followed by the application of the PCA for analyzing images. Then

the fusion rule that combines the PCA weighted scheme and selecting

maximum intensity pixels was presented in this chapter. This algorithm has a

very good performance both in terms of the visual quality of the fused image

evaluated subjectively, and also in terms of quality metrics.

Next the PCA Gaussian fusion scheme was described. This new

fusion technique designed makes use of the principal components as the

weights for the additive fusion. The performance of these two methods is

compared in Table 4.1. From the parameters it is seen that the PCA Gaussian

method gives a higher SSIM index. Also from the output image it can be

observed that this rule gives an output image of better quality when the source

images have very high intensity pixels. In case of source images with an

evenly distributed histogram the two algorithms give an equivalent

performance. These results are presented in section 6.3. A comparative study

of the fusion schemes proposed in this thesis along with existing algorithms is

presented in chapter 6.

Also the average results obtained for a set of 40 images shows a

higher value of the SSIM based measure for the two new algorithms

compared to the existing techniques. The cross entropy (CE) of the existing

methods is better and so is the standard deviation (SD). However, these two

metrics determine only the amount of information transferred from the source

images to the fused output. The observation of the fused output images

correlates the SSIM results on the efficiency of the new PCA algorithms.