Pattern Recognition 38 (2005) 459–471www.elsevier.com/locate/patcog

Implementation of real-time constrained linear discriminantanalysis to remote sensing image classification

Qian Dua,∗, Reza Nekoveib

aDepartment of Electrical and Computer Engineering, Mississippi State University, MS 39762, USAbDepartment of Electrical Engineering and Computer Science, Texas A&M University-Kingsville, TX 78363, USA

Received 26 June 2003; received in revised form 26 July 2004; accepted 15 September 2004

Abstract

In this paper, we investigate the practical implementation issues of the real-time constrained linear discriminant analysis(CLDA) approach for remotely sensed image classification. Specifically, two issues are to be resolved: (1) what is the bestimplementation scheme that yields lowest chip design complexity with comparable classification performance, and (2) howto extend CLDA algorithm for multispectral image classification. Two limitations about data dimensionality have to berelaxed. One is in real-time hyperspectral image classification, where the number of linearly independent pixels received forclassification must be larger than the data dimensionality (i.e., the number of spectral bands) in order to generate a non-singularsample correlation matrixR for the classifier, and relaxing this limitation can help to resolve the aforementioned first issue.The other is in multispectral image classification, where the number of classes to be classified cannot be greater than the datadimensionality, and relaxing this limitation can help to resolve the aforementioned second issue. The former can be solvedby introducing a pseudo inverse initiate of sample correlation matrix forR−1 adaptation, and the latter is taken care of byexpanding the data dimensionality via the operation of band multiplication. Experiments on classification performance usingthese modifications are conducted to demonstrate their feasibility. All these investigations lead to a detailed ASIC chip designscheme for the real-time CLDA algorithm suitable to both hyperspectral and multispectral images. The proposed techniquesto resolving these two dimensionality limitations are instructive to the real-time implementation of several popular detectionand classification approaches in remote sensing image exploitation.� 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Keywords:Real-time processing; Constrained linear discriminant analysis (CLDA); Classification; Hyperspectral imagery; Multispectralimagery

1. Introduction

Remote sensing images provide a repetitive and consis-tent view of the earth that is invaluable to monitoring theearth system and the effect of human activities on the earth.

∗ Corresponding author. Tel.: +1 662 325 2035;fax: +1 662 325 2298.

E-mail address:du@ece.msstate.edu(Q. Du).

0031-3203/$30.00� 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.doi:10.1016/j.patcog.2004.09.008

With the rapid advance in remote sensing instruments, itsapplication has expanded into many areas, such as environ-mental assessment and monitoring, global change detectionand monitoring, agriculture, non-renewable resource explo-ration, meteorology, natural disaster assessment, militarysurveillance and reconnaissance, etc[1]. Our research is fo-cused on the spectrum region from visible to near-infrared(typically 0.4.2.5�m), i.e., optical reflected region. Twodifferent types of sensors are used to capture the mate-rial reflectance for image construction: hyperspectral and

460 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

multispectral sensors. The former usually acquires hundredsof images in very narrow spectral bands (normally 10 nmwide) for the same area, while the latter contains only severalspectral channels in the same spectral coverage. Because ofhigher spectral resolution and resultant contiguous spectralsignatures, hyperspectral data are capable of providing moreaccurate identification of surface materials and are particu-larly useful in national defense related applications, whilemultispectral data are more frequently adopted in civilianapplications. In some cases, such as national disaster assess-ment, law enforcement activities and military applications,real-time data processing is inevitable so as to process dataon-line and provide the information for immediate response[2,3].

We have developed constrained linear discriminant analy-sis (CLDA) algorithm for hyperspectral image classification[4]. In CLDA the original high-dimensional data is projectedonto a low-dimensional space as done by Fisher’s LDA. Butdifferent classes are forced to be along different directions inthis low-dimensional space. Thus all classes were expectedto be better separated and the classification is achieved si-multaneously with the CLDA transform. The transforma-tion matrix in CLDA maximizes the ratio of inter-class dis-tance to intra-class distance while satisfying the constraintthat the means of different classes are aligned with differ-ent directions, which can be constructed by using an or-thogonal subspace projection (OSP) method coupled with adata whitening process. The experimental results in Ref.[4]demonstrated that the CLDA algorithm could provide moreaccurate classification results than other popular methods inhyperspectral image processing, such as OSP classifier[5]and constrained energy minimization (CEM) operator[6].In order to implement the CLDA algorithm in real-time, wemodified the formula and adaptively adjusted�−1, the in-verse sample covariance matrix, in our recent paper[7]. Inour research, we assume that an image is acquired from leftto right and from top to bottom. Two real-time processingfashions were discussed. One was pixel-by-pixel process-ing, and the other was line-by-line processing. In the pixel-by-pixel fashion, a pixel was processed right after it wasreceived and the analysis result was generated within an ac-ceptable delay. While in the line-by-line fashion, a line ofpixels was processed after the entire line was received. Inboth fashions, the analysis result of a pixel would not bechanged after the pixel had been processed. The experimen-tal results in Ref.[7] demonstrated the effectiveness of thereal-time CLDA algorithm.

In this paper, we investigate the practical implementationissues of the real-time CLDA for remotely sensed imageclassification. In order to determine the best implementationscheme with smallest processing delay and lowest chip de-sign complexity, we will solve the dimensionality limitationwhen the real-time CLDA algorithm is applied to hyper-spectral image classification, i.e., the real-time processingcannot get started until enough number of linearly indepen-dent pixels are received in order to generate a non-singular

data sample correlation matrixR for the real-time�−1 up-date. For instance, when classifying a 210-band HYDICEimage, we need to have at least 210 independent pixels tocalculate theR with full rank. It is not a problem for offlineprocessing since the entire image is available for theR cal-culation and the total number of pixels are much larger thanthe number of bands. But in real-time on-line processing,this limitation not only increases the processing delay, butalso complicates the chip design since a big data buffer, arank calculator, and a matrix inversion module are needed.In Ref. [7] we showed that�−1 adaptation is based onR−1

adaptation. IfR−1 can be used to replace�−1, which is tobe investigated in this paper, the processing delay can beshortened and chip design can be simplified.

The second task for practical implementation is to extendthe real-time CLDA algorithm to multispectral image clas-sification, where another dimensionality limitation shouldbe overcome, i.e, the number of classes to be classified can-not excess the data dimensionality (the number of bands).For instance, using the 6-band LANDSAT image (with ther-mal band being removed), we can classify no more than sixclasses using the CLDA algorithm. If the number of differ-ent classes present in the image scene is more than six, theCLDA algorithm cannot achieve the classification. When theCLDA algorithm is applied to hyperspectral image classifi-cation, this is not a problem since the number of bands is avery large value and generally much larger than the numberof classes. A nonlinear band generation approach will be in-troduced. In particular, we will investigate the efficiency ofa simple operation such as band multiplication that not onlyincreases the data dimensionality but also improve the classseparability.

Quantitative studies are conducted to demonstrate the per-formance of the modified real-time CLDA algorithm, basedon which a detailed ASIC chip design scheme is presented.It should be noted that the results in this paper are instructiveto the implementation of several other detection and clas-sification approaches in real-time, in particular, when theysuffer one of or both aforementioned dimensionality limita-tions [11]. For instance, the well-known RX algorithm[8]contains the calculation of�−1 and CEM requires the cal-culation ofR−1, so the real-time adaptation technique de-veloped for the CLDA algorithm can be directly adoptedand the way to overcome the limitation that the number ofreceived pixels must be larger than the data dimensionalityis also applicable to them; when the popular OSP technique,filter vector algorithm (FVA)[9] and generalized likelihoodratio test (GLRT)[10] are applied to multispectral images,they also suffer the data dimensionality limitation and thedata dimensionality expansion method to be applied to theCLDA algorithm is suitable to them.

The remainder of this paper is organized as follows. Sec-tion 2 briefly reviews the CLDA algorithm and its real-timeimplementation. Section 3 describes how to reduce the pro-cessing delay and computational complexity in hyperspec-tral image classification for chip design. Section 4 focuses

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 461

on how to apply the CLDA algorithm to multispectral im-age classification. In Section 5, HYDICE and SPOT dataexperiments are used to demonstrate the performance ofthe modified real-time CLDA algorithm in hyperspectal andmultispectral image classification. A detailed ASIC designdiagram is presented and discussed in Section 6. Finally,Section 7 concludes this paper.

2. CLDA algorithm and its real-time implementation

Assume that there arec classes and thekth class containsNk patterns. LetN = N1 + · · · + Nc be the number ofpixels. Thej th pattern in thekth class, denoted byxk

j=

[xk1j , xk2j , . . . , xkLj ]T is anL-dimensional vector (L is thenumber of spectral bands, i.e., data dimensionality). Let�=1/N

∑ck=1

∑Nkj=1 xk

jbe the global mean of training samples

and �k = 1/Nk∑Nkj=1 xk

jthe mean of thekth class. We

definedJ (F ) to be the ratio of the inter-class distance to theintra-class distance after a linear transformationF , which isgiven by

J (F )= 2/(c(c − 1))∑c−1i=1

∑cj=i+1 ‖F(�i )− F(�j )‖2

1/cN∑ck=1

[∑Nkj=1 ‖F(xk

j)− F(�k)‖2

]

(1)

and

F(x)= (WL×c)Tx = [w1,w2, . . . ,wc]Tx, (2)

wherewi is anL× 1 vector.The optimal linear transformationF ∗ is the one that max-

imizes

J (F ) subject totk = F(�k) for all k, (3)

wheretk=(0 · · · 0 1︸︷︷︸k

0 · · · 0)T is ac×1 unit column vector

with one in thekth component and zeros elsewhere.F ∗ canbe determined by

w∗i = �

Ti P⊥

Ui, (4)

where

P⊥Ui

= I − Ui (UTi Ui )

−1UTi (5)

with Ui = [�1 · · · �j · · · �c]j �=i and I the identity matrix.

The “∧” specifies the whitened data, i.e.,x = PTwx, where

Pw is the data whitening operator.Let Sdenote the entire target signature matrix, i.e.,c class

means. Without loss of generality, we assume the target tobe classified is the first signature and denoted asd, thenS= [d U]. It was proved in Ref.[7] that the CLDA-based

classifier using Eqs. (4)–(5) could be equivalently expressedas

PTc = [1 0T](ST�−1S)−1ST�−1, (6)

where0 is a (c − 1)× 1 vector with all the elements equalto 0.

The major advantage of using Eq. (6) is the capability ofreal-time implementation since the data whitening processis avoided. The key is the adaptation of�−1. The samplecorrelation matrixR is defined asR = 1/N

∑Ni=1 xi · xT

i,

which can be related to� and sample mean� by

� = R − � · �T. (7)

Using the data matrixX, Eq. (7) can be written asN · � =X · XT − N · � · �T. If � denotesN · �, R denotesN · R,and � denotesN · �, then

� = R − 1

N· � · �

T. (8)

Supposed that at timet we receive the pixel vectorxt .The data matrixXt including all the pixels received up tothe timet is Xt=[x1, x2, . . . , xt ] with Nt pixel vectors. Thesample mean, sample correlation, and covariance matricesat time t are denoted as�t , Rt , and�t , respectively. ThenEq. (8) becomes

�t = Rt − 1

Nt· �t · �

Tt . (9)

The following Woodbury’s formula[12] can be used to

update�−1t

(A + BCD)−1 = A−1 − A−1B(C−1 + DA−1B)−1

× DA−1, (10)

whereA andC are two positive-definite matrices, and thesizes of matricesA, B, C andD allow the operation(A +BCD). It should be noted that Eq. (10) is for the most generalcase. Actually,A, B, C, andD can be reduced to vector orscalar as long as Eq. (10) is applicable.

Comparing Eq. (9) with Eq. (10),A = Rt , B = �t , C =−1/Nt , D = �

Tt . Then �

−1t can be calculated using the

variables at time(t − 1) as

�−1t = R−1

t + R−1t ut (Nt − uT

t R−1t ut )−1uT

t R−1t . (11)

The �t can be updated by

�t = �t−1 + xt . (12)

SinceRt andRt−1 can be related as

Rt = Rt−1 + xt · xTt (13)

R−1t in Eq. (13) can be updated by using the Woodbury’s

formula again:

R−1t = R−1

t−1 − R−1t−1xt (1 + xT

t R−1t−1xt )−1xT

t R−1t−1, (14)

462 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

whereRt−1, xt , 1 andxTt correspond toA, B, C andD in

Eq. (10), respectively.Eqs. (12)–(14) are for pixel-by-pixel processing. If the

data processing is carried out line-by-line from left to rightand top to bottom in an image, the line received at timet

forms a data matrixYt = [xt1xt2 · · · xtM ], whereM is thetotal number of pixels in each line. Assume that the numberof lines received up to timet is Kt , then Eq. (11) remainsalmost the same as

�−1t = R−1

t + R−1t ut (KtM − uT

t R−1t ut )−1uT

t R−1t . (15)

Eq. (12) becomes

�t = �t−1 +M∑i=1

xt i , (16)

wherext i is theith pixel in the line received at timet . AndEq. (13) becomes

R−1t = R−1

t−1 − R−1t−1Yt (IM×M + YT

t R−1t−1Yt )−1

× YTt R−1

t−1, (17)

whereIM×M is anM ×M identity matrix.

3. Modified real-time CLDA for hyperspectral imageclassification

The major objective of algorithm modification is to sim-plify the resultant chip design scheme. Two ways are to beproposed, which also enable the reduction of processing de-lay, i.e., the time between a pixel (or a line of pixels forthe line-by-line processing) is received and its processingresult is generated. It is expected that this time is as shortas possible.

3.1. R−1 adaptation initiative

When the CLDA is applied to hyperspectral images, onelimitation related to the data dimensionality is that the num-ber of independent pixel vectors in processing must begreater than the number of bands such thatR is full rankand R−1 exists. In offline processing it is not a problem,because the number of pixel vectors in an image scene gen-erally is very large. However, when pixels are analyzed rightafter they are received in on-line processing, the number ofpixels may not be enough. Eqs. (11)–(17) show that in the

update of�−1t the most important is the update ofR−1

t . In

order to calculate theR−1t at each timet , we must know

the first initial R−10 in Eq. (14) or Eq. (17). In the real-time

version in Ref.[7], we waited for enough number of lin-early independent pixels being received to calculateR0 andR−1

0 . This means extra processing delay was introduced.Such a processing structure will also complicate the chip

design because extra data buffer, rank and matrix inversioncalculation modules are required.

An easy way to deal with this problem is to use a pseudoinverse correlation matrixR−1

0 with full rank for initializingthe real-time algorithm. Then the on-line processing canbe performed upon the arrival of the first pixel. A simplechoice ofR−1

0 = �−1I , where� is a constant andI is theidentity matrix of sizeL×L. When the first pixel is received,R1 = x1xT

1 ≈ R0 + x1xT1 = �I + x1xT

1 . According to the

Woodbury’s formula in Eq. (10),R−11 ≈ R−1

0 − R−10 x1(1+

xT1 R−1

0 x1)−1xT

1 R−10 = �−1I − �−2x1(1+ �−1xT

1x1)−1xT

1 ,

where the approximation holds whendiag(x1xT1 )?�. Since

the pixel values correspond to solar reflectances, which arewithin [0, 1], this condition can be satisfied by letting�being small values. In our experiments, we find�=0.001 isa good choice, which will not change the final classificationresults.

3.2. Practical consideration about the implementation ofmatrix inversion

The chip implementation of matrix inversion is very ex-pensive, in particular, when the dimension of a matrix islarge. In line-by-line processing, a matrix inversion modulerequired for the term(IM×M+YT

t R−1t−1Yt )−1 in Eq. (17) is

difficult to achieve since theM can be a large number (e.g.,M is 64 in the HYDICE experiment, and 256 in the SPOTexperiment). In pixel-by-pixel processing, when a pseudoinverseR−1

0 = �−1I is used, there is no need for any ma-

trix inversion module during theR−1 adaptation. The onlymatrix inversion module is the one for(ST�−1S)−1 in Eq.(6) in classification step, whose size isc× c. Sincec is thenumber of classes which usually is not a large number, itschip implementation is achievable. Therefore, we will focuson pixel-by-pixel processing thereafter.

3.3. R−1 versus�−1

According to Section 2,R−1 update only includes onestep (see Eq. (14)), while�−1 update has three steps (seeEqs. (14), (12) and (11)). The number of multiplicationssaved by usingR−1 is 5L2 for each update. UsingR−1

instead of�−1 can also reduce the number of modules inthe chip. Then Eq. (7) will be changed to

PTc = [1 0T](STR−1S)−1STR−1. (18)

From the image processing point of view, the functions ofR−1 and�−1 in the operator are both to suppressing the un-desired background interference before applying the matchfilter ST. Based on our experience on different hyperspec-tral/multispectral image scenes, usingR−1 generates veryclose results to using�−1.

In summary, the modified CLDA-based real-time hy-perspectral image classification algorithm can be de-

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 463

scribed as follows, based on which an ASIC chip is to bedesigned.

(1) SetR−10 = �−1I . Construct the signature matrixS.

(2) After a new pixelxt is received at timet , updateR−1t

using Eq. (14).(3) Classify the pixelxt using PT

c xt in Eq. (18), and thendisplay the partial result. Lett = t + 1, and go to step 2for classifying next pixel.

It is worth noting that usingR−1t instead ofR−1

t will

not change the result, since the constant included inR−1t

(i.e.,Nt , the number of pixels being received at timet) iscancelled in Eq. (18).

4. Real-time CLDA for multispectral imageclassification

When the CLDA algorithm is applied to multispectralimages, another limitation related to the data dimensionalityis that the number of classes to be classified cannot excessthe number of bands. Otherwise, the term(ST�−1S) in Eq.(7) or (STR−1S) in Eq. (18) is singular and the CLDAalgorithm is not applicable. It may not be a problem for ahyperspectral image, since the number of bandsL is verylarge, and the number of classes present in the image scenec

usually is much less thanL. However, if it is a multispectralimage, then we have to find a way to solve this problem.

A simplest solution is to use a dimensionality expan-sion process to generate a set of artificial images so thatthere are enough dimensionalities to accommodate a certainnumber of classes[13]. Its basic idea was to use nonlin-ear functions to generate new bands, which may offer ad-ditional information for object detection and classification.But there was no specific rule for nonlinear function se-lection. We find out that a nonlinear function that can en-large or emphasize the discrepancy between original spectralmeasurements will help to improve classification perfor-mance, because the techniques we apply here are based onspectral information. A simplest but effective choice is mul-tiplication. When two original imagesBi andBj are mul-tiplied together (each pair of corresponding pixels is mul-tiplied), a new image is generated, i.e.,{BiBj }L

i,j=1,i �=j .Here multiplication acts as match filtering. When the mul-tiplicant and multiplier are equal, the product is maximal(at the quantity level of the multiplicant and multiplier). Sothe operation of such a band multiplication emphasizes thesimilarity between two original spectral measurements inBi

and Bj , which is equivalent to emphasizing their dissimi-larity. Multiplication can be also used for a single band, i.e.,{BiBi}L

i=1, which is to emphasize itself, and also equivalentto enlarge the spectral difference from other original spec-tral measurements. Adding these two sets of artificial bands,

there are totallyl+ l+(l2

)= l2/2+3l/2 bands for process-

ing. According to our experience, these two sets of spectralbands can provide useful information about class character-istics. Another good feature of multiplication is that a mul-tiplier can be very easily implemented in chip design.

Table 1shows four three-dimensional vectors, which areassumed to represent four different class means. The firstpixel vectorxT

1 = [0.2 0.4 0.6], andx2, x3, x4 were gen-erated by adding a random value within[−0.1 + 0.1] toeach element ofx1. So the spectral signatures of these fourvectors are very close. Since there are only three bands,we need to generate more for classification. The multipli-cation operation was used to generate additional six bands.The spectral difference can be measured by spectral anglemapper (SAM)[14], which calculates the angle betweentwo unit vectors, i.e.,d(xi , xj )=cos−1(xT

ixj /|xi | · |xj |). If

xi=xj , thend(xi , xj )=0; if xi ⊥ xj , thend(xi , xj )=1. Forall other cases, 0<d(xi , xj )<1. A larged(xi , xj ) meansxi andxj are very dissimilar and easy to be differentiatedfrom each other.Tables 2and 3 list the spectral distancesbetween each pair of vectors before and after data dimen-sionality expansion. We can see that the distances of con-structed 9-dimensional vectors are larger than their coun-terparts, which means using multiplication can enhance thespectral differences. In other words, the class separability isimproved.

Another intuitive way to expand the data dimensionalityis to apply an interpolation method between two adjacentbands, such as cubic spline[15]. In Table 4the original four3D vectors were interpolated, three additional bands weregenerated by interpolatingB1 andB2, denoted as I(B1B2)1,I(B1B2)2, and I(B1B2)3, and another three bands were gen-erated by interpolatingB2 and B3, denoted as I(B2B3)1,I(B2B3)2, and I(B2B3)3. The distances among this new setof 9D vectors were calculated and listed inTable 5. Com-pare toTable 2, they are no larger than the distances amongthe original vectors. This is because the linear nature of sucha band generation technique does not bring about new in-formation for differentiating these vectors from each other.Interestingly, all the distances inTable 5are even smallerthan their counterparts inTable 2. This is due to the fact thatit is more difficult to separate two similar classes when thedata dimensionality goes high since more data samples arerequired for comparable classification[16,17].

By incorporating this nonlinear dimensionality expansionprocess into the modified real-time CLDA algorithm in Sec-tion 3, the on-line multispectral classification algorithm canbe described as follows:

(1) SetR−10 = �−1I . Construct the signature matrixS.

(2) After a new pixelxt is received at timet , expand thedimensionality via band multiplication. The resultingpixel vector is denoted asxt .

(3) UpdateR−1t using xt with Eq. (14).

(4) Classify the pixelxt usingPTc xt in Eq. (18), and then

display the result. Lett = t + 1, and go to step 2 forclassifying next pixel.

464 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

Table 1Four three-dimensional pixel vectors with six generated bands

B1 B2 B3 B1B2 B2B3 B1B3 (B1)2 (B2)2 (B3)2

x1 0.2000 0.4000 0.6000 0.8000 0.2400 0.1200 0.0400 0.1600 0.3600x2 0.2900 0.3462 0.6214 0.1004 0.2151 0.1802 0.0841 0.1199 0.3861x3 0.1913 0.3037 0.6643 0.0581 0.2017 0.1271 0.0366 0.0922 0.4413x4 0.1889 0.4231 0.6584 0.0799 0.2786 0.1244 0.0357 0.1790 0.4335

Table 2SAM-based spectral distances between four original 3-D vectors

x1 x2 x3 x4

x1 0 0.1388 0.1544 0.0374x2 0 0.1512 0.1617x3 0 0.1394x4 0

Table 3SAM-based spectral distances between four nonlinearly generated9-D vectors

x1 x2 x3 x4

x1 0 0.1540 0.1796 0.0563x2 0 0.1697 0.1797x3 0 0.1611x4 0

5. Experiments

In this section, experiments were conducted to demon-strate the feasibility of the modified CLDA algorithm pro-posed in Section 3 for chip design simplification and theone in Section 4 for multispectral image classification. Thesame HYDICE image scene as in Ref.[7] was used, butthe purpose here is to conduct quantitative study since thepixel-level ground truth is available.

Table 4Four three-dimensional pixel vectors with six generated bands

B1 B2 B3 I (B1B2)1 I (B1B2)2 I (B1B2)3 I (B2B3)1 I (B2B3)2 I (B2B3)3

x1 0.2000 0.4000 0.6000 0.2500 0.3000 0.3500 0.4500 0.5000 0.5500x2 0.2900 0.3462 0.6214 0.2836 0.2908 0.3117 0.3945 0.4564 0.5321x3 0.1913 0.3037 0.6643 0.1961 0.2165 0.2523 0.3706 0.4530 0.5509x4 0.1889 0.4231 0.6584 0.2474 0.3059 0.3644 0.4818 0.5406 0.5995

5.1. Hyperspectral experiment

The image scene of size 128× 64 shown inFig. 1(a)was collected in Maryland in 1995 from the flight altitudeof 10,000 ft with approximately 1.5 m spatial resolution in0.4.2.5�m spectral region. The atmospheric water bandswith low signal-to-noise ratio were removed reducing thedata dimensionality from 210 to 169. The image scene has128 lines and the number of pixel in each lineM is 64, sothe total number of pixel vector is 128× 64= 8192. Thisscene includes 30 panels arranged in a 10× 3 matrix. Eachelement in this matrix is denoted bypij with row indexedby i = 1, . . . ,10 and column indexed byj = a, b, c. Thethree panels in the same rowpia, pib, pic were made fromthe same material of size 3 m× 3 m, 2 m× 2 m, 1m× 1 m,respectively, which could be considered as one class, Pi. Theground truth map inFig. 1(b) shows the precise locationsof the panel center pixels. One of pure pixels in each panelin the leftmost column is used as class signature inS. Inaddition, there were two nature objects: grass field as thebackground and tree line to the left. Therefore, the matrixScontains 12 signatures.

This image was classified by using offline processing andpixel-by-pixel on-line processing with the original�−1 up-date scheme (Eq. (6)) and�−1 being replaced withR−1

(Eq. (18)), respectively. In order to compare with the pixel-level ground truth, the generated gray scale classificationmaps were normalized into [0,1] dynamic range and con-verted into binary images using a threshold 0.5. The num-bers of correctly classified pure panel pixelsND in differentcases were counted and listed inTables 6and7. Here thenumber of false alarm pixelsNF = 0 in all the cases, whichmeans the ten panel classes were well separated. We can see

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 465

Table 5SAM-based spectral distances between four linearly generated 9-Dvectors

x1 x2 x3 x4

x1 0 0.1083 0.1495 0.0315x2 0 0.1397 0.1245x3 0 0.1317x4 0

Fig. 1. (a) A HYDICE image scene which contains 30 panels. (b)Spatial locations of 30 panels provided by ground truth.

Table 6Classification accuracyND when using inverse sample covariancematrix �−1 in Eq. (7)

# of Offline On-line Modifiedpure pixels processing processing on-line processing

P1 3 2 2 3P2 3 2 2 2P3 4 3 3 3P4 3 2 2 2P5 6 5 6 5P6 3 2 2 2P7 4 3 3 4P8 4 3 3 3P9 4 3 3 3P10 4 3 3 3Total 38 28 29 30

In all cases, the number of false alarm pixelsNF = 0.

that classification performances provided in different casesare comparable. On-line real-time CLDA algorithm can cor-rectly classify 29 to 30 out of 38 panel pixels, while theoffline CLDA algorithm can correctly classify 28 out of38 panel pixels. Interestingly, on-line processing yieldedslightly better results, where the major difference is somepanel pixelspic at the rightmost column were classified bythe online versions but missed by offline version. These pan-

Table 7Classification accuracyND when using inverse sample correlationmatrix R−1 in Eq. (18)

# of Offline On-line Modifiedpure pixels processing processing on-line processing

P1 3 2 3 3P2 3 2 2 2P3 4 3 3 3P4 3 2 2 2P5 6 5 5 5P6 3 2 2 2P7 4 3 3 4P8 4 3 3 3P9 4 3 3 3P10 4 3 3 3Total 38 28 29 30

In all cases, the number of false alarm pixelsNF = 0.

els are embedded at the sub-pixel level (1 m× 1 m size issmaller than 1.5 m × 1.5 m pixel size). Whether they canbe classified partially depends on how well the backgroundsuppression (the function ofR−1 or �−1) is, i.e., if theR or� can reflect the local background statistics. In the on-lineclassifierR−1 or �−1 is generated and updated based on thereceived pixels instead of all the pixels in the entire imagescene as in the offline case, so they may have a chance tobetter reflect the background statistics surrounding thepicpixels. As a result, in some cases online version might offerslightly better classification.

This experiment demonstrates that the pseudoR−10 pro-

posed in Section 3 to overcome the dimensionality limita-tion is feasible, which can save the processing time withoutaffecting the classification accuracy. In order to further fas-ten the real-time processing, the�−1 update in the originalCLDA algorithm can be replace byR−1 without affectingthe overall classification accuracy.

5.2. Multispectral experiments

5.2.1. Example 1: SPOT experiment—a qualitative studyThe multispectral image used in the first example is a

SPOT image about Falls Church, Virginia of size 256×256shown inFig. 2 (left column), which contains only threebands (Green, Red, Near-IR) with 20 m spatial resolu-tion. According to prior information, there are at leastfour classes present in this image scene: {“buildings”,“highway”, “vegetation”, “water”}, whose class meanswere manually determined by selecting samples in smallregions. In order to make the CLDA algorithm applicable,we generated three additional bands by multiplying eachpair of bands. So totally there were six band images used inthe real-time processing as shown inFig. 2 (right column).It should be noted that such generation was also performedin real-time. Fig. 3 shows the classification result using

466 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

Fig. 2. The original three band images (Band 1–3) and three gen-erated artificial band images (Band 4–6).

OSP approach[5], a popular technique in remote sensingimage analysis, where “highway” was not detected and theclassification maps for “buildings” and “water” containedpixels belonging to “vegetation”. The real-time CLDAclassification results are depicted inFig. 4, which weregenerated by using Eq. (18) for each class. We can see thatall the four classes were more correctly classified exceptthat some man-made “buildings” were classified into theclass of “highway” for their reflectance (based upon thematerials they made from) may be close to each other.

5.2.2. Example 2: simulated multispectral experiment—aquantitative study

In order to conduct a quantitative study by taking advan-tage of the ground truth in HYDICE data, in the secondexample we generated an artificial multispectral HYDICEimage by uniformly selecting 10 bands from the originalHYDICE data in 5.1. Since there are 12 signatures present,more bands must be generated. Each adjacent pair of bands

Fig. 3. The OSP classification results of the SPOT image.

Fig. 4. The CLDA classification results of the SPOT image.

was multiplied to generate a new band. These 9 additionalbands along with 10 original bands were used for CLDAclassification. Then on-line pixel-by-pixel processing fash-ion was used forR−1 update only in Eq. (18) with pseudoR−1

0 = �−1I . The offline processing was also conducted for

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 467

Table 8Classification accuracy in the artificial multispectral experiment

# of pure pixels Offline (19 bands) On-line (19 bands) Offline (55 bands) On-line (55 bands)

ND NF ND NF ND NF ND NF

P1 3 2 25 3 13 2 0 3 0P2 3 2 306 2 215 2 0 2 5P3 4 4 5144 4 5955 3 0 3 0P4 3 2 225 2 307 2 0 2 0P5 6 5 0 5 0 5 0 5 0P6 3 3 5703 3 7077 1 0 1 0P7 4 4 5621 4 2721 3 2 3 1P8 4 4 4715 4 4613 2 0 2 0P9 4 4 299 4 318 3 0 3 0P10 4 3 26 3 21 3 0 3 0Total 38 33 22064 34 21240 26 2 27 6

comparison purpose, and the number of false alarm pixelsNF and correctly detected panel pixelsND were calculatedand listed inTable 8. We can see that only P1, P5, and P10were well classified because of the smallNF ’s. All otherseven panel classes were not classified because theNF ’swere too large (although the TotalND was large, all thesepanel pixels were submerged in the background pixels). Inorder to improve the classification performance, each pair

of bands were multiplied and(

102

)= 45 new bands were

generated. Now the classification was greatly improved bygenerating more artificial bands andNF were significantlyreduced to 2 and 6, which means a much better backgroundsignature suppression. Also, the on-line classification per-formance is comparable to the offline case.

These experiments demonstrate that the dimensionalityexpansion process described in Section 4 is effective in mul-tispectral image classification when the number of originalband images is not enough to accommodate different classes.It was successfully integrated into the real-time CLDA al-gorithm for on-line classification. Even when the number ofbands is greater than the number of classes to be classifiedand CLDA is directly applicable, using the dimensionalityexpansion process the algorithm performance may be im-proved since more artificial bands provide more additionalspectral information. However, its performance is limited bythe content in the original images because it can only “digout” the information hidden in the original data but can-not create brand new information. This is also why in theExample 1 “highway” cannot be completely extracted from“buildings” no matter how many nonlinear bands were gen-erated. Their spectral difference in the original images wastoo tiny to work out.

6. ASIC chip design scheme

The efficient ASIC implementation of a classificationalgorithm requires intensive exploitation of its intrinsic

Update −1~

R

CLDA TcP

Classify 2−t

Tc xP

S1 S2 S3

Classified Pixel

tx

1~−R S 21:: −− ttt xxx

Loaded at initialization Memory banks

0

Fig. 5. Classification pipeline stages.

parallelisms and pipelines. Based on the aforementionedCLDA algorithm, we propose a three-stage pipeline struc-ture as is shown inFig. 5. Before the classification begins,target signature matrixS and initial inverse correlationmatrix R−1

0 are loaded into the memory banks. The firstcomputational stage updates the inverse correlation matrixR−1 after the arrival of each pixel (after band multiplica-tion for multispectral images). The pixel is also stored ina temporary circular buffer for use during the third stage.Using the updatedR−1 and the storedS, the CLDA-basedtarget classifier is calculated in the second stage. Finally,the third stage classifies the pixel, which arrived twostages earlier and was queued. Each stage of this pipelinecontains further optimizations, which is described asfollows.

6.1. Selection of numbering system and bit-depth

Selection of numbering system and the bit-depth to beused has a major effect on the system’s performance andclassification accuracy. Although, floating-point operationsrequire more chip area than fixed-point operations, during

468 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

* + A1 Reg. B1 Reg. MUX2

Sel2

* + A2 Reg. B2 Reg. MUX2

Sel2

MUX1

Sel1

* + AL Reg.

BL Reg. MUX2

Sel2

MUX1

Sel1

DEMUX

Sel1 Inner Reg.

DEMUX

C Reg.

Sel3

*

*

*

*

II

�

�

∗ Outer

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

MAC1 PMVM with Inner Product

Fig. 6. Architecture for inverse correlation matrix adaptation.

the R−1 adaptation it becomes necessary to represent verylarge and very small values. On one hand, such a com-plex operation with fixed-point arithmetic becomes too ex-pensive as the bit-widths required for maintaining both theadequate precision and range increases. However, on theother hand, full precision IEEE floating-point standard 754[17] is a general-purpose representation and its large bit-widths will significantly raise the hardware complexity. Con-sequently, a customized floating-point representation simi-lar to the IEEE with much smaller bit-widths is sufficientto achieve needed precision with fewer hardware resources.Next, we need to assign the appropriate bit-width to ex-ponent and mantissa (fraction), noting that assigning morebits to the exponent results in a higher data range whileassigning more bits to the mantissa results in better preci-sion. Optimizing these trade-offs for the classification al-gorithm resulted in a sixteen-bit custom floating point for-mat, which is represented by a one-bit mantissa sign-bit(s),a six-bit exponent(e), and a nine-bit mantissa magnitude(m). The exponent is in 2’s complement and it is normal-ized so that the mantissa is between 0.5 and 1.0 with themost significant bit not stored, since it is always one afternormalization. Hence, the value for a given representationcan be calculated by(−1)sm2e, which is analogous to the

IEEE standard and is simple to convert to and from IEEEformat.

6.2. First stage: inverse correlation matrix adaptation

The first stage calculates the new inverse correlation ma-trix R−1

t by updatingR−1t−1 as dataxt becomes available ac-

cording to the Eq. (14). During initializationR−10 is loaded.

Next, it is iteratively updated based on the input data. Eq.(14) requires three matrix–vector multiplications, an innervector product and an outer vector product. However, allthe matrix–vector multiplications are calculating the samevalues due to the symmetric nature ofR−1

t (i.e. xTt R−1

t−1 =((R−1

t−1)Txt )T). Additionally, the second term in Eq. (14)

will result in a scalar value due to the vector inner product.Therefore, Eq. (14) can be reorganized to the following formR−1t = R−1

t−1 − �(R−1t−1xt )(R

−1t−1xt )T, where scalar value

� = (1 + xTt R−1

t−1xt )−1is the increment inverse of the in-ner product between the vector-matrix multiplication result,R−1t−1xt , and the input data. This stage’s hardware consists

of three parallel pipeline sub-stages, a parallel matrix–vectormultiplier (PMVM) with an inner product generator, a par-allel outer product generator, and a stage to updateR−1

t in

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 469

MMM 1~ −RST

MMM SRS )

~( 1−T

MI 11 )

~( −− SRST

MMM 111 ~

)~

( −−− RSSRS TT

S1 S2 S3 S4

Buffer ( 1~ −RST )

Fig. 7. Pipelined calculation stages for three MMM and a MI.

INPUT VECTORS

O U T P U T V E C T O R S

MAC0,1 MAC0,L-1Mult 0

MAC1,1 MAC1,N-1Mult 1

MACM-1,1 MACM-1,N-1MulM-1

. . .

. . .

. . .

.

.

.

.

.

.

.

.

.

. . .

Sel

*

Mult

DEMUX

Reg

Sel

*

MAC

DEMUX

+ Reg

Fig. 8. Architecture for the general scalable MMM.

the memory.Fig. 6 shows the first two stages of the pro-posed architecture. It consists ofL rows of hardware foran L × L inverse correlation matrix. The PMVM is usedto computeR−1

t−1xt and store the results in registersB1

throughBL. Next, the inner product(xTt )(R

−1t−1xt ) is cal-

culated by setting the multiplexers to pipeline all the paral-lel multiply and accumulate (MAC) units. The final valueis stored in theInner Register. The parallel outer productgenerator pre-scales its multiplication by� during theLcycles required to generate the outer product matrix. The� value is calculated by passing the inner product throughan increment inverse module(II ). The II module is anoptimized floating-point adder and divider simplified forfast computation of(1 + A)−1. Here,A = xT

t R−1t−1xt is a

scalar.

6.3. Second stage: CLDA classifier construction

This is the most computational intensive stage and cor-responds to calculation ofPT

c using Eq. (18). It requiresthree matrix–matrix multiplications (MMM) and one ma-trix inversion (MI) sub-stages as illustrated inFig. 7. Whenc is the number of classes to classify andL is the data di-mension size, the first sub-stage is a(c × L)(L × L) mul-tiplier which calculatesSTR−1, the second sub-stage is a(c × L)(L × c) multiplier calculating the(c × c) resultingmatrix, the third sub-stage inverses the(c × c) matrix, andfinally the last stage is a(c×c)(c×L)multiplier calculating(STR−1S)−1STR−1. We designed a scalable fully parallelpipelined matrix multiplier to perform MMM; its generalarchitecture is shown inFig. 8. This scalable approach al-

470 Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471

lows for reducing the fully parallel pipelined multiplier to apartial parallel pipelined multiplier by decreasing hardwarerows (reusing the hardware) as the trade-off between the de-lay versus power and area requirements becomes known forselected fabrication technology. Unfortunately in the thirdsub-stage, the algorithm requires an expensive matrix inver-sion. However, it should be noted that the matrix to be in-verted(STR−1S)−1 has dimensionc which is much smallerthan that of sample correlation matrix. This matrix is in-verted using a fast systolic matrix inversion circuit in Ref.[18].

6.4. Third stage: pixel classification

At this stage each pixel is classified usingPTc xt−2. A cir-

cular buffer is used to store values ofxt , which is neededduring the classification stage. This buffer queues three pixelvectors before reusing the memory. A parallel matrix–vectormultiplier similar to the one inFig. 6, without the multi-plexers being used to calculate the inner product, is the onlyhardware required for this stage.

7. Conclusion

This paper investigates the implementation issues of thereal-time CLDA algorithm in remote sensing image classi-fication. Two data dimensionality limitations are resolved.One limitation is in online hyperspectral image classifica-tion, where the number of pixels received for processingmust be greater than the number of bands. A pseudoR−1

0can be used to initialize the real-time adaptation ofR−1,and then the real-time processing can get started upon thearrival of the first pixel so as to reduce the processing delayand future chip design complexity. The other limitation isin multispectral image classification, where the number ofclasses to be classified cannot be greater than the numberof bands. The dimensionality expansion process can be ap-plied to generate nonlinear artificial bands such that thereare enough dimensions to accommodate different classes.We find out that a simply operation as band multiplicationcan accomplish this task, which can also enhance the classseparability. Such an operation can be easily integrated intothe real-time CLDA algorithm and implemented in chip de-sign. The HYDICE and SPOT experiments show the ef-fectiveness of our proposed techniques for relaxing thesetwo data dimensionality limitations. We suggest the pixel-by-pixel processing because it yields the lowest chip designcomplexity.

The results from this paper are suitable to several other re-mote sensing image detection and classification approaches,whereR−1 or �−1 update is involved, and one of or bothdimensionality limitations need to be relaxed.

Acknowledgments

This research was supported by National Geospatial-intelligence Agency (NGA) through Grant NMA401-02-1-2015. The authors would like to thank Professor Chein-IChang at University of Maryland Baltimore County forproviding the image data used in the experiments.

References

[1] R.A. Schowengerdt, Remote Sensing, Models and Methodsfor Image Processing, second ed., Academic Press, New York,1997.

[2] C.M. Stellman, G.G. Hazel, F. Bucholtz, J.V. Michalowicz,Real-time hyperspectral detection and cuing, Opt. Eng. (2000)1928-1935.

[3] C.-I Chang, H. Ren, S.S. Chiang, Real-time processingalgorithms for target detection and classification inhyperspectral imagery, IEEE Trans. Geosci. Remote Sensing39 (4) (2001) 760–768.

[4] Q. Du, C.-I Chang, Linear constrained distance-baseddiscriminant analysis for hyperspectral image classification,Pattern Recognition 34 (2) (2001) 361–373.

[5] J.C. Harsanyi, C.-I Chang, Hyperspectral image classificationand dimensionality reduction: an orthogonal subspaceprojection, IEEE Trans. Geosci. Remote Sensing 32 (4) (1994)779–785.

[6] W.H. Farrand, J.C. Harsanyi, Mapping the distribution of minetailing in the coeur d’Alene river valley, Idaho through theuse of constrained energy minimization technique, RemoteSensing Environ. 59 (1997) 64–76.

[7] Q. Du, H. Ren, Real-time constrained linear discriminantanalysis to target detection and classification in hyperspectralimagery, Pattern Recognition 36 (1) (2003) 1–12.

[8] I.S. Reed, X. Yu, Adaptive multuple-band CFAR detection ofan optical pattern with unknown spectral distribution, IEEETrans. Acoustic Speech Signal Process. 38 (10) (1990) 1760–1770.

[9] J. Bowles, P. Palmadesso, J. Antoniades, M. Baumback, Useof filter vectors in hyperpectral data analysis, Proc. SPIE 2553(1995) 148–152.

[10] L. Scharf, B. Friedlander, Matched subspace detector, IEEETrans. Signal Process. 42 (8) (1994) 2146–2157.

[11] Q. Du, B. Shia, Realtime online processing for remote sensingimagery, Proceedings of SPIE’s 17th International Symposiumon AeroSense, vol. 5093, SPIE, Orlando, Florida, April 2003,pp. 686–693.

[12] G.H. Golub, C.F. Van Loan, Matrix Computations, third ed.,Johns Hopkins University Press, Baltimore, MD, 1996.

[13] H. Ren, C.-I Chang, A generalized orthogonal subspaceprojection approach to unsupervised multispectral imageclassification, IEEE Trans. Geosci. Remote Sensing 38 (6)(2000) 2525–2528.

[14] F.A. Kruse, A.B. Lefkoff, J.W. Boardman, K.B. Heidebrecht,A.T. Shapiro, P.J. Barloon, A.F.H. Goetz, The spectralimage processing system (SIPS)-interactive visualization andanalysis of imaging spectrometer data, Remote SensingEnviron. 44 (1993) 145–163.

Q. Du, R. Nekovei / Pattern Recognition 38 (2005) 459–471 471

[15] G.F. Hughes, On the mean accuracy of statistical patternrecognizers, IEEE Trans. Inf. Theory IT-14 (1968) 55–63.

[16] D. Landgrebe, Hyperspectral image data analysis, IEEE SignalProcess. Mag. 19 (1) (2002) 17–28.

[17] IEEE Standard Board and ANSI, IEEE Standard for BinaryFloating-Point Arithmetic, IEEE Std 754-1985, 1985.

[18] A. El-Amawy, A systolic architecture for fast dense matrixinversion, IEEE Trans. Comput. 38 (3) (1989) 449–455.

About the Author —QIAN DU received her Ph.D. degree in Electrical Engineering from University of Maryland Baltimore County in2000. She is currently an Assistant Professor at the Department of Electrical and Computer Engineering, Mississippi State University.Before joining MSU, she was with the Department of Electrical Engineering and Computer Science, Texas A&M University-Kingsville.Her research interests include signal and image processing, pattern recognition, neural networks and remote sensing. She is a member ofIEEE, SPIE, ASPRS and the honor society of Phi Kappa Phi.

About the Author —REZA NEKOVEI received his Ph.D. degree in Electrical Engineering from The University of Rhode Island in 1994. Heis currently an Associate Professor at the Department of Electrical Engineering and Computer Science, Texas A&M University-Kingsville.He was previously with the Remote Sensing Laboratory, Graduate School of Oceanography at The University of Rhode Island. His researchinterests include VLSI Architectures, Distributed Systems, Remote Sensing, and Medical Imaging. He is a member of IEEE and Etta KappaNu honor society.