image fusion with some emphasis on cwd - lehigh ece r.s. blum - lehigh university 16 nmdb image...

Image Fusion with some Emphasis on CWD

R. S. [email protected]

ECE Dept., Lehigh University

This material is based on work supported by the U. S. Army Research Office under grant number DAAD19-00-1-0431. The content of the information does not necessarily reflect the position or the policy of the federal government, and no official endorsement should be inferred.

Several Graduate Students of the SPCRL Lab at Lehigh contributed greatly to this work:

Zhong Zhang Zhiyun Xue

Jinzhong YangFor more information and papers please see

our website at: http://www.ece.lehigh.edu/SPCRL/spcrl.htm

06/24/2002 R.S. Blum - Lehigh University 3

Outline

Introduction to Image FusionMethods for Image FusionPerformance Testing for CWD A Signal Processing Approach Other Lehigh Contributions Conclusions


Image SensorsOptical cameras, millimeter wave (MMW) cameras, infrared (IR) cameras, x-ray imagers, radar imagersOptimized for different operating conditionsDifferent characteristics of the generated image data

Introduction


Concept of Image FusionA process of combining information from multiple images to generate a single image that contains a more accurate description of the scene than any of the individual source images

Introduction


Advantages of Image FusionMore completeMore accurateMore robustIn less timeAt a lower costImprove human visual perceptionImprove automatic computer analysis

Introduction


Applications of Image FusionConcealed weapon detectionDigital Camera Remote sensingIntelligent robotsMedical diagnosisDefect inspectionsurveillance

Introduction


(a) Image from CCD (b) Image from MMW (c) Fused image

Concealed Weapon Detection

*The source images were obtained from Thermotex Corporation

Our current focus (see our website)


Digital Camera

(a) Focus on the left (b) Focus on the right (c) Fused image (all-focus)

A past focus (see our website)


IntroductionCategory of Fusion

Signal-level fusionPixel-level fusion = Image Fusion (focus)Feature-level fusionDecision-level fusion (see our website: we have an extensive set of papers.)


IntroductionMethods of Image Fusion

Non-multiscale-decomposition-based (NMDB) methodsMultiscale-decomposition-based (MDB) methods


Methods of Image FusionNon-Multiscale-Decomposition-Based (NMDB) Methods

Pixel-level weighted averagingNonlinear methodOpponent Color Processing Artificial neural networkEstimation theory based methods


Pixel-level weighted averagingMethodology

To take the weighted average of the pixel intensity of the two source images

ExamplePrinciple component analysis (PCA): O. Rockinger, and T. Fechner, "Pixel-level image fusion: The case of image sequences", Proceedings of SPIE, vol. 3374, 1998Adaptive weight averaging (AWA): E. Lallier and M. Farooq, “A real time pixel-level based image fusion via adaptive weight averaging”, ISIF 2000

NMDB Fusion Methods

*Example methods chosen selected for CWD tests shown later.


NMDB Fusion MethodsNonlinear Method

MethodologySeparate images into low-pass & high-pass componentsadaptively modify each component, fuse, then addlow-pass: enhance each local luminance mean & fuse the by nonlinear mappinghigh-pass: fuse by weighted averaging

ExampleC. W. Therrien and W. K. Krebs, “An adaptive technique for the enhanced fusion of low-light visible with uncooledthermal infrared imagery”, ICIP 1997


NMDB Image Fusion MethodsOpponent Color Processing

MethodologyUse biological models of opponent-color processing to fuse low-light visible and thermal IR imagery, and render it in color

ExampleA. M. Waxman, M. Aguilar, R. A. Baxter, et.al. “Opponent-color fusion of multi-sensor imagery: visible, IR and SAR”, Proceedings of IRIS Passive Sensors, vol.1, pp.43-61, 1998


NMDB Image Fusion MethodsArtificial Neural Networks

MethodologyMotivated by the fusion of different sensor signals in biological systemsMulti-layer perceptron neural networks and pulse-coupled neural networks

ExampleT. Fechner and G. Godlewski, “Optimal fusion of TV and infrared images using artificial neural networks”, Proceedings of SPIE, vol.2492, pp.919-925, 1995J. M. Kinser, “Pulse-coupled image fusion”, Optical Engineering, vol.36, no.3, pp.737-742, 1997


NMDB Image Fusion MethodsEstimated Theory Based Methods

MethodologyImage formation model and the prior modelsMAP estimate, ML estimate, others. Bayes formula (s=scene, z=observations)

ExampleR. K. Sharma, T. K. Leen, and M. Pavel, “Bayesian sensor image fusion using local linear generative models”, Optical Engineering, vol.40, no.7, pp.1364-1376, July 2001

)()()|(

)|(zp

spszpzsp =


Methods of Image FusionMultiscale Decomposition-Based (MDB) Methods

IMST

MST

MST Image 1

Image 2

Fused Image

Fusion

Images/coefficents for different scalesMST = Multiscale transform


Multiscale-Decomposition-Based (MDB) Methods

Methods of Image Fusion

Z. Zhang, and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application”, Proceedings of IEEE, vol. 87, no. 8, pp. 1315-1326, 1999.


Laplacian Pyramid TransformDecomposition

Reconstruction

MDB Fusion Methods

IG =0

[ ] 21 ↓−∗= kk GG ω 1,...,0 −= Nk[ ] 214 ↑+∗−= kkk GGL ω 1,...,0 −= Nk

NN GG =ˆ

[ ] 21ˆ4ˆ

↑+∗+= kkk GLG ω

0ˆˆ GI =

1,...,0 −= Nk


MDB Fusion MethodsDiscrete Wavelet Transform

One stage of a 2-D DWT decomposition


MDB Fusion MethodsDiscrete Wavelet Transform

One stage of a 2-D DWT reconstruction


MDB Fusion MethodsDiscrete Wavelet Frame


MDB Fusion Methods

MSD Structures


Non-Multiscale-Decomposition-Based (NMDB) Methods

Principle component analysis (PCA): O. Rockinger, and T. Fechner, "Pixel-level image fusion: The case of image sequences", Proceedings of SPIE, vol. 3374, 1998Adaptive weight averaging (AWA): E. Lallier and M. Farooq, “A real time pixel-level based image fusion via adaptive weight averaging”, ISIF 2000Pixel-level choosing maximum (MAX): simply to take the maximum value of the source images pixel by pixelNonlinear method (NONL): C. W. Therrien and W. K. Krebs, “An adaptive technique for the enhanced fusion of low-light visible with uncooled thermal infrared imagery”, ICIP 1997

Performance Testing for CWD :Visual and IR images

Pixel-level weighted averaging

NL


Multiscale-Decomposition-Based (MDB) Methods (similar with Laplacian & DWF)

DWT-1 : DWT, coefficient-based, single-scale grouping, choose max, no verification (2-level, average for the last LL band)DWT-2 : DWT, window-based (max), no grouping, choose max, no verification (2-level, average for the last LL band)DWT-3 : DWT, window-based (weighted average), no grouping, choose max, no verification (2-level, average for the last LL band)DWT-4 : DWT, window-based (weighted average), no grouping, weighted average, no verification (2-level, average for the last LL band)DWT-5: DWT, window-based (max), no grouping, choose max, consistency verification (2-level, average for the last LL band)

Performance Testing for CWD


Performance Testing for CWDDWF-1: DWF, window-based, modified Burt’s method, no grouping, consistency verification (DWT and Laplacian also)

Modified Burt’s method (A=visual image)

Attempt to maintain high resolution in the fused image by preserving the high-resolution in the visual image except where the two images are dissimilar

),,(),,(

),,(),,(2),,(

),,(),,(

;),,,(),,(),,(,),,,(

),,(),,(,),,,(

),,(

2

kjiSkjiS

knjmiCknjmiCkjiM

knjmiCkjiS

MifkjiCkjiSkjiSandMifkjiC

kjiSkjiSandMifkjiC

kjiC

BA

m nBA

AB

m n

ABA

BAABB

BAABA

c

+

++⋅++=

++=

><≤≥≤

=

∑∑∑∑

ααα


Evaluation Methods Requiring a Reference Image

Root Mean Square Error (RMSE)

Correlation

Peak Signal to Noise Ratio (PSNR)

2

1 1),(),(

1 ∑ ∑= =−=

N

i

M

jjiZjiR

NMRMSE

∑∑∑∑∑∑= == == =

===+

=N

i

M

jZR

N

i

M

jZ

N

i

M

jR

ZR

ZR jiZjiRRjiZRjiRRRR

RCORR

1 1,

1 1

2

1 1

2, ),(),(,),(,),(,2

−=

∑ ∑= =

2

1 1

2

10

),(),(1log10

N

i

M

jjiZjiR

NM

LPSNR

Evaluation Methods


Evaluation Methods Not Requiring a Reference Image

Standard Deviation (SD)

Entropy

Overall Cross Entropy (CE)

( ) ∑∑==

×=−=L

i

L

i

ihiiihii00

2)(,)(σ

∑=

−=L

i

ihihH0

2 )(log)(

∑=

=+

=L

i Z

XX ih

ihihZXCE

ZYCEZXCEZYXCE

02 )(

)(log)();(,

2);();(

);,(

Evaluation Methods


Experimental Tests

Test Images

img1 – Visual img2 – Visual img3 – Visual img4 – Visual img5 – Visual

img1 – IR img2 – IR img3 – IR img4 – IR img5 – IR


Experimental Tests

Test Images

img6 – Visual img7 – Visual img8 – Visual img9 – Visual

img6 – IR img7 – IR img8 – IR img9 – IR


Experimental Tests

Image Fusion Results (Image-1)

Visual IR Reference

PCA AWA Max Nonlinear Laplacian


Experimental Tests

Image Fusion Results (Image-1)

DWT-2 DWT-3 DWT-4 DWT-5 DWF-1

FSD Contrast Gradient Morphological DWT-1


RMSE

00.050.1

0.150.2

0.250.3

0.350.4

0.45

A B C D E F G H I J K L M N O VI IR

CORR

00.10.20.30.40.50.60.70.80.9

1


PSNR

02468

101214161820


Image - 1SD

05

101520253035404550


Entropy

6.6

6.7

6.8

6.9

7

7.1

7.2

7.3

7.4


CE

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J K L M N O


Visual EvaluationFor most of the images, all of the methods are acceptable except method j (DWT-1)Method c (Pixel-level maximum) and method o (DWF-1) are best for producing a detailed view of the person’s face and a recognizable view of the object of interest (the gun)

Discussion


Discussion

Quantitative Evaluation

Often differs greatly from the visual evaluation making the use of these measures questionable. More work needed on automatic evaluation methods.


A Signal Processing Approach

Motivation

Image formation model

EM fusion algorithm

Experiments and results

Summary


MotivationAttempt to capitalize on the well developed theories of statistics and estimation theoryEstimate true scene from images with different sensor typesDifficult to provide a well-justified, fixed statistical model for imagesCan solve this with an adaptive signal processing approach for a general modelCan get a particularly efficient processing structure


Image Formation ModelImage Formation Model

indexes the sensors denotes the pyramid

coefficient, is the pixel coordinate, m is the level z is the sensor images is the true scene image

is the sensor selectivity factor

is the non-Gaussian distortion

nFor ease of explanation we present a simple model.nWe have also studied more complicated models.

MST Pyramid

m=M

m=2

m=1

y

x

)()()()( jjsjjz iii εβ +=

qi ,...,1=),,( myxj ≡

),( yx

0,1±=β

ε


Image Formation Model (Cont.)is the random distortion modeled using a K-

term mixture of Gaussian probability density functions (pdfs) as

Can represent Gaussian or non-Gaussian distortion

1λ

21σ

2λ

22σ

Two terms

v , get Gaussian

vGroup data into two sets

v = Prob (sample from set i)

11 =λ 02 =λ

iλ

)( jiε

∑=

−=

K

k ik

i

ik

ikij jj

jjjf

i1

2,

2

2,

,)( )(2)(

exp)(2

1)())((

σε

πσλεε


EM Fusion AlgorithmEM used to estimate

Start with an initial set of estimates ( , …) and observed data ( , …). Then produce updated estimates ( , …) each iteration.Estimates converge to maximum likelihood estimates (typically need only 3 to 5 iterations).

)1(1z

jijjjjjjs iKiiKii ,allfor)(),...,();(),...,();();( 2,

2,1,,1 σσλλβ

)1(s

)1('s


Using a Window of dataTo estimate parameters for coefficient j, use a window of coefficients around j

Coefficient j

window

h

h

vPyramid level m

vWindow contains coefficients

v Number coefficient in window

v and the parameters of distortion pdf constant over the window

hhL ×=

Ll ,...,1=

β


Developing Iterative Estimation Approach

EM algorithm produces a sequence of estimates that increase in likelihoodAt each step you must find the estimate that maximizes a “likelihood-like” cost functionSkip details to give the iterative algorithm results Use SAGE algorithm: one variable at a time


Iterative Fusion ProcedureFirst compute (classifier)

Update . Choose from set 0,-1,+1 to maximize

Q is the “likelihood-like” function we need to maximize. Note Q is computed using the window

This procedure gives estimates for coefficient j at the center of window.

( )

( )LlqiKk

lslz

lslz

lzgK

p ip

ii

ip

ip

ik

ii

ik

ik

iilk ,...,1;,...,1;,...,1,

2)()(

exp2

2)()(

exp2

)]([

12

,

2

2,

,

2,

2

2,

,

, ===

−−

−−

=

∑= σ

β

πσ

λ

σβ

πσ

λ

iβ

( ) ( ) [ ]∑∑∑= = =

⋅

−+−=

q

i

L

l

K

kiilk

ik

iiik lzg

lslzQ

1 1 1,2

,

2'2, )(

)()(ln

21

σβ

σ


EM Fusion Procedure (Cont.)Update true scene (Generalization of Gaussian when )

Update and

Processing similar to radial basis function neural networks but comes from estimation theory

1)]([, ≠lzg iilk)(ls

ik ,λ 2,ikσ

[ ]

[ ] Lllzg

lzglz

lsq

i

K

k ik

iilki

q

i

K

k ik

iilkii

,...,1,)(

'

)(')(

)('

1 12,

,2

1 12,

,

==

∑∑

∑∑

= =

= =

σβ

σβ

[ ]∑=

===L

liilkik qiKklzg

L 1,, ,...,1;,...,1,)(

1'λ

( ) [ ]

[ ]qiKk

lzg

lzglslz

L

liilk

L

liilkii

ik ,...,1;,...,1,)(

)()('')('

1,

1,

2

2, ==

−=

∑

∑

=

=

βσ


Initialization for EM Fusion Algorithm

Initialize as the weighted average of sensor images:

are determined by salience measure:

The salience measure:

v is the weight for each coefficient around

∑=

==q

iii Lllzlwls

1

,...,1)()()(

)(lwi

qilllwq

jjii ,...,1)(/)()(

1

=ΩΩ= ∑=

qimyyxxzyxplx y

ii ,...,1),','()','()(2

2'

2

2'

2 =++=Ω ∑ ∑−= −=

),,( myxl =

)(ls

)','( yxp


Initialization for EM Fusion Algorithm (Cont.)

InitializeInitialize and

Initialize

qii ,...,1,1 ==β

qiKiKi ,...,1),1/(2.0,,2 =−=== λλ L

[ ]∑∑==

−=L

li

K

kikik Llslz

1

2

1

2,, /)()(σλso that

8.0,1 =iλ

Kkqiikik ,...,2;,...,1,2,1

2, === −γσσ


Force of coefficient j to be consistent with its

neighboring coefficients ( l = 1,…, L) in the same

pyramid level m

Define and

Pick to minimize

Can minimize component by component to simplify

Consistency Verification

∑=

−L

llj

1

ßßjß

β

( )jqjj ββ ,...,1=ß ( ) Lllqll ,...,1,,...,1 == ββß


EM Fusion for CWD (1)

Visual Image MMW Image

EM Fusion Averaging Selecting Maximum Laplacian Fusion

vImage size: 256×256

vNumber of parameter levels: 5

vGaussian mixture terms: 2

vWindow size: 3×3


Visual Image IR Image





vWindow size: 5×5

EM Fusion for CWD (2)


EM Fusion for autonomous landing guidance (ALG)

Long Wave Medium Wave





vWindow size: 5×5


Research on Optimizing the Procedure

Number of MST pyramid levels has slight effect on the fused result Window size should be chosen carefully: 3×3 or 5×5 is best in most applicationsCan improve fusion using multi-frame sensor images


SummaryPresented a statistical signal processing based image fusion methodApplied the EM fusion method to concealed weapon detection and autonomous landing guidance applications with good results


New Image Registration Algorithm for Image Fusion

Details in:

Z. Zhang and R. S. Blum, ``A Hybrid Image Registration Technique for a Digital Camera Image Fusion Application'', Information Fusion, pp. 135-149, June 2001.

Z. Zhang and R. S. Blum, ``Image registration for Multi-focus image fusion'', SPIE AeroSense, Conference on Battlefield Digitization and Network Centric Warfare (4396-39), Orlando, FL, April 2001.


Conclusion

We hope this talk was useful.We are seeking out collaborators. Working with people who have practical knowledge of real world applications of image fusion is highly desirable to us.Please contact us to initiate collaboration: [email protected]


ReferencesJ. Yang, R. S. Blum,`` A Statistical Signal Processing Approach to image fusion for concealed weapon detection'', IEEE International Conference on Image Processing, Rochester, NY, Sept. 2002.Z. Xue, R. S. Blum and Y. Li , ``Fusion of visual and IR images for concealed weapon detection'',

International Conference on Information Fusion (Fusion 2002), Annapolis, Maryland, July 2002. Z. Zhang and R. S. Blum, `À Hybrid Image Registration Technique for a Digital Camera Image Fusion Application'', Information Fusion, pp. 135-149, June 2001.Z. Zhang and R. S. Blum, `À hybrid image registration technique for a digital camera image fusion application'', SPIE AeroSense, Conference on Battlefield Digitization and Network Centric Warfare (4396-39), Orlando, FL, April 2001. Z. Zhang, and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application”, Proceedings of IEEE, vol. 87, no. 8, pp. 1315-1326, 1999.Z. Zhong and R. S. Blum, `Èxtraction of 3-D coordinates from fusion of Omnicamera images,'' Asilomar Conference on Signals, Systems, and Computers, pp. 397-401, Monterey, CA, Nov. 1999. Z. Zhang and R. S. Blum, "Image fusion for a digital camera application" Asilomar Conference on Signals, Systems, and Computers, pp. 603-607, Monterey, CA, Nov. 1998.R. S. Blum, ``Decision and data fusion research at Lehigh University,'' National Symposium on Data Fusion (sponsored by AFOSR) and held at Georgia Tech. Atlanta, Georgia, March 1998. Z. Zhong and R. S. Blum, `À region-based image fusion scheme for concealed weapon detection,'‘ 30th Annual Conference on Information Sciences andSystems, Johns Hopkins University, pp. 168-173, Baltimore, MD, March 1997.Z. Zhang and R. S. Blum, `Òn estimating the quality of noisy images,'' 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2897-2901, Seattle, WA, May 1998.Z. Zhang and R. S. Blum, "Multisensor image fusion using a region-based wavelet transform approach", Proc. of the DARPA IUW, pp. 1447-1451, 1997.

image fusion with some emphasis on cwd - lehigh ece r.s. blum - lehigh university 16 nmdb image...

Documents