Spatial and Spectral Evaluation of Image Fusion Methods

Sascha KlonusManfred Ehlers

Institute for Geoinformatics and Remote SensingUniversity of Osnabrück

Content

Introduction Image Fusion

Test Site

Fusion Results

Color Distortions

Evaluation Methods and Results

Ehlers Fusion

Conclusions and Future Work

Remote sensors have different spatial resolution for panchromatic and multispectral imagery

The ratios vary between 1:2 and 1:5

For multisensor fusion the ratios can exceed 1:30(e.g. Ikonos/Landsat)

Data Fusion: Why is it Necessary?

Objectives of Image Fusion

Sharpen images Improve geometric corrections Provide stereo-viewing capabilities Enhance certain features Complement data sets Detect changes Substitute missing information Replace defective data

Pohl & van Genderen (1998)

Meaning of Pan-Sharpening

Spatial Spectral +

panchromatic &high geometric resolution

multi-/hyperspectral image &low geometric resolution

multi-/hyperspectral &high geometric resolution

Fusion Methods

Color Transformations Modified IHS Transformation

Statistical Methods Principal Component Merge

Numerical Methods Brovey CN Spectral Sharpening Gram-Schmidt Spectral Sharpening Wavelet based Fusion

Combined Methods Ehlers Fusion

Test Site

Original Data

Quickbird Multispectral image (2004-09-04)

Quickbird Panchromatic image (2004-09-04)

Formosat Multispectral image (2004-01-30)

Ikonos Multispectral image (2005-08-03)

Single Sensor Fusion: Quickbird

Quickbird Multispectral imageFused with BroveyFused with CN Spectral SharpeningFused with EhlersFused with WaveletFused with Gram-Schmidt Fused with PCFused with modified IHS

Multisensor Fusion: Ikonos

Ikonos Multispectral imageFused with BroveyFused with CN Spectral SharpeningFused with EhlersFused with modified IHSFused with PCFused with Gram-Schmidt Fused with Wavelet

Multisensor Fusion: Formosat

Formosat Multispectral imageFused with BroveyFused with CN Spectral SharpeningFused with EhlersFused with modified IHSFused with PCFused with Gram-Schmidt Fused with Wavelet

Panchromatic band has a different spectral sensitivity

Multisensoral differences (e.g. Ikonos and SPOT merge)

Multitemporal (seasonal) changes between pan and ms image data

Fusion Problem: Color Distortion

Inconsistent panchromatic information is fused into the multispectral bands

Spectral Comparison Methods (1)

fusedorg

fusedorg

xxbias

ss

biasRMSE

22

s = standard deviation

org = Original image

fused = Fused image

x = Mean

RMSE

Correlation coefficients

)()(

),(),(

yVarxVar

yxCovyxKor

Visual (Structure and Colour Preservation)

Results RMSE

Quickbird Ikonos Formosat

Mod. IHS 7.1822 11.0714 1.7792

PC 42.5058 23.8979 23.3095

Brovey 222.1732 143.8303 14.3147

CN-Sharpening 46.7389 123.5038 70.4803

Gram-Schmidt 5.6883 5.1425 0.4078

Wavelet 3.1915 2.5379 0.0290

Ehlers 1.0028 4.3645 0.4201

Results Correlation Coefficients

Quickbird Ikonos Formosat

Mod. IHS 0.8737 0.5731 0.5892

PC 0.8811 0.6512 0.6199

Brovey 0.8611 0.6020 0.5677

CN-Sharpening 0.8611 0.6020 -0.0546

Gram-Schmidt 0.8690 0.6774 0.6110

Wavelet 0.9698 0.9719 0.9720

Ehlers 0.9510 0.8094 0.9324

Spectral Comparison Methods (2)

Per Pixel Deviation

Degrade

Degraded to ground resolution of original image(Formosat = 8m)

Original multispectral image (Formosat 8m) Band 1 2.56

Band 2 2.92

Band 3 3.49

Band 4 3.35

Result: Vector containing the deviation per pixel

Fused image (Formosat 2m)

Mean Per Pixel Deviation

Quickbird Ikonos Formosat

IHS 27.28 42.95 11.93

PC 48.80 53.27 22.52

Brovey 209.15 136.62 19.48

CN-Sharpening 48.73 117.61 70.71

Gram-Schmidt 30.35 45.29 11.99

Wavelet 7.20 13.47 3.04

Ehlers 17.28 25.86 4.29

Spatial Comparison Methods (1)

Edge Detection

-

--

Band 1 91.16 %

Band 2 92.10 %

Band 3 92.64 %

Mean 91.96 %

Results Edge Detection

Quickbird Ikonos Formosat

Mod. IHS 92.71 % 92.44 % 95.54 %

PC 95.10 % 93.28 % 93.44 %

Brovey 94.69 % 95.16 % 97.87 %

CN-Sharpening 94.69 % 95.16 % 90.69 %

Gram-Schmidt 95.02 % 95.53 % 97.82 %

Wavelet 85.00 % 83.82 % 84.81 %

Ehlers 91.85 % 90.35 % 94.40 %

Spatial Comparison Methods (2)

Highpass Filtering

Correlation

Band 1 0.8012

Band 2 0.7820

Band 3 0.7912

Mean 0.7918

Highpass Correlation Results

Quickbird Ikonos Formosat

Mod. IHS 0.9336 0.9149 0.9420

PC 0.9900 0.0021 0.8073

Brovey 0.9715 0.9765 0.9895

CN-Sharpening 0.9714 0.9764 -0.0170

Gram-Schmidt 0.9857 0.9879 0.9652

Wavelet 0.3976 0.3627 0.3799

Ehlers 0.8997 0.8689 0.9349

FFT Filter Based Data Fusion (Ehlers Fusion)

Panchromatic Image

Multispectral Image

R

G

B

Basis: IHS Transform and Filtering in the Fourier Domain

FFT FourierSpectrum

FFT FourierSpectrum

HPFPanHP

LPFILPI

H

S

R‘

G‘

B‘

IHS-1

ILP+PanHP

H

S

FFT-1

Panchromatic image and its spectrum

Original panchromatic image

Panchromatic Spectrum

Filtersetting effects

Frequency

Intensity

Cut-off Frequencyfn

Filtered Panchromatic Spectrum

Effects in the spatial domain

Filtered panchromatic image Fused image

Filtersetting effects

Frequency

Intensity

Cut-off Frequencyfn

Filtered Panchromatic Spectrum

Effects in the spatial domain

Filtered panchromatic image Fused image

Filtersetting effects

Filtered Panchromatic Spectrum

Frequency

Intensity

Cut-off Frequencyfn

Effects in the spatial domain

Filtered panchromatic image Fused image

Results

Ehlers Fusion shows the best overall results in all images

It works also if the panchromatic Information does not match the spectral sensitivity of the merged bands (multitemporal and multisensoral fusion)

Its performance is superior to standard fusion techniques (IHS, Brovey Transform, PC Merge)

Wavelet preserves the spectral characteristics at the cost of spatial improvement

Ehlers Fusion is integrated in a commercial image processing system (Erdas Imagine 9.1)

Future Work

Fusion of radar- and optical Data

Development of one method to evaluate the spatial and spectral quality of an fused image

Comparison with the algorithm of Zhang (PCI Geomatica)

Research on automation for filter design

Thanks for your Attention

Questions???

Ehlers Fusion Program

Ehlers Fusion Program

Ehlers Fusion Program

Ehlers Fusion Program

Ehlers Fusion Program

Multispectral image and its spectrum

Original multispectral intensity

Multispectral intensity spectrum

Filtersetting effects

Filtered multispectral spectrum

Frequency

Intensity

Cut-off Frequencyfn

Filtersetting effects

Filtered multispectral spectrum

Frequency

Intensity

Cut-off Frequencyfn

Filtersetting effects

Filtered multispectral spectrum

Frequency

Intensity

Cut-off Frequencyfn