Improved F-transform Based Image Fusion�

Marek Vajgl and Irina Perfilieva

Institute for Research and Applications of Fuzzy ModelingCentre of Excellence IT4InnovationsUniversity of Ostrava, Czech Republic

{marek.vajgl,irina.perfilieva}@osu.cz

Abstract. The article summarizes current approaches to image fusionproblem using fuzzy transformation (F-transform) with their weak pointsand proposes improved version of the algorithm which suppress them.The first part of this contribution brings brief theoretical introductioninto problem domain. Next part analyses weak points of current imple-mentations. Last part introduces improved algorithm and compares itwith the previous ones.

Keywords: Image processing, image fusion, F-transform.

1 Introduction

The contribution focus on the problem of image fusion, what is one of the im-portant research areas in image processing field. The main aim of image fusion isthe comparison and integration of the distorted scenes into result image, whichcontains the best part extracted from the each input scene. The problem is howto define, select and extract the ”best” pixel from all input scenes. The definitionof the ”best” pixel depends on the requested fusion result. In this contributionwe aim at the multi focus images, which differ by focused area (e.g. see Fig.1 (a),(b)). Moreover, if each pixel is ”best” in the some input image, we arespeaking about mosaic multi focus.

A measurement of the local focus is the main task, typically based on eval-uation of the high frequencies in the image. The idea comes from the fact thatblurry parts (which are unfocused) suppress high frequencies, so focus measuredecreases with scene blur.

There already exist many approaches used to solve this problem. They differsby mathematical fields – statistical methods(e.g., usage of aggregation operators([1]), estimation theory ([2]), fuzzy methods ([3], [4]), approaches based on theoptimization using genetic algorithms ([5]), etc.).

In this article approaches based on the fuzzy transform (called F-transformfor short) will be discussed. F-transform is integral transformation, for which

� This work was supported by the European Regional Development Fund in theIT4Innovations Centre of Excellence project (CZ.1.05/1.1.00/02.0070). This workwas also supported by SGS14/PF/2013 project and ”SGS/PF/2014 – Vyzkum aaplikace technik soft-computingu ve zpracovanı obrazu” project.

A. Laurent et al. (Eds.): IPMU 2014, Part II, CCIS 443, pp. 153–162, 2014.c© Springer International Publishing Switzerland 2014

154 M. Vajgl and I. Perfilieva

motivation came from fuzzy modeling ([6], [7]) and which was successfully ap-plied in multiple areas of image processing, like image compression ([11]), edgedetection ([12]) or image reconstruction ([13]).

The contribution presents image fusion solutions using F-transform as themain tool for image fusion([9], [10], [14]). After the experiments, further researchwas done to obtain further improvements. Those improvements are presented inthis paper.

2 F-transform

As presented in the introduction, all methods explained in this contributionrefers to the F-transform technique, which will be explained only briefly. For fulldescription see [6].

F-transform performs linear mapping from the set of ordinary continuous ordiscrete functions over domain P into a set of discrete functions defined on afuzzy partition of P using direct and inverse F-transform.

Let image function u is discrete function P → R of two variables, defined overthe set of pixels P = {(i, j)|i = 1, ..., N ; j = 1, ...,M}. Let fuzzy sets Ak × Bl,k = 1, . . . , n, l = 1, . . . ,m, (in the meaning defined in [6]) establish a fuzzypartition of [1, N ] × [1,M ]. The (direct) F-transform of u (with respect to thechosen partition) is an image of the map F [u] : {A1, . . . , An}×{B1, . . . , Bm} → R

defined by

F [u](Ak ×Bl) =

∑Ni=1

∑Mj=1 u(i, j)Ak(i)Bl(j)

∑Ni=1

∑Mj=1 Ak(i)Bl(j)

, (1)

where k = 1, . . . , n, l = 1, . . . ,m. The value F [u](Ak × Bl) is called an F-transform component of u and is denoted by F [u]kl. The components F [u]klcan be arranged into the matrix Fnm[u].

The inverse F-transform of u is a function on P representing inversed im-age unm obtained by the following inversion formula, where i = 1, . . . , N, j =1, . . . ,M :

unm(i, j) =

n∑

k=1

m∑

l=1

F [u]klAk(i)Bl(j). (2)

It can be shown that the inverse F-transform unm approximates the originalfunction u on the domain P . The proof can be found in [6,7].

Processing of direct and inverse F-transform applied sequentially is calledfilter F-transform. Values N , M are referred as number-of-components – noc,minimal value is 2. Number of pixels covered by one basis function Ak or Bl iscalled as points-per-base – ppb and minimal value is 3.

2.1 F-transform Application in Image Processing

The main approach used in almost all of the image processing application ar-eas using F-transform is calculation of residuals, which can be obtained as a

Improved F-transform Based Image Fusion 155

subtraction between original input and input processed by direct and inverseF-transform.

As the main property of the F-transform is removal of high frequencies fromthe input image, the reconstructed image do not contain high frequency artifacts,like noise, or sharp edges. The amount of removed frequencies depends on thepoints–per–base/number-of-components values.

2.2 Image Fusion Using F-transform

The main aim of the image fusion is integration of multiple input images intonew one, where new image is in some way ”better” than original input ones.In case of the multifocus area, ”better” image means the image containing themost focused parts from the all input images.

Generally, for multifocus analysis, let u represents ideal image and c1, . . . , cKare acquired (input) images. Then the relation between each ci and u can beexpressed by

ci(x, y) = di(u(x, y)) + ei(x, y), i = 1, . . . ,K

where di is an unknown operator describing the image degradation, and ei issome random noise. The aim of the fusion is to obtain fused image u such thatit is closer to u (and therefore ”better”) than any of c1, . . . , cK .

The basic idea of usage of F-transform for image decomposition. We assumethat the image u is a discrete real function u = u(x, y) defined on the N×M arrayof pixels P = {(i, j) | i = 1, . . . , N, j = 1, . . . ,M} so that u : P → R. Moreover,let fuzzy sets Ak ×Bl, k = 1, . . . , n, l = 1, . . . ,m, where 2 ≤ n ≤ N, 2 ≤ m ≤ Mestablish a fuzzy partition of [1, N ]× [1,M ].

We begin with the following representation of u on P :

u(x, y) = unm(x, y) + e(x, y), (3)

e(x, y) = u(x, y)− unm(x, y), (4)

where 0 < n ≤ N, 0 < m ≤ M , and unm is the inverse F-transform of u ande is the respective first difference. Value e represents residuals of the image u.If we replace e in (3) by its inverse F-transform eNM with respect to the finestpartition of [1, N ] × [1,M ], the above representation can then be rewritten asfollows:

u(x, y) = unm(x, y) + eNM (x, y), ∀(x, y) ∈ P. (5)

We call (5) a one-level decomposition of u on P .If function u is smooth, then the function eNM is small, and the one-level

decomposition (5) is sufficient for our fusion algorithm (see Section 3.2). How-ever, for complex images there may be need to process next level decompositionof first difference e in (3) in following manner: we decompose e into its inverseF-transform en′m′ (with respect to a finer fuzzy partition of [1, N ]× [1,M ] withn′ : n < n′ ≤ N and m′ : m < m′ ≤ M basic functions, respectively) and thesecond difference e′. Thus, we obtain the second-level decomposition of u on P :

156 M. Vajgl and I. Perfilieva

u(x, y) = unm(x, y) + en′m′(x, y) + e′(x, y),e′(x, y) = e(x, y)− en′m′(x, y).

In the same manner, we can obtain a higher-level decomposition of u on P(see [14]).

In recursion, the whole iteration can be repeated to achieve such e decompo-sition fulfilling algorithm requirements.

3 Previous Implementations

3.1 Original (CA) Algorithm

Complete algorithm (CA for short) was the first approach researched in the im-age fusion based on the F-transform. The algorithm processes iteratively theinput images where each iteration increases number-of-components value (andtakes into account higher frequencies) and processes residuals from the previ-ous iteration until stopping condition (too high number-of-components value orresiduals are not significant).

As CA algorithm was described precisely in [14] with its behavior, and isreplaced by ESA algorithm due to its memory and time consumption, it will notbe explained.

3.2 Simple (SA, ESA) Algorithm

The main idea of simple algorithm (SA) is to process input image via direct andinverse F-transform to obtain residuals and then create final image by takingpixels from the image with the higher residual value (further referred as great-est). This algorithm is based on the idea of the minimization time and memoryconsumption.

Enhanced version (called ESA) of this algorithm (presented at [14]) did im-prove behavior. The main improvement is that result pixel is calculated asweighted sum of all input image pixel and weights represented by first leveldecomposition residuals (further referred as weighted) instead of picking one in-put image (referred as greatest), what suppress disturbing effect (compare Fig2a vs. Fig. 2b).

The main advantage is lowered time and memory consumption. The maindisadvantage of this approach is requirement to set input variable points-per-base by some expert, in wide range 3 – 50, where invalid selection has negativeeffect on the result. Another disadvantages are occurrences of artifacts as can beseen in 2. Some of them may be avoided by correct ppb selection, but some willremain.

3.3 Results Achieved by Presented Algorithms

Objective comparison of CA and ESA algorithm can be seen in Table 1 andTable 2 (only best result combinations are presented).

Improved F-transform Based Image Fusion 157

(a) Input image, back-ground focus

(b) Input image, fore-ground focus

(c) Non-mosaic, ”greatest”residuals

Fig. 1. ESA algorithm - fusion results, ppb=25

(a) ESA ”greatest” ppb=5 – low ppp (b) ESA ”weighted” ppb=25 – ”lake” ar-tifact

(c) ESA ”greatest” ppb=25 – ”lake” arti-fact

(d) CA – ”ghost” artifact

Fig. 2. Fusion results – artifacts

From the subjective point of view (except enormously complex scenes) ESAalgorithm provides comparable results to CA algorithm in shorter time. Howeverinvalid ppb selection will produces low quality results.

Moreover, both implementations still create artifacts called as a ”ghost/lakes”,where blur edge is propagated from the blurred image into the final result (seeFig. 2).

4 Improved Implementation Based on ESA Algorithm -IESA

The main aim of improved ESA algorithm is to produce better results than ESAalgorithm within acceptable time/memory consumption and to remove depen-dency on expert decision about points-per-base.

158 M. Vajgl and I. Perfilieva

4.1 Ghost Issue Artifact Description

”Ghost” artifact occurs near the significant edges in the result image. They arecaused by very blurry edges, which affect the ”focus” detection in the image (seeFig. 2), because:

– Exactly at the edge, the more ”focused” image has sharp intensity changebetween both sides of the edge (e.g. between plane wing and sky). Lessfocused image has only slight color change (due to blur over the edge) andtherefore more focused image seems to be more representative.

– Near the edge, the more ”focused” image may have no color change (e.g.near the plane wing is only sky). However, less focused image contains ”fade”between the edge colors (e.g. sky near plane contains fade between sky andwing) and therefore the less focused image seems to be more representative.

As a result invalid image is preferred to obtain result. Therefore there is needto suppress this behavior by spreading influence of the edge out.

4.2 Improved Fusion

Improved algorithm is built on the idea that edge is taken as significant only ifthere is no more significant edge in the neighbor on different source scene. There-fore the first step is to influence edge to its neighbor by blurring residuals usingF-transform. Second step is to fuse images together using one of the followingtechniques:

– Greatest - the image with the highest difference at the processed pixel is thewinner and result image will be taken from the winning image only;

– Greatest soften - the image with the highest difference at the processed pixelis the winner (see Fig. 3). However, winning indices (weights) are blurred(using F-transform) to achieve slight fades between pixels from differentimages.

– Weighted - same as in ESA, but with threshold noise reduction.

– Square-weighted - same as previous, but with second power of the weights.

In implementation the ”greatest” and ”greatest-soften” approaches are calcu-lated as weighted approaches with adequate weights set to zero.

4.3 Low Residual Values Reduction

Second part of the algorithm is the low residual values reduction - this occur whenresidual values are to low, so there is no significant edge. This is easily achievedby setting threshold value for residuals - if residual value is lower than threshold,it is set to zero. Otherwise there will be no change (see Fig. 3). Threshold valuewill be referred as residual-threshold with typical value set to 0.9.

Improved F-transform Based Image Fusion 159

(a) ”Evolution” of weights during IESA

Fig. 3. Converting residuals to weights for ”greatest soften” option

4.4 Final Algorithm

Final algorithm combines updated SA algorithm together with new featuresdescribed above. It can be simply described as (numerical values are set accordingto experimental results):

1. Calculate F-transform filter and residual for each input image (see section2.2).

2. Blur residuals using F-transform filter (optimal ppb value to do this is 25).3. Remove low residual values (typical threshold value should be 0.9).4. For ”greatest”/”greatest-soften” approaches convert residuals (〈0; 1〉) into

weights ({0; 1}).5. For ”greatest-soften” blur the weights using F-transform filter (that’s the

”soften”, for small ppb=5).6. Calculate result image as combination of input images and weights.

(a) Non-mosaic, greatest (b) Non-mosaic, greatest soften

(c) Non-mosaic, weighted (d) Mosaic image

Fig. 4. IESA algorithm - cut-out fusion results, ppb=5, residual threshold=0.9

160 M. Vajgl and I. Perfilieva

Cut-out of fusion result examples are shown in Fig. 4 to demonstrate artifactselimination.

As shown in the following section, presented IESA algorithm achieves betterresults than preceding CA and ESA algorithms with comparable time consump-tion requirements.

IESA algorithm has two initial parameters - points-per-base and residual-threshold. Experiments show that both parameters are in most cases independenton input images and preferred values for ppb=6 and for residual-threshold=0.9.

5 Comparison to the Existing Algorithms

From the subjective point of view, results achieved by IESA algorithm are betterthan results achieved by ESA algorithm and CA algorithm.

For objective evaluation, two characteristics describing image ”quality” areused: mean square error – MSE and peak signal to noise ratio – PSNR.

For fused image uf and optimal (original) image u, MSE is defined as:

MSEu,uf=

1

M ×N

∑

m∈M,n∈N

(u(m,n)− uf (m,n))2 (6)

The higher the MSE value is, the bigger is the difference between optimal imageu and the image created by fusion uf , so lower values are better. ”Error” ofMSE is evaluated against ”optimal” image for mosaic multifocus, or against thebest-available image created by expert in photo editor software for non-mosaicimages.

Peak signal-to-noise ratio is calculated from MSE using formula:

PSNRu,uf= 20 · log10 255

√MSEu,uf

(7)

Higher value in PSNR means lower noise information, so higher values are better.Results for the algorithms can be seen in Table 1. IESA algorithm gives ob-

jectively much better results than so far best results of ESA algorithm (CAalgorithm results were worse than SA (ppb=25) results and therefore were notpresented).

It is very important to notice that approaches based on the greatest-residual(that are ”greatest” and ”greatest-soften”) give better results than approachesbased on weights. This supports idea that its better to pick pixel from specificimage than to calculate pixel as weight of input images.

Computation requirements can be described by measurable characteristics:computation time and consumed memory, however they are very dependent onhardware and software. Moreover number of direct or inverse F-transform evalu-ations - NoFT characteristics is presented, describing number of direct of inverseF-transforms evaluations. Results for the algorithms can be seen in table 2.

Computation time of IESA algorithm is higher than ESA algorithm due toadditional F-transform operations and more sophisticate residual processing.However, for testing image (cca 11.2 MP) the time is still at level of seconds.

Improved F-transform Based Image Fusion 161

Table 1. Comparison by characteristics for selected image, two input images4129 × 2726 pixels (for CA algorithm – PPB column defines how ppb value is increasedper iteration)

Algorithm PPB Other settings Non-mosaic image Mosaic imageMSE PSNR MSE PSNR

CA 2i – – – 6.036 40.300CA i3 – – – 25.036 34.145ESA 5 weighted 45.071 31.591 23.364 34.445ESA 5 greatest 59.688 30.371 30.968 33.221ESA 25 weighted 29.577 33.421 7.217 39.546ESA 25 greatest 29.113 33.489 4.263 41.832IESA 5 greatest 5.301 40.886 1.386 46.710IESA 5 greatest-soften 4.507 41.591 0.921 48.485IESA 5 weighted 5.243 40.934 2.504 44.143IESA 5 squared-weights 9.408 38.395 1.600 46.089

Table 2. Comparison by resource consumption for selected image, two input images4129 × 2726 pixels (for CA algorithm – PPB column defines how ppb value is increasedper iteration; for CA algorithm – images are half sized as full size images processingdid run out of memory)

Algorithm PPB Other settings NoFT Time (s) Memory (MB)

CA 2i – 11 25.504 781CA i3 – 4 5.234 369ESA 5 weighted 2 · n 3.202 193ESA 5 greatest 2 · n 3.151 193ESA 25 weighted 2 · n 2.873 128ESA 25 greatest 2 · n 1.854 128IESA 5 greatest 4 · n 3.939 286IESA 5 greatest-soften 6 · n 4.447 214IESA 5 weighted 4 · n 3.281 286IESA 5 squared-weights 4 · n 3.099 279

6 Conclusion

In this paper we presented results about research of effective fusion algorithmsbased on previous research ([9] and [14]). Improved method presented in paperbrings better results within time and memory resources limits and suppressionof artifact occurring in result fused image; moreover dependency on input pa-rameter is removed.

References

1. Blum, R.S.: Robust image fusion using a statistical signal processing approach.Information Fusion 6(2), 119–128 (2005)

162 M. Vajgl and I. Perfilieva

2. Loza, A., Bull, D., Canagarajah, N., Achim, A.: Non-gaussian model-based fusionof noisy images in the wavelet domain. Computer Vision and Image Understand-ing 114(1), 54–65 (2010)

3. Singh, H., Raj, J., Kaur, G., Meitzler, T.: Image fusion using fuzzy logic andapplications. In: Proceedings of the 2004 IEEE International Conference on FuzzySystems, vol. 1, pp. 337–340 (2004)

4. Ranjan, R., Singh, H., Meitzler, T., Gerhart, G.: Iterative image fusion techniqueusing fuzzy and neuro fuzzy logic and applications. In: Annual Meeting of the NorthAmerican Fuzzy Information Processing Society, NAFIPS 2005, pp. 706–710 (2005)

5. Mumtaz, A., Majid, A.: Genetic algorithms and its application to image fusion.In: 4th International Conference on Emerging Technologies, ICET 2008, pp. 6–10(2008)

6. Perfilieva, I.: Fuzzy transforms: Theory and applications. Fuzzy Sets and Sys-tems 157, 993–1023 (2006)

7. Perfilieva, I.: Fuzzy transforms: A challenge to conventional transforms. In: Hawkes,P.W. (ed.) Advances in Images and Electron Physics, vol. 147, pp. 137–196. ElsevierAcademic Press, San Diego (2007)

8. Perfilieva, I., Dankova, M.: Image fusion on the basis of fuzzy transforms. In: Proc.8th Int. FLINS Conf., Madrid, pp. 471–476 (2008)

9. Perfilieva, I., Dankova, M., Hodakova, P., Vajgl, M.: The Use of F-Transform forImage Fusion Algorithms. In: Proc. Intern. Conf. of Soft Computing and PatternRecognition, SoCPaR 2010, pp. 472–477 (2010)

10. Hodakova, P., Perfilieva, I., Dankova, M., Vajgl, M.: F-transform based imagefusion. In: Ukimura, O. (ed.) Image Fusion, pp. 3–22. InTech (2011),http://www.intechopen.com/articles/show/title/

f-transform-based-image-fusion

11. Perfilieva, I., Pavliska, V., Vajgl, M., De Baets, B.: Advanced image compression onthe basis of fuzzy transforms. In: Proc. Conf. IPMU 2008, Torremolinos, Malaga,Spain, pp. 1167–1174 (2008)

12. Perfilieva, I., Hodakova, P., Hurtık, P.: F 1-transform Edge Detector Inspired byCanny’s Algorithm. In: Greco, S., Bouchon-Meunier, B., Coletti, G., Fedrizzi, M.,Matarazzo, B., Yager, R.R. (eds.) IPMU 2012, Part I. CCIS, vol. 297, pp. 230–239.Springer, Heidelberg (2012)

13. Perfiljeva, I., Vlasanek, P., Wrublova, M.: Fuzzy transform for image reconstruc-tion. In: Uncertainty Modeling in Knowledge Engineering and Decision Making,pp. 615–620. World Scientific, Singapore (2012) ISBN 978-987-4417-73-0

14. Prefiljeva, I., Vajgl, M.: Novel Image Fusion Based on F-transform. In: 2nd WorldConference on Soft Computing Proceedings, pp. 165–171. Letterpress PublishingHouse (2012) ISBN 9789952452372