Stavri Nikolov1, Tim Dixon2, John Lewis1, Nishan Canagarajah1, Dave Bull1, Tom Troscianko2, Jan Noyes21Centre for Communications Research, University of Bristol, UK2Department of Experimental Psychology, University of Bristol, UK

How Multi-Modality DisplaysAffect Decision MakingNATO ARW 2006, 21 - 25 October 2006, Velingrad, Bulgaria

OverviewMulti-Sensor Image FusionMulti-Modality Fused Image/Video DisplaysTarget Detection in Fused Images with Short Display Times (results)Scanpath Assessment of Fused VideosMulti-Modality Image SegmentationSummary

How Does Image/Video Fusion Affect Decision MakingExperiment 1: Target Detection in Fused Images with Short Display Times; Decision: is the target present or not?Experiment 2: Target Tracking in Fused Videos (+ secondary task); Decision: where to look to follow the target? Experiment 3: Image Segmentation (decomposing an image into meaningful regions/object) in Fused Images; Decision: which objects to segment and how?

Multi-Sensor Image Fusion

Multi-Sensor Image Fusion: Definitionthe process by which several images coming from different sensors, or some of their features, are combined together to form a fused image

the aim of the fusion process is to create a single image (or visual representation) that will capture most of the important and complementary information in the input images and will resolve better any uncertainties, inconsistencies or ambiguities.

Multi-Sensor Image Fusion: ExampleVisible and IR images courtesy of Octec Ltd, UKAn example

Multi-Sensor Image Fusion: ApplicationsMany different applications of image fusion:remote sensingsurveillancedefencecomputer visionroboticsmedical imagingmicroscopic imagingart

Multi-Sensor Image Fusion: ApplicationsImage fusion is used in:night vision systemsbinocular vision3-D scene model building from multiple viewsimage/photo mosaicsdigital cameras and microscopes to extend the effective depth of field by combining multi-focus imagestarget detection

Multi-Sensor Image Fusion: Different LevelsImage fusion can be performed at different levels of the information representation:signal levelpixel levelfeature / region levelobject levelsymbolic level

Multi-Modality Image Displays

Multi-Modality Image DisplaysAdjacent (side-by-side) displays (*)Window displaysFade in/out displaysCheckerboard displays (*)Gaze-contingent multi-modality displays (*)Hybrid fused displays (*)Interleaved video displays

Adjacent and Checkerboard DisplaysImages from the Eden Project Multi-Sensor Data Set

Gaze-Contingent Multi-Modal DisplaysDemo of a gaze-contingent multi-modal display (GCMMD) using aerial photographs and maps of England (from Multimap.com).Multi-Modality Gaze-Contingent Displays for Image Fusion", S. G. Nikolov, M. G. Jones, I. D. Gilchrist, D. R. Bull, C. N. Canagarajah, Proceedings of Fusion 2002

Hybrid Fused Image Displays (1.0,0.0) (0.8,0.2) (0.6,0.4)(0.4,0.6) (0.2,0.8) (0.0,1.0)Hybrid Fused Displays: Between Pixel- and Region-Based Image Fusion", S. G. Nikolov, J. J. Lewis, R. J. OCallaghan, D. R. Bull and C. N. Canagarajah, Proceedings of Fusion 2004

The results of image fusion are: either used for presentation to a human observer for easier and enhanced interpretation or subjected to further computer analysis or processing, e.g. target detection or tracking, with the aim of improved accuracy and more robust performanceFinding an optimal fused image is a very difficult problem since in most cases this is task and application dependent.Fused Image Assessment

it depends what we want to do with it, i.e. the task we have!Which Fused Image is Better?Original Visible and IR UN Camp images courtesy of TNO Human Factors

Categories of Fused Image Assessment Metrics A

Binput images

FUSION

F

fused image

A number of image quality metrics have been proposed in the past but all require a reference imageIn practice an ideal fused is rarely known and is application and task specificother metrics try to estimate what information is transferred from the input images to the fused imagetwo such metrics that we used in our study to assess the quality of the fused images are Piella's image quality index (IQI) [03] and Petrovic's edge-based Q^AB/F metric [00,03] (both of which are IFIMs)Fused Image Assessment Metrics

Experiment 1: Target Detection in Fused Images

Decision: Is the target present or not?

Experiment 1, Task 1: Objective Human Task PerformanceTesting 3 fusion schemes: AVR, CP & DT-CWT, and 3 JPEG2000 compression rates: clean, low (.3bpp) and high (.2bpp).Using a signal detection paradigm to assess Ps ability to detect presence of the soldier (target) in briefly displayed images.

Task 1: MethodFixation point + shown for 750ms, an image presented for 15ms, followed by an inter-stimulus interval of 15ms, and a mask for 250ms.

Experiment 1, Task 2: Subjective Image AssessmentShow pairs of images, ask Ps to rate both out of 5 (5 = Best quality, 1 = Worst quality). Images paired:

Target Detection in Fused Images: Main ResultsThe results showed a significant effect for fusion but not compression in JPEG2000 imagesSubjective ratings differed for JPEG2000 images, whilst metric results for both JPEG (different study) and JPEG2000 showed similar trendsCharacterisation of Image Fusion Quality Metrics for Surveillance Applications over Bandlimited Channels", E. F. Canga, T. D. Dixon, S. G. Nikolov, D. R. Bull, C. N. Canagarajah, J. M. Noyes, T. Troscianko, Proceedings of Fusion 2005

Experiment 2: Target Tracking in Fused Videos

Decision: Where to look to follow the target?

Experiment 2Applying an eye-tracking paradigm to the fused image assessment process.Moving beyond still images: assessing participants ability to accurately track a figure.Using footage taken recently at the Eden Project Biome.Videos of a soldier walking through thick foliage filmed in both visible light and IR, and at two natural luminance levels.All videos registered using our Video Fusion Toolbox (VFT)

Original Videos UsedHigh Luminance (HL)

Low Luminance (LL)Videos from the Eden Project Multi-Sensor Data Set

Fused Videos UsedLow Luminance:Fused AverageFused DWTFused DT-CWTHigh Luminance:Fused AverageFused DWTFused DT-CWT

Tasks + MethodsParticipants asked to visually track the solider as accurately as possible throughout video sequence.Tobii x50 Eye-Tracker used to record eye movements.Participants also asked to press SPACE at specific points in the two sequences (when soldier walked past features of the scene).10 Ps (5m, 5f): mean age = 27.1 (s.d. = 6.76).Each shown 6 displays: Viz, IR, Viz+IR*, AVE, DWT, DT-CWT.All Ps shown each condition in 3 separate sessions.Half shown above order first, half reverse order. Order switched for 2nd and switch back for 3rd sessions.Eye position and reaction times recorded.

Accuracy Results IEye position translated onto target box for each participant.Calculated an accuracy ratio, hits:total views for each condition.Also considered Tobii accuracy coding.

Accuracy Results IIVideos from the Eden Project Multi-Sensor Data Set

Results (High Luminance) Accuracy Scores revealed:Main effect display modality (p = .001).No main effect of session (p > .05).No interaction (p > .05).Post hoc tests revealed differences between Viz and: AVE, DWT, CWT.IR and: AVE, DWTRT Scores revealed:No significant effects

Scanpath Analysis of Fused Multi-Sensor Images with Luminance Change", T.D. Dixon, S.G. Nikolov, J.J. Lewis, J. Li, E.F. Canga, J.M. Noyes, T. Troscianko, D.R. Bull and C.N. Canagarajah, Proceedings of Fusion 2006

Results (Low Luminance) Accuracy Scores revealed:Main effect display modality (p < .001).No main effect of session (p > .05).No interaction (p > .05).Post hoc tests revealed differences between Viz and: IR, AVE, DWT, CWT.RT Scores revealed:Main effect of fusion: IR significantly closer to ideal timing.

Target Tracking in Fused Videos: Conclusions IThe current experimental results reveal two methods for differentiating between fusion schemes: the use of scanpath accuracy and RTs.Fused videos with higher (perceived) quality do not necessarily lead to better tracking performanceThe AVE and DWT fusion methods were found to perform best in the 2.1_i tracking task. From a subjective point, the DWT appeared to create a sequence that was much noisier and with more artefacts than the CWT method.

Target Tracking in Fused Videos: Conclusions IIAll of the fusion methods performed significantly better than the inputs, highlighting the advantages of using a fused sequence even when luminance levels are high.Results suggest that when luminance is low, any method of attaining additional information regarding the target location will significantly improve upon a visible light camera alone.

Experiment 3: Multi-Modal Image Segmentation

Decision: Which objects to segment and how?

Multi-Modal Image SegmentationMulti-modal sensorsMulti-sensor systemsMany applications need good segmentationHow best to segment a set of multi-modal images?To study how fusion affects segmentationPrevious evaluation methodsSubjectivebased on ground truthNeed for objective measure of quality of segmentation techniques

sets of multi-modal images}

Joint Vs. Uni-Modal SegmentationTwo approaches investigated:Uni-modal segmentationS1 = (I1),, SN = (IN)Each image segmented separatelyDifferent segmentations for each image in the setJoint segmentationSjoint = (I1 IN)All images in the set contribute a single segmentationSegmentation accounts for all features from all input images

Uni-Modal and Joint Image SegmentationOriginal IR image in red Original Visible Image in green Joint SegmentationUnimodal Segmetation Unimodal Segmentation Union of Unimodal Segmentations

Multi-Sensor Image Segmentation Data SetTo enable objective comparison of different segmentation techniquesNeed some method of finding a ground truth of natural imagesThe human visual system is good at segmenting imagesThe Berkeley Segmentation Database1000 natural images12000 human segmentations[Martin et al., A Database of Human Segmented natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics, ICCV, 2001]

Multi-Sensor Image Segmentation Data Set11 Sets of multi-modal images14 IR and 11 grey scale images33 fused images from 3 pixel-based fusion algorithmsContrast pyramidsDiscrete wavelets transformDual tree complex wavelet transformAll images have been segmented by the techniques described using the same good parameters across the whole data set

Image Data Set: ExamplesImages from the Multi-Sensor Image Segmentation Data Set

Experimental Setup63 subjects The instructions were toDivide each image into pieces, most important pieces first, where each piece represents a distinguished thing in the image. The number of things in each image is completely up to you. Something between 2 and 20 is usually reasonable. Take care and try and be as accurate as possible.5 images segmented eachImages pseudo-randomly distributed so that:Each subject sees only one image from each setThey see at least one IR, one visible and one fused imageAn image is not distributed a second time unless all images have been distributed once; etc.

The Segmentation ToolThe Berkeley Segmentation Tool (SegTool)

The Human Segmentations315 human segmentation produced~20 rejected as obviously wrong 5-6 segmentations for each image1 expert segmentation for each image The human segmentations are available to download from www.ImageFusion.org

Examples of Human SegmentationsUser 5User 61User 54User 39User 35User 15Human Segmentations of UN Camp CWT Fused Image

Segmentation Error Measure IWe adopt the approach used with the Berkley Segmentation DatasetPrecision, P, fraction of detections that are true positives rather than false positivesRecall, R, fraction of true positives that are detected rather than missedF-measure is a weighted harmonic meanF = PR/(R+(1- )P) = 0.5 used

Segmentation Error Measure IICorrespondences computed byComparing the segmentation to each human segmentation of that imageCorrespondence computed as a minimum cost bipartite assignment problemScores averaged to give a single P, R and F value for each imageTolerates localization errorsFinds explicit correspondences only

Analysis of Human Segmentations

Examples of Automatic and Human Segmentations IImages from the Multi-Sensor Image Segmentation Data Set

Examples of Automatic and Human Segmentations IIImages from the Multi-Sensor Image Segmentation Data Set

Joint Vs Uni-Modal Segmentation (Original Images)

Multi-Sensor Image Segmentation: ResultsUsing the human segmentations as ground truth for evaluationFound UoB_Uni to give best segmentations of uni-modal techniquesFound joint segmentations to be better than the uni-modal segmentations of the original imagesFound the joint segmentations to be at least as good as the uni-modal segmentations of the fused imagesThe relevance of these results to region-based fusion confirmed

Joint- versus Uni-Modal Segmentation for Region-Based Image Fusion", J. J. Lewis, S. G. Nikolov, A. Toet, D. R. Bull and C. N. Canagarajah, Proceedings of Fusion 2006

Multi-Sensor Image Segmentation: Work in ProgressRecent results indicate that schemes for fusion of visible and IR imagery should prioritise terrain features from the visible imagery and man-made targets from the IR imagery in the fusion process, in order to produce a fused image that is optimally tuned to human visual cognition and decision makingBy comparing the human segmentations of the input images to the human segmentations of the fused images we can hopefully study how image fusion affects segmentation decisions

Summary IMulti-sensor image fusion affects decision making in various waysBy applying tasks to the image fusion assessment process, it has been found that DT-CWT fusion can lead to better target detection human performance than AVE, pyramid and DWT methodsIn addition, the objective tasks utilised have been shown to produce very different patterns of results to comparative subjective tasks.

Summary IIFused videos with higher (perceived) quality do not necessarily lead to better tracking performanceIn most cases there are significant advantages of using a fused video sequence for target tracking even in HL levels and more so in LL levelsUsing the Multi-Sensor Segmentation Data Set we are trying to produce fused images that are optimally tuned to human visual cognition and decision making and to study how image fusion affects segmentation decisions

AcknowledgementsNATO and the ARW organisersThe Data and Information Fusion Defence Technology Centre (DIF-DTC), UK, for partially funding this researchThe Image Fusion Toolbox (IFT) and the Video Fusion Toolbox (VFT) development team at the University of BristolLex Toet (TNO Defence and Security, The Netherlands), Dave Dwyer (Octec Ltd, UK) and Equinox Corp (USA) for providing some of the images sequences used in this study (all these image sequences are available through www.ImageFusion.org)The Eden Project in Cornwall

C1-OpenCV, C2-JSEG, C3-UoBuni, C4-UoBjoint, C5- human segsC1-OpenCV, C2-JSEG, C3-UoBuni, C4-UoBjoint, C5- human segs