1

Multi-Channel CNN-based Object Detection forEnhanced Situation Awareness

Shuo Liu, Student Member, IEEE, Zheng Liu, Senior Member, IEEE

Abstract—Object Detection is critical for automatic militaryoperations. However, the performance of current object detectionalgorithms is deficient in terms of the requirements in militaryscenarios. This is mainly because the object presence is hardto detect due to the indistinguishable appearance and dramaticchanges of object’s size which is determined by the distance to thedetection sensors. Recent advances in deep learning have achievedpromising results in many challenging tasks. The state-of-the-art in object detection is represented by convolutional neuralnetworks (CNNs), such as the fast R-CNN algorithm. These CNN-based methods improve the detection performance significantlyon several public generic object detection datasets. However,their performance on detecting small objects or undistinguishableobjects in visible spectrum images is still insufficient. In thisstudy, we propose a novel detection algorithm for military objectsby fusing multi-channel CNNs. We combine spatial, temporal andthermal information by generating a three-channel image, andthey will be fused as CNN feature maps in an unsupervisedmanner. The backbone of our object detection framework isfrom the fast R-CNN algorithm, and we utilize cross-domaintransfer learning technique to fine-tune the CNN model ongenerated multi-channel images. In the experiments, we validatedthe proposed method with the images from SENSIAC (MilitarySensing Information Analysis Centre) database and compared itwith the state-of-the-art. The experimental results demonstratedthe effectiveness of the proposed method on both accuracy andcomputational efficiency.

Index Terms—Object Detection, Image Fusion, ConvolutionalNeural Networks, Military Applications.

I. INTRODUCTION

Automatic target detection (ATD) is a key technologyfor automatic military operations and surveillance missions.In military mission, sensors can be placed on the ground,mounted on unmanned aerial vehicles(UAVs) and unmannedground vehicles (UGVs) to acquire sensory data. Then the datawill be processed using ATD algorithms which aim to findbounding boxes where the targets may be located. Fast andaccurate object detector can increase lethality and survivabilityof the weapons platform/soldier. Whether the tactical scenariois the onslaught of an array of combat vehicles coming throughthe Fulda Gap, which was feared during the Cold War [1],or the identification of humans with intent to kill in anurban scene, the identification of the threat for avoidance andengagement is paramount to survival and threat neutralization.

Numerous ATD algorithms have been proposed during pastdecades. Generally, these algorithms can be classified into twomain categories: 1) background modeling approaches, 2) andlearning-based approaches.

S. Liu and Z. Liu (Corresponding author) are with Department of School ofEngineering — Faculty of Applied Science, University of British ColumbiaOkanagan Campus, Kelowna, Canada (email: zheng.liu@ubc.ca).

Fig. 1. Left: the appearance of target is undistinguishable from backgroundenvironment. Right: the scale of target is various dramatically.

Background modeling approaches assume that backgroundpixels have a similar color (or intensity) over time in a fixedcamera, and the background model is built on this assumption.The background is abstracted from the input image, andthe foreground (moving objects) region is determined bymarking the pixels in which a significant difference occurs.In[2], the authors modeled the background using a Kerneldensity estimation (KDE) method over a joint domain-rangerepresentation of image pixels. Multilayer codebook-basedbackground subtraction (MCBS) model was proposed in[3],which can remove most of the non-stationary background andsignificantly increase the processing efficiency. Reference[4]proposed a motion detection model based on probabilisticneural networks. Above methods are designated for the sta-tionary camera scenario. In the works of[5][6][7], the authorsproposed several schemes that can handle the problems inthe moving camera scenario. The background modeling basedmethods are effective for detecting moving objects, whereaswhen the objects are still or moving slowly, those methodswill always be unsatisfying.

Another popular category is the learning-based approaches.Traditionally, hand-engineered features like SIFT [8] or HOG[9] are firstly extracted and then fed into a classifier, suchas boosting[10], support vector machine (SVM)[11] and ran-dom forest [12]. The typical work in this paradigm is thedeformable part models (DPM) [13]. More recently, Convolu-tional Neural Networks (CNN) raise a significant impact onATD research community, which helped achieve promisingresults in many difficult object detection challenges [14] [15][16]. Overfeat [17] firstly utilized CNN models in a slidingwindow fashion on ATD task successfully. Where it has twoCNNs, one for classifying if a window contains an object andthe other for predict the bounding box of the object. Afterthat, the most popular CNN-based ATD framework appeared,R-CNN [18], which uses a pre-trained CNN to extract features

arX

iv:1

712.

0007

5v1

[cs

.CV

] 3

0 N

ov 2

017

2

from box proposals generated by selective search [19], andthen it classifies them with class specific linear SVMs. Thesignificant advantage of this work is derived from replacinghand-engineered features by CNN feature extractor. Then, thevariants of R-CNN were proposed to mainly solve the problemwith computational burden [20] [21] [22].

Nevertheless, above mentioned ATD methods are only ap-plicable to the general natural scene, and many challengescome up from the military scenario. First, the environment ofbattlefields is extremely complex. As shown in Figure 1, theappearance of the object includes color and texture is similarto the background in left example, because soldiers alwaysattempt to decorate themselves or their vehicles similar tothe environment in order to be invisible. And due to the vastbattlefield, the scale of objects always dramatically changeswith their distance to sensors. Thus, those environmentalfactors will always limit the ability of generic object detectionalgorithm. Second, military ATD application always runs onthe embedded platform whose computational and memoryresources are limited. In this case, the ability to run at highframe rates with relatively high accuracy becomes a crucialissue for military ATD.

Several image fusion based methods were proposed toenhance target representation in literature [23] [24] [25] [26].Multiple images acquired with different range of electro-magnetic spectrum were fused into one image by pixel-level image fusion algorithms such as PCA-based weightedfusion [26]and Discrete Wavelet Transform (DWT) [24], andthen fed into an ATD system. When the fused images areused in ATD tasks, there are still deficient. To address theserious limitation, we propose a novel image fusion approachto improving detector performance in the military scenario,which exploits the significant advantage of the unsupervisedfeature learning characteristic of CNNs. Compared with high-level image fusion, the proposed method can achieve a higheraccuracy and computational efficiency. In addition, we adoptedthe state-of-the-art generic object detection framework into themilitary scenario and used a cross-domain transfer learningtechniques to cover the shortage of insufficient data. In thisway, the proposed framework achieved promising results onSENSIAC military dataset. To sum up, our contributions inthis work are as follows:

1) Both spatial-temporal and multiple spectral bands infor-mation are employed into ATD task.

2) An unsupervised learning-based image fusion approach isproposed, which can automatically learn how to fuse theessential information from the different type of imagesinto a set of discriminative feature maps.

3) The proposed framework has the capability to transferlearning from visible images to our created complex fusedimages. Moreover, the proposed method can achieve98.34% average precision and 98.90% top1 accuracy onSENSIAC military dataset.

II. METHODOLOGY

In this paper, we proposed a framework, namely, imagefused object detector (IFOD). As illustrated in Figure 2, the

whole system is composed of four modules: 1) an image fusionmodule, which can fuse three different type of images into aBGR image; 2) a CNN feature extractor, used for extractinghigh-level semantic representations from the fused image; 3)a region of interest (ROI) proposal module manipulated onfused image is utilized for generating hundreds or thousandsof candidate bounding boxes, for each ROI on feature mapproduced by feature extractor module; and 4) an ROI classi-fication and regression is performed to obtain fine boundingboxes and corresponding class.

A. Image Fusion

1) Image Selection: Multi-sensor data often provide com-plementary information for context enhancement, which mayfurther enhance the performance of object detection. In ourwork, we investigated two type of images from differentsensors, mid-wave infrared image (MWIR) and visible image(VI), respectively. In addition to the images acquired fromthese two sensors, we also incorporate motion image generatedfrom two consecutive visible frames in order to complementsufficient description of objects.

MWIR: Depending on the different range of electromagneticspectrum, the infrared (IR) spectrum can be divided intodifferent spectral bands. Basically, the IR bands include theactive IR band and the passive (thermal) IR band. The maindifference between active and passive infrared bands is thatthe passive IR image can be acquired without any externallight source whereas the active IR required that. The passive(thermal) IR band is further divided into the mid-wave infrared(3-5 um) and the long-wave infrared (7-14 um).In general, themid-wave infrared (MWIR) cameras can sense temperaturevariations over targets and background at a long distance, andproduce thermograms in the form of 2D images. Its abilityto present large contrasts between cool and hot surfaces isextremely useful for many computer vision tasks such asimage segmentation and object detection. However, the MWIRsensor is not sensitive to cool background. And due to low-resolution sensor arrays and the possible absence of auto-focuslens capabilities, high-frequency content of the objects likeedges and texture are mostly missed.

VI: The range of the electromagnetic spectrum of visibleimage is from 380 nm to 750 nm. This type of image can beeasily and conveniently acquired via various kinds of generalcameras. In comparison with MWIR image, the VI imageis sensitive to illumination changes, preserve high-frequencyinformation and can provide a relatively clear perspectiveof the environment. In most of the computer vision topics,the VI image has become major studying object for manydecades. Thus, there are a large number of public VI datasetsacross many research areas. On the other hand, the significantdrawbacks of VI image are that it has poor quality in theharsh environmental conditions with unfavourable lightingand pronounced shadows, and there is no dramatic contrastbetween background and foreground when the environment isextremely complicated such as the battlefield.

Motion image: In general, the moving objects are the targetsin the battle fields. Therefore, estimating the motion of objects

3

Fig. 2. The pipeline of proposed object detector framework, which include four main components:1) image fusion, 2) ROI proposal, 3) CNN feature extractor,and 4) ROI classification and regression.

Fig. 3. The procedure of motion estimation: where t is the current frame andt-5 is the previous 5th frame, and the motion is what our algorithm estimate.

can provide significant cues to segment those targets. Variousmotion estimation algorithms have been proposed in recentdecades, such as dense optical flow methods, points corre-spondence methods, and background subtraction. And eachof them has shown effectiveness on many computer visiontasks. However, considering the trade-off between accuracyand computational complexity, we do not opt for any ofthe complicated motion estimation approaches but utilize astraightforward and easier to be implemented method. Themethod is illustrated in Fig 3, we estimate the motion mapbased on two consecutive frames. To be specific, the objectiveimages are sampled at every 5 frames, and then force thecurrent frame to subtract the last frame, the resulting image isthe desired motion image. The method can be formulated asfollow:

Mn(x, y) = |In(x, y)− In−5(x, y)| (1)

where Mn(x, y) represents the motion value of frame n atpixel point (x, y) and In(x, y) denotes the pixel value of framen at pixel point (x,y).

In this way, we do not need to consider multiple frames toestimate background, like the background subtraction meth-ods, and only the subtraction operator is employed in thisprocedure, which is more efficient that other state-of-the-artmethods. Even though this method can bring lots of noisein the motion image, this image still can provide enoughcomplementary information in the later fusion stage.

2) Fusion Methodology: Here, we formalize the possibleconfigurations of information fusion for object detection intothree categories, namely, pixel-, feature- and decision- fusionarchitecture. An illustration is shown in Figure 4. Havingthese possibilities in mind will help to highlight the importantbenefits of our proposed fusion method in terms of efficiencyand effectiveness.

Pixel-level fusion: A typical pixel-level fusion architectureis illustrated in Figure 4(a). This configuration of imagefusion is the lowest level techniques dealing with the pixelsobtained from the sensor directly and tries to improve thevisual enhancement. Typically, multiple images from differentsources are combined into one single image in pixel-wisemanner, after which it is fed into the object detection systemto generate final results. One of the main advantages of thepixel-level fusion is their low computational complexity andeasy implementation.

Feature-level fusion: As an higher level fusion system toFigure 4(a), one might pursue Figure 4(b), in which differenttype of images are simultaneously fed into their independentlower part of the entire object detection system, which istypically called feature extractor. For instance, this lower-levelsystem might be the hand-engineered feature extractor for thetraditional object detection system, and high-level convolutionlayer for CNN-based system. After which the concatenatedfeatures produced by the various independent lower systemare fed into one upper (decision-making) system to producethe final results. Although this feature-level fusion is usuallyrobust to noise and misregistration, it always require almostdouble memory and computing resource to deal with featurefusion procedure in a parallel fashion, especially for the CNN-based methods.

Decision-level fusion: The decision-level fusion schemeillustrated in Figure 4(c) operates on the highest level, andrefers to fusing discriminate results from different systemsdesigned for various type images. Note that for an object

4

Fig. 4. Illustration of different image fusion architectures: (a) pixel-level fusion architecture; (b) feature-level fusion architecture; (c) decision-level fusionarchitecture.

detection system which usually based on machine learningalgorithms, this high-level fusion probably could not establishwell relationship of interior characteristics between differenttype of images.In addition, this method might also be prac-tically challenging to implement as this duplication wouldmultiply the amount of resources and running time.

In our framework, we proposed a novel image fusion ap-proach which is similar to pixel-level fusion style, we combinemultiple images into one single image while we do not conductfusion algorithm in pixel-level stage. As you can see in theimage fusion module in Figure 2, firstly, the three type of rawimages (MWIR, VI and Motion image) are concatenated into aBGR-style three-channel image where MWIR in red channel,motion image in green channel and VI in blue channel. Thenwe obtain this three-channel fused image, and it is worthynoted that we do not modify any pixel values of the rawimages but just put them into independent channels of finalfused image. After the fused image is obtained, we fed it intoconvolutional neural network (CNN) to training our objectdetector in end-to-end manner, meanwhile, the feature fromdifferent source images can be fused together in the internalof CNN in an unsupervised learning fashion. Therefore, com-pared with feature-level and decision-level fusion methods,our approach is easier to implement and low computationalcomplexity. And to pixel-level fusion, we employ unsupervisedlearning style to fuse images from different sources instead ofutilizing hand-engineered pixel-level fusion algorithms suchas discrete wavelet transform (DWT) pixel-level image fusionmethodologies.

B. Regions of Interest Proposal

As you can see in the ROIs proposal module in Figure 2,given an image, the ROIs proposal algorithms can output aset of class-independent locations that are likely to containobjects. Different from the exhaustive search ”sliding win-dow” paradigm which will propose every possible candidatelocations and generate around 104 − 105 windows per image,ROIs proposal methods try to reach high object recall withconsiderably fewer windows. In the most popular object de-

tectors such as R-CNN [18] and fast R-CNN [20], they selectSelective Search [19] method as their ROIs proposal module.

The Selective Search [19] is a ROIs proposal that com-bines the intuitions of bottom-up segmentation and exhaustivesearch. The whole algorithm can be simplified as follows.Firstly, [27] algorithm is adopted to create initial regions. Thenthe similarities between all neighbor regions are calculatedand the two most similar regions are grouped together. Afterthat, the new similarities are calculated between the resultingregion and its neighbors. In this iterative manner, the processof grouping the most similar regions is repeated until the wholeimage becomes a single region. Finally, the object locationboxes can be extracted from each region. Because of thishierarchical grouping process, the generated locations comefrom all scales.

C. Network Architecture

The great success of Convolutional Neural Networks(CNNs) in recent years aroused broader interest in CNNs-based generic object detection among researchers. In typically,a CNN comprises a stack of convolutional and pooling lay-ers. The convolutional layer can generate feature maps byconvolving the input feature maps or image with a set oflearnable kernels. And the pooling layer can pool informationof a given region on output feature maps in order to achievedown sampling and expansion of the receptive field.

The most typical CNNs-based object detector is the R-CNN[18], which utilize Selective Search method to generate a setof ROIs proposal from input image and then feed each ROIto a CNN to obtain final results. However, this paradigmis slow, because lots of heavily overlapped ROIs have togo through the CNN separately and thus a large amount ofredundant computation is consumed. SPP-net [21] and fastR-CNN [20] successful solved this problem by proposing anSpatial Pyramid Pooling (SPP) and ROI pooling, respectively.They suggested the whole image can go through CNN onceand the final decision is made at the last feature maps producedby the CNN by using their proposed pooling strategies.

Our proposed framework is illustrated in Figure 2. Thefused three channel image is firstly fed into the CNN feature

5

extractor to generate conv feature maps. It should be noted thatthe final conv feature maps in our project are also the fusingresults of the three types of images by unsupervised learning.After which, for each ROIs generated by the ROIs proposal,we conduct an ROI pooling process directly on the convfeature maps instead of an input image to extract a fixed lengthfeature vector. The reason to choose ROI pooling instead ofSPP is that the gradients can propagate to the CNN layersin training stage and this can help CNN learn how to fusethe multiple channel-independent images in an unsupervisedfashion. Finally, the extracted vector need to be sent to a fullyconnected neural network which has two output ports whereone is for classification and another one is for bounding boxesregression.

Taking the trade-off between accuracy and computationalcomplexity into account, the VGGM from [28] is selectedas the CNN feature extractor in our framework. Specifically,The VGGM is a shallow version of VGG16 [29] and widerversion of AlexNet[30], but faster than VGG16 as well asmore accurate than AlexNet. More detail about the VGGMconfiguration can be seen in Table I.

D. Training Details

1) Transfer Learning: Transferring general information be-tween different data source for related tasks is an effectivetechniques to help deal with insufficient training data and over-fitting in deep learning scenario. For instance, In the variants ofR-CNN, they firstly train the model on the large ImageNet [14]dataset and then finetune it on their domain-specific dataset.However, they were only limited to transferring informationbetween RGB(visible image) models.

The target training dataset includes the visible images, IRimages, and generated motion maps, whose data distributionis significantly different to the large-scale public visible imagedatasets, such as ImageNet [14]. Our goal is to leverage theCNN model gain essential common knowledge from a large-scale visible dataset and then transfer these information foraccelerating training in the domain-specific dataset as well asboosting overall performance.

Based on the general transfer learning techniques, theVGGM model is pre-trained on the large-scale RGB imagedataset, ImageNet, which contains most common objects andscenes in daily life. Before training the network on the fuseddataset, the weights of conv1 to conv5 in the network areinitialized by transferring learned weights. Unlike some priorwork, we do not freeze the lower layers of CNN and allow ournetwork to adapt the new data distribution in an end-to-endlearning setting.

2) Loss Function: As shown in Table I, the network has twooutput heads. The first is for classifying each ROI, which willoutput a discrete probability distribution over two categories(background and target). And the second is for regressing thebounding box offsets of ROI where for each category, it willoutput a tuple of (tx, ty, tw, th), the elements indicate the shiftvalue relative to the central coordinate, height and width oforiginal proposal ROI.

Same to [20], for classification, we use the negative loglikelihood objective:

Lcls(p, u) = −log(pu) (2)

where, p represents the predicted probability of one of cate-gories and u is the ground truth class.

For regression, the smooth L1 loss function is used:

Lbbox(tu, v) =

∑i∈{x,y,w,h}

smoothL1(tui − vi) (3)

in which tu is the bounding box offsets of the u class. And vis the true offsets.

In training stage, both of them will be jointed together asfollow:

L(p, u, tu, v) = Lcls(p, u) + λ[u = 1]Lbbox(tu, v) (4)

u = 1 means only when the class is target, the bounding boxregression can be trained. The λ is used to control balancebetween classification and regression, we set it as 1 in allexperiments.

III. EXPERIMENTS

A. Dataset

Fig. 5. Appearance of targets in training dataset and testing dataset.

We evaluate our approach on a public released ATRdatabase from Military Sensing Information Analysis Cen-ter (SENSIAC). This database package contains 207 GB ofMWIR imagery and 106 GB of visible imagery along withground truth data. All imagery was taken using commercialcameras operating in the MWIR and visible bands. The typesof targets are various, which include people, foreign militaryvehicles, and civilian vehicles. The datasets were collectedduring both daytime and night and the distance betweencameras and targets varied from 500 to 5000 meters.

In our experiments, we only consider the objects of vehicle,and split 5 types of vehicles as training targets and 3 typesof vehicles as testing targets, their name and appearance areshowed in Figure 5. And we select each type of vehicles with 3different range of distance between cameras and targets (1000,1500 and 2000 meters). It should be noted that no matter how

6

TABLE INETWORK CONFIGURATION: THE COMPLETE NETWORK ARCHITECTURE CONTAINS TWO MODULES, FIRST MODULE IS CALLED CNN feature extractorWHICH INCLUDES 5 CONVOLUTIONAL LAYERS (CONV 1-5), SECOND MODULE IS THE ROI classification and regression WHICH HAS AN ROI POOLING

LAYER AND 4 FULLY CONNECTED LAYERS.

Name Conv1 Norm1 Pool1 Conv2 Norm2 Pool2 Conv3 Conv4 Conv5 ROI Pooling FC6 FC7 Cls BboxInput Channels 3 96 96 96 256 256 256 512 512 512 36 4096 1024 1024

Output Channels 96 96 96 256 256 256 512 512 512 36 4096 1024 2 8Kernel Size 7× 7 3× 3 5× 5 3× 3 3× 3 3× 3 3× 3 6× 6

Type conv LRN max-pool conv LRN max-pool conv conv conv ROI-pool fc fc fc fcStride 2 2 2 2 1 1 1Pad 1 1 1 1

Activation function relu relu relu relu reluDropout X X

many fine-grained types of vehicle it has, we treated themas one class, ”vehicle”. Thus, the problem become a binary(vehicle and background) object detection problem. Moreover,because the format of raw data is video, we sampled theimages at every 5 frames to maximize the difference betweeneach frame. In total, we used 4573 images as training data and2812 images as testing data.

B. Experimental SetupOur framework is implemented by using Caffe deep learning

toolbox [31]. For the training machine, we use a normalcomputer with a NVIDIA GeForce GTX 1080 GPU, a IntelCore i7 CPU and 32 GB Memory. For the setup of hyper-parameters, we also follow the fast R-CNN [20], where wetrain all the networks each for 40000 iterations with initiallearning rate 0.001 and 0.0001 for the last 10000 iterations,momentum 0.9 and weight decay 0.0005.

C. Evaluation1) Metrics: For all the metrics, we considered the detection

as true or false positives based on whether the area of overlapwith ground truth bounding boxes exceed 0.5. The overlaparea can be calculated by the below equation:

ao =area(Bp ∩Bgt)

area(Bp ∪Bgt)(5)

Where ao denotes the overlap area, Bp and Bgt denotethe predicted bounding box and ground truth bounding box,respectively.

Mean Average Precision(mAP) is a golden standard metricfor evaluating the performance of an object detection algo-rithm, where it first calculates the average precision (AP)of each class and then average all the obtained AP values.Because there is only one class (vehicle) in our experiments,we select AP as one of the evaluation metrics. The APvalue can be easily obtained by computing the area under theprecision-recall curve.

Top1 Precision is a metric that is widely used in clas-sification tasks, where the probability of multiple classes ispredicted and one having the highest score is selected, then thetop1 precision score is computed as the numbers a predictedlabel matched the target label, divided by the number of wholedata. In our case, there is only one target in each image.Thus, we can employ top1 precision metric in experimentsto evaluate the performance of our framework in a practicalscenario.

Fig. 6. Average precision (AP) comparison between different experimentaldesigns. Independent input of visible, MWIR and Motion image, fusion imageof visible and MWIR image (Visible-MWIR), fusion image of visible, MWIRand Motion (3-Channels) and decision-level fusion, respectively.

2) Results and Analysis: We perform six incremental ex-periment designs to examine the effectiveness of our fusionmethod. And at the beginning, we attempt to see the perfor-mance of the detection algorithms on three type of images(Visible, MWIR and Motion) independently. Because all ofthe independent images are single-channel format and theinput format requirement of CNN is three-channels image,we generate the desired images by duplicating the single-channel image in three times. After that, we fuse visibleand MWIR images together and examine whether the fusedimage without short-temporal information can boost the over-all performance or not. To meet the requirements of CNN,we duplicate visible channel for twice. Then, we incorporatethe short-temporal information, Motion image, and generateour complete fused image, 3-channels image. In addition, wealso test the decision-level fusion method, where we combinethe three outputs of three single networks on three type ofindependent images.

Figure 6 shows the AP curves of the six incrementalexperiments.In independent image experiments, we can seethat the CNN-based detector performed well enough in overall,especially for the single visible image which achieved 97.31%average precision and 98.04% top1 accuracy, as shown inaccuracy column of Table II. The visible-MWIR fused imageget a better result than the best performance of single image.

7

TABLE IIPERFORMANCE COMPARISON ON ACCURACY AND TIME COST OF

DIFFERENT METHODS.

Methods Accuracy (%) Running Time (s/image)

AP Top1 ROIs Proposal Networks Overall

Visible 97.31 98.04 1.378 0.164 1.542MWIR 95.63 96.91 1.144 0.069 1.213Motion 91.64 92.39 1.167 0.038 1.205

Visible-MWIR 97.37 98.18 1.505 0.248 1.7533-Channels 98.34 98.90 1.272 0.235 1.507

Decision-level Fusion 97.52 97.93 3.690 0.271 3.961

It should be noted that our 3-channels image achieve boththe highest average precision(98.34%) and top1 accuracy(98.90%) which means our method only false detect 16 framesin the totally 2812 testing frames. It is also interesting that eventhought the average precision of the decision fusion method ishigher than the best single image method, but when it comes topractical application, its top1 accuracy is lower than the singlevisible image approach and it is extremely time-consuming inrunning time (3.961s).

To further verify the effectiveness of our unsupervisedimage fusion method, we visualize the feature map of thelast convolutional layer and the final output of our frameworkin Figure 7. The feature map is the output of CNN featureextractor in Figure 2, and for the fused image, it is the fusedhigh-level features. It could be reasoned that if the object infeature map is segmented clearly, the framework will get abetter result. In the examples of Figure 7, we can see thatthe 3-channels can well fuse the complementary informationfrom the three independent images and make its feature mapget enhancement. And its final output also verify the fact thatthe enhanced feature map can boost the performance.

IV. CONCLUSION

In this work, an unsupervised learning based image fusionmethod is proposed to integrate the ATD network, whichfused visible, MWIR and motion information effectively. Wefurther adopted state-of-the-art generic object detector for thebattle field object detection. We also utilized cross-domaintransfer learning techniques to deal with the insufficient databy training the model on large-scale visible image datasetfirstly and then fine-tuning on the small-scale fused imagedataset. The proposed framework was evaluated with theSENSIAC dataset. It achieved 98.34% average precision and98.90% top1 accuracy. However, the processing time is stilltoo long for real-time applications. This remains a topic forour future work.

REFERENCES

[1] J. A. Ratches, “Review of current aided/automatic target acquisitiontechnology for military target acquisition tasks,” Optical Engineering,vol. 50, DOI 10.1117/1.3601879, no. 7, pp. 072 001–072 001, 2011.

[2] Y. Sheikh and M. Shah, “Bayesian Object Detection in DynamicScenes,” 2005.

[3] J.-m. Guo, S. Member, C.-h. Hsia, M. Ieee, Y.-f. Liu, and S. Member,“Fast Background Subtraction Based on a Multilayer Codebook Modelfor Moving Object Detection,” vol. 23, no. 10, pp. 1809–1821, 2013.

[4] B.-h. Chen and S.-c. Huang, “Probabilistic neural networksbased moving vehicles extraction algorithm for intelligent trafficsurveillance systems,” Information Sciences, vol. 299, DOI10.1016/j.ins.2014.12.033, pp. 283–295, 2015. [Online]. Available:http://dx.doi.org/10.1016/j.ins.2014.12.033

[5] K. Yun, J. Lim, and J. Y. Choi, “Scene conditional background updatefor moving object detection in a moving camera,” Pattern RecognitionLetters, vol. 88, DOI 10.1016/j.patrec.2017.01.017, pp. 57–63, 2017.

[6] W.-C. Hu, C.-H. Chen, T.-Y. Chen, D.-Y. Huang, and Z.-C. Wu, “Movingobject detection and tracking from video captured by moving camera,”DOI 10.1016/j.jvcir.2015.03.003, 2015. [Online]. Available: http://ac.els-cdn.com/S104732031500053X/1-s2.0-S104732031500053X-main.pdf? tid=58e2ce5c-249b-11e7-a3f7-00000aab0f6c&acdnat=1492563708 eaac5d28180332af8f627dbfe1c446ba

[7] . Francisco, J. Lopez-Rubio, and E. Lopez-Rubio, “Foregrounddetection for moving cameras with stochastic approximation,” PatternRecognition Letters, vol. 68, DOI 10.1016/j.patrec.2015.09.007,pp. 161–168, 2015. [Online]. Available: http://ac.els-cdn.com/S0167865515003177/1-s2.0-S0167865515003177-main.pdf? tid=249b8706-249b-11e7-abfc-00000aacb362&acdnat=14925636217d6892646917decce5a8df7f2bb255ca

[8] S.-i. Keypoints and D. G. Lowe, “Distinctive Image Featuresfrom,” International Journal of Computer Vision, vol. 60, DOI10.1023/B:VISI.0000029664.99615.94, no. 2, pp. 91–110, 2004.

[9] N. Dalal, B. Triggs, O. Gradients, D. Cordelia, N. Dalal, and B. Triggs,“To cite this version : Histograms of Oriented Gradients for HumanDetection,” pp. 886–893, 2010.

[10] M. J. P. Viola, “Rapid Object Detection Using A Boosted Cascadeof Simple Features,” Proceedings of the 2001 IEEE Computer So-ciety Conference on Computer Vision and Pattern Recognition, DOI10.1109/CVPR.2001.990517, no. 1, pp. 511–518, 2001.

[11] J. A. K. Suykens and J. Vandewalle, “Least Squares SupportVector Machine Classifiers,” Neural Processing Letters 9, DOI10.1023/A:1018628609742, pp. 293–300, 1999.

[12] L. Breiman, “Randomforest2001,” DOI10.1017/CBO9781107415324.004, pp. 1–33, 2001.

[13] P. F. Felzenszwalb, R. B. Girshick, D. Mcallester, and D. Ramanan,“Object Detection with Discriminatively Trained Part Based Models,”IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. 32, DOI 10.1109/TPAMI.2009.167, no. 9, pp. 1–20, 2009.[Online]. Available: http://cs.brown.edu/∼pff/papers/lsvm-pami.pdf

[14] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,L. Fei-Fei, O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge.”[Online]. Available: https://arxiv.org/pdf/1409.0575.pdf

[15] M. Everingham, L. Van Gool, C. K. I Williams, J. Winn, A. Zisserman,M. Everingham, L. K. Van Gool Leuven, B. CKI Williams, J. Winn, andA. Zisserman, “The PASCAL Visual Object Classes (VOC) Challenge,”Int J Comput Vis, vol. 88, DOI 10.1007/s11263-009-0275-4, pp. 303–338, 2010.

[16] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick,J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dolı,“Microsoft COCO: Common Objects in Context.” [Online]. Available:https://arxiv.org/pdf/1405.0312.pdf

[17] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. Lecun,“OverFeat: Integrated Recognition, Localization and Detection usingConvolutional Networks,” 2014.

[18] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich featurehierarchies for accurate object detection and semantic segmentation.”[Online]. Available: http://www.cs.berkeley.edu

[19] J. R. R. Uijlings, K. E. A. Van De Sande, T. Gevers, and A. W. M.Smeulders, “Selective Search for Object Recognition,” Int J ComputVis, vol. 104, DOI 10.1007/s11263-013-0620-5, pp. 154–171, 2013.[Online]. Available: http://disi.

[20] R. Girshick, “Fast R-CNN,” in Proceedings of the IEEE InternationalConference on Computer Vision, DOI 10.1109/ICCV.2015.169, 2016.

[21] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial Pyramid Pooling inDeep Convolutional Networks for Visual Recognition,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, vol. 37, DOI10.1109/TPAMI.2015.2389824, no. 9, pp. 1904–1916, 2015.

[22] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.”

[23] G. Bhatnagar and Z. Liu, “A novel image fusion framework for night-vision navigation and surveillance,” Signal, Image and Video Processing,DOI 10.1007/s11760-014-0740-6, 2015.

8

[24] Y. Niu, S. Xu, L. Wu, and W. Hu, “Airborne infrared and visible imagefusion for target perception based on target region segmentation anddiscrete wavelet transform,” Mathematical Problems in Engineering, vol.2012, DOI 10.1155/2012/275138, 2012.

[25] J. Han and B. Bhanu, “Fusion of color and infrared videofor moving human detection,” Pattern Recognition, vol. 40,DOI 10.1016/j.patcog.2006.11.010, pp. 1771–1784, 2007. [Online].Available: https://www.researchgate.net/profile/Bir Bhanu/publication/222827442 Fusion of color and infrared video for moving humandetection/links/0912f50b0b010e4b15000000.pdf

[26] M. A. Smeelen, P. B. W. Schwering, A. Toet, and M. Loog, “Semi-hidden target recognition in gated viewer images fused with thermal IRimages,” Information Fusion, vol. 18, DOI 10.1016/j.inffus.2013.08.001,pp. 131–147, 2014. [Online]. Available: http://ac.els-cdn.com/S156625351300081X/1-s2.0-S156625351300081X-main.pdf?tid=ea9a0eba-24a9-11e7-bc7a-00000aab0f26&acdnat=1492569966af49728e4b8bc2c1eb9379e1c1606efb

[27] P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient Graph-Based Image Segmentation,” International Journal of ComputerVision, vol. 59, no. 2, pp. 167–181, 2004. [Online]. Avail-able: http://fcv2011.ulsan.ac.kr/files/announcement/413/IJCV(2004)%20Efficient%20Graph-Based%20Image%20Segmentation.pdf

[28] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return ofthe Devil in the Details: Delving Deep into Convolutional Nets,” arXivpreprint arXiv: . . . , DOI 10.5244/C.28.6, pp. 1–11, 2014. [Online].Available: http://arxiv.org/abs/1405.3531

[29] K. Simonyan and A. Zisserman, “VERY DEEP CONVOLUTIONALNETWORKS FOR LARGE-SCALE IMAGE RECOGNITION,” 2015.[Online]. Available: https://arxiv.org/pdf/1409.1556/

[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNetClassification with Deep Convolutional Neural Networks,”Advances In Neural Information Processing Systems, DOIhttp://dx.doi.org/10.1016/j.protcy.2014.09.007, pp. 1–9, 2012.

[31] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture forFast Feature Embedding *.” [Online]. Available: https://arxiv.org/pdf/1408.5093.pdf

9

Fig. 7. Example visualizing results of our framework. Example 1 and 2 demonstrate the performance of different type of inputs in system on large and smallobject detection, respectively. Different columns denote different types of input image. The raw input image, generated feature map and the final output areshowed in consecutive rows. In the final output image, the green bounding box represent the position of object predicted by system.