review article literature survey on stereo vision
TRANSCRIPT
Review ArticleLiterature Survey on Stereo Vision Disparity Map Algorithms
Rostam Affendi Hamzah12 and Haidi Ibrahim1
1School of Electrical amp Electronic Engineering Universiti SainsMalaysia Engineering Campus 14300 Nibong Tebal Penang Malaysia2Fakulti Teknologi Kejuruteraan (FTK) Universiti Teknikal Malaysia Melaka Kampus Teknologi Hang Tuah Jaya76100 Durian Tunggal Melaka Malaysia
Correspondence should be addressed to Haidi Ibrahim haidi ibrahimieeeorg
Received 21 August 2015 Accepted 11 November 2015
Academic Editor Eduard Llobet
Copyright copy 2016 R A Hamzah and H Ibrahim This is an open access article distributed under the Creative CommonsAttribution License which permits unrestricted use distribution and reproduction in any medium provided the original work isproperly cited
This paper presents a literature survey on existing disparitymap algorithms It focuses on fourmain stages of processing as proposedby Scharstein and Szeliski in a taxonomy and evaluation of dense two-frame stereo correspondence algorithms performed in 2002To assist future researchers in developing their own stereomatching algorithms a summary of the existing algorithms developed forevery stage of processing is also providedThe survey also notes the implementation of previous software-based and hardware-basedalgorithms Generally the main processing module for a software-based implementation uses only a central processing unit Bycontrast a hardware-based implementation requires one or more additional processors for its processingmodule such as graphicalprocessing unit or a field programmable gate array This literature survey also presents a method of qualitative measurement thatis widely used by researchers in the area of stereo vision disparity mappings
1 Introduction
Computer vision is currently an important field of researchIt includes methods such as image acquisition processinganalysis and understanding [1] Computer vision techniquesattempt tomodel a complex visual environment using variousmathematical methods One of the purposes of computervision is to define the world that we see based on one ormore images and to restructure its properties such as itsillumination shape and color distributions Stereo vision isan area within the field of computer vision that addresses animportant research problem which is the reconstruction ofthe three-dimensional coordinates of points for depth esti-mation A system of stereo vision system consists of a stereocamera namely two cameras placed horizontally (ie one onthe left and the other on the right) The two images capturedsimultaneously by these cameras are then processed for therecovery of visual depth information [2] The challenge is todetermine the best method of approximating the differencesbetween the views shown in the two images to map (ieplot) the correspondence (ie disparity) of the environment
Intuitively a disparity map represents corresponding pixelsthat are horizontally shifted between the left image andright image New methods and techniques for solving thisproblem are developed every year and exhibit a trend towardimprovement in accuracy and time consumption
Another device that is used to acquire depth informationis a time-of-flight (ToF) or structured light sensor Such adevice is a type of active sensor unlike a classic stereo visioncamera Devices of this type such as the Microsoft Kinectare cheap and have led to increased interest in computervision applications However these active sensors suffer fromcertain characteristic problems [3] First they are subject tosystematic errors such as noise and ambiguity which arerelated to the particular sensor that is used Second theyare subject to nonsystematic errors such as scattering andmotion blur According to the comparative analyses per-formed by Foix et al [4] Kim et al [5] and Zhang et al[6] ToF devices perform satisfactorily only up to amaximumdistance of approximately 5ndash7 meters and are too sensitive tobe used in outdoor environments especially in very brightareas Because of these limitations of ToF sensors stereo
Hindawi Publishing CorporationJournal of SensorsVolume 2016 Article ID 8742920 23 pageshttpdxdoiorg10115520168742920
2 Journal of Sensors
Quantity of original papers from ScienceDirect
22 23 39
32
33
43
50
58
41
47
YearTrendline
0
10
20
30
40
50
60
70
Qua
ntity
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Journalbook
(a)
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
YearTrendline
Quantity of original papers from IEEE Xplore
92
144
144
165
198
207
186
224
168
171
0
50
100
150
200
250
Qua
ntity
Conferencejournal
(b)
Figure 1 The yearly distributions of the number of papers published from 2005 to 2014 in the area of stereo vision from the database of (a)ScienceDirect and (b) IEEE Xplore
vision sensors (ie passive sensors) are more reliable androbust they are able to produce high-resolution disparitymaps and are suitable for both indoor and outdoor environ-ments [7]
In stereo vision disparity map processing the num-ber of calculations required increases with an increasingnumber of pixels per image This phenomenon causes thematching problem to be computationally complex [8] Theimprovements to and reduction in computational complexitythat have been achieved with recent advances in hardwaretechnology have been beneficial for the advancement ofresearch in the stereo vision field Thus the main motivationfor hardware-based implementation is to achieve real timeprocessing [9] In real time stereo vision applications suchas autonomous driving 3D gaming and autonomous roboticnavigation fast but accurate depth estimations are required[10] Additional processing hardware is therefore necessaryto improve the processing speed
An updated survey on stereo vision disparity map algo-rithms would be valuable to those who are interested in thisresearch area Figures 1(a) and 1(b) illustrate the quantityof original contributions published in this area over thepast ten years (ie 2005ndash2014) from the databases of Sci-enceDirect and IEEE Xplore The keywords used were stereovisionstereo vision algorithm and the components that weresearchedwere the title abstract and keywordsindex terms ofthe papers in the databases All of these papers may representcontributions to fundamental algorithm development analy-sis or application of stereo vision algorithms In both figuresthe trendlines are increasing indicating that the field ofstereo vision remains active in research and development andhas become an interesting and challenging area of researchThis paper provides a brief introduction to the state-of-the-art developments accomplished in the context of suchalgorithms This work reviews the latest published stereovision algorithms and categorizes them into different stagesof processing which are based on the taxonomy proposedby Scharstein and Szeliski [11] This paper also discusses
two types of implementation platforms for these algorithms(ie software-based and hardware-based) In software-basedplatforms the techniques are implemented only on a standardCPU without any other additional processing hardware Incontrast to hardware-based platforms the algorithms areexecuted on a CPU with a GPU or FPGA as a standalonesystem
The remainder of the paper is organized as followsPrevious review papers related to stereo vision disparity mapalgorithms are discussed in Section 2Then the taxonomy forthe stages of processing performed in stereo vision disparitymap algorithms is presented in Section 3 It consists of foursubsections (ie matching cost computation cost aggre-gation disparity selection and optimization and disparityrefinement) Section 4 presents a review of algorithms imple-mented through software-based platforms and Section 5discusses real time stereo vision disparity map algorithmsbased on additional hardware (ie FPGAs and GPUs) Amethod ofmeasuring the accuracy of stereo vision algorithmsis explained in Section 6 and the conclusion is presented inSection 7
2 Previous Reviews of Stereo VisionDisparity Map Algorithms
Numerous methods of implementation for stereo visiondisparity mapping have been established in the past fewyears This can be observed from the review papers listedin Table 1 The contents of these review papers are alsosummarized in this table Among these review papers themain focus was to summarize and compare the accuracylevel and execution time of each cited algorithm Howevernone of these reviews provided a detailed discussion of thestages of implementation based on the taxonomy proposedby Scharstein and Szeliski [11] as does the survey presentedin this paper Furthermore this paper also reviews thelatest algorithms implemented using two different types ofplatforms (ie software-based and hardware-based)
Journal of Sensors 3
Table 1 Previous review papers on stereo vision disparity map algorithms
Year Author Focus
2002 Scharstein and Szeliski [11]
Proposed a taxonomy for vision algorithms and provided a quality metric to compare andevaluate multiple blocks of algorithms as shown in Figure 1 They have also provided a test bed formeasurable evaluation of stereo depth map algorithms The test bed or benchmarking datasetconsists of four images (Tsukuba Venus Teddy and Cones) which are available athttpwwwmiddleburyedustereo
2003 Brown et al [19] Reviewed advances in stereo vision disparity map algorithms regarding correspondence methodsand occlusion handling methods for real time implementations
2008 Tombari et al [77] Presented a survey and compared the different methods of cost aggregation for stereocorrespondence through accuracy and computational requirements
2008 Lazaros et al [12]Reviewed developments in stereo vision algorithms implemented via software and hardwarecategorized in terms of their major attributes The comparison of local and global methodsprovided by previously developed algorithms implemented on software and hardware basedplatforms was presented in this work
2011 Tombari et al [101] Contributed an evaluation of stereo vision depth map algorithms in terms of their 3D objectrecognition ability
2013 Tippetts et al [8]Reviewed stereo vision algorithms and their suitability for resource-limited systems They havecompiled and presented an accuracy and runtime performance data for all stereo vision disparitymap algorithms in the past decade with an emphasis on real time performance
Inputimages
Step 1
Matchingcost
computation
Step 2
Costaggregation
Step 3
Disparityselection
Step 4
Disparityrefinement
Disparitymap
Figure 2 A framework for the development of a stereo vision algorithm
3 A Taxonomy for the Processing Stages ofStereo Vision Disparity Map Algorithms
Most stereo vision disparitymap algorithms have been imple-mented using multistage techniques These techniques ascodified by Scharstein and Szeliski consist of four main stepsas shown in Figure 2 [11] In this figure the input images areobtained from stereo vision sensors (ie from at least twocameras) Commonly these cameras are arranged horizon-tally and set up which produce two or more correspondingimages For the explanation or the process as described byadopted taxonomy these input images are assumed to berectified images Next the image pair to be analysed will passthrough all of the blocks in sequence beginning with Step1 and ending with Step 4 The output of this process shouldbe a smooth disparity map In essence each block representsone ormore algorithms whose performance can bemeasuredbased on the expected outputThis taxonomyhas beenwidelyused by many current developers of stereo vision disparitymap algorithms [8 12]
In general stereo vision disparity map algorithms can beclassified into local or global approaches A local approachis also known as area based or window based approachThis is because the disparity computation at a given point(or pixel) depends only on the intensity values within apredefined support window Thus such method considersonly local information and therefore has a low computationalcomplexity and a short run time Local methods include all
four steps of the taxonomy Examples of implementation ofsuch methods are provided by the work of Mattoccia et al[13] Arranz et al [14] and Xu et al [15] The disparity mapvalue assignment is achieved through winner take all (WTA)optimization For each pixel the corresponding disparityvalue with the minimum cost is assigned to that pixel Thematching cost is aggregated via a sum or an average over thesupport window
By contrast a global method treats disparity assignmentas a problem of minimizing a global energy function forall disparity values Such a method is formulated as anenergy minimization process with two terms in the objectivefunction (ie a data term which penalizes solutions thatare inconsistent with the target data and a smoothness termwhich enforces the piecewise smoothing assumption withneighboring pixels) The smoothness term is designed toretain smoothness in disparity among pixels in the sameregion The disparity map is produced by assigning similardepth values to neighboring pixels Global methods producegood results but are computationally expensive Thereforethey are impractical for use in real time systems Globalmethods typically skip Step 2 of the taxonomy depicted inFigure 2 (ie they do not perform cost aggregation andtherefore contain only three steps) [16ndash18] Markov randomfield (MRF) modelling is the approach that is most commonapproach used in globalmethodsThis type ofmodelling usesan iterative framework to ensure smooth disparity maps andhigh similarity between matching pixels
4 Journal of Sensors
P
Il Ir
pl pre e
OlOr
Figure 3 Epipolar geometry The 3D image of target scene at point119875
31 Matching Cost Computation All stereo matching algo-rithms require a cost criterion tomeasure the extent ofmatch-ing between two pixelsThematching cost computation is thestage in which whether the values of two pixels correspond tothe same point in a scene is determinedTherefore the stereomatching cost computation can be defined as a method ofdetermining the parallax values of each point between the leftand right images [19] The matching cost is computed at eachpixel for all pixels under considerationThe difference in pixelintensity between a pair of the matching pixels in two imagesis called the disparity and can be associated with depth valuesthrough three-dimensional (3D) projection
The matching points must lie on epipolar lines 119890 asshown in Figure 3 This matching can be performed via aone-dimensional horizontal search if the stereo pairs areaccurately calibrated [20] A target point 119875 is viewed fromthe optical centers 119874119897 and 119874119903 of the two cameras It producesone left image plane 119868119897 and one corresponding right imageplane 119868119903 one from each of the two cameras The points 119901119897and 119901119903 represent the matching pixel intensities in the left andright image planes respectively for the same scene at point119875Therefore the targeted left and right matching points mustbe located at the same coordinates on the horizontal line 119890In a stereo vision system this requirement is imposed in theform of an epipolar constraint [21] This epipolar constraintplays a significant role in stereo matchingThis is because thesearch for correspondences can be limited to a line insteadof the entire image space thereby reducing the required timeand search range Early stereo vision disparity map algorithmfor the matching cost computation task uses pixel basedtechnique [12] These algorithms are the methods of absolutedifferences (AD) squared differences (SD) adaptations ofthe former include the methods of sampling-insensitiveabsolute differences and truncated absolute differencesThesealgorithms can be applied to grayscale or color images
Area based or window based techniques are capableof offering richer data than matching techniques basedon individual pixels or features Such techniques can be
more accurate because the matching process considers theentire set of pixels associated with image regions Commonalgorithms for window based techniques include the sum ofabsolute differences (SAD) the sum of squared differences(SSD) normalized cross correlation (NCC) rank transforms(RT) and census transforms (CT) [12] The matching cost iscalculated over a support region This support region whichis commonly referred to as support or aggregating windowmay be square or rectangular and may be fixed or adaptivein sizeThemajor shortcoming of window based technique isthat these approaches commonly assume that all pixels withina support window have similar disparity values This is notnecessarily true for pixels near depth discontinuities or edgesHence an improper selection of the size and shape of thematching window can lead to poor depth estimations
311 Absolute Differences (AD) The AD algorithm aggre-gates the differences in luminance (or intensity values)between the pixel in the left image 119868119897 and the correspondingpixels in the right image 119868119903 as given by
AD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (1)
In this equation (119909 119910 119889) represents the disparitymap coordi-nates where (119909 119910) are the coordinates of the pixel of interestand 119889 is the disparity (or depth) value Typically in thematching process 119868119897 is used as the reference image and theright image 119868119903 represents the target (or candidate) imageThe AD algorithm is the simplest among matching cost algo-rithms Because of its low complexity Wang et al [22] usedthis algorithm for real time stereo matching using graphicshardware (GPU) The AD algorithm functions satisfactorilyin regions with little texture but for highly textured imagesthis algorithm is not capable of producing a smooth disparitymap To overcome this difficulty the truncated version ofthe AD algorithm was developed The truncated absolutedifference (TAD) algorithm as implemented by Min et al[23] and Pham and Jeon [24] is able to minimize the errorsin disparity maps Furthermore the TAD algorithm usesthe colors and gradients at matching pixels to improve itsrobustness against variations in illumination
312 Squared Differences (SD) The SD algorithm aggregatesthe squared differences between the reference pixels in 119868119897 andthe candidate pixels in 119868119903 as described in
SD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (2)
Yang et al [25] implemented the SD algorithm for theirmatching cost computation in a subpixel estimation methodfor disparity mapping Considerable noise was generated atthe boundaries in their initial disparity maps However Yanget al applied a bilateral filter (ie a type of edge preservingfilter) to improve the flattening of edges and to smooth areasnear depth discontinuities Recently Miron et al [26] testedvarious matching cost functions in their stereo disparitymap algorithms for intelligent vehicle applications Theyconcluded that the SD algorithm produced the largest errorThe errors that occur due to the SD algorithm are highlysensitive to brightness and noise especially in real timeenvironment
Journal of Sensors 5
313 Feature Based Techniques The matching cost func-tion can also be constructed using feature based tech-niques This approach attempts to establish correspondencesonly for similar feature points that can be unambiguouslymatched Common methods of feature extraction includethose based on visual features (eg edges shapes texturessegmentation and gradient peaks) statistical characteristics(eg minima medians and histograms) and transformationfeatures (eg Hough transforms wavelet transforms andGabor transforms) [27] As an example Sharma et al [28]developed a new disparity map algorithm using featuresderived from the scale invariant feature transform (SIFT)for autonomous vehicle navigation In their implementationthey modified the SIFT algorithm to use self-organizing mapto achieve more efficient performance in feature matchingprocess Their results indicate that the computation time oftheir method is reduced compared with the conventionalSIFT algorithm Because only feature points are correlatedalthough the computational cost is significantly reducedthe complete disparity map cannot be obtained The featurematching accuracy is still low Sparse disparity maps areproduced because only matching points derived from thetargeted object features are used [29] Furthermore featurebased matching techniques exhibit low accuracy and arerather insensitive to occlusion and textureless areas Liuet al [30] used a combination of image segmentation andedge detection for the matching cost function The timeimplementation was fast but the accuracy level also remainedlow in regions of discontinuity Ekstrand et al [31] used asegmentation process that created one-dimensional segmentscontaining information on color and edge coordinates Asegment correspondence estimation was performed for everyrow of the image to reduce the possibility of mismatchedsegments However errors still occurred as a result of usinglimited vertical support segments during the matching pro-cess Thus feature based techniques are not preferred andtheir usage by researchers for the development of disparitymap algorithms remains low
314 Sum of Absolute Differences (SAD) TheSADalgorithmdefined in (3) considers the absolute difference between theintensity of each pixel in the reference block and that of thecorresponding pixel in the target block
SAD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (3)
Thedifferences are summed over the aggregated support win-dow 119908 to generate a simple metric of block similarity knownas a disparity map The SAD algorithm is a well-known algo-rithm for matching cost computation The SAD algorithm isable to function in real time implementation because of itslow computational complexity This was proven by Tippettset al [32] who calculated and evaluated SAD performancesfor real time human pose images in a resource limited systemLee and Sharma [33] implemented real time disparity mapalgorithm estimations using the sliding window techniqueto calculate matching costs using the SAD algorithm Theiralgorithm uses parallel processing via a graphical processing
unit (GPU) By virtue of applying this new technique at thematching cost stage the accuracy of stereo vision process-ing can be increased while simultaneously improving thespeed
Gupta and Cho [34] implemented a new techniqueusing two different sizes of correlation windows in the SADalgorithm At the first level the initial cost aggregation isdetermined and at the second level the object boundariesare improved using a smaller window size compared withthe first-level implementationThe results produced aremoreaccurate in terms of pixel matching but still require theimplementation of multiple loops at different window sizesThe SAD algorithm is fast but the quality of the initialdisparity map that is produced is low because of the noise atobject boundaries and in textureless regions
315 Sum of Squared Differences (SSD) Equation (4) pre-sents the SSD algorithm in which the summation is per-formed over the squared differences in pixel intensity valuesbetween two corresponding pixels in the aggregated supportwindow 119908
SSD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (4)
An early implementation of the SSD algorithm for thematching cost calculation stage was achieved by Fusielloet al [36] They tested the SSD algorithm on multiple fixedwindow blocks to reduce the incidence of occlusion errorsThe purpose of using multiple blocks of windows is tosearch for the smallest error to select an appropriate pixelof interest in the disparity map Yang and Pollefeys [37]used technique similar to that presented in [36] but theiralgorithm was implemented on platform with a GPU Theyachieved good results in terms of speed compared withFusiellorsquos work Currently there is still relatively little researchon the use of the SSD algorithm in for stereo vision disparitymap algorithms compared with that on other matching costalgorithms This is evident from the previous review papers[8 12] on stereo vision disparity map algorithms and is alsoshown in Table 2
316 Normalized Cross Correlation (NCC) The NCC algo-rithm is another method of determining the correspondencebetween two windows around a pixel of interest The nor-malization within the window compensates for differences ingain and bias [38] Equation (5) specifies the formula for theNCC technique
NCC (119909 119910 119889)
=
sum(119909119910)isin119908119868119897 (119909 119910) sdot 119868119903 (119909 minus 119889 119910)
radicsum(119909119910)isin119908 1198682
119897(119909 119910) sdot sum
(119909119910)isin1199081198682119903(119909 minus 119889 119910)
(5)
However the NCC algorithm tends to blur regions of dis-continuity more than other matching cost algorithms [38]This is because any outliers lead to large errors in the NCCcalculations A new method for low-dimensional image fea-tures matching using NCC has been proposed by Satoh [39]
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
2 Journal of Sensors
Quantity of original papers from ScienceDirect
22 23 39
32
33
43
50
58
41
47
YearTrendline
0
10
20
30
40
50
60
70
Qua
ntity
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
Journalbook
(a)
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
YearTrendline
Quantity of original papers from IEEE Xplore
92
144
144
165
198
207
186
224
168
171
0
50
100
150
200
250
Qua
ntity
Conferencejournal
(b)
Figure 1 The yearly distributions of the number of papers published from 2005 to 2014 in the area of stereo vision from the database of (a)ScienceDirect and (b) IEEE Xplore
vision sensors (ie passive sensors) are more reliable androbust they are able to produce high-resolution disparitymaps and are suitable for both indoor and outdoor environ-ments [7]
In stereo vision disparity map processing the num-ber of calculations required increases with an increasingnumber of pixels per image This phenomenon causes thematching problem to be computationally complex [8] Theimprovements to and reduction in computational complexitythat have been achieved with recent advances in hardwaretechnology have been beneficial for the advancement ofresearch in the stereo vision field Thus the main motivationfor hardware-based implementation is to achieve real timeprocessing [9] In real time stereo vision applications suchas autonomous driving 3D gaming and autonomous roboticnavigation fast but accurate depth estimations are required[10] Additional processing hardware is therefore necessaryto improve the processing speed
An updated survey on stereo vision disparity map algo-rithms would be valuable to those who are interested in thisresearch area Figures 1(a) and 1(b) illustrate the quantityof original contributions published in this area over thepast ten years (ie 2005ndash2014) from the databases of Sci-enceDirect and IEEE Xplore The keywords used were stereovisionstereo vision algorithm and the components that weresearchedwere the title abstract and keywordsindex terms ofthe papers in the databases All of these papers may representcontributions to fundamental algorithm development analy-sis or application of stereo vision algorithms In both figuresthe trendlines are increasing indicating that the field ofstereo vision remains active in research and development andhas become an interesting and challenging area of researchThis paper provides a brief introduction to the state-of-the-art developments accomplished in the context of suchalgorithms This work reviews the latest published stereovision algorithms and categorizes them into different stagesof processing which are based on the taxonomy proposedby Scharstein and Szeliski [11] This paper also discusses
two types of implementation platforms for these algorithms(ie software-based and hardware-based) In software-basedplatforms the techniques are implemented only on a standardCPU without any other additional processing hardware Incontrast to hardware-based platforms the algorithms areexecuted on a CPU with a GPU or FPGA as a standalonesystem
The remainder of the paper is organized as followsPrevious review papers related to stereo vision disparity mapalgorithms are discussed in Section 2Then the taxonomy forthe stages of processing performed in stereo vision disparitymap algorithms is presented in Section 3 It consists of foursubsections (ie matching cost computation cost aggre-gation disparity selection and optimization and disparityrefinement) Section 4 presents a review of algorithms imple-mented through software-based platforms and Section 5discusses real time stereo vision disparity map algorithmsbased on additional hardware (ie FPGAs and GPUs) Amethod ofmeasuring the accuracy of stereo vision algorithmsis explained in Section 6 and the conclusion is presented inSection 7
2 Previous Reviews of Stereo VisionDisparity Map Algorithms
Numerous methods of implementation for stereo visiondisparity mapping have been established in the past fewyears This can be observed from the review papers listedin Table 1 The contents of these review papers are alsosummarized in this table Among these review papers themain focus was to summarize and compare the accuracylevel and execution time of each cited algorithm Howevernone of these reviews provided a detailed discussion of thestages of implementation based on the taxonomy proposedby Scharstein and Szeliski [11] as does the survey presentedin this paper Furthermore this paper also reviews thelatest algorithms implemented using two different types ofplatforms (ie software-based and hardware-based)
Journal of Sensors 3
Table 1 Previous review papers on stereo vision disparity map algorithms
Year Author Focus
2002 Scharstein and Szeliski [11]
Proposed a taxonomy for vision algorithms and provided a quality metric to compare andevaluate multiple blocks of algorithms as shown in Figure 1 They have also provided a test bed formeasurable evaluation of stereo depth map algorithms The test bed or benchmarking datasetconsists of four images (Tsukuba Venus Teddy and Cones) which are available athttpwwwmiddleburyedustereo
2003 Brown et al [19] Reviewed advances in stereo vision disparity map algorithms regarding correspondence methodsand occlusion handling methods for real time implementations
2008 Tombari et al [77] Presented a survey and compared the different methods of cost aggregation for stereocorrespondence through accuracy and computational requirements
2008 Lazaros et al [12]Reviewed developments in stereo vision algorithms implemented via software and hardwarecategorized in terms of their major attributes The comparison of local and global methodsprovided by previously developed algorithms implemented on software and hardware basedplatforms was presented in this work
2011 Tombari et al [101] Contributed an evaluation of stereo vision depth map algorithms in terms of their 3D objectrecognition ability
2013 Tippetts et al [8]Reviewed stereo vision algorithms and their suitability for resource-limited systems They havecompiled and presented an accuracy and runtime performance data for all stereo vision disparitymap algorithms in the past decade with an emphasis on real time performance
Inputimages
Step 1
Matchingcost
computation
Step 2
Costaggregation
Step 3
Disparityselection
Step 4
Disparityrefinement
Disparitymap
Figure 2 A framework for the development of a stereo vision algorithm
3 A Taxonomy for the Processing Stages ofStereo Vision Disparity Map Algorithms
Most stereo vision disparitymap algorithms have been imple-mented using multistage techniques These techniques ascodified by Scharstein and Szeliski consist of four main stepsas shown in Figure 2 [11] In this figure the input images areobtained from stereo vision sensors (ie from at least twocameras) Commonly these cameras are arranged horizon-tally and set up which produce two or more correspondingimages For the explanation or the process as described byadopted taxonomy these input images are assumed to berectified images Next the image pair to be analysed will passthrough all of the blocks in sequence beginning with Step1 and ending with Step 4 The output of this process shouldbe a smooth disparity map In essence each block representsone ormore algorithms whose performance can bemeasuredbased on the expected outputThis taxonomyhas beenwidelyused by many current developers of stereo vision disparitymap algorithms [8 12]
In general stereo vision disparity map algorithms can beclassified into local or global approaches A local approachis also known as area based or window based approachThis is because the disparity computation at a given point(or pixel) depends only on the intensity values within apredefined support window Thus such method considersonly local information and therefore has a low computationalcomplexity and a short run time Local methods include all
four steps of the taxonomy Examples of implementation ofsuch methods are provided by the work of Mattoccia et al[13] Arranz et al [14] and Xu et al [15] The disparity mapvalue assignment is achieved through winner take all (WTA)optimization For each pixel the corresponding disparityvalue with the minimum cost is assigned to that pixel Thematching cost is aggregated via a sum or an average over thesupport window
By contrast a global method treats disparity assignmentas a problem of minimizing a global energy function forall disparity values Such a method is formulated as anenergy minimization process with two terms in the objectivefunction (ie a data term which penalizes solutions thatare inconsistent with the target data and a smoothness termwhich enforces the piecewise smoothing assumption withneighboring pixels) The smoothness term is designed toretain smoothness in disparity among pixels in the sameregion The disparity map is produced by assigning similardepth values to neighboring pixels Global methods producegood results but are computationally expensive Thereforethey are impractical for use in real time systems Globalmethods typically skip Step 2 of the taxonomy depicted inFigure 2 (ie they do not perform cost aggregation andtherefore contain only three steps) [16ndash18] Markov randomfield (MRF) modelling is the approach that is most commonapproach used in globalmethodsThis type ofmodelling usesan iterative framework to ensure smooth disparity maps andhigh similarity between matching pixels
4 Journal of Sensors
P
Il Ir
pl pre e
OlOr
Figure 3 Epipolar geometry The 3D image of target scene at point119875
31 Matching Cost Computation All stereo matching algo-rithms require a cost criterion tomeasure the extent ofmatch-ing between two pixelsThematching cost computation is thestage in which whether the values of two pixels correspond tothe same point in a scene is determinedTherefore the stereomatching cost computation can be defined as a method ofdetermining the parallax values of each point between the leftand right images [19] The matching cost is computed at eachpixel for all pixels under considerationThe difference in pixelintensity between a pair of the matching pixels in two imagesis called the disparity and can be associated with depth valuesthrough three-dimensional (3D) projection
The matching points must lie on epipolar lines 119890 asshown in Figure 3 This matching can be performed via aone-dimensional horizontal search if the stereo pairs areaccurately calibrated [20] A target point 119875 is viewed fromthe optical centers 119874119897 and 119874119903 of the two cameras It producesone left image plane 119868119897 and one corresponding right imageplane 119868119903 one from each of the two cameras The points 119901119897and 119901119903 represent the matching pixel intensities in the left andright image planes respectively for the same scene at point119875Therefore the targeted left and right matching points mustbe located at the same coordinates on the horizontal line 119890In a stereo vision system this requirement is imposed in theform of an epipolar constraint [21] This epipolar constraintplays a significant role in stereo matchingThis is because thesearch for correspondences can be limited to a line insteadof the entire image space thereby reducing the required timeand search range Early stereo vision disparity map algorithmfor the matching cost computation task uses pixel basedtechnique [12] These algorithms are the methods of absolutedifferences (AD) squared differences (SD) adaptations ofthe former include the methods of sampling-insensitiveabsolute differences and truncated absolute differencesThesealgorithms can be applied to grayscale or color images
Area based or window based techniques are capableof offering richer data than matching techniques basedon individual pixels or features Such techniques can be
more accurate because the matching process considers theentire set of pixels associated with image regions Commonalgorithms for window based techniques include the sum ofabsolute differences (SAD) the sum of squared differences(SSD) normalized cross correlation (NCC) rank transforms(RT) and census transforms (CT) [12] The matching cost iscalculated over a support region This support region whichis commonly referred to as support or aggregating windowmay be square or rectangular and may be fixed or adaptivein sizeThemajor shortcoming of window based technique isthat these approaches commonly assume that all pixels withina support window have similar disparity values This is notnecessarily true for pixels near depth discontinuities or edgesHence an improper selection of the size and shape of thematching window can lead to poor depth estimations
311 Absolute Differences (AD) The AD algorithm aggre-gates the differences in luminance (or intensity values)between the pixel in the left image 119868119897 and the correspondingpixels in the right image 119868119903 as given by
AD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (1)
In this equation (119909 119910 119889) represents the disparitymap coordi-nates where (119909 119910) are the coordinates of the pixel of interestand 119889 is the disparity (or depth) value Typically in thematching process 119868119897 is used as the reference image and theright image 119868119903 represents the target (or candidate) imageThe AD algorithm is the simplest among matching cost algo-rithms Because of its low complexity Wang et al [22] usedthis algorithm for real time stereo matching using graphicshardware (GPU) The AD algorithm functions satisfactorilyin regions with little texture but for highly textured imagesthis algorithm is not capable of producing a smooth disparitymap To overcome this difficulty the truncated version ofthe AD algorithm was developed The truncated absolutedifference (TAD) algorithm as implemented by Min et al[23] and Pham and Jeon [24] is able to minimize the errorsin disparity maps Furthermore the TAD algorithm usesthe colors and gradients at matching pixels to improve itsrobustness against variations in illumination
312 Squared Differences (SD) The SD algorithm aggregatesthe squared differences between the reference pixels in 119868119897 andthe candidate pixels in 119868119903 as described in
SD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (2)
Yang et al [25] implemented the SD algorithm for theirmatching cost computation in a subpixel estimation methodfor disparity mapping Considerable noise was generated atthe boundaries in their initial disparity maps However Yanget al applied a bilateral filter (ie a type of edge preservingfilter) to improve the flattening of edges and to smooth areasnear depth discontinuities Recently Miron et al [26] testedvarious matching cost functions in their stereo disparitymap algorithms for intelligent vehicle applications Theyconcluded that the SD algorithm produced the largest errorThe errors that occur due to the SD algorithm are highlysensitive to brightness and noise especially in real timeenvironment
Journal of Sensors 5
313 Feature Based Techniques The matching cost func-tion can also be constructed using feature based tech-niques This approach attempts to establish correspondencesonly for similar feature points that can be unambiguouslymatched Common methods of feature extraction includethose based on visual features (eg edges shapes texturessegmentation and gradient peaks) statistical characteristics(eg minima medians and histograms) and transformationfeatures (eg Hough transforms wavelet transforms andGabor transforms) [27] As an example Sharma et al [28]developed a new disparity map algorithm using featuresderived from the scale invariant feature transform (SIFT)for autonomous vehicle navigation In their implementationthey modified the SIFT algorithm to use self-organizing mapto achieve more efficient performance in feature matchingprocess Their results indicate that the computation time oftheir method is reduced compared with the conventionalSIFT algorithm Because only feature points are correlatedalthough the computational cost is significantly reducedthe complete disparity map cannot be obtained The featurematching accuracy is still low Sparse disparity maps areproduced because only matching points derived from thetargeted object features are used [29] Furthermore featurebased matching techniques exhibit low accuracy and arerather insensitive to occlusion and textureless areas Liuet al [30] used a combination of image segmentation andedge detection for the matching cost function The timeimplementation was fast but the accuracy level also remainedlow in regions of discontinuity Ekstrand et al [31] used asegmentation process that created one-dimensional segmentscontaining information on color and edge coordinates Asegment correspondence estimation was performed for everyrow of the image to reduce the possibility of mismatchedsegments However errors still occurred as a result of usinglimited vertical support segments during the matching pro-cess Thus feature based techniques are not preferred andtheir usage by researchers for the development of disparitymap algorithms remains low
314 Sum of Absolute Differences (SAD) TheSADalgorithmdefined in (3) considers the absolute difference between theintensity of each pixel in the reference block and that of thecorresponding pixel in the target block
SAD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (3)
Thedifferences are summed over the aggregated support win-dow 119908 to generate a simple metric of block similarity knownas a disparity map The SAD algorithm is a well-known algo-rithm for matching cost computation The SAD algorithm isable to function in real time implementation because of itslow computational complexity This was proven by Tippettset al [32] who calculated and evaluated SAD performancesfor real time human pose images in a resource limited systemLee and Sharma [33] implemented real time disparity mapalgorithm estimations using the sliding window techniqueto calculate matching costs using the SAD algorithm Theiralgorithm uses parallel processing via a graphical processing
unit (GPU) By virtue of applying this new technique at thematching cost stage the accuracy of stereo vision process-ing can be increased while simultaneously improving thespeed
Gupta and Cho [34] implemented a new techniqueusing two different sizes of correlation windows in the SADalgorithm At the first level the initial cost aggregation isdetermined and at the second level the object boundariesare improved using a smaller window size compared withthe first-level implementationThe results produced aremoreaccurate in terms of pixel matching but still require theimplementation of multiple loops at different window sizesThe SAD algorithm is fast but the quality of the initialdisparity map that is produced is low because of the noise atobject boundaries and in textureless regions
315 Sum of Squared Differences (SSD) Equation (4) pre-sents the SSD algorithm in which the summation is per-formed over the squared differences in pixel intensity valuesbetween two corresponding pixels in the aggregated supportwindow 119908
SSD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (4)
An early implementation of the SSD algorithm for thematching cost calculation stage was achieved by Fusielloet al [36] They tested the SSD algorithm on multiple fixedwindow blocks to reduce the incidence of occlusion errorsThe purpose of using multiple blocks of windows is tosearch for the smallest error to select an appropriate pixelof interest in the disparity map Yang and Pollefeys [37]used technique similar to that presented in [36] but theiralgorithm was implemented on platform with a GPU Theyachieved good results in terms of speed compared withFusiellorsquos work Currently there is still relatively little researchon the use of the SSD algorithm in for stereo vision disparitymap algorithms compared with that on other matching costalgorithms This is evident from the previous review papers[8 12] on stereo vision disparity map algorithms and is alsoshown in Table 2
316 Normalized Cross Correlation (NCC) The NCC algo-rithm is another method of determining the correspondencebetween two windows around a pixel of interest The nor-malization within the window compensates for differences ingain and bias [38] Equation (5) specifies the formula for theNCC technique
NCC (119909 119910 119889)
=
sum(119909119910)isin119908119868119897 (119909 119910) sdot 119868119903 (119909 minus 119889 119910)
radicsum(119909119910)isin119908 1198682
119897(119909 119910) sdot sum
(119909119910)isin1199081198682119903(119909 minus 119889 119910)
(5)
However the NCC algorithm tends to blur regions of dis-continuity more than other matching cost algorithms [38]This is because any outliers lead to large errors in the NCCcalculations A new method for low-dimensional image fea-tures matching using NCC has been proposed by Satoh [39]
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 3
Table 1 Previous review papers on stereo vision disparity map algorithms
Year Author Focus
2002 Scharstein and Szeliski [11]
Proposed a taxonomy for vision algorithms and provided a quality metric to compare andevaluate multiple blocks of algorithms as shown in Figure 1 They have also provided a test bed formeasurable evaluation of stereo depth map algorithms The test bed or benchmarking datasetconsists of four images (Tsukuba Venus Teddy and Cones) which are available athttpwwwmiddleburyedustereo
2003 Brown et al [19] Reviewed advances in stereo vision disparity map algorithms regarding correspondence methodsand occlusion handling methods for real time implementations
2008 Tombari et al [77] Presented a survey and compared the different methods of cost aggregation for stereocorrespondence through accuracy and computational requirements
2008 Lazaros et al [12]Reviewed developments in stereo vision algorithms implemented via software and hardwarecategorized in terms of their major attributes The comparison of local and global methodsprovided by previously developed algorithms implemented on software and hardware basedplatforms was presented in this work
2011 Tombari et al [101] Contributed an evaluation of stereo vision depth map algorithms in terms of their 3D objectrecognition ability
2013 Tippetts et al [8]Reviewed stereo vision algorithms and their suitability for resource-limited systems They havecompiled and presented an accuracy and runtime performance data for all stereo vision disparitymap algorithms in the past decade with an emphasis on real time performance
Inputimages
Step 1
Matchingcost
computation
Step 2
Costaggregation
Step 3
Disparityselection
Step 4
Disparityrefinement
Disparitymap
Figure 2 A framework for the development of a stereo vision algorithm
3 A Taxonomy for the Processing Stages ofStereo Vision Disparity Map Algorithms
Most stereo vision disparitymap algorithms have been imple-mented using multistage techniques These techniques ascodified by Scharstein and Szeliski consist of four main stepsas shown in Figure 2 [11] In this figure the input images areobtained from stereo vision sensors (ie from at least twocameras) Commonly these cameras are arranged horizon-tally and set up which produce two or more correspondingimages For the explanation or the process as described byadopted taxonomy these input images are assumed to berectified images Next the image pair to be analysed will passthrough all of the blocks in sequence beginning with Step1 and ending with Step 4 The output of this process shouldbe a smooth disparity map In essence each block representsone ormore algorithms whose performance can bemeasuredbased on the expected outputThis taxonomyhas beenwidelyused by many current developers of stereo vision disparitymap algorithms [8 12]
In general stereo vision disparity map algorithms can beclassified into local or global approaches A local approachis also known as area based or window based approachThis is because the disparity computation at a given point(or pixel) depends only on the intensity values within apredefined support window Thus such method considersonly local information and therefore has a low computationalcomplexity and a short run time Local methods include all
four steps of the taxonomy Examples of implementation ofsuch methods are provided by the work of Mattoccia et al[13] Arranz et al [14] and Xu et al [15] The disparity mapvalue assignment is achieved through winner take all (WTA)optimization For each pixel the corresponding disparityvalue with the minimum cost is assigned to that pixel Thematching cost is aggregated via a sum or an average over thesupport window
By contrast a global method treats disparity assignmentas a problem of minimizing a global energy function forall disparity values Such a method is formulated as anenergy minimization process with two terms in the objectivefunction (ie a data term which penalizes solutions thatare inconsistent with the target data and a smoothness termwhich enforces the piecewise smoothing assumption withneighboring pixels) The smoothness term is designed toretain smoothness in disparity among pixels in the sameregion The disparity map is produced by assigning similardepth values to neighboring pixels Global methods producegood results but are computationally expensive Thereforethey are impractical for use in real time systems Globalmethods typically skip Step 2 of the taxonomy depicted inFigure 2 (ie they do not perform cost aggregation andtherefore contain only three steps) [16ndash18] Markov randomfield (MRF) modelling is the approach that is most commonapproach used in globalmethodsThis type ofmodelling usesan iterative framework to ensure smooth disparity maps andhigh similarity between matching pixels
4 Journal of Sensors
P
Il Ir
pl pre e
OlOr
Figure 3 Epipolar geometry The 3D image of target scene at point119875
31 Matching Cost Computation All stereo matching algo-rithms require a cost criterion tomeasure the extent ofmatch-ing between two pixelsThematching cost computation is thestage in which whether the values of two pixels correspond tothe same point in a scene is determinedTherefore the stereomatching cost computation can be defined as a method ofdetermining the parallax values of each point between the leftand right images [19] The matching cost is computed at eachpixel for all pixels under considerationThe difference in pixelintensity between a pair of the matching pixels in two imagesis called the disparity and can be associated with depth valuesthrough three-dimensional (3D) projection
The matching points must lie on epipolar lines 119890 asshown in Figure 3 This matching can be performed via aone-dimensional horizontal search if the stereo pairs areaccurately calibrated [20] A target point 119875 is viewed fromthe optical centers 119874119897 and 119874119903 of the two cameras It producesone left image plane 119868119897 and one corresponding right imageplane 119868119903 one from each of the two cameras The points 119901119897and 119901119903 represent the matching pixel intensities in the left andright image planes respectively for the same scene at point119875Therefore the targeted left and right matching points mustbe located at the same coordinates on the horizontal line 119890In a stereo vision system this requirement is imposed in theform of an epipolar constraint [21] This epipolar constraintplays a significant role in stereo matchingThis is because thesearch for correspondences can be limited to a line insteadof the entire image space thereby reducing the required timeand search range Early stereo vision disparity map algorithmfor the matching cost computation task uses pixel basedtechnique [12] These algorithms are the methods of absolutedifferences (AD) squared differences (SD) adaptations ofthe former include the methods of sampling-insensitiveabsolute differences and truncated absolute differencesThesealgorithms can be applied to grayscale or color images
Area based or window based techniques are capableof offering richer data than matching techniques basedon individual pixels or features Such techniques can be
more accurate because the matching process considers theentire set of pixels associated with image regions Commonalgorithms for window based techniques include the sum ofabsolute differences (SAD) the sum of squared differences(SSD) normalized cross correlation (NCC) rank transforms(RT) and census transforms (CT) [12] The matching cost iscalculated over a support region This support region whichis commonly referred to as support or aggregating windowmay be square or rectangular and may be fixed or adaptivein sizeThemajor shortcoming of window based technique isthat these approaches commonly assume that all pixels withina support window have similar disparity values This is notnecessarily true for pixels near depth discontinuities or edgesHence an improper selection of the size and shape of thematching window can lead to poor depth estimations
311 Absolute Differences (AD) The AD algorithm aggre-gates the differences in luminance (or intensity values)between the pixel in the left image 119868119897 and the correspondingpixels in the right image 119868119903 as given by
AD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (1)
In this equation (119909 119910 119889) represents the disparitymap coordi-nates where (119909 119910) are the coordinates of the pixel of interestand 119889 is the disparity (or depth) value Typically in thematching process 119868119897 is used as the reference image and theright image 119868119903 represents the target (or candidate) imageThe AD algorithm is the simplest among matching cost algo-rithms Because of its low complexity Wang et al [22] usedthis algorithm for real time stereo matching using graphicshardware (GPU) The AD algorithm functions satisfactorilyin regions with little texture but for highly textured imagesthis algorithm is not capable of producing a smooth disparitymap To overcome this difficulty the truncated version ofthe AD algorithm was developed The truncated absolutedifference (TAD) algorithm as implemented by Min et al[23] and Pham and Jeon [24] is able to minimize the errorsin disparity maps Furthermore the TAD algorithm usesthe colors and gradients at matching pixels to improve itsrobustness against variations in illumination
312 Squared Differences (SD) The SD algorithm aggregatesthe squared differences between the reference pixels in 119868119897 andthe candidate pixels in 119868119903 as described in
SD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (2)
Yang et al [25] implemented the SD algorithm for theirmatching cost computation in a subpixel estimation methodfor disparity mapping Considerable noise was generated atthe boundaries in their initial disparity maps However Yanget al applied a bilateral filter (ie a type of edge preservingfilter) to improve the flattening of edges and to smooth areasnear depth discontinuities Recently Miron et al [26] testedvarious matching cost functions in their stereo disparitymap algorithms for intelligent vehicle applications Theyconcluded that the SD algorithm produced the largest errorThe errors that occur due to the SD algorithm are highlysensitive to brightness and noise especially in real timeenvironment
Journal of Sensors 5
313 Feature Based Techniques The matching cost func-tion can also be constructed using feature based tech-niques This approach attempts to establish correspondencesonly for similar feature points that can be unambiguouslymatched Common methods of feature extraction includethose based on visual features (eg edges shapes texturessegmentation and gradient peaks) statistical characteristics(eg minima medians and histograms) and transformationfeatures (eg Hough transforms wavelet transforms andGabor transforms) [27] As an example Sharma et al [28]developed a new disparity map algorithm using featuresderived from the scale invariant feature transform (SIFT)for autonomous vehicle navigation In their implementationthey modified the SIFT algorithm to use self-organizing mapto achieve more efficient performance in feature matchingprocess Their results indicate that the computation time oftheir method is reduced compared with the conventionalSIFT algorithm Because only feature points are correlatedalthough the computational cost is significantly reducedthe complete disparity map cannot be obtained The featurematching accuracy is still low Sparse disparity maps areproduced because only matching points derived from thetargeted object features are used [29] Furthermore featurebased matching techniques exhibit low accuracy and arerather insensitive to occlusion and textureless areas Liuet al [30] used a combination of image segmentation andedge detection for the matching cost function The timeimplementation was fast but the accuracy level also remainedlow in regions of discontinuity Ekstrand et al [31] used asegmentation process that created one-dimensional segmentscontaining information on color and edge coordinates Asegment correspondence estimation was performed for everyrow of the image to reduce the possibility of mismatchedsegments However errors still occurred as a result of usinglimited vertical support segments during the matching pro-cess Thus feature based techniques are not preferred andtheir usage by researchers for the development of disparitymap algorithms remains low
314 Sum of Absolute Differences (SAD) TheSADalgorithmdefined in (3) considers the absolute difference between theintensity of each pixel in the reference block and that of thecorresponding pixel in the target block
SAD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (3)
Thedifferences are summed over the aggregated support win-dow 119908 to generate a simple metric of block similarity knownas a disparity map The SAD algorithm is a well-known algo-rithm for matching cost computation The SAD algorithm isable to function in real time implementation because of itslow computational complexity This was proven by Tippettset al [32] who calculated and evaluated SAD performancesfor real time human pose images in a resource limited systemLee and Sharma [33] implemented real time disparity mapalgorithm estimations using the sliding window techniqueto calculate matching costs using the SAD algorithm Theiralgorithm uses parallel processing via a graphical processing
unit (GPU) By virtue of applying this new technique at thematching cost stage the accuracy of stereo vision process-ing can be increased while simultaneously improving thespeed
Gupta and Cho [34] implemented a new techniqueusing two different sizes of correlation windows in the SADalgorithm At the first level the initial cost aggregation isdetermined and at the second level the object boundariesare improved using a smaller window size compared withthe first-level implementationThe results produced aremoreaccurate in terms of pixel matching but still require theimplementation of multiple loops at different window sizesThe SAD algorithm is fast but the quality of the initialdisparity map that is produced is low because of the noise atobject boundaries and in textureless regions
315 Sum of Squared Differences (SSD) Equation (4) pre-sents the SSD algorithm in which the summation is per-formed over the squared differences in pixel intensity valuesbetween two corresponding pixels in the aggregated supportwindow 119908
SSD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (4)
An early implementation of the SSD algorithm for thematching cost calculation stage was achieved by Fusielloet al [36] They tested the SSD algorithm on multiple fixedwindow blocks to reduce the incidence of occlusion errorsThe purpose of using multiple blocks of windows is tosearch for the smallest error to select an appropriate pixelof interest in the disparity map Yang and Pollefeys [37]used technique similar to that presented in [36] but theiralgorithm was implemented on platform with a GPU Theyachieved good results in terms of speed compared withFusiellorsquos work Currently there is still relatively little researchon the use of the SSD algorithm in for stereo vision disparitymap algorithms compared with that on other matching costalgorithms This is evident from the previous review papers[8 12] on stereo vision disparity map algorithms and is alsoshown in Table 2
316 Normalized Cross Correlation (NCC) The NCC algo-rithm is another method of determining the correspondencebetween two windows around a pixel of interest The nor-malization within the window compensates for differences ingain and bias [38] Equation (5) specifies the formula for theNCC technique
NCC (119909 119910 119889)
=
sum(119909119910)isin119908119868119897 (119909 119910) sdot 119868119903 (119909 minus 119889 119910)
radicsum(119909119910)isin119908 1198682
119897(119909 119910) sdot sum
(119909119910)isin1199081198682119903(119909 minus 119889 119910)
(5)
However the NCC algorithm tends to blur regions of dis-continuity more than other matching cost algorithms [38]This is because any outliers lead to large errors in the NCCcalculations A new method for low-dimensional image fea-tures matching using NCC has been proposed by Satoh [39]
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
4 Journal of Sensors
P
Il Ir
pl pre e
OlOr
Figure 3 Epipolar geometry The 3D image of target scene at point119875
31 Matching Cost Computation All stereo matching algo-rithms require a cost criterion tomeasure the extent ofmatch-ing between two pixelsThematching cost computation is thestage in which whether the values of two pixels correspond tothe same point in a scene is determinedTherefore the stereomatching cost computation can be defined as a method ofdetermining the parallax values of each point between the leftand right images [19] The matching cost is computed at eachpixel for all pixels under considerationThe difference in pixelintensity between a pair of the matching pixels in two imagesis called the disparity and can be associated with depth valuesthrough three-dimensional (3D) projection
The matching points must lie on epipolar lines 119890 asshown in Figure 3 This matching can be performed via aone-dimensional horizontal search if the stereo pairs areaccurately calibrated [20] A target point 119875 is viewed fromthe optical centers 119874119897 and 119874119903 of the two cameras It producesone left image plane 119868119897 and one corresponding right imageplane 119868119903 one from each of the two cameras The points 119901119897and 119901119903 represent the matching pixel intensities in the left andright image planes respectively for the same scene at point119875Therefore the targeted left and right matching points mustbe located at the same coordinates on the horizontal line 119890In a stereo vision system this requirement is imposed in theform of an epipolar constraint [21] This epipolar constraintplays a significant role in stereo matchingThis is because thesearch for correspondences can be limited to a line insteadof the entire image space thereby reducing the required timeand search range Early stereo vision disparity map algorithmfor the matching cost computation task uses pixel basedtechnique [12] These algorithms are the methods of absolutedifferences (AD) squared differences (SD) adaptations ofthe former include the methods of sampling-insensitiveabsolute differences and truncated absolute differencesThesealgorithms can be applied to grayscale or color images
Area based or window based techniques are capableof offering richer data than matching techniques basedon individual pixels or features Such techniques can be
more accurate because the matching process considers theentire set of pixels associated with image regions Commonalgorithms for window based techniques include the sum ofabsolute differences (SAD) the sum of squared differences(SSD) normalized cross correlation (NCC) rank transforms(RT) and census transforms (CT) [12] The matching cost iscalculated over a support region This support region whichis commonly referred to as support or aggregating windowmay be square or rectangular and may be fixed or adaptivein sizeThemajor shortcoming of window based technique isthat these approaches commonly assume that all pixels withina support window have similar disparity values This is notnecessarily true for pixels near depth discontinuities or edgesHence an improper selection of the size and shape of thematching window can lead to poor depth estimations
311 Absolute Differences (AD) The AD algorithm aggre-gates the differences in luminance (or intensity values)between the pixel in the left image 119868119897 and the correspondingpixels in the right image 119868119903 as given by
AD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (1)
In this equation (119909 119910 119889) represents the disparitymap coordi-nates where (119909 119910) are the coordinates of the pixel of interestand 119889 is the disparity (or depth) value Typically in thematching process 119868119897 is used as the reference image and theright image 119868119903 represents the target (or candidate) imageThe AD algorithm is the simplest among matching cost algo-rithms Because of its low complexity Wang et al [22] usedthis algorithm for real time stereo matching using graphicshardware (GPU) The AD algorithm functions satisfactorilyin regions with little texture but for highly textured imagesthis algorithm is not capable of producing a smooth disparitymap To overcome this difficulty the truncated version ofthe AD algorithm was developed The truncated absolutedifference (TAD) algorithm as implemented by Min et al[23] and Pham and Jeon [24] is able to minimize the errorsin disparity maps Furthermore the TAD algorithm usesthe colors and gradients at matching pixels to improve itsrobustness against variations in illumination
312 Squared Differences (SD) The SD algorithm aggregatesthe squared differences between the reference pixels in 119868119897 andthe candidate pixels in 119868119903 as described in
SD (119909 119910 119889) = 1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (2)
Yang et al [25] implemented the SD algorithm for theirmatching cost computation in a subpixel estimation methodfor disparity mapping Considerable noise was generated atthe boundaries in their initial disparity maps However Yanget al applied a bilateral filter (ie a type of edge preservingfilter) to improve the flattening of edges and to smooth areasnear depth discontinuities Recently Miron et al [26] testedvarious matching cost functions in their stereo disparitymap algorithms for intelligent vehicle applications Theyconcluded that the SD algorithm produced the largest errorThe errors that occur due to the SD algorithm are highlysensitive to brightness and noise especially in real timeenvironment
Journal of Sensors 5
313 Feature Based Techniques The matching cost func-tion can also be constructed using feature based tech-niques This approach attempts to establish correspondencesonly for similar feature points that can be unambiguouslymatched Common methods of feature extraction includethose based on visual features (eg edges shapes texturessegmentation and gradient peaks) statistical characteristics(eg minima medians and histograms) and transformationfeatures (eg Hough transforms wavelet transforms andGabor transforms) [27] As an example Sharma et al [28]developed a new disparity map algorithm using featuresderived from the scale invariant feature transform (SIFT)for autonomous vehicle navigation In their implementationthey modified the SIFT algorithm to use self-organizing mapto achieve more efficient performance in feature matchingprocess Their results indicate that the computation time oftheir method is reduced compared with the conventionalSIFT algorithm Because only feature points are correlatedalthough the computational cost is significantly reducedthe complete disparity map cannot be obtained The featurematching accuracy is still low Sparse disparity maps areproduced because only matching points derived from thetargeted object features are used [29] Furthermore featurebased matching techniques exhibit low accuracy and arerather insensitive to occlusion and textureless areas Liuet al [30] used a combination of image segmentation andedge detection for the matching cost function The timeimplementation was fast but the accuracy level also remainedlow in regions of discontinuity Ekstrand et al [31] used asegmentation process that created one-dimensional segmentscontaining information on color and edge coordinates Asegment correspondence estimation was performed for everyrow of the image to reduce the possibility of mismatchedsegments However errors still occurred as a result of usinglimited vertical support segments during the matching pro-cess Thus feature based techniques are not preferred andtheir usage by researchers for the development of disparitymap algorithms remains low
314 Sum of Absolute Differences (SAD) TheSADalgorithmdefined in (3) considers the absolute difference between theintensity of each pixel in the reference block and that of thecorresponding pixel in the target block
SAD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (3)
Thedifferences are summed over the aggregated support win-dow 119908 to generate a simple metric of block similarity knownas a disparity map The SAD algorithm is a well-known algo-rithm for matching cost computation The SAD algorithm isable to function in real time implementation because of itslow computational complexity This was proven by Tippettset al [32] who calculated and evaluated SAD performancesfor real time human pose images in a resource limited systemLee and Sharma [33] implemented real time disparity mapalgorithm estimations using the sliding window techniqueto calculate matching costs using the SAD algorithm Theiralgorithm uses parallel processing via a graphical processing
unit (GPU) By virtue of applying this new technique at thematching cost stage the accuracy of stereo vision process-ing can be increased while simultaneously improving thespeed
Gupta and Cho [34] implemented a new techniqueusing two different sizes of correlation windows in the SADalgorithm At the first level the initial cost aggregation isdetermined and at the second level the object boundariesare improved using a smaller window size compared withthe first-level implementationThe results produced aremoreaccurate in terms of pixel matching but still require theimplementation of multiple loops at different window sizesThe SAD algorithm is fast but the quality of the initialdisparity map that is produced is low because of the noise atobject boundaries and in textureless regions
315 Sum of Squared Differences (SSD) Equation (4) pre-sents the SSD algorithm in which the summation is per-formed over the squared differences in pixel intensity valuesbetween two corresponding pixels in the aggregated supportwindow 119908
SSD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (4)
An early implementation of the SSD algorithm for thematching cost calculation stage was achieved by Fusielloet al [36] They tested the SSD algorithm on multiple fixedwindow blocks to reduce the incidence of occlusion errorsThe purpose of using multiple blocks of windows is tosearch for the smallest error to select an appropriate pixelof interest in the disparity map Yang and Pollefeys [37]used technique similar to that presented in [36] but theiralgorithm was implemented on platform with a GPU Theyachieved good results in terms of speed compared withFusiellorsquos work Currently there is still relatively little researchon the use of the SSD algorithm in for stereo vision disparitymap algorithms compared with that on other matching costalgorithms This is evident from the previous review papers[8 12] on stereo vision disparity map algorithms and is alsoshown in Table 2
316 Normalized Cross Correlation (NCC) The NCC algo-rithm is another method of determining the correspondencebetween two windows around a pixel of interest The nor-malization within the window compensates for differences ingain and bias [38] Equation (5) specifies the formula for theNCC technique
NCC (119909 119910 119889)
=
sum(119909119910)isin119908119868119897 (119909 119910) sdot 119868119903 (119909 minus 119889 119910)
radicsum(119909119910)isin119908 1198682
119897(119909 119910) sdot sum
(119909119910)isin1199081198682119903(119909 minus 119889 119910)
(5)
However the NCC algorithm tends to blur regions of dis-continuity more than other matching cost algorithms [38]This is because any outliers lead to large errors in the NCCcalculations A new method for low-dimensional image fea-tures matching using NCC has been proposed by Satoh [39]
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 5
313 Feature Based Techniques The matching cost func-tion can also be constructed using feature based tech-niques This approach attempts to establish correspondencesonly for similar feature points that can be unambiguouslymatched Common methods of feature extraction includethose based on visual features (eg edges shapes texturessegmentation and gradient peaks) statistical characteristics(eg minima medians and histograms) and transformationfeatures (eg Hough transforms wavelet transforms andGabor transforms) [27] As an example Sharma et al [28]developed a new disparity map algorithm using featuresderived from the scale invariant feature transform (SIFT)for autonomous vehicle navigation In their implementationthey modified the SIFT algorithm to use self-organizing mapto achieve more efficient performance in feature matchingprocess Their results indicate that the computation time oftheir method is reduced compared with the conventionalSIFT algorithm Because only feature points are correlatedalthough the computational cost is significantly reducedthe complete disparity map cannot be obtained The featurematching accuracy is still low Sparse disparity maps areproduced because only matching points derived from thetargeted object features are used [29] Furthermore featurebased matching techniques exhibit low accuracy and arerather insensitive to occlusion and textureless areas Liuet al [30] used a combination of image segmentation andedge detection for the matching cost function The timeimplementation was fast but the accuracy level also remainedlow in regions of discontinuity Ekstrand et al [31] used asegmentation process that created one-dimensional segmentscontaining information on color and edge coordinates Asegment correspondence estimation was performed for everyrow of the image to reduce the possibility of mismatchedsegments However errors still occurred as a result of usinglimited vertical support segments during the matching pro-cess Thus feature based techniques are not preferred andtheir usage by researchers for the development of disparitymap algorithms remains low
314 Sum of Absolute Differences (SAD) TheSADalgorithmdefined in (3) considers the absolute difference between theintensity of each pixel in the reference block and that of thecorresponding pixel in the target block
SAD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816 (3)
Thedifferences are summed over the aggregated support win-dow 119908 to generate a simple metric of block similarity knownas a disparity map The SAD algorithm is a well-known algo-rithm for matching cost computation The SAD algorithm isable to function in real time implementation because of itslow computational complexity This was proven by Tippettset al [32] who calculated and evaluated SAD performancesfor real time human pose images in a resource limited systemLee and Sharma [33] implemented real time disparity mapalgorithm estimations using the sliding window techniqueto calculate matching costs using the SAD algorithm Theiralgorithm uses parallel processing via a graphical processing
unit (GPU) By virtue of applying this new technique at thematching cost stage the accuracy of stereo vision process-ing can be increased while simultaneously improving thespeed
Gupta and Cho [34] implemented a new techniqueusing two different sizes of correlation windows in the SADalgorithm At the first level the initial cost aggregation isdetermined and at the second level the object boundariesare improved using a smaller window size compared withthe first-level implementationThe results produced aremoreaccurate in terms of pixel matching but still require theimplementation of multiple loops at different window sizesThe SAD algorithm is fast but the quality of the initialdisparity map that is produced is low because of the noise atobject boundaries and in textureless regions
315 Sum of Squared Differences (SSD) Equation (4) pre-sents the SSD algorithm in which the summation is per-formed over the squared differences in pixel intensity valuesbetween two corresponding pixels in the aggregated supportwindow 119908
SSD (119909 119910 119889) = sum(119909119910)isin119908
1003816100381610038161003816119868119897 (119909 119910) minus 119868119903 (119909 minus 119889 119910)1003816100381610038161003816
2 (4)
An early implementation of the SSD algorithm for thematching cost calculation stage was achieved by Fusielloet al [36] They tested the SSD algorithm on multiple fixedwindow blocks to reduce the incidence of occlusion errorsThe purpose of using multiple blocks of windows is tosearch for the smallest error to select an appropriate pixelof interest in the disparity map Yang and Pollefeys [37]used technique similar to that presented in [36] but theiralgorithm was implemented on platform with a GPU Theyachieved good results in terms of speed compared withFusiellorsquos work Currently there is still relatively little researchon the use of the SSD algorithm in for stereo vision disparitymap algorithms compared with that on other matching costalgorithms This is evident from the previous review papers[8 12] on stereo vision disparity map algorithms and is alsoshown in Table 2
316 Normalized Cross Correlation (NCC) The NCC algo-rithm is another method of determining the correspondencebetween two windows around a pixel of interest The nor-malization within the window compensates for differences ingain and bias [38] Equation (5) specifies the formula for theNCC technique
NCC (119909 119910 119889)
=
sum(119909119910)isin119908119868119897 (119909 119910) sdot 119868119903 (119909 minus 119889 119910)
radicsum(119909119910)isin119908 1198682
119897(119909 119910) sdot sum
(119909119910)isin1199081198682119903(119909 minus 119889 119910)
(5)
However the NCC algorithm tends to blur regions of dis-continuity more than other matching cost algorithms [38]This is because any outliers lead to large errors in the NCCcalculations A new method for low-dimensional image fea-tures matching using NCC has been proposed by Satoh [39]
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
6 Journal of Sensors
Table2Summaryof
stereovisio
ndisparity
map
algorithm
simplem
entedon
different
compu
tatio
nalplatfo
rms
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2006
Wangetal[22]
Local
AD
MW
WTA
Medianfilter
Multip
lecostaggregation
CPUIntelP
4andGPU
ATI
800
2006
Yoon
andKw
eon[102]Lo
cal
TAD
ASW
WTA
Left-rig
htconsistency
check
Adaptiv
esup
portweight
mdash
2007
Ciglae
tal[70]
Local
Edge
based
Arbitraryshaped
basedAW
WTA
Ang
lesw
eep
Multiv
iewregion
-based
mdash
2008
Hirs
chmuller[71]
Global
Semiglobal
matchingpixel
based
mdashDP
Left-rig
htconsistency
check
+medianfilterin
gSemiglobalm
atching
CPUOpteron
22G
Hz
2009
Yang
etal[63]
Global
Color
weighted
correlation
mdashBP
Left-rig
htconsistency
check
Hierarchicalloo
pybelief
prop
agation
mdash
2009
Salm
enetal[16]
Global
SAD
mdashDP
Left-rig
htconsistency
check
Optim
ized
dynamic
programming
CPUwith
18GHz
2010
Mattocciaetal[13]
Local
TAD
Jointb
ilateral
filterin
gbased
ASW
WTA
Left-rig
htconsistency
check
Jointb
ilateralstereo
CPUIntelC
oreD
uo214GHz
2010
Nalpantidisand
Gasteratos[5983]
Local
AD
ASW
WTA
mdashLC
DM
mdash
2010
Kalarotand
Morris
[87]
Global
Pixelbased
mdashDP
mdashSymmetric
dynamic
programming
CPUIntelQ
uadCore
23G
Hza
ndGPU
GeForce
GTX
280FP
GAAltera
Stratix
III
2011
Leea
ndSharma[
33]
Local
SAD+RT
Multiresolution
basedMW
WTA
mdashRa
nktransfo
rm+
multiresolution
Person
alCom
putera
ndGPU
(detailn
a)
2011
Meietal[45]
Local
AD+CT
Crossb
ased
AWWTA
Cross-checking
ADcensus
+dynamiccross
CPUIntelC
ore2
Duo
220
GHza
ndGPU
GeForce
GTX
480
2011
Min
etal[23]
Local
TAD
Jointh
istogram-
MW
based
aggregation
WTA
Boxfilterin
gHistogram-based
aggregation
CPU(detailn
a)
2011
Pereza
ndSanchez
[17]
Global
Pixelbased
mdashBP
Left-rig
htconsistency
check
Mem
ory-effi
cientb
elief
prop
agation
mdash
2011
Sharmae
tal[28]
Local
Featureb
ased
mdashWTA
mdashSIFT
Autono
mou
svehicle
navigatio
n
2011
Psotae
tal[78]
Local
SAD
ASW
WTA
Medianfilter
Iterativ
erefinement
adaptiv
esup
portweight
mdash
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 7Ta
ble2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2011
Tipp
ettsetal[32]
Local
Gaussianfilter+
profi
leshape
MW
WTA
Medianfilter
Profi
leshapem
atching
CPUAMDPh
enom
II28G
Hz
2011
Bann
oandIkeuchi
[68]
Global
Pixelbased
mdashBP
Dire
cted
aniso
tropic
diffu
sion
3Ddirected
aniso
tropic
diffu
sion
mdash
2011
Zhao
andTaub
in[103]
Local
SAD
Multiresolution
basedAW
WTA
Cross-checking
Multiresolutionadaptiv
ewindo
ws
CPU(detailn
a)a
ndGPU
GeForce
GTX
280
2011
Nalpantidisetal[20]
Local
AD
Gaussianweighted
+cellu
lara
utom
ata
WTA
mdashEffi
cienth
ierarchical
Stratix
IIGXFP
GA
2012
Arranze
tal[14
]Global
Pixelbased
mdashMRF
based
optim
ization
mdashMultiresolutionenergy
minim
ization
CPUInteli734G
Hzw
ith7G
BRA
M
2012
Xiangetal[9]
Global
Color
weighted
pixelbased
mdashHierarchical
belief
prop
agation
Uniqu
eness
constraint
+sim
ilarity
constraint
Hierarchicalbelief
prop
agation
CPUIntel27GHza
ndGPU
Geforce
GTX
260
2012
Yang
[79]
Local
Edge
based
Minim
umspanning
tree
WTA
Guidedfilter
Minim
umspanning
tree
based
CPUInteli718
GHzw
ith4G
BRA
M2013
Chen
andLai[73]
Global
Pixelbased
mdashGC
mdashParallelG
CCP
UInteli7(detailn
a)
2013
Chen
andSu
[54]
Local
Areab
ased
Shapea
daptivelow
complexity
based
AWWTA
Cross-checking
Shapea
daptivelow
complexity
CPUInteli526G
Hz
2013
Wangetal[18]
Global
Pixelbased
mdashGC
mdashHierarchicalbilateral
disparity
structure
(HBD
S)CP
UIntelC
ore2
Duo
233
GHzw
ith2G
BRA
M
2014
Chengetal[40
]Lo
cal+
glob
alZN
CCMatch-edgeb
ased
MW
WTA
Globaledge
constraint
Globaledgec
onstr
aint
na
2013
ChoiandPark
[96]
Global
SAD
mdashGC
Trun
catedlin
ear
functio
nEffi
cientG
PU-based
GC
CPU(detailn
a)a
ndGPU
GeForce
GTX
580
2013
Wangetal[62]
Global
TAD
mdashGC
Medianfilterin
g+
interpolationand
depth
discon
tinuity
adjustment
Reliablep
ointsb
ased
GC
CPUIntelC
ore2
Duo
20G
Hzw
ith2G
BRA
M
2013
Michaeletal[66
]Lo
cal+
glob
alAD
mdashWTA
Left-rig
htconsistency
check
Realtim
esem
iglobal
CPUIntelX
eonandGPU
GeForce
GTX
480
2013
Ciglaa
ndAlatan
[46]
Local
SAD+CT
Successiv
eweighted
summationbased
ASW
WTA
Perm
eabilitybased
filterin
gSeparables
uccessive
weightedsummation
CPUInteli73G
Hzw
ith6G
BRA
Mand
GPU
GeForce
GTX
480
2013
Gup
taandCh
o[34]
Local
SAD
MW
WTA
Left-rig
htconsistency
check
Dou
blew
indo
wbased
CPU(detailn
a)
2013
Hosni
etal[58]
Local
TAD(color
+gradient)
ASW
WTA
Weightedbilateral
medianfilterin
gASW
+patchmatch
stereo
CPUIntelC
ore2
Quad
24G
HZandGPU
GeForce
GTX
480
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
8 Journal of Sensors
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2013
Zhangetal[57]
Local
TAD(color
+gradient)
ASW
WTA
mdashRe
altim
eGPU
CPUInteli7andGPU
GeForce
GT5
40M
2013
Kowalczuk
etal[104]
Local
SAD
Iterativ
eASW
+tempo
ralA
SWWTA
Medianfilterin
gIte
rativ
erefinementA
SWCP
UAMDPh
enom
IIX6
30G
Hza
ndGPU
GeForce
GTX
580
2013
Leee
tal[60]
Local
Three-mod
ecross
CTASW
WTA
Left-rig
htconsistency
check
ASW
+three-mod
ecross
census
mdash
2013
Linetal[105]
Local
TAD(color
+gradient)
Hierarchical
cluste
rbased
ASW
WTA
Left-rig
htconsistency
check
Hierarchicalcluste
ring
algorithm
CPUIntelC
ore2
Duo
299
GHza
ndGPU
GeForce
9500GT
2013
Zhangetal[47]
Local
SAD+ALD
Crossb
ased
AWWTA
Left-rig
htconsistency
check
SAD+arm-le
ngth
differences
(ALD
)
CPUIntelC
ore2
Duo
300
GHza
ndGPU
GeForce
GTX
570
2013
Pham
andJeon
[24]
Local
TAD
MW
WTA
Weightedmedian
filterin
gDom
aintransfo
rmation
CPUAMDAthlon
64X2
DualC
ore2
GHza
ndGPU
GeForce
GTX
460
2014
Vijayanagare
tal[65]
Local
Segm
entb
ased
MW
WTA
Multiresolutio
naniso
tropic
diffu
sion
Multiresolutionaniso
tropic
diffu
sion
CPUrunn
ingat32G
Hz
with
8GBRA
MandGPU
AMDFireProV4900
2013
Xuetal[1580]
Local
TAD
ASW
WTA
mdashAd
aptiv
eweightselectio
nmdash
2014
Wangetal[75]
Global
Pixelbased
mdashMRF
based
optim
ization
Jointb
ilateral
filterin
gMRF
basedop
timization
CPUrunn
ingat3G
Hz
with
3GBRA
M
2013
Mattoccia[99]
Local
Segm
entb
ased
ASW
WTA
Guidedfilter
Multip
lealgorithm
s(FW
ASW
)FP
GAXilin
xSpartan6
2013
Zhangetal[6]
Global
BT+AD
mdashBP
Color
imaged
epth
mattin
g+2D
polyno
mial
regressio
n
MultiscaleBP
CPUInteli7with
4GBRA
MandGPU
GeForce
GTX
295
2013
Fowerse
tal[91]
Local
SAD
MW
WTA
mdashSA
D
CPUIntelQ
uadCore
300
GHza
ndGPU
GeForce
GTX
295FP
GA
Altera
Stratix
IIIE
260
2013
Wangetal[94]
Local
AD+CT
AWWTA
mdashRe
altim
eHQ-FPG
AFP
GAAltera
Stratix
IV
2013
Mae
tal[4467]
Local
TAD
ASW
WTA
Weightedmedian
filterin
g+left-rig
htconsistency
check
Con
stant
timew
eighted
medianfilterin
gCP
UIntelX
eon283
GHz
andGPU
GeForce
GTX
580
2014
Miro
netal[26]
Local
CTCr
ossb
ased
AWWTA
Anisotro
piclocal
polyno
mial
approxim
ation
Cross-comparis
oncensus
mdash
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 9
Table2Con
tinued
Year
Author
Metho
dTaxono
mysta
geAlgorith
mCom
putatio
nalplatfo
rmStep1
Step2
Step3
Step4
2014
Yang
etal[49]
Local
TAD
AWWTA
Adaptiv
eguided
filter
Fastadaptiv
eguidedfilter
CPUIntelC
oreD
uo316GHzw
ith2G
BRA
MandGPU
NvidiaT
esla
C2050
2014
Colod
ro-C
onde
etal
[100]
Local
SAD+CT
MW
WTA
Medianfilterin
gSA
D+CT
Xilin
xFP
GA
2014
JinandMaruyam
a[92]
Local
CEN+AD
AWWTA
mdashCA
+FL
CXilin
xXC
6VLX
240T
2004
Park
etal[69]
Local+
glob
alEd
gebased
FWMRF
based
Boxfilterin
gSegm
entatio
nbased
mdash
2014
Motae
tal[95]
Global
Symmetry
mdashMRF
based
mdashSymStereo
CPUInteli73G
Hza
ndGeforce
GTX
680
2014
Tipp
ettsetal[106]
Local
Edge
MW
WTA
Verticalfilterin
gProfi
leshapem
atching
Xilin
xVirtex
4FX
60
2014
Xuetal[89]
Local
Pixelbased
FWWTA
mdashPy
ramidal
AMDOpteron
processor
6366HE
IntelX
eon
processorE
5-2620N
vidia
GeForce
GTX
770
2014
Wernere
tal[84]
Local
NCC
mdashWTA
Crossb
ilateralfilterRe
altim
efullH
DFP
GA
2015
Ekstr
andetal[31]
Local
Segm
entatio
nMW
WTA
Medianfilter
Realtim
esegmentatio
nXilin
xFP
GA-
Spartan
3A-1800A
2015
Matsuoetal[82]
Local
AD+edge
FWWTA
Weightedjoint
bilateralfi
lter
In-Loo
pFeedback
Matching
mdash
2015
Yaoetal[97]
Global
Feature-based
mdashMRF
mdashRe
altim
eadaptivem
eshing
CPUXe
onandNvidia
Tesla
K2
2015
Plou
mpise
tal[76]
Global
Feature+
particle
filter
mdashMRF
based
Histogram
+segm
entatio
nParticlefilter+
scattered
controllandm
arks
CPUIntelC
ore2
Duo
2015
Chen
etal[56]
Local
Medianfilter+
AD
ASW
WTA
mdashTrilateral
CPUInteli73G
Hzw
ith8G
RAM
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
10 Journal of Sensors
The NCC algorithm was chosen because of its robustness tointensity offsets and changes in contrastThe results achievedin Satohrsquos work exhibit high accuracy but considerablecomputational resources are required Additionally Chenget al [40] implemented their matching cost calculation usinga zero mean normalized cross correlation (ZNCC) in whichthe pixels at which edges are located are manipulated via amultiple-window strategy This method relies on a neutralnetwork model Furthermore the least-mean-square deltarule is used for training and for the determination of theproper window shape and size for each support regionThesetechniques offer improved accuracy but their computationalrequirements are still high
317 Rank Transform (RT) Thematching cost for RT is givenby (6) and is calculated based on the absolute differencebetween two ranks (ie Rankref from the reference image andRanktar from the target image)
RT (119909 119910 119889)
= sum
(119909119910)isin119908
1003816100381610038161003816Rankref (119909 119910) minus Ranktar (119909 minus 119889 119910)1003816100381610038161003816
(6)
In this equation Rankref and Ranktar are calculated as shownin
Rank (119909 119910) = sum(119894119895)(119909119910)
119871 (119894 119895) (7)
with 119871(119894 119895) as defined in
119871 (119894 119895) =
0 119868 (119894 119895) lt 119868 (119909 119910)
1 otherwise(8)
where (119894 119895) are the coordinates of a neighboring pixel and(119909 119910) are the coordinates of the pixel of interest Equation(8) computes the number of neighboring pixels 119868(119894 119895) thathave values larger than that of the central pixel 119868(119909 119910) Anew model of a disparity map algorithm that uses the RTalgorithm to achieve improved accuracy has been proposedby Gac et al [41] Through careful selection of the windowsizes a reliable initial disparity map is efficiently obtainedAt the time of its publication this method demonstrated thehighest correct initial matching rate of any matching costalgorithm in date The RT algorithm is typically effectivefor coping with brightness differences and image distortionsSometimes when the RT algorithm is used a matching pixelmay look extremely similar to a neighboring pixel leadingto matching ambiguity In [42] a new extension of the RTapproach was developed to reduce this matching ambiguityusing a Bayesian model This model considers not only thesimilarities between the left and right image pixels similaritiesbut also the level of ambiguity within each image inde-pendently The results of experiments on images exhibitingvariations in intensity and brightness differences as reportedby authors of that study indicate a reduction in matchingambiguities
318 Census Transform (CT) The CT algorithm translatesthe results of comparisons between a center pixel and itsneighboring pixels within a window into a bit string as shown
Census (119909 119910) = Bitstring(119894119895)isin119908(119868 (119894 119895) ge 119868 (119909 119910)) (9)
This algorithm is calculated using the Hamming distancesbetween the census bit strings of on the correspondingmatchcandidates as given by
CT (119909 119910 119889) = sum(119909119910)isin119908
Hamming (Censusref (119909 119910)
minus Censustar (119909 minus 119889 119910)) (10)
where Censusref represents the census bit string from thereference image andCensustar represents the census bit stringfrom the target image The CT algorithm is rather robustto the disparity discontinuities because of its good outliertolerance as described byHumenberger et al [43]This claimwas proven by performance comparisons between the CTalgorithm and the SAD algorithm The disparity maps pro-duced by the CT algorithm exhibited highermatching qualityat object borders than those produced by the SAD algorithmThe disadvantage of the CT algorithm is its tendency to pro-duce incorrect matches in regions with repetitive structuresThis shortcoming was mitigated by Ma et al [44] throughtheir modifications to the CT algorithm They implementedadditional bits to represent the differences between the pixelof interest and the neighborhood pixels According to theirresults the accuracy of the disparity map was improved andthe incorrect matching problem was alleviated by this mod-ification In addition the proposed algorithm demonstratedgreater robustness when applied to a noisy image comparedwith the conventional CT algorithm
Several researchers have also developed matching costmethods based on a combination of two algorithms A com-bination of the AD and CT algorithms as shown by Mei et al[45] successfully reduces the occurrence of errorsThe reasonfor combining these two methods is to compensate for theirrespective limitations The CT algorithm tends to produceincorrect matches in regions with repetitive local structureswhereas the AD algorithm does not performance well onlarge textureless regions Similarly a combination of the SADand CT algorithms will also lead to higher performance butwill incur an increase in computational complexity [46] TheSAD and CT cost measures are obtained individually and thefinal cost function is constructed as a linear combination ofboth cost measures based on a weighting factorThe accuracyimprovement achieved by Zhang et al [47] was accomplishedbymeans of a costmeasure combining the SAD approach andarm length differences (ALD) The use of ALD was inspiredby the similarity of the matching pixels support regions inthe vertical direction as a result of the pixels being locatedon the same horizontal line This combination is able toreduce errors in most regions especially those containingrepeated color and shapes Lee et al [48] combined the CTand gradient difference approaches to achieve a highermatching cost quality However according to themmatchingambiguities can occur in certain regions as a result of similaror repetitive texture patterns
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 11
(a) (b)
(d)
(c)
Target pixel
Subw
indow
Figure 4 Cost aggregation windows (a) 5times 5 pixel square window(b) adaptive window (c) windowwith adaptive support weights and(d) all six possible resulting shapes of adaptive windows
32 Cost Aggregation Cost aggregation is the most impor-tant stage for determining the general performance of astereo vision disparity map algorithm especially for localmethods The purpose of cost aggregation is to minimizematching uncertainties Cost aggregation is needed becausethe information obtained for a single pixel upon calculatingthe matching cost is not sufficient for precise matching Localmethods aggregate the matching cost by summing themover a support region [11] This support region is typicallydefined by a square window centered on the current pixel ofinterest as shown in Figure 4(a) The most straightforwardaggregation method is to apply a simple low-pass filter inthe square support window The fixed-size window (FW)technique (eg binomial or Gaussian uniform (box filters))suffers an increased error rate when the size of the supportwindow is increased over a certain threshold Moreover thismethod requires the parameters to be set to values suitablefor the particular input dataset Otherwise it tends to blurobject boundaries [49] To avoid fattening artifacts near depthdiscontinuities methods using shifting window or multiplewindows (MW) as well as methods using adaptive windows(AW) windows with adaptive sizes or adaptive supportweights (ASW) have been developed
In the MW technique multiple windows are selectedfrom among a number of candidates based on the supportwindows that produce smaller matching costs This methodwas implemented by Hirschmuller et al [50] and Veksler [51]in their previous studies of real time stereo vision disparitymap algorithms However their experimental results revealdifficulties in preserving dedicated pixel arrangements indisparity maps especially at object boundaries This occursbecause of the shape of the support windows This approachis imperfect for a small number of candidates To resolvethis problem the AW technique was developed to reduce theerrors in the disparity map caused by boundary problems Inthis method the support regions are constructed as approx-imations to the local image structures Figure 4(b) illustrates
the application of this method with five subwindows withdimensions of 3 times 3 These subwindows must be located nearthe target pixel as shown in Figure 4 The cost aggregationwith the minimum matching cost value for this pixel iscalculated For example the cost can be calculated as the sum-mation over the target pixel subwindow and any two otheradjacent subwindowsThe chosen shape of the validmatchingwindows for aggregation can therefore be any of the shapesshown in Figure 4(d) In practice the shape of the adaptivewindow is adaptively varied to reflect the local image contentsuch as corners and edges
TheAW technique was implemented by Lu et al [52] whoachieved high quality results both near depth discontinuitiesand in homogenous regions Lursquos work was improved uponby Zhang et al [53] through a modification to the conceptof adaptive support regions They developed support regionswith arbitrarily adaptive shapes and implemented the algo-rithmon aGPU for real time applicationsThe shapes of thesesupport regions are more flexible and are not restricted to berectangles These authors achieved high matching accuracywith real time implementation In this AW technique thealgorithm attempts to find support windows that fit the shapeor size of each region while preventing them from crossingobject boundaries Furthermore this technique is able toreduce computational costs as discussed by Chen and Su[54] These authors proposed a shape adaptive low com-plexity technique for eliminating computational redundancybetween stereo image pairs for pixelsmatchingThey groupedpixels with the same depth value to reduce the number ofcomputations
A comparative study of the use of different support regiontechniques in the cost aggregation stage was performed byFang et al [55] This study addressed the FW AW and ASWapproaches The authors concluded that the most advanta-geous technique for cost aggregation is the ASW approachIn this technique each pixel in the support region is assigneda support weight which depends on its intensity dissimilarityand spatial distance from the anchor pixel as shown inFigure 4(c) The target pixel which is located at the center isassigned different weight depending on distance as indicatedby the different tone of colors Generally for typical ASWtechniques (11) is used to aggregate the matching costs1198621015840(119901 119889) at pixel 119901 and disparity 119889
1198621015840(119901 119889) = sum
119902isin119908119901
119882(119901 119902) 119862 (119902 119889) (11)
where 119908119901 is a square support window centered on pixel 119901The window size is a user defined parameter The value ofthe119882(119901 119902) function represents the possibility that a pixel 119902will possess a disparity value similar to that of the windowrsquoscenter pixel 119901 119862(119902 119889) represents a target pixel 119902 with adisparity value 119889 Ideally119882(119901 119902) should return a value of ldquo1rdquoif pixels119901 and 119902have equal disparity values and ldquo0rdquo otherwiseChen et al [56] developed a trilateral filter based on theASW approach with using a bilateral filter They also added anew weighted term to increase the robustness against objectboundaries
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
12 Journal of Sensors
Essentially in ASW application a higher weight will beallocated to a pixel if its intensity is more similar to thatof the anchor pixel and if it is located at a smaller distancefrom the anchor pixel as implemented by Zhang et al [57]This method is able to produce a disparity map in which theobject boundaries are well preserved and the accuracy is veryhigh compared with the previous methods reported in theirliterature Hosni et al [58] presented an extensive evaluationof ASW regions They performed their test on a GPU toevaluate whether the speed and computational efficiencywere sufficient for real time responsesTheir evaluations indi-cated that the ASW approach produces outstanding resultsin terms of both computational efficiency and the qualityof the generated disparity maps Nalpantidis and Gasteratos[59] developed a new approach based on the ASW techniqueThey combined it with the quantified gestalt law to calculatea weighting factor In general a correlation weight reflectsthe proximity similarity and continuity between both inputimages (ie left and right images)
33 Disparity Computation and Optimization Generally astereo matching algorithm represents one of the two majoroptimizations approaches the local approach or the globalapproach In the local approach when the final disparities arecomputed the disparity for each pixel is essentially selectedusing a local winner takes all (WTA) strategy as define by
119889119901 = arg min119889isin119863
1198621015840(119901 119889) (12)
The disparity associated with the minimum aggregated cost119889119901 at each pixel is chosen 1198621015840(119901 119889) represents the aggregatecost obtained after the matching cost calculation and 119863denotes the set of all allowed discrete disparities The WTAstrategy is utilized in this stage in local algorithms such asthose implemented by Cigla and Alatan [46] Zhang et al[53] and Lee et al [60] According to their findings thedisparity maps obtained at this stage still contain errors inthe form of unmatched pixels or occluded regions Becausethe aggregation in local methods is performed throughsummation or averaging over support regions their accu-racy is sensitive to noise and unclear regions This occursbecause only local information from a small number of pixelssurrounding the pixel of interest is utilized to make eachdecision Therefore the accuracy of a local method at thisstage depends on the matching cost computation and costaggregation stages Subsequently in the disparity refinementstage the errors will be reduced using several filteringtechniques
By contrast in a global approach certain assumptionsare made about the depth of field of the scene which areusually expressed in an energyminimization frameworkThebulk of the effort in a global method is expended during thedisparity computation phase and the aggregation step is oftenskipped [11]Themost commonly adopted assumption is thatthe scene is locally smooth except for object boundaries andthus neighboring pixels should have very similar disparitiesThis constraint is referred to as a smoothness constraintin the stereo vision literature In the typical global stereo
vision formulation the objective is to find an optimal energydisparity assignment function 119889 = 119889(119909 119910) that minimizes
119864 (119889) = 119864data (119889) + 120573119864smooth (119889) (13)
where 119864data(119889) represents the matching costs at the coordi-nates (119909 119910) the smoothness energy 119864smooth(119889) encouragesneighboring pixels to have similar disparities based on theprevious stated assumptions and 120573 is a weighting factor
A global method such as the belief propagation (BP)approach requires large amounts of computational resourcesand memory for the storage of the image data and theexecution of the algorithm For the improvement to theBP technique achieved implemented by Liang et al [61]which was implemented on the GPU the time required forprocessing is still large compared with that required by thelocal method strategy Wang et al [62] implemented globalapproach using a graph cut (GC) algorithm to optimize theenergy functionTheir method selects disparity values with alower energy value Another well-known global technique isa dynamic programming (DP) DP is executed for each scanline (row) independently resulting in polynomial complexityThe assumption adopted DP is that of an ordering constraintbetween neighboring pixels of the same row Recently themultiresolution energy minimization framework introducedby Arranz et al [14] achieved real time performance whilemaintaining the resolution of producing disparity maps Theadvantage of this framework is the reduction in computa-tional complexity that is achieved through the multiresolu-tion technique However for images of higher resolution andwith many more different levels of disparity the frameworkis unable to perform at a real time frame rate (30 frames persecond)
34 Disparity Map Refinement The purpose of the disparityrefinement stage is to reduce noise and improve the disparitymaps Typically the refinement step consists of regularizationand occlusion filling or interpolation The regularizationprocess will reduce the overall noise through the filteringof inconsistent pixels and small variations among pixels ondisparitymapTheocclusion filling or interpolation process isresponsible for approximating the disparity values in areas inwhich the disparity is unclear Typically occluded regions arefilled with disparities similar to those of the background ortextureless areas Usually the occlusion regions are detectedby using left-right consistency check such as those imple-mented by Yang et al [63] andHeo et al [64] If the matchingalgorithm rejects disparities with low confidence then theinterpolation algorithm will estimate approximations to thecorrect disparities based on the local neighborhoodsThe dis-parity refinement step normally combines local informationfrom the local neighborhood near each measurement with aconfidence metric
Two classic and common techniques for local disparityrefinement are Gaussian convolution and the median filterIn Gaussian convolution disparities are estimated in com-bination with those of neighboring pixels in accordancewith weights defined by a Gaussian distribution The pri-mary purpose of this method is to reduce the noise in
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 13
CPU
Libraries(OpenCV OpenGL
OpenCL etc)
(a)
CPU
Libraries
GPU
(CUDA OpenCVOpenGL OpenCL etc)
(b)
CPU
FPGA
VHDLVerilog etc
(c)
Figure 5 Block diagrams of the implementation of stereo vision disparity map algorithms (a) Software-based one (eg only a computer ora laptop) (b) hardware-based onewith a GPU implementation (eg a computer or a laptop built with a GPU unit) and (c) hardware-basedone with FPGA implementation connected to a computer to burn-in the codes and become a standalone system
the disparity map but the Gaussian filter also reduces theamount of fine detail present in the final disparity mapA technique developed by Vijayanagar et al [65] uses theweights defined by aGaussian filter to improve disparitymapsby approximating missing disparity values based nearby highconfidence disparity pixels as a guide to prevent filteringacross object boundaries Meanwhile themedian filter is ableto remove small isolated mismatches in disparity by virtueof its edge preserving property and it is suitable for real timeimplementation because of its low computational complexityThis filter selects the median value within window of pixelsas the final result for the central pixel In a study by Michaelet al [66] a disparitymap refinement approach usingmedianfilteringwas developed for a real time stereo vision algorithmFurthermore the median filter was modified by Ma et al[67] using the constant time weighted technique Theirmodification achieves high accuracy in removing noise anderror while maintaining the edges in disparity maps
The diffusion technique performs a function similar tothat of Gaussian convolutionMoreover there exists an adap-tation of this approach called anisotropic diffusion UnlikeGaussian convolution which destroys edges and fine detailsanisotropic diffusion applies smoothing without crossing anyedges as implemented by Banno and Ikeuchi [68] in the dis-parity map refinement stage of their disparity map algorithmTheir approach was improved upon by Vijayanagar et al [65]yielding a method called multiresolution anisotropic diffu-sion In thismethod the disparitymap is downsampled usingthree different resolution factors At each resolution 35 itera-tions of the anisotropic diffusion process are performed Theresult of the proposed algorithm is free of occlusion errorsand the edges in the disparitymap have been refined Anotherapproach which was developed by Zhang et al [6] employsa two-step process to further refine the estimated disparitymapThe authors presented the results they achieved througha color image guided depth matting process in a frameworkbased on Bayesian matting and 2D polynomial regressionsmoothing techniques This technique was found to be usedto be effectively preserving the discontinuities at objectboundaries while achieving smoothing in flat regions
4 Software-Based Stereo VisionDisparity Map Algorithms
This section reviews several software-based implementationprocesses of global and local methods for the generation ofdisparity maps These algorithms were developed and testedusing only a CPU as the processing hardware as shown inFigure 5 A software-based implementation is designed to usethe CPU to interface with API software The API softwareprovides a set of libraries such as Open Computer Vision(OpenCV) Open Computing Language (OpenCL) librariesand Open Graphic Language (OpenGL) A previous sum-mary of software-based stereo vision disparitymap algorithmand their performances was presented by Brown et al [19]The discussion also addressed the corresponding methodsand occlusion handling techniques In essence the differ-ences in these algorithms lie in the cost aggregation stage andin the optimization of the building blocks which determinethe main characteristics of the developed algorithms Severalresearchers such as Park et al [69] and Cigla et al [70] havedeveloped new algorithms by taking advantages of local andglobal methods for handling occlusions object boundariesand untextured regions These techniques follow an iterativeprocess for allocating disparities that spread into certain seg-ments by applying pixel similarity constraints consideringoverlapping regions enforcing smoothness between similarlycolored neighboring segments and penalizing occlusionsSeveral other researchers have applied other global tech-niques with various modifications resulting in a semiglobalapproachThismethod involves dynamic programming opti-mization such as that implemented by Hirschmuller [71]and Salmen et al [16] The results reported by these authorsindicate low computational complexity
41 Global Approaches The best known approaches amongthe global methods are belief propagation (BP) and graphcut (GC) algorithms Perez and Sanchez [17] used the BPapproach to develop a real time high-definition algorithmthat outperformed classical BP by implementing two BP algo-rithms in their 3D telepresence systems The first instance of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
14 Journal of Sensors
BP performs a classification of the pixels into areas designatedas reliable containing occlusion errors and textureless toreduce the numbers of memory accesses required for thesethree groups of pixels The second BP process is used todecreasememory traffic by generating the final disparity mapwith a reduced number of iterations due to information fromprevious BP iterations The experimental results demon-strated improved performance The authors compared thisapproach with classical BP and observed a 90 improvementin efficiency Wang and Yang [72] implemented groundcontrol points (GCP) influenced by the MRF model Intheir method GCP-based regularization for the optimizationframework is performed using a Bayesian rule Meanwhilethe energy minimization technique for finding an optimalsolution to the inference problem is implemented using theGC approach Evaluations of this method demonstrated itseffectiveness at improving the disparity map reconstructionto regularize problems of incorrect stereo matching Theapproaches developed by Perez and Sanchez [17] and WangandYang [72] both produce accurate disparitymaps but theircomputational costs remain high However Wang et al [18]proposed a hierarchical bilateral disparity structure (HBDS)algorithm based on a GC technique to reduce computa-tional complexity and improve the accuracy of the generateddisparity maps These authors divide all disparity levelshierarchically into a series of bilateral disparity structuresto increase the fineness of the disparity map During therefinement stage any fattened regions are recalibrated basedon the disparity values of all nearby pixels The evaluationresults indicated good performance with reduced processingtime and improved disparity map accuracy
A new technique was proposed by Chen and Lai [73]based on augmenting paths and the adoption of a push-relabelling scheme The augmenting path algorithm func-tions by using multiple threads to calculate each block indi-vidually and to select another completed nearby block withwhich to merge The proposed method identifies the inde-pendent processing loops in theGC approach and isolates thecomputation of each loop Each image is sliced into smallerimage segments which will then be processed in parallelThe proposed method enables a remarkable decrease in exe-cution time by a factor of 47 compared with the originalGC approach but considerable computational programmingefforts are required Kolmogorov et al [74] developed fourdifferent smoothness terms (ie data smoothness occlusionand uniqueness) to improve the accuracy of their resultsTheobjectives of theirmethod are to reduce the errors in occludedareas and increase the efficiency during postprocessingWanget al [75] developed an algorithm using the MRF frameworkto eliminate holes and misaligned pixels Their work pro-duced high quality disparity maps but also required complexcomputational programming Ploumpis et al [76] developeda new stereo matching approach based on particle filters andscattered control landmarks The proposed method consistsof three steps First multiple disparity maps are used toacquire a set of features or landmarks and then segment theimages Afterward to estimate the best disparity values scanline particle filtering is applied In the last step a Markovchain model is employed to reduce the computational
redundancy of the particle filtering process Using thismethod high quality disparity maps can be produced
42 Local Approaches In local methods pixel correspon-dences are generated by measuring the correspondenceand similarity between image regions and very effectiveimplementation can be produced using this approach [77]The assignment of disparity values is achieved by applyingWTA strategy after calculating each candidate disparity valueindividually The matching cost function is aggregated viaa summation or an averaging over a support region Thedisparity value with the minimum cost for each pixel isassigned to that pixel An algorithm based on an efficient costaggregation strategy was proposed by Mattoccia et al [13]These authors used joint bilateral filtering and expanded thecalculation structures that allow for the efficient and accurategeneration of disparity maps The idea behind adoptingthe selected bilateral filtering approach in the developedalgorithm is to combine a geometric constraint (ie a spatialfilter) with color proximity constraint (ie a range filter)The performance of the proposed approach was tested fornoise and accuracy using the Middlebury dataset Anotherrefinement technique was proposed by Psota et al [78] whichdoes not use image segmentation or plane fitting Insteadthe algorithm performs iterative refinement of the resultsof adaptive weight stereo matching In each iteration ofdisparity refinement the algorithm uses the ASW approachto penalize disparity differences in local windows A total ofeight iterations on the Middlebury dataset were performedby Psota et al and the correspondence error percentage wasobserved to decrease from 146 to 083 A new techniquefor local cost aggregation for stereo matching was proposedby Yang [79] In this technique the matching cost values areaggregated adaptively based on a tree structure The nodesof this tree consist of all of the image pixels and the treecontains all edges between nearest neighboring pixels Aspanning tree can be computed by removing unwanted edgesEdge with high weights will be removed during spanningtree construction Then the minimum spanning tree MSTapproach is applied to obtain the sum of the minimumvalues of all spanning trees Yangrsquos method offers a lowcomputational complexity and high accuracy but has notbeen tested for use in real time implementation
Xu et al [80] proposed an algorithm that calculates theaggregation cost via the join optimization of both the leftand right matching costs The authors assign reasonableweighting coefficient and exclude occluded pixels whilepreserving sufficient support windows for accuratematchingThe result is the ability to reduce unwanted pixels in theforeground and increase accuracy in highly textured regionsFurthermore Lee et al [60] developed an algorithm basedon local approach with no iteration using three-mode crossCT with noise buffering to increase the robustness againstimage noise in textureless areas This technique also providestwo bits of cross CTwithin threemodes of implementation toincrease the reliability of the census measure Most disparitymethods encounter difficulties when confronted with fast-moving objects but Leersquos algorithm addresses this problemby using the concept of optical flow to support weight
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 15
Table 3 Previous evaluation papers on CPU FPGA and GPUYear Author Focus
2008 Gac et al [41]
Presented a performance evaluation of different target architectures which are FPGA (XilinxVirtex 2 Pro) GPU (Nvidia Geforce 8880) and CPU (Xeon dual core 3GHz) performance basedon back projection technique with several ways to speed up They have reduced the computationaltime for these three different architectures through memory parallelized architecture
2010 Kalarot and Morris[87]
Reviewed performance on FPGA (Altera Stratix III) and GPU (Nvidia Geforce GTX 280)performance based on strengths and limitations The same DP algorithm has been applied toFPGA and GPUThe evaluations have been made on internal clocks memory space and disparityrange of 128 and 256
2012 Pauwels et al [88]Reviewed performance on FPGA (Xilinx Virtex 4) and GPU (Nvidia Geforce GTX 7900)performance based on optical flow stereo vision and local image features including energyorientation and phase
2012 Russo et al [90]Presented performance comparison on FPGA (cyclone II) and GPU (Nvidia Geforce GTX 295)for image convolution processing The performance is measured through clock cycle ratio andexecution time
2013 Fowers et al [91]
Presented performance comparison on FPGA (Altera Stratix III) GPU (Nvidia Geforce GTX 295)and CPU (Xeon Quad Core W3520) for sliding window applications They have used SADalgorithm for all three architectures The performance comparison is based on energy efficiencyand time consuming for image processing Multiple windows sizes (4 times 4 9 times 9 16 times 16 25 times 25and 45 times 45) have been used to evaluate the best performance architecture
2014 Xu et al [89] Presented the speed performance of CPU (AMD Opteron processor 6366HE Intel Xeonprocessor E5-2620) and GPU (Nvidia GeForce GTX 770) on the pyramidal stereo algorithm
computations within a localized window Koo et al [81]used a gradient based matching technique to reduce theradiometric errors and improve the matching cost functionby using a Gaussian-based weighting functionThe referenceimage is divided into two difference images corresponding tolow and high frequencies Then the Difference of Gaussian(DoG) function is employed to reduce the errors that ariseduring the matching process The authors demonstrated areduction of errors on a sample set of images acquired inan outdoor environment Matsuo et al [82] used a localapproach based on the AD algorithm and the Sobel operatorin the matching cost calculation stage and box filtering withthe WTA optimization in the cost aggregation stage Theyused a weighted joint bilateral filter (JBF) in the refinementstage They produced accurate disparity maps using severaliterations and a fixedwindow size for the JBFNalpantidis andGasteratos [83] developed a new stereo matching algorithmthat employs the AD algorithm and performs aggregationby considering the gestalt laws of proximity similarity andcontinuity within a psychophysically based weight assign-ment framework Their proposed algorithm yielded accurateresults when applied to the Middlebury dataset
5 Real Time Stereo Vision Disparity MapAlgorithms Using Additional Hardware
The ability to implement stereo matching algorithms in realtime represents a new research area in the field of computervision The results of the online Middlebury benchmarkingsystem established by Scharstein and Szeliski [35] indi-cate that algorithms developed using a parallel processingapproach or additional hardware are able to deliver pro-cessing times among the best ones achieved on a standardbenchmarking dataset Real time stereo vision algorithms
are able to achieve rates of greater than 30 frames persecond in their disparity mapping output In this section thediscussion is limited to platforms that use FPGAs and GPUsfor real time implementation of stereo vision algorithmsFigure 5 shows a basic block diagram for such hardware-based implementation on FPGA or GPUThe FPGA consistsof a CPU a multi-inputoutput port and a large set ofconfigurable logic blocks (CLB) that can be configuredaccording to the developerrsquos preferred design The featuresof FPGA include versatility and the flexibility to operateeither as standalone systems or as coprocessors on expansioncards for computer FPGAs are most often programmedusing hardware description language (HDL) [84] and Baconet al [85] By contrast a GPU is a dedicated coprocessorwith a fixed architecture that enables the acceleration of therendering of 2D and 3D graphics by offloading the relatedprocesses from the CPU Recent GPU designs have evolvedfrom being dedicated graphics rendering processors to moregeneral parallel processors A GPU consists of multipleprocessors with more than hundred cores depending on themodel A GPU is able to operate in combination with CPUand open-source libraries such as the OpenCL OpenCV andComputeUnifiedDeviceArchitecture (CUDA) libraries [86]
51 Comparative Studies of FPGAs and GPUs Several papersrelated to performance evaluations of FPGAs GPUs andCPUs were reviewed for this literature survey as specified inTable 3 An early comparison among the FPGA GPU andCPU was conducted by Gac et al [41] The results indicatedthat the GPU achieved the highest absolute performancein terms of reconstruction time The authors applied theback projection technique in their global algorithm via 3Dtomography image reconstruction Kalarot and Morris [87]compared the performance of theDP algorithm implemented
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
16 Journal of Sensors
on the FPGA and GPU when applied to their own rectifiedimages for different disparity range Their results indicatedthat the FPGA offered faster processing than the GPU for adisparity range below 128 but that the FPGA was unable tohandle a disparity range of greater than 256 unlike the GPUThis finding can be attributed to the memory limitationsof standalone FPGA systems which prevent their use forprocessing large images Another structured evaluation andcomparison of the FPGAandGPUwas performedbyPauwelset al [88] in the context of a real time analysis of opticalflow local image features and stereo vision applications Theauthors applied their method to the Middlebury dataset Thecomparison was performed based on the hardware architec-ture speed data dependency accuracy and time required todesign the structure of the algorithms The presented resultsdemonstrated that the GPU implementation was superior inall respects and yieldedmore accurate and faster results whenimplemented as real time stereo vision systems comparedwith the FPGA implementation Xu et al [89] compared thespeed performances achieved using CPU and a GPU for theirpyramidal stereo algorithm Their results indicated that theGPU was 12x faster than the CPU
Russo et al [90] performed a performance comparisonof FPGA and a GPU for image convolution processing Theyreported that the GPU exhibited the better performance interms of execution time and number of clock cycles Thisfinding was attributed to the fact that the GPU tended to bebetter able to explore the extremely large amounts of datacontained within the high-resolution images Moreover thecharacteristic features of a GPU such as multiple pipelinesand high bandwidth assist in enhancing its performanceFowers et al [91] conducted a performance and energycomparison of a GPU FPGA and a CPU with a multicorearchitecture They used the SAD algorithm with the slidingwindow technique as the algorithm implemented for thecomparison It was found that the FPGA provided the bestenergy efficiency whereas the GPU delivered the best per-formance Jin andMaruyama [92] compared implementationof their algorithm on the FPGA and a GPU based on speedand accuracy They also proposed a method of improv-ing the circuit design for FPGAs to reduce the requiredmemory resources while maintaining accuracy However theresults reported in all of the reviewed papers are subjectto processing performance which depend on the availablehardware resources and the computational requirements ofthe considered task A recent study of FPGAs was conductedby Lentaris et al [93] for their ongoing projects SPARTANSEXTANT and COMPASS to improve the behaviour ofautonomous planetary exploration rovers The study focusedon the potential use of FPGAs for implementing a varietyof stereo correspondence feature extraction and visualodometry algorithms
52 Global Approaches Several global methods have beenimplemented on FPGAs and GPUs for the developmentof real time stereo vision disparity map algorithms Aglobal optimization algorithm for stereo matching based onimprovement to the BP approach implemented on a GPUwas presented by Xiang et al [9] Their technique involves
the integration of color-weighted correlations to improvehierarchical BP Occlusion problems are resolved by combin-ing a uniqueness constraint and a similarity constraint forthe detection of occluded regions The approach of Xiang etal outperforms other BP methods with regard to their realtime implementation on GPUs However its results in dis-continuous regions of the disparity maps are somewhat poorand it requires more complex computations The approachalso suffers an increased time delay when the algorithm isattempting to generate accurate results for such discontin-uous regions An improvement to the poor quality of thedisparity maps was achieved by Wang et al [94] They usedthe AD-CT algorithm in the matching cost calculation stagewith a semiglobal optimization framework on FPGA boardSemiglobal optimization involves optimizing the smoothnessof the disparity map along different directions separatelyThe designated directions are along lines traveling to theright bottom bottom right and bottom left This systemwas found to be able to adjust the image resolution anddegree of parallelism to achieve maximum efficiency Theresult was the ability to produce high quality disparity mapsfrom high-definition images In [95] a new cost functionwas developed for the matching of corresponding pixels Theauthors proposed a parallel approach to a variant of a globalmatching cost calculationmethod implemented on aGPU forsymmetric stereo images A bank of log-Gabor wavelets wasdeveloped for the analysis of such symmetric images in thespectral domain Using a GPU the authors achieved real timedisparity estimations for high-resolution images
Implementation of a new algorithmusing theGCmethodon a GPU was presented by Choi and Park [96] Theybuilt their algorithm to operate in three stages using thegraph construction method to accelerate the convergence ofthe GC calculation A reordering heuristic and initializationmethod were employed to further increase the executionspeed based on the proposed graph construction methodThen a repetitive block based push and relabel method wasused to increase the data transfer efficiency Finally theyused low-overhead global relabelling algorithm to increasethe GPU occupancy They achieved an improved executiontime compared with typical global methods at the cost ofconsiderable programming efforts Yao et al [97] treatedimage warping as an energy minimization problem Firstthey developed a sparse disparity map by means of stereomatching process Then the map was warped an energyminimization function with three independent terms (ie adisparity constraint a structure constraint and a temporalconstraint) They applied their method to test images withdifferent resolutions and evaluated the results that are basedon the execution times required for the GPU and CPUimplementation The GPU runtime was 24x faster than theCPUprocessing time satisfying the requirements of real timeoperation
53 Local Approach Current FPGA technology offers thou-sands of small logic blocks embedded in the connectionmatrix This allows arbitrary computation blocks to be con-structed frombasic computing blocks through parallel circuitconnections Detailed summary of information regarding
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 17
the advantages and disadvantages of real time implementa-tion of stereo vision algorithms on FPGAs has been pro-vided by Samarawickrama [98] Kalarot and Morris [87]implemented an algorithm on FPGA using a fast and simpleapproach by combining the distortion removal and alignmentcorrection tasks in a single step by means of lookup tablesHowever a problem was encountered in the case of images ofmore than 1 megapixel in size at which the FPGAwas unableto process because of its very limited onboard memory Thismemory limitation makes FPGAs unsuitable for the process-ing of high-definition images unless external memory is usedto support itMattoccia [99] performed a comparison of threedifferent algorithms namely a fixed window algorithm anASW algorithm and a semiglobal algorithm implementedon FPGAs Mattocciarsquos results demonstrated that the outputdisparity maps were fairly accurate for all tested algorithmsRecently Colodro-Conde et al [100] implemented an areabased stereo matching algorithm on FPGA board and testedits performance using the Middlebury dataset They devel-oped the algorithm to use the SAD approach for thematchingcost calculation and the median filter to refine the disparitymaps Their architecture design involved multiple buffers fortemporary memory storage In this design when the windowsize is increased the buffers also need to be increased forthe parallel processing of the allocated memory Howeverthe memory size and inherent frequency of FPGA limitits suitability for such tasks and applications especially forreal time applications Nalpantidis et al [20] used FPGAto prepare an efficient implementation of their hierarchicalmatching algorithm on uncalibrated stereo vision images Intheir approach two-dimensional correspondence search isperformed using a hierarchical technique Then the inter-mediate results are refined by three-dimensional cellularautomata (CA)The final disparity value is defined in terms ofthe distance between the matching positions This proposedalgorithm is able to process uncalibrated and nonrectifiedstereo images when implemented on the FPGA
Excessive time consumption is the main challenge fac-ing real time algorithm implementation because of theircomputational complexity The reasons that real time visionalgorithms are generally suitable for implementation onGPUs have been explained and discussed by Kim et al [107]A GPU unit is able to run the same instructions on multiplesets of data simultaneously Based on the functionalityMei et al [45] developed a stereo matching algorithm forimplementation on a GPU with good performance in termsof both accuracy and speed The matching cost value wasinitialized using the AD measure and CT The cost wasaggregated in dynamic cross based regions and updated ina multidirection scan line optimization Several researchershave developed algorithms based on domain transforma-tions This technique was previously initiated by Gastal andOliveira [108] who used a transformation technique thatenables the aggregation of 2D cost data using a sequence of 1Dfilters This technique was improved upon by Pham and Jeon[24] by means of dimensionality reduction technique Theadvantage of this technique is that it reduces the complexityof the computational requirements compared with a 2Dcost aggregation calculation A multiresolution anisotropic
diffusion approach based on a disparity refinement algorithmthat can be executed in a real time environment was proposedby Vijayanagar et al [65] This algorithm exploits the imagepyramid concept to gradually enhance the disparity mapat different levels of resolution and to align the objectboundaries in color imagesThis technique allows smoothingto be achieved without loss of edges making it a useful toolfor improving image segmentation
A novel local method for stereo matching using a GPUwas presented byKowalczuk et al [104]The algorithmbeginswith an approximation based on ASW aggregation and alow-complexity iterative disparity refinement technique Theprobabilistic framework combines the summation term intoa matching cost minimization via a series of approximationsand facilitates interactive processing to improve the accuracyof the disparity map The refinement algorithm operatesby calculating the estimated disparity value of each pixelduring the current iteration using nearby pixel disparitiesfrom previous iterationsThe implementation of this methodof cost aggregation and iterative disparity refinement wasperformed by Yoon and Kweon [102] Instead of searchingfor the matching window with the optimal size and shapeit is possible to aggregate costs after local smoothing withina corresponding window to reduce matching noise Usuallynoise can be effectively reduced by applying a linear filter suchas a Gaussian filter but the resulting disparity map alwaysexhibits edge fattening Therefore to address this problem ofmismatching pixels or noise around regions of discontinuitydisparity maps Lin et al [105] proposed a new algorithmbased on an edge preserving filter for the ASW methodcomputed using a hierarchical clustering algorithm Thisalgorithm used a novel cost aggregation block to computecorresponding response for all the corresponding pixels in aset of sampling points Tippetts et al [106] used an intensityprofile shape-matching algorithm implemented on an FPGAto achieve real time estimations for microscale unmannedvehicles (ie helicopter) The algorithm consists of threesteps In the first stage filtering is performed using aGaussiankernel Then the shapes of target objects are identified on arow-by-row basis In the last stage after the entire image hasbeen processed a vertical smoothing filter is applied to reducethe remaining noise The authors also presented designs forFPGA blocks for each stage of implementation
54 The Challenges of Implementing Stereo Vision Algorithmson GPUs and FPGAs Over the past decade developmentsin computing architectures have exhibited a clear trendtoward increased heterogeneity and parallelism with mostmainstream microprocessors now possessing multiple coresand robust system architectures [109] At the same time theincreasing number of accelerator options has considerablyincreased the complexity of application design because ofthe need to perform an extensive exploration of the availabledesign space when attempting to choose a suitable deviceAlthough GPUs with CUDA have come into common use asaccelerators because of their low cost ready availability andsimple programming model comparable to that of FPGAsEkstrand et al [110] Perez-Patricio et al [111] Stein [112]and Long et al [113] have all presented results that different
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
18 Journal of Sensors
Left image Right image Ground-truth image
Tsukuba
Venus
Teddy
Cones
Figure 6 Standard benchmarking images provided in the Middlebury Stereo Vision datasets in [35]
devices are better suited for different applications Thereforesufficient exploration of the different available devices fordifferent applications is critical to prevent researchers fromselecting unsuitable devices during the design phase Inthis survey for the summary of performance comparisonsbetween GPUs and FPGAs the results vary among differ-ent implementation and application domains None of theplatforms appears to be universally superior The preferreddesign depends on the specifications of the target platformHowever the use of GPUs and FPGAs can facilitate increased
speeds and reduced execution times The challenge for a newresearcher in this field is to determine how to develop analgorithm that is appropriate to a specific application and themost suitable platform
6 Accuracy Measurement
There are several academic research centers that providequalitative accuracy assessments of disparity maps throughonline submissions The datasets used for these assessments
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 19
PM-F
ores
tSe
gAgg
rLC
UA
DC
ensu
sPM
FSu
bPix
Sear
chSO
SSu
bPix
Dou
bleB
PRD
PPa
tchM
atch
CCRA
DA
RLA
MC-
DSM
Adap
tiveG
FH
istA
ggrS
lant
Mul
tiAgg
Tree
Filte
rRe
altim
eED
PLM
3CCr
ossL
MF
Hist
oAgg
r2
impr
NLC
AD
TAgg
r-P
Hyb
ridTr
eeSe
gmen
tTre
eRe
lativ
eGra
d
HEB
FBS
M
AlgorithmLocal
AllNonocc Disc
Cros
sTre
es+
SP
Segm
+vi
sib
TF_A
SW
000200400600800
1000
Bad
pixe
l (
)
Figure 7The performance of local algorithms in terms of bad pixelpercentages for the nonocc all and disc which are cited from [35]
can also be downloaded from the associated web pagesfor example the Middlebury Computer Vision pages [35]KITTI Vision Benchmark [114] and DIBRIS [115] Thesedatasets include both static and dynamic scenes Most of thearticles reviewed in this survey use the qualitative accuracymeasurements provided by [35] Thus this paper uses thesame resource to report the accuracy performance of a stereovision algorithm in terms of the percentage of bad pixelsAccording to Scharstein and Szeliski [11] the evaluation of theaccuracy level for each image is based on three attributes (iethe percentages of bad pixels among all pixels in nonoccludedregions (nonocc) all pixels detected as valid pixels (all) andpixels in regions near depth discontinuities and occludedregions (disc)) Four standard benchmarking images are usedin this evaluation these images are Tsukuba Venus Teddyand Cones and the original images and ground-truth imagesfor each are shown in Figure 6 Figures 7 and 8 show thenonocc all and disc results obtained on these images forapproximately 60 algorithms selected based on both local andglobal optimizations methods The bad pixel percentages ofthese algorithms are among the lowest values represented inthe database of [35]
Because this section discusses only accuracy measure-ments cited from [35] which are based on online submissionsthe implementation of the algorithms in Figures 7 and 8is not specified as either software-based or hardware-basedThere is some possibility that accuracy improvement can beachieved through implementation on additional hardwareas shown by Pauwels et al [88] However as reported byKalarot andMorris [87] the primary advantage of hardware-based (ie FPGA andGPU) implementation is that the speedor execution time can be tremendously improved comparedwith implementation using only a CPU Figure 9 showsthe average errors of local and global methods Here thealgorithms are represented by numbers which correspondto the algorithms represented at the same 119909-axis positionsin Figure 7 (ie for local methods) and 8 (ie for globalmethods) This figure shows only the accuracy performancesof existingmethods and is intended as a guidance or referencefor those who wish to develop their own algorithms
Global
AllNonocc Disc
LLR
CSM
TwoS
tep
IGSM
ARA
P
PMBP
TSG
O
GC+
LSL
SCoB
eP
MVS
egBP
GCP
MC
War
pMat
SegT
reeD
P
Dou
bleB
P
FastN
LGC
CVW
-RM
Plan
eFitB
P
Obj
ectS
tere
o
Art
istic
GC
JSO
SP+
GCP
Glo
balG
CP
Adap
tingB
P
Mul
tiRes
GC
Surfa
ceSt
ereo
Coo
pReg
ion
Real
timeH
D
Inte
riorP
tLP
Out
lierC
onf
Impr
oveS
ubPi
x
BP+
Dire
cted
Diff
Algorithm
000200400600800
100012001400
Bad
pixe
l (
)
Figure 8 The performance of global algorithms in terms of badpixel percentages for the nonocc all and disc which are cited from[35]
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29Algorithm
Total averageLocalGlobal
000100200300400500600700800
Bad
pixe
l (
)
Figure 9 The average errors in terms of bad pixel percentage forlocal and global methods
7 Conclusion
The stereo matching problem remains a challenge for com-puter vision researchers A literature survey of the latest stereovision disparity map algorithms is provided here and all citedalgorithms are categorized according to the processing stepswith which they are associated in the taxonomy of Scharsteinand Szeliski Becoming familiar with the state-of-the-artalgorithms for stereo vision disparity mapping is a timeconsuming task In this survey of the latest developmentsin the area of stereo matching algorithms the processingsteps composing such an algorithm and their software-basedas well as hardware-based implementation was thereforeperformed and presented to assist in this taskThe qualitativemeasurement of the accuracy of such algorithmswas also dis-cussed To assist the reader in navigating the numerous workspresented Table 2 is presented as a summary It specifies thesteps and computational platforms used in each approach asa reference for the development of new algorithms
Nomenclature
119868119897 Left image119868119903 Right image119889 Disparity value(119909 119910) Pixel coordinates
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
20 Journal of Sensors
119908 Support windowisin Element of(119894 119895) Neighboring pixel coordinates119908119901 Support window centered on
pixel 119901119901 Pixel of interest119889119901 Smallest pixel value119863 Set of disparity values120573 Weighting factor119864 Energy function119902 Neighboring pixels in the
support window
Conflict of Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This work was supported in part by the Universiti SainsMalaysia Research University Individual (RUI) with Accountno 1001PELECT814169 and the Universiti Teknikal Malay-sia Melaka
References
[1] R Szeliski Computer Vision Algorithms and ApplicationsSpringer 2010
[2] B H Bodkin Real-time mobile stereo vision [MS thesis] Uni-versity of Tennessee 2012
[3] M Hansard S Lee O Choi and R P Horaud Time-of-FlightCameras Principles Methods and Applications Springer 2012
[4] S Foix G Alenya and C Torras ldquoLock-in time-of-flight (ToF)cameras a surveyrdquo IEEE Sensors Journal vol 11 no 9 pp 1917ndash1926 2011
[5] M Y Kim S M Ayaz J Park and Y Roh ldquoAdaptive 3D sensingsystem based on variable magnification using stereo vision andstructured lightrdquo Optics and Lasers in Engineering vol 55 pp113ndash127 2014
[6] S Zhang C Wang and S C Chan ldquoA new high resolutiondepth map estimation system using stereo vision and depthsensing devicerdquo in Proceedings of the IEEE 9th InternationalColloquium on Signal Processing and Its Applications (CSPA rsquo13)pp 49ndash53 IEEE Kuala Lumpur Malaysia March 2013
[7] W Kazmi S Foix G Alenya and H J Andersen ldquoIndoor andoutdoor depth imaging of leaves with time-of-flight and stereovision sensors analysis and comparisonrdquo ISPRS Journal ofPhotogrammetry and Remote Sensing vol 88 pp 128ndash146 2014
[8] B Tippetts D J Lee K Lillywhite and J Archibald ldquoReviewof stereo vision algorithms and their suitability for resource-limited systemsrdquo Journal of Real-Time Image Processing pp 1ndash212013
[9] X Xiang M Zhang G Li Y He and Z Pan ldquoReal-time stereomatching based on fast belief propagationrdquoMachine Vision andApplications vol 23 no 6 pp 1219ndash1227 2012
[10] L Nalpantidis and A Gasteratos ldquoStereovision-based fuzzyobstacle avoidance methodrdquo International Journal of HumanoidRobotics vol 8 no 1 pp 169ndash183 2011
[11] D Scharstein and R Szeliski ldquoA taxonomy and evaluation ofdense two-frame stereo correspondence algorithmsrdquo Interna-tional Journal of ComputerVision vol 47 no 1ndash3 pp 7ndash42 2002
[12] N Lazaros G C Sirakoulis and A Gasteratos ldquoReview ofstereo vision algorithms from software to hardwarerdquo Interna-tional Journal of Optomechatronics vol 2 no 4 pp 435ndash4622008
[13] S Mattoccia S Giardino and A Gambini ldquoAccurate andefficient cost aggregation strategy for stereo correspondencebased on approximated joint bilateral filteringrdquo in Proceedingsof the Asian Conference on Computer Vision (ACCV rsquo10) vol 38pp 371ndash380 Xirsquoan China September 2010
[14] A Arranz A Sanchez and M Alvar ldquoMultiresolution energyminimisation framework for stereo matchingrdquo IET ComputerVision vol 6 no 5 pp 425ndash434 2012
[15] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 Beijing China May 2013
[16] J Salmen M Schlipsing J Edelbrunner S Hegemann and SLuke ldquoReal-time stereo vision making more out of dynamicprogrammingrdquo inComputer Analysis of Images and Patterns pp1096ndash1103 Springer Berlin Germany 2009
[17] J M Perez and P Sanchez ldquoReal-time stereo matching usingmemory-efficient Belief Propagation for high-definition 3Dtelepresence systemsrdquo Pattern Recognition Letters vol 32 no16 pp 2250ndash2253 2011
[18] Y-C Wang C-P Tung and P-C Chung ldquoEfficient disparityestimation using hierarchical bilateral disparity structure basedgraph cut algorithm with a foreground boundary refinementmechanismrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 23 no 5 pp 784ndash801 2013
[19] M Z Brown D Burschka and G D Hager ldquoAdvances incomputational stereordquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 25 no 8 pp 993ndash1008 2003
[20] L Nalpantidis A Amanatiadis G C Sirakoulis and A Gaster-atos ldquoEfficient hierarchical matching algorithm for processinguncalibrated stereo vision images and its hardware architec-turerdquo IET Image Processing vol 5 no 5 pp 481ndash492 2011
[21] T Song B Tang M Zhao and L Deng ldquoAn accurate 3-D firelocation method based on sub-pixel edge detection and non-parametric stereo matchingrdquo Measurement vol 50 no 1 pp160ndash171 2014
[22] L Wang M Gong M Gong and R Yang ldquoHow far can we gowith local optimization in real-time stereo matchingrdquo in Pro-ceedings of the 3rd International Symposium on 3DData Process-ing Visualization and Transmission pp 129ndash136 Chapel HillNC USA June 2006
[23] D Min J Lu and M N Do ldquoA revisit to cost aggregation instereo matching how far can we reduce its computationalredundancyrdquo in Proceedings of the IEEE International Confer-ence on Computer Vision (ICCV rsquo11) pp 1567ndash1574 BarcelonaSpain November 2011
[24] C C Pham and J W Jeon ldquoDomain transformation-based effi-cient cost aggregation for local stereo matchingrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 23 no 7pp 1119ndash1130 2013
[25] Q Yang R Yang J Davis and D Nister ldquoSpatial-depth superresolution for range imagesrdquo in Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo07)pp 1ndash8 IEEE Minneapolis Minn USA June 2007
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 21
[26] A Miron S Ainouz A Rogozan and A Bensrhair ldquoA robustcost function for stereomatching of road scenesrdquoPattern Recog-nition Letters vol 38 no 1 pp 70ndash77 2014
[27] H Huang and Q Wang ldquoA region and feature-based matchingalgorithm for dynamic object recognitionrdquo in Proceedings ofthe IEEE International Conference on Intelligent Computing andIntelligent Systems (ICIS rsquo10) pp 735ndash739 Xiamen China Octo-ber 2010
[28] K Sharma K Jeong and S-G Kim ldquoVision based autonomousvehicle navigation with self-organizing map feature matchingtechniquerdquo in Proceedings of the International Conference onControl Automation and Systems pp 946ndash949 Gyeonggi-doRepublic of Korea October 2011
[29] L Wang Novel dense stereo algorithms for high-quality depthestimation from images [Theses and DissertationsmdashComputerScience] University of Kentucky 2012
[30] J Liu X Sang C Jia N Guo Y Liu andG Shi ldquoEfficient stereomatching algorithm with edge-detectingrdquo in OptoelectronicImaging andMultimedia Technology III vol 9273 of Proceedingsof SPIE p 7 November 2014
[31] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoHigh-speed segmentation-driven high-resolution matchingrdquoin Seventh International Conference on Machine Vision (ICMV2014) p 94451Y International Society forOptics and Photonics2015
[32] B J Tippetts D-J Lee J K Archibald and K D LillywhiteldquoDense disparity real-time stereo vision algorithm for resource-limited systemsrdquo IEEE Transactions on Circuits and Systems forVideo Technology vol 21 no 10 pp 1547ndash1555 2011
[33] S H Lee and S Sharma ldquoReal-time disparity estimation algo-rithm for stereo camera systemsrdquo IEEE Transactions on Con-sumer Electronics vol 57 no 3 pp 1018ndash1026 2011
[34] R K Gupta and S-Y Cho ldquoWindow-based approach for faststereo correspondencerdquo IET Computer Vision vol 7 no 2 pp123ndash134 2013
[35] D Scharstein and R Szeliski Middlebury Stereo EvaluationmdashVersion 2 2015
[36] A Fusiello U Castellani and V Murino ldquoRelaxing symmetricmultiple windows stereo using Markov Random Fieldsrdquo inComputer Vision and Pattern Recognition vol 2134 of LectureNotes in Computer Science pp 91ndash105 Springer 2001
[37] R Yang and M Pollefeys ldquoMulti-resolution real-time stereoon commodity graphics hardwarerdquo in Proceedings of the IEEEComputer Society Conference on Computer Vision and PatternRecognition vol 1 pp I211ndashI217 Madison Wis USA June2003
[38] H Hirschmuller and D Scharstein ldquoEvaluation of cost func-tions for stereomatchingrdquo in Proceedings of the IEEEConferenceon Computer Vision and Pattern Recognition (CVPR rsquo07) pp 1ndash8 Minneapolis Minn USA June 2007
[39] S Satoh ldquoSimple low-dimensional features approximatingNCC-based image matchingrdquo Pattern Recognition Letters vol32 no 14 pp 1902ndash1911 2011
[40] F Cheng H Zhang D Yuan and M Sun ldquoStereo matching byusing the global edge constraintrdquo Neurocomputing vol 131 pp217ndash226 2014
[41] N Gac S Mancini M Desvignes and D Houzet ldquoHigh speed3D tomography on CPU GPU and FPGArdquo EURASIP Journalon Embedded Systems vol 2008 Article ID 930250 12 pages2008
[42] G Zhao Y K Du and Y D Tang ldquoA new extension of the ranktransform for stereo matchingrdquo Advanced Engineering Forumvol 2 pp 182ndash187 2011
[43] M Humenberger C Zinner M Weber W Kubinger and MVincze ldquoA fast stereo matching algorithm suitable for embed-ded real-time systemsrdquo Computer Vision and Image Under-standing vol 114 no 11 pp 1180ndash1202 2010
[44] L Ma J Li J Ma and H Zhang ldquoA modified census transformbased on the neighborhood information for stereo matchingalgorithmrdquo in Proceedings of the 7th International Conferenceon Image and Graphics (ICIG rsquo13) pp 533ndash538 IEEE QingdaoChina July 2013
[45] X Mei X Sun M Zhou S Jiao H Wang and X Zhang ldquoOnbuilding an accurate stereo matching system on graphics hard-warerdquo in Proceedings of the IEEE International Conference onComputer VisionWorkshops (ICCVWorkshops rsquo11) pp 467ndash474Barcelona Spain November 2011
[46] C Cigla and A A Alatan ldquoInformation permeability for stereomatchingrdquo Signal Processing Image Communication vol 28 no9 pp 1072ndash1088 2013
[47] N Zhang H Wang and J Cr ldquoA near real-time color stereomatching method for GPUrdquo in Proceedings of the 3rd Interna-tional Conference on Advanced Communications and Computa-tion pp 27ndash32 Lisbon Portugal November 2013
[48] S Lee J H Lee J Lim and I H Suh ldquoRobust stereo matchingusing adaptive random walk with restart algorithmrdquo Image andVision Computing vol 37 pp 1ndash11 2015
[49] Q Yang P Ji D Li S Yao andM Zhang ldquoFast stereo matchingusing adaptive guided filteringrdquo Image and Vision Computingvol 32 no 3 pp 202ndash211 2014
[50] H Hirschmuller P R Innocent and J Garibaldi ldquoReal-timecorrelation-based stereo vision with reduced border errorsrdquoInternational Journal of Computer Vision vol 47 no 1 pp 229ndash246 2002
[51] O Veksler ldquoFast variable window for stereo correspondenceusing integral imagesrdquo in Proceedings of the IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition(CVPR rsquo03) vol 1 pp I-556ndashI-561 IEEE Madison Wis USAJune 2003
[52] J Lu G Lafruit and F Catthoor ldquoAnisotropic local high-confidence voting for accurate stereo correspondencerdquo in ImageProcessing Algorithms and Systems VI vol 6812 of Proceedingsof SPIE pp 1ndash12 San Jose Calif USA January 2008
[53] K Zhang J Lu Q Yang G Lafruit R Lauwereins and L VanGool ldquoReal-time and accurate stereo a scalable approach withbitwise fast voting onCUDArdquo IEEETransactions onCircuits andSystems for Video Technology vol 21 no 7 pp 867ndash878 2011
[54] K-H Chen and C-L Su ldquoReducing computation complexityfor disparity matchingrdquo in Proceedings of the IEEE InternationalSymposium on Circuits and Systems (ISCAS rsquo13) pp 2916ndash2919Beijing China May 2013
[55] J Fang A L Varbanescu J Shen H Sips G Saygili and LVan Der Maaten ldquoAccelerating cost aggregation for real-timestereo matchingrdquo in Proceedings of the 18th IEEE InternationalConference on Parallel andDistributed Systems (ICPADS rsquo12) pp472ndash481 IEEE Singapore December 2012
[56] D Chen M Ardabilian and L Chen ldquoA fast trilateral filter-based adaptive support weight method for stereo matchingrdquoIEEE Transactions on Circuits and Systems for Video Technologyvol 25 no 5 pp 730ndash743 2015
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
22 Journal of Sensors
[57] J Zhang J-F Nezan M Pelcat and J-G Cousin ldquoReal-timeGPU-based local stereomatchingmethodrdquo in Proceedings of the7th Conference on Design and Architectures for Signal and ImageProcessing (DASIP rsquo13) pp 209ndash214 Cagliari Italy October2013
[58] A Hosni M Bleyer and M Gelautz ldquoSecrets of adaptive sup-port weight techniques for local stereo matchingrdquo ComputerVision and Image Understanding vol 117 no 6 pp 620ndash6322013
[59] L Nalpantidis and A Gasteratos ldquoStereo vision for roboticapplications in the presence of non-ideal lighting conditionsrdquoImage and Vision Computing vol 28 no 6 pp 940ndash951 2010
[60] Z Lee J Juang and T Q Nguyen ldquoLocal disparity estimationwith three-moded cross census and advanced support weightrdquoIEEE Transactions on Multimedia vol 15 no 8 pp 1855ndash18642013
[61] C-K Liang C-C Cheng Y-C Lai L-G Chen and H HChen ldquoHardware-efficient belief propagationrdquo IEEE Transac-tions on Circuits and Systems for Video Technology vol 21 no5 pp 525ndash537 2011
[62] H-Q Wang M Wu Y-B Zhang and L Zhang ldquoEffectivestereo matching using reliable points based graph cutrdquo in Pro-ceedings of the IEEE International Conference on Visual Com-munications and Image Processing (VCIP rsquo13) pp 1ndash6 KuchingMalaysia November 2013
[63] Q Yang L Wang R Yang H Stewenius and D Nister ldquoStereomatching with color-weighted correlation hierarchical beliefpropagation and occlusion handlingrdquo IEEE Transactions onPattern Analysis andMachine Intelligence vol 31 no 3 pp 492ndash504 2009
[64] Y S Heo K M Lee and S U Lee ldquoJoint depth map and colorconsistency estimation for stereo images with different illumi-nations and camerasrdquo IEEE Transactions on Pattern Analysisand Machine Intelligence vol 35 no 5 pp 1094ndash1106 2013
[65] K R Vijayanagar M Loghman and J Kim ldquoReal-time refine-ment of kinect depth maps using multi-resolution anisotropicdiffusionrdquo Mobile Networks and Applications vol 19 no 3 pp414ndash425 2014
[66] M Michael J Salmen J Stallkamp and M Schlipsing ldquoReal-time stereo vision optimizing semi-global matchingrdquo in Pro-ceedings of the IEEE Intelligent Vehicles Symposium (IV rsquo13) pp1197ndash1202 Gold Coast Australia June 2013
[67] ZMa KHe YWei J Sun and EWu ldquoConstant timeweightedmedian filtering for stereo matching and beyondrdquo in Proceed-ings of the 14th IEEE International Conference on ComputerVision (ICCV rsquo13) pp 49ndash56 IEEE Sydney Australia Decem-ber 2013
[68] A Banno and K Ikeuchi ldquoDisparity map refinement and 3Dsurface smoothing via directed anisotropic diffusionrdquo Com-puter Vision and Image Understanding vol 115 no 5 pp 611ndash619 2011
[69] S Y Park S H Lee and N I Cho ldquoSegmentation based dispar-ity estimation using color and depth informationrdquo in Proceed-ings of the International Conference on Image Processing (ICIPrsquo04) vol 5 pp 3275ndash3278 Singapore October 2004
[70] C Cigla X Zabulis and A Alatan ldquoRegion-based dense depthextraction from multi-view videordquo in Proceedings of the 14thIEEE International Conference on Image Processing (ICIP rsquo07)vol 5 pp V213ndashV216 IEEE San Antonio Tex USA September2007
[71] H Hirschmuller ldquoStereo processing by semiglobal matchingandmutual informationrdquo IEEETransactions on PatternAnalysisand Machine Intelligence vol 30 no 2 pp 328ndash341 2008
[72] L Wang and R Yang ldquoGlobal stereo matching leveraged bysparse ground control pointsrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo11) pp 3033ndash3040 Providence RI USA June 2011
[73] B-Y Chen and B-C Lai ldquoA high performance parallel graphcut optimization for depth estimationrdquo inAdvances in IntelligentSystems and Applications vol 2 pp 311ndash320 Springer 2013
[74] V Kolmogorov P Monasse and P Tan ldquoKolmogorov andZabihrsquos graph cuts stereo matching algorithmrdquo Image ProcessingOn Line vol 4 pp 220ndash251 2014
[75] Y Wang F Zhong Q Peng and X Qin ldquoDepth map enhance-ment based on color and depth consistencyrdquo Visual Computervol 30 no 10 pp 1157ndash1168 2014
[76] S Ploumpis A Amanatiadis and A Gasteratos ldquoA stereomatching approach based on particle filters and scattered con-trol landmarksrdquo Image and Vision Computing vol 38 pp 13ndash232015
[77] F Tombari S Mattoccia L D Stefano and E AddimandaldquoClassification and evaluation of cost aggregation methods forstereo correspondencerdquo in Proceedings of the 26th IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR rsquo08)pp 1ndash8 Anchorage Alaska USA June 2008
[78] E T Psota J Kowalczuk J Carlson and L C Perez ldquoA localiterative refinement method for adaptive support-weight stereomatchingrdquo in Proceedings of the International Conference onImage Processing Computer Vision and Pattern Recognition pp271ndash277 Las Vegas Nev USA July 2011
[79] Q Yang ldquoA non-local cost aggregation method for stereomatchingrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition (CVPR rsquo12) pp 1402ndash1409Providence RI USA June 2012
[80] L Xu O C Au W Sun et al ldquoStereo matching by adaptiveweighting selection based cost aggregationrdquo in Proceedings ofthe IEEE International Symposium on Circuits and Systems(ISCAS rsquo13) pp 1420ndash1423 IEEE Beijing China May 2013
[81] J Koo Y-H Kim and S Lee ldquoRobust stereo matching algo-rithm for advanced driver assistance systemrdquo in Proceedingsof the IEEE International Conference on Consumer Electronics(ICCE rsquo15) pp 268ndash269 Las Vegas Nev USA January 2015
[82] T Matsuo S Fujita N Fukushima and Y Ishibashi ldquoEffi-cient edge-awareness propagation via single-map filtering foredge-preserving stereo matchingrdquo inThree-Dimensional ImageProcessing Measurement (3DIPM) and Applications vol 9393of Proceedings of SPIE International Society for Optics andPhotonics San Francisco Calif USA March 2015
[83] L Nalpantidis andA Gasteratos ldquoBiologically and psychophys-ically inspired adaptive support weights algorithm for stereocorrespondencerdquoRobotics and Autonomous Systems vol 58 no5 pp 457ndash464 2010
[84] M Werner B Stabernack and C Riechert ldquoHardware imple-mentation of a full HD real-time disparity estimation algo-rithmrdquo IEEE Transactions on Consumer Electronics vol 60 no1 pp 66ndash73 2014
[85] D F Bacon R Rabah and S Shukla ldquoFPGA programming forthe massesrdquo Communications of the ACM vol 56 no 4 pp 56ndash63 2013
[86] N Wilt The Cuda Handbook A Comprehensive Guide to GpuProgramming Pearson Education 2013
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
Journal of Sensors 23
[87] R Kalarot and J Morris ldquoComparison of FPGA and GPUimplementations of real-time stereo visionrdquo in Proceedings ofthe IEEE Computer Society Conference on Computer Vision andPattern Recognition Workshops pp 9ndash15 IEEE San FranciscoCalif USA June 2010
[88] K Pauwels M Tomasi J Dıaz Alonso E Ros and M M VanHulle ldquoA Comparison of FPGA and GPU for real-time phase-based optical flow stereo and local image featuresrdquo IEEE Trans-actions on Computers vol 61 no 7 pp 999ndash1012 2012
[89] T Xu P Cockshott and S Oehler ldquoAcceleration of stereo-matching on multi-core CPU and GPUrdquo in Proceedings of theIEEE International Conference on High Performance Computingand Communications pp 108ndash115 IEEE Paris France August2014
[90] LM Russo E C Pedrino E Kato and V O Roda ldquoImage con-volution processing a GPU versus FPGA comparisonrdquo in Pro-ceedings of the 8th Southern Programmable Logic Conference(SPL rsquo12) pp 1ndash6 Bento Goncalves Brazil March 2012
[91] J Fowers G Brown J Wernsing and G Stitt ldquoA performanceand energy comparison of convolution on GPUs FPGAs andmulticore processorsrdquo ACM Transactions on Architecture andCode Optimization vol 9 no 4 article 25 2013
[92] M Jin andTMaruyama ldquoFast and accurate stereo vision systemon FPGArdquoACMTransactions on Reconfigurable Technology andSystems vol 7 no 1 article 3 2014
[93] G Lentaris I Stamoulias D Diamantopoulos et al ldquoSPAR-TANSEXTANTCOMPASS advancing space rover vision viareconfigurable platformsrdquo in Applied Reconfigurable Comput-ing pp 475ndash486 Springer 2015
[94] W Wang J Yan N Xu Y Wang and F-H Hsu ldquoReal-timehigh-quality stereo vision system in FPGArdquo in Proceedingsof the 12th International Conference on Field-ProgrammableTechnology (FPT rsquo13) pp 358ndash361 Kyoto JapanDecember 2013
[95] VMota G FalcaoMAntunes J Barreto andUNunes ldquoUsingthe GPU for fast symmetry-based dense stereo matching inhigh resolution imagesrdquo in Proceedings of the IEEE InternationalConference on Acoustics Speech and Signal Processing (ICASSPrsquo14) pp 7520ndash7524 Florence Italy May 2014
[96] Y-K Choi and I K Park ldquoEfficient GPU-based graph cutsfor stereo matchingrdquo in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo13) pp 642ndash648 IEEE Portland Ore USA June 2013
[97] S-J Yao L-H Wang C-L Lin and M Zhang ldquoReal-timestereo to multi-view conversion system based on adaptivemeshingrdquo Journal of Real-Time Image Processing pp 1ndash19 2015
[98] M G Samarawickrama Performance Evaluation of VisionAlgorithms on FPGA Universal-Publishers 2010
[99] S Mattoccia ldquoStereo vision algorithms for FPGAsrdquo in Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW rsquo13) pp 636ndash641 PortlandOre USA June 2013
[100] C Colodro-Conde F J Toledo-Moreo R Toledo-Moreo J JMartınez-Alvarez J Garrigos Guerrero and J M Ferrandez-Vicente ldquoEvaluation of stereo correspondence algorithms andtheir implementation on FPGArdquo Journal of Systems Architec-ture vol 60 no 1 pp 22ndash31 2014
[101] F Tombari F Gori and L Di Stefano ldquoEvaluation of stereoalgorithms for 3D object recognitionrdquo in Proceedings of the IEEEInternational Conference on Computer VisionWorkshops (ICCVrsquo11) pp 990ndash997 IEEE Barcelona Spain November 2011
[102] K-J Yoon and I S Kweon ldquoAdaptive support-weight approachfor correspondence searchrdquo IEEE Transactions on Pattern Anal-ysis and Machine Intelligence vol 28 no 4 pp 650ndash656 2006
[103] Y Zhao and G Taubin ldquoReal-time stereo on GPGPU usingprogressive multi-resolution adaptive windowsrdquo Image andVision Computing vol 29 no 6 pp 420ndash432 2011
[104] J Kowalczuk E T Psota and L C Perez ldquoReal-time stereomatching on CUDA using an iterative refinement method foradaptive support-weight correspondencesrdquo IEEE Transactionson Circuits and Systems for Video Technology vol 23 no 1 pp94ndash104 2013
[105] Y Lin N Lu X Lou F Zou Y Yao and Z Du ldquoMatchingcost filtering for dense stereo correspondencerdquo MathematicalProblems in Engineering vol 2013 Article ID 654139 11 pages2013
[106] B Tippetts D J Lee K Lillywhite and J K ArchibaldldquoHardware-efficient design of real-time profile shape matchingstereo vision algorithm on FPGArdquo International Journal ofReconfigurable Computing vol 2014 Article ID 945926 12pages 2014
[107] J-S Kim M Hwangbo and T Kanade ldquoParallel algorithms toa parallel hardware designing vision algorithms for a GPUrdquoin Proceedings of the IEEE 12th International Conference onComputer Vision Workshops (ICCV rsquo09) pp 862ndash869 KyotoJapan October 2009
[108] E S L Gastal and M M Oliveira ldquoDomain transform foredge-aware image and video processingrdquo ACM Transactions onGraphics vol 30 no 4 article 69 2011
[109] J Fowers G Brown P Cooke and G Stitt ldquoA perfor-mance and energy comparison of FPGAs GPUs and multi-cores for sliding-window applicationsrdquo in Proceedings of theACMSIGDA International Symposium on Field ProgrammableGate Arrays (FPGA rsquo12) pp 47ndash56 ACMMonterey Calif USAFebruary 2012
[110] F Ekstrand C Ahlberg M Ekstrom and G SpampinatoldquoTowards an embedded real-time high resolution vision sys-temrdquo in Advances in Visual Computing pp 541ndash550 SpringerBerlin Germany 2014
[111] M Perez-Patricio A Aguilar-Gonzalez J L Camas-Anzuetoand M Arias-Estrada ldquoA fuzzy logic approach for stereomatching suited for real-time processingrdquo International Journalof Computer Applications vol 113 no 2 pp 1ndash8 2015
[112] F Stein ldquoThe challenge of putting vision algorithms into acarrdquo in Proceedings of the IEEE Computer Society Conference onComputer Vision and Pattern Recognition Workshops (CVPRWrsquo12) pp 89ndash94 IEEE Providence RI USA June 2012
[113] Q Long Q Xie S Mita H Tehrani K Ishimaru and CGuo ldquoReal-time dense disparity estimation based onmulti-pathviterbi for intelligent vehicle applicationsrdquo in Proceedings of theBritish Machine Vision Conference pp 1ndash13 BMVA Press 2014
[114] A Geiger P Lenz C Stiller and R Urtasun ldquoTheKITTI VisionBenchmark Suiterdquo 2015
[115] M Chessa F Solari and S P Sabatini ldquoDatasets for disparityand optic flow evaluationrdquo 2015 httpwwwpspcunigeit
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of
International Journal of
AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014
RoboticsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Active and Passive Electronic Components
Control Scienceand Engineering
Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
International Journal of
RotatingMachinery
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporation httpwwwhindawicom
Journal ofEngineeringVolume 2014
Submit your manuscripts athttpwwwhindawicom
VLSI Design
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Shock and Vibration
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawi Publishing Corporation httpwwwhindawicom
Volume 2014
The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014
SensorsJournal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
Navigation and Observation
International Journal of
Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014
DistributedSensor Networks
International Journal of