LearningImageProcessingwithOpenCV

TableofContents

LearningImageProcessingwithOpenCV

Credits

AbouttheAuthors

AbouttheReviewers

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmore

Whysubscribe?

FreeaccessforPacktaccountholders

Preface

Whatthisbookcovers

Whatyouneedforthisbook

Whothisbookisfor

Conventions

Readerfeedback

Customersupport

Downloadingtheexamplecode

Downloadingthecolorimagesofthisbook

Errata

Piracy

Questions

1.HandlingImageandVideoFiles

AnintroductiontoOpenCV

DownloadingandinstallingOpenCV

GettingacompilerandsettingCMake

ConfiguringOpenCVwithCMake

Compilingandinstallingthelibrary

ThestructureofOpenCV

CreatinguserprojectswithOpenCV

Generalusageofthelibrary

Toolstodevelopnewprojects

CreatinganOpenCVC++programwithQtCreator

Readingandwritingimagefiles

ThebasicAPIconcepts

Imagefile-supportedformats

Theexamplecode

Readingimagefiles

Eventhandlingintotheintrinsicloop

Writingimagefiles

Readingandwritingvideofiles

Theexamplecode

User-interactionstools

Trackbars

Mouseinteraction

Buttons

Drawinganddisplayingtext

Summary

2.EstablishingImageProcessingTools

Basicdatatypes

Pixel-levelaccess

Measuringthetime

Commonoperationswithimages

Arithmeticoperations

Datapersistence

Histograms

Theexamplecode

Theexamplecode

Summary

3.CorrectingandEnhancingImages

Imagefiltering

Smoothing

Theexamplecode

Sharpening

Theexamplecode

Workingwithimagepyramids

Gaussianpyramids

Laplacianpyramids

Theexamplecode

Morphologicaloperations

Theexamplecode

LUTs

Theexamplecode

Geometricaltransformations

Affinetransformation

Scaling

Theexamplecode

Translation

Theexamplecode

Imagerotation

Theexamplecode

Skewing

Theexamplecode

Reflection

Theexamplecode

Perspectivetransformation

Theexamplecode

Inpainting

Theexamplecode

Denoising

Theexamplecode

Summary

4.ProcessingColor

Colorspaces

Conversionbetweencolorspaces(cvtColor)

RGB

Theexamplecode

Grayscale

Examplecode

CIEXYZ

Theexamplecode

YCrCb

Theexamplecode

HSV

Theexamplecode

HLS

Theexamplecode

CIEL*a*b*

Theexamplecode

CIEL*u*v*

Theexamplecode

Bayer

Theexamplecode

Color-space-basedsegmentation

HSVsegmentation

YCrCbsegmentation

Colortransfer

Theexamplecode

Summary

5.ImageProcessingforVideo

Videostabilization

Superresolution

Stitching

Summary

6.ComputationalPhotography

High-dynamic-rangeimages

CreatingHDRimages

Example

Tonemapping

Alignment

Exposurefusion

Seamlesscloning

Decolorization

Non-photorealisticrendering

Summary

7.AcceleratingImageProcessing

OpenCVwiththeOpenCLinstallation

AquickrecipetoinstallOpenCVwithOpenCL

ChecktheGPUusage

Acceleratingyourownfunctions

CheckingyourOpenCL

Thecodeexplanation

YourfirstGPU-basedprogram

Thecodeexplanation

Goingrealtime

Thecodeexplanation

Theperformance

Summary

Index

LearningImageProcessingwithOpenCV

LearningImageProcessingwithOpenCVCopyright©2015PacktPublishing

Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthors,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

Firstpublished:March2015

Productionreference:1230315

PublishedbyPacktPublishingLtd.

LiveryPlace

35LiveryStreet

BirminghamB32PB,UK.

ISBN978-1-78328-765-9

www.packtpub.com

CreditsAuthors

GloriaBuenoGarcía

OscarDenizSuarez

JoséLuisEspinosaAranda

JesusSalidoTercero

IsmaelSerranoGracia

NoeliaVállezEnano

Reviewers

WalterLucetti

AndrédeSouzaMoreira

MarvinSmith

CommissioningEditor

JulianUrsell

AcquisitionEditor

SamWood

ContentDevelopmentEditor

KirtiPatil

TechnicalEditor

FaisalSiddiqui

CopyEditor

StutiSrivastava

ProjectCoordinator

NidhiJoshi

Proofreaders

MartinDiver

MariaGould

SamanthaLyon

Indexer

TejalSoni

Graphics

AbhinashSahu

ProductionCoordinator

ConidonMiranda

CoverWork

ConidonMiranda

AbouttheAuthorsGloriaBuenoGarcíaholdsaPhDinmachinevisionfromCoventryUniversity,UK.Shehasexperienceworkingastheprincipalresearcherinseveralresearchcenters,suchasUMR7005researchunitCNRS/LouisPasteurUniv.Strasbourg(France),GilbertGilkes&GordonTechnology(UK),andCEITSanSebastian(Spain).Sheistheauthoroftwopatents,oneregisteredtypeofsoftware,andmorethan100refereedpapers.Herinterestsarein2D/3Dmultimodalityimageprocessingandartificialintelligence.SheleadstheVISILABresearchgroupattheUniversityofCastilla-LaMancha.ShehascoauthoredabookonOpenCVprogrammingformobiledevices:OpenCVessentials,PacktPublishing.

Thisisdedicatedtooursonsforthetimewehavenotbeenabletoplaywiththemandourparentsfortheirunconditionalsupportduringourlifetime.ThanksfromGloriaandOscar.

OscarDenizSuarez‘sresearchinterestsaremainlyfocusedoncomputervisionandpatternrecognition.Heistheauthorofmorethan50refereedpapersinjournalsandconferences.Hereceivedtherunner-upawardforthebestPhDworkoncomputervisionandpatternrecognitionbyAERFAIandtheImageFileandReformattingSoftwareChallengeAwardbyInnocentiveInc.Hehasbeenanationalfinalistforthe2009CorBaayenaward.Hisworkisusedbycutting-edgecompanies,suchasExistor,Gliif,Tapmedia,E-Twenty,andothers,andhasalsobeenaddedtoOpenCV.Currently,heworksasanassociateprofessorattheUniversityofCastilla-LaManchaandcontributestoVISILAB.HeisaseniormemberofIEEEandisaffiliatedwithAAAI,SIANI,CEA-IFAC,AEPIA,andAERFAI-IAPR.HeservesasanacademiceditorofthePLoSONEjournal.HehasbeenavisitingresearcheratCarnegieMellonUniversity,ImperialCollegeLondon,andLeicaBiosystems.HehascoauthoredtwobooksonOpenCVpreviously.

JoséLuisEspinosaArandaholdsaPhDincomputersciencefromtheUniversityofCastilla-LaMancha.HehasbeenafinalistforCertamenUniversitarioArquímedesdeIntroducciónalaInvestigacióncientíficain2009forhisfinaldegreeprojectinSpain.Hisresearchinterestsinvolvecomputervision,heuristicalgorithms,andoperationalresearch.HeiscurrentlyworkingattheVISILABgroupasanassistantresearcheranddeveloperincomputervisiontopics.

Thisisdedicatedtomyparentsandmybrothers.

JesusSalidoTercerogainedhiselectricalengineeringdegreeandPhD(1996)fromUniversidadPolitécnicadeMadrid(Spain).Hethenspent2years(1997and1998)asavisitingscholarattheRoboticsInstitute(CarnegieMellonUniversity,Pittsburgh,USA),workingoncooperativemultirobotsystems.SincehisreturntotheSpanishUniversityofCastilla-LaMancha,hespendshistimeteachingcoursesonroboticsandindustrialinformatics,alongwithresearchonvisionandintelligentsystems.Overthelast3years,hiseffortshavebeendirectedtodevelopvisionapplicationsonmobiledevices.HehascoauthoredabookonOpenCVprogrammingformobiledevices.

ThisisdedicatedtothosetowhomIoweallIam:myparents,SagrarioandMaria.

IsmaelSerranoGraciareceivedhisdegreeincomputersciencein2012fromthe

UniversityofCastilla-LaMancha.Hegotthehighestmarksforhisfinaldegreeprojectonpersondetection.ThisapplicationusesdepthcameraswithOpenCVlibraries.Currently,heisaPhDcandidateatthesameuniversity,holdingaresearchgrantfromtheSpanishMinistryofScienceandResearch.HeisalsoworkingattheVISILABgroupasanassistantresearcheranddeveloperondifferentcomputervisiontopics.

Thisisdedicatedtomyparents,whohavegivenmetheopportunityofeducationandhavesupportedmethroughoutmylife.Itisalsodedicatedtomysupervisor,Dr.OscarDeniz,whohasbeenafriend,guide,andhelper.Finally,itisdedicatedtomyfriendsandmygirlfriend,whohavealwayshelpedmeandbelievedthatIcoulddothis.

NoeliaVállezEnanohaslikedcomputerssinceherchildhood,thoughshedidn’thaveonebeforehermid-teens.In2009,shefinishedherstudiesincomputerscienceattheUniversityofCastilla-LaMancha,whereshegraduatedwithtophonors.ShestartedworkingattheVISILABgroupthroughaprojectonmammographyCADsystemsandelectronichealthrecords.Sincethen,shehasobtainedamaster’sdegreeinphysicsandmathematicsandhasenrolledforaPhDdegree.Herworkinvolvesusingimageprocessingandpatternrecognitionmethods.Shealsolikesteachingandworkinginotherareasofartificialintelligence.

AbouttheReviewersWalterLucetti,knownontheInternetasMyzhar,isanItaliancomputerengineerwithaspecializationinroboticsandroboticsperception.Hereceivedthelaureadegreein2005whilestudyingatResearchCenter“E.Piaggio”inPisa(Italy),wherehewroteathesisabout3Dmappingoftherealworldusinga2Dlasertiltedwithaservomotor,fusing3DwithRGBdata.Whilewritingthethesis,heencounteredOpenCVforthefirsttime—itwasearly2004andOpenCVwasatitslarvalstage.

Afterthelaurea,hestartedworkingassoftwaredeveloperforlow-levelembeddedsystemsandhigh-leveldesktopsystems.HegreatlyimprovedhisknowledgeofcomputervisionandmachinelearningasaresearcheratGustavoStefaniniAdvancedRoboticsCenterinLaSpezia(Italy),aspinoffofPERCROLaboratoryofScuolaSuperioreSant’AnnaofPisa(Italy).

Currently,heisworkinginthesoftwareindustry,writingfirmwareforembeddedARMsystems,softwarefordesktopsystemsbasedontheQtframework,andintelligentalgorithmsforvideosurveillancesystemsbasedonOpenCVandCUDA.

Heisalsoworkingonapersonalroboticproject:MyzharBot.MyzharBotisatrackedgroundmobilerobotthatusescomputervisiontodetectobstaclesandtoanalyzeandexploretheenvironment.TherobotisguidedbyalgorithmsbasedonROS,CUDA,andOpenCV.Youcanfollowtheprojectonthiswebsite:http://myzharbot.robot-home.it.

AndrédeSouzaMoreirahasamaster’sdegreeincomputerscience,withemphasisoncomputergraphicsfromthePontificalCatholicUniversityofRiodeJaneiro(Brazil).

HegraduatedwithabachelorofcomputersciencedegreefromUniversidadeFederaldoMaranhão(UFMA)inBrazil.Duringhisundergraduatedegree,hewasamemberofLabmint’sresearchteamandworkedwithmedicalimaging,specifically,breastcancerdetectionanddiagnosisusingimageprocessing.

Currently,heworksasaresearcherandsystemanalystatInstitutoTecgraf,oneofthemajorresearchanddevelopmentlabsincomputergraphicsinBrazil.HehasbeenworkingextensivelywithPHP,HTML,andCSSsince2007,andnowadays,hedevelopsprojectsinC++11/C++14,alongwithSQLite,Qt,Boost,andOpenGL.Moreinformationabouthimcanbeacquiredonhispersonalwebsiteatwww.andredsm.com.

MarvinSmithiscurrentlyasoftwareengineerinthedefenseindustry,specializinginphotogrammetryandremotesensing.HereceivedhisBSdegreeincomputersciencefromtheUniversityofNevadaReno.Histechnicalinterestsincludehighperformancecomputing,distributedimageprocessing,andmultispectralimageryexploitation.Priortoworkingindefense,MarvinheldinternshipswiththeIntelligentRoboticsGroupattheNASAAmesResearchCenterandtheNevadaAutomotiveTestCenter.

www.PacktPub.com

Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.

DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<service@packtpub.com>formoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

https://www2.packtpub.com/books/subscription/packtlib

DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.

Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser

FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.

PrefaceOpenCV,arguablythemostwidelyusedcomputervisionlibrary,includeshundredsofready-to-useimagingandvisionfunctionsandisusedinbothacademiaandindustry.Ascamerasgetcheaperandimagingfeaturesgrowindemand,therangeofapplicationsusingOpenCVincreasessignificantly,bothfordesktopandmobileplatforms.

Thisbookprovidesanexample-basedtourofOpenCV’smainimageprocessingalgorithms.WhileotherOpenCVbookstrytoexplaintheunderlyingtheoryorprovidelargeexamplesofnearlycompleteapplications,Thisbookisaimedatpeoplewhowanttohaveaneasy-to-understandworkingexampleassoonaspossible,andpossiblydevelopadditionalfeaturesontopofthat.

Thebookstartswithanintroductorychapterinwhichthelibraryinstallationisexplained,thestructureofthelibraryisdescribed,andbasicimageandvideoreadingandwritingexamplesaregiven.Fromthis,thefollowingfunctionalitiesarecovered:handlingofimagesandvideos,basicimageprocessingtools,correctingandenhancingimages,color,videoprocessing,andcomputationalphotography.Lastbutnotleast,advancedfeaturessuchasGPU-basedaccelerationsarealsoconsideredinthefinalchapter.Newfunctionsandtechniquesinthelatestmajorrelease,OpenCV3,areexplainedthroughout.

WhatthisbookcoversChapter1,HandlingImageandVideoFiles,showsyouhowtoreadimageandvideofiles.Italsoshowsbasicuser-interactiontools,whichareveryusefulinimageprocessingtochangeaparametervalue,selectregionsofinterest,andsoon.

Chapter2,EstablishingImageProcessingTools,describesthemaindatastructuresandbasicproceduresneededinsubsequentchapters.

Chapter3,CorrectingandEnhancingImages,dealswithtransformationstypicallyusedtocorrectimagedefects.Thischaptercoversfiltering,pointtransformationsusingLookUpTables,geometricaltransformations,andalgorithmsforinpaintinganddenoisingimages.

Chapter4,ProcessingColor,dealswithcolortopicsinimageprocessing.Thischapterexplainshowtousedifferentcolorspacesandperformcolortransfersbetweentwoimages.

Chapter5,ImageProcessingforVideo,coverstechniquesthatuseavideoorasequenceofimages.Thischapterisfocusedonalgorithms’implementationforvideostabilization,superresolution,andstitching.

Chapter6,ComputationalPhotography,explainshowtoreadHDRimagesandperformtonemappingonthem.

Chapter7,AcceleratingImageProcessing,coversanimportanttopicinimageprocessing:speed.ModernGPUsarethebestavailabletechnologytoacceleratetime-consumingimageprocessingtasks.

WhatyouneedforthisbookThepurposeofthisbookistoteachyouOpenCVimageprocessingbytakingyouthroughanumberofpracticalimageprocessingprojects.Thelatestversion,Version3.0ofOpenCV,willbeused.

Eachchapterprovidesseveralready-to-useexamplestoillustratetheconceptscoveredinit.Thebookis,therefore,focusedonprovidingyouwithaworkingexampleassoonaspossiblesothattheycandevelopadditionalfeaturesontopofthat.

Tousethisbook,onlyfreesoftwareisneeded.AlltheexampleshavebeendevelopedandtestedwiththefreelyavailableQtCreatorIDEandGNU/GCCcompiler.TheCMaketoolisalsousedtoconfigurethebuildprocessoftheOpenCVlibraryonthetargetplatform.Moreover,thefreelyavailableOpenCLSDKisrequiredfortheGPUaccelerationexamplesshowninChapter7,AcceleratingImageProcessing.

WhothisbookisforThisbookisintendedforreaderswhoalreadyknowC++programmingandwanttolearnhowtodoimageprocessingusingOpenCV.Youareexpectedtohaveaminimalbackgroundinthetheoryofimageprocessing.Thebookdoesnotcovertopicsthataremorerelatedtocomputervision,suchasfeatureandobjectdetection,tracking,ormachinelearning.

ConventionsInthisbook,youwillfindanumberofstylesoftextthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestyles,andanexplanationoftheirmeaning.

Codewordsintext,foldernames,filenames,fileextensions,pathnames,systemvariables,URLs,anduserinputareshownasfollows:“Eachmodulehasanassociatedheaderfile(forexamplecore.hpp).”

Ablockofcodeissetasfollows:

#include<opencv2/core/core.hpp>

#include<opencv2/highgui/highgui.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,char*argv[])

{

Matframe;//Containerforeachframe

Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:

#include<opencv2/core/core.hpp>

#include<opencv2/highgui/highgui.hpp>

#include<iostream>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,char*argv[])

{

Anycommand-lineinputoroutputiswrittenasfollows:

C:\opencv-buildQt\install

Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ClickingtheNextbuttonmovesyoutothenextscreen.”

NoteWarningsorimportantnotesappearinaboxlikethis.

TipTipsandtricksappearlikethis.

ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.

Tosendusgeneralfeedback,simplye-mail<feedback@packtpub.com>,andmentionthebook’stitleinthesubjectofyourmessage.

Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.

CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.

DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.

DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefrom:https://www.packtpub.com/sites/default/files/downloads/ImageProcessingwithOpenCV_Graphics.pdf

ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.

Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.

PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.

Pleasecontactusat<copyright@packtpub.com>withalinktothesuspectedpiratedmaterial.

Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.

QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<questions@packtpub.com>,andwewilldoourbesttoaddresstheproblem.

Chapter1.HandlingImageandVideoFilesThischapterisintendedasafirstcontactwithOpenCV,itsinstallation,andfirstbasicprograms.Wewillcoverthefollowingtopics:

AbriefintroductiontoOpenCVforthenovice,followedbyaneasystep-by-stepguidetotheinstallationofthelibraryAquicktourofOpenCV’sstructureaftertheinstallationintheuser’slocaldiskQuickrecipestocreateprojectsusingthelibrarywithsomecommonprogrammingframeworksHowtousethefunctionstoreadandwriteimagesandvideosFinally,wedescribethelibraryfunctionstoaddrichuserinterfacestothesoftwareprojects,includingmouseinteraction,drawingprimitives,andQtsupport

AnintroductiontoOpenCVInitiallydevelopedbyIntel,OpenCV(OpenSourceComputerVision)isafreecross-platformlibraryforreal-timeimageprocessingthathasbecomeadefactostandardtoolforallthingsrelatedtoComputerVision.Thefirstversionwasreleasedin2000underBSDlicenseandsincethen,itsfunctionalityhasbeenverymuchenrichedbythescientificcommunity.In2012,thenonprofitfoundationOpenCV.orgtookonthetaskofmaintainingasupportsitefordevelopersandusers.

NoteAtthetimeofwritingthisbook,anewmajorversionofOpenCV(Version3.0)isavailable,stillonbetastatus.Throughoutthebook,wewillpresentthemostrelevantchangesbroughtwiththisnewversion.

OpenCVisavailableforthemostpopularoperatingsystems,suchasGNU/Linux,OSX,Windows,Android,iOS,andsomemore.ThefirstimplementationwasintheCprogramminglanguage;however,itspopularitygrewwithitsC++implementationasofVersion2.0.NewfunctionsareprogrammedwithC++.However,nowadays,thelibraryhasafullinterfaceforotherprogramminglanguages,suchasJava,Python,andMATLAB/Octave.Also,wrappersforotherlanguages(suchasC#,Ruby,andPerl)havebeendevelopedtoencourageadoptionbyprogrammers.

Inanattempttomaximizetheperformanceofcomputingintensivevisiontasks,OpenCVincludessupportforthefollowing:

MultithreadingonmulticorecomputersusingThreadingBuildingBlocks(TBB)—atemplatelibrarydevelopedbyIntel.AsubsetofIntegratedPerformancePrimitives(IPP)onIntelprocessorstoboostperformance.ThankstoIntel,theseprimitivesarefreelyavailableasofVersion3.0beta.InterfacesforprocessingonGraphicProcessingUnit(GPU)usingComputeUnifiedDeviceArchitecture(CUDA)andOpenComputingLanguage(OpenCL).

TheapplicationsforOpenCVcoverareassuchassegmentationandrecognition,2Dand3Dfeaturetoolkits,objectidentification,facialrecognition,motiontracking,gesturerecognition,imagestitching,highdynamicrange(HDR)imaging,augmentedreality,andsoon.Moreover,tosupportsomeofthepreviousapplicationareas,amodulewithstatisticalmachinelearningfunctionsisincluded.

DownloadingandinstallingOpenCVOpenCVisfreelyavailablefordownloadathttp://opencv.org.Thissiteprovidesthelastversionfordistribution(currently,3.0beta)andolderversions.

NoteSpecialcareshouldbetakenwithpossibleerrorswhenthedownloadedversionisanonstablerelease,forexample,thecurrent3.0betaversion.

Onhttp://opencv.org/downloads.html,suitableversionsofOpenCVforeachplatformcanbefound.Thecodeandinformationofthelibrarycanbeobtainedfromdifferentrepositoriesdependingonthefinalpurpose:

Themainrepository(athttp://sourceforge.net/projects/opencvlibrary),devotedtofinalusers.Itcontainsbinaryversionsofthelibraryandready-to‑compilesourcesforthetargetplatform.Thetestdatarepository(athttps://github.com/itseez/opencv_extra)withsetsofdatatotestpurposesofsomelibrarymodules.Thecontributionsrepository(athttp://github.com/itseez/opencv_contrib)withthesourcecodecorrespondingtoextraandcutting-edgefeaturessuppliedbycontributors.Thiscodeismoreerror-proneandlesstestedthanthemaintrunk.

TipWiththelastversion,OpenCV3.0beta,theextracontributedmodulesarenotincludedinthemainpackage.Theyshouldbedownloadedseparatelyandexplicitlyincludedinthecompilationprocessthroughtheproperoptions.Becautiousifyouincludesomeofthosecontributedmodules,becausesomeofthemhavedependenciesonthird‑partysoftwarenotincludedwithOpenCV.Thedocumentationsite(athttp://docs.opencv.org/master/)foreachofthemodules,includingthecontributedones.Thedevelopmentrepository(athttps://github.com/Itseez/opencv)withthecurrentdevelopmentversionofthelibrary.Itisintendedfordevelopersofthemainfeaturesofthelibraryandthe“impatient”userwhowishestousethelastupdateevenbeforeitisreleased.

RatherthanGNU/LinuxandOSX,whereOpenCVisdistributedassourcecodeonly,intheWindowsdistribution,onecanfindprecompiled(withMicrosoftVisualC++v10,v11,andv12)versionsofthelibrary.EachprecompiledversionisreadytobeusedwithMicrosoftcompilers.However,iftheprimaryintentionistodevelopprojectswithadifferentcompilerframework,weneedtocompilethelibraryforthatspecificcompiler(forexample,GNUGCC).

TipThefastestroutetoworkingwithOpenCVistouseoneoftheprecompiledversionsincludedwiththedistribution.Then,abetterchoiceistobuildafine-tunedversionofthe

librarywiththebestsettingsforthelocalplatformusedforsoftwaredevelopment.ThischapterprovidestheinformationtobuildandinstallOpenCVonWindows.FurtherinformationtosetthelibraryonLinuxcanbefoundathttp://docs.opencv.org/doc/tutorials/introduction/linux_installandhttps://help.ubuntu.com/community/OpenCV.

GettingacompilerandsettingCMakeAgoodchoiceforcross‑platformdevelopmentwithOpenCVistousetheGNUtoolkit(includinggmake,g++,andgdb).TheGNUtoolkitcanbeeasilyobtainedforthemostpopularoperatingsystems.OurpreferredchoiceforadevelopmentenvironmentconsistsoftheGNUtoolkitandthecross‑platformQtframework,whichincludestheQtlibraryandtheQtCreatorIntegratedDevelopmentEnvironment(IDE).TheQtframeworkisfreelyavailableathttp://qt-project.org/.

NoteAfterinstallingthecompileronWindows,remembertoproperlysetthePathenvironmentvariable,addingthepathforthecompiler’sexecutable,forexample,C:\Qt\Qt5.2.1\5.2.1\mingw48_32\binfortheGNU/compilersincludedwiththeQtframework.OnWindows,thefreeRapidEnvironmentEditortool(availableathttp://www.rapidee.com)providesaconvenientwaytochangePathandotherenvironmentvariables.

TomanagethebuildprocessfortheOpenCVlibraryinacompiler-independentway,CMakeistherecommendedtool.CMakeisafreeandopensourcecross‑platformtoolavailableathttp://www.cmake.org/.

ConfiguringOpenCVwithCMakeOncethesourcesofthelibraryhavebeendownloadedintothelocaldisk,itisrequiredthatyouconfigurethemakefilesforthecompilationprocessofthelibrary.CMakeisthekeytoolforaneasyconfigurationofOpenCV’sinstallationprocess.Itcanbeusedfromthecommandlineorinamoreuser‑friendlywaywithitsGraphicalUserInterface(GUI)version.

ThestepstoconfigureOpenCVwithCMakecanbesummarizedasfollows:

1. Choosethesource(let’scallitOPENCV_SRCinwhatfollows)andtarget(OPENCV_BUILD)directories.Thetargetdirectoryiswherethecompiledbinarieswillbelocated.

2. MarktheGroupedandAdvancedcheckboxesandclickontheConfigurebutton.3. Choosethedesiredcompiler(forexample,GNUdefaultcompilers,MSVC,andso

on).4. Setthepreferredoptionsandunsetthosenotdesired.5. ClickontheConfigurebuttonandrepeatsteps4and5untilnoerrorsareobtained.6. ClickontheGeneratebuttonandcloseCMake.

ThefollowingscreenshotshowsyouthemainwindowofCMakewiththesourceandtargetdirectoriesandthecheckboxestogroupalltheavailableoptions:

ThemainwindowofCMakeafterthepreconfigurationstep

NoteForbrevity,weuseOPENCV_BUILDandOPENCV_SRCinthistexttodenotethetargetandsourcedirectoriesoftheOpenCVlocalsetup,respectively.Keepinmindthatalldirectoriesshouldmatchyourcurrentlocalconfiguration.

Duringthepreconfigurationprocess,CMakedetectsthecompilerspresentandmanyotherlocalpropertiestosetthebuildprocessofOpenCV.ThepreviousscreenshotdisplaysthemainCMakewindowafterthepreconfigurationprocess,showingthegroupedoptionsinred.

Itispossibletoleavethedefaultoptionsunchangedandcontinuetheconfigurationprocess.However,someconvenientoptionscanbeset:

BUILD_EXAMPLES:ThisissettobuildsomeexamplesusingOpenCV.BUILD_opencv_<module_name>:Thisissettoincludethemodule(module_name)in

thebuildprocess.OPENCV_EXTRA_MODULES_PATH:Thisisusedwhenyouneedsomeextracontributedmodule;setthepathforthesourcecodeoftheextramoduleshere(forexample,C:/opencv_contrib-master/modules).WITH_QT:ThisisturnedontoincludetheQtfunctionalityintothelibrary.WITH_IPP:Thisoptionisturnedonbydefault.ThecurrentOpenCV3.0versionincludesasubsetoftheIntelIntegratedPerformancePrimitives(IPP)thatspeeduptheexecutiontimeofthelibrary.

TipIfyoucompilethenewOpenCV3.0(beta),becautiousbecausesomeunexpectederrorshavebeenreportedrelatedtotheIPPinclusion(thatis,withthedefaultvalueofthisoption).WerecommendthatyouunsettheWITH_IPPoption.

IftheconfigurationstepswithCMake(loopthroughsteps4and5)don’tproduceanyfurthererrors,itispossibletogeneratethefinalmakefilesforthebuildprocess.ThefollowingscreenshotshowsyouthemainwindowofCMakeafteragenerationstepwithouterrors:

CompilingandinstallingthelibraryThenextstepafterthegenerationprocessofmakefileswithCMakeisthecompilationwiththepropermaketool.Thistoolisusuallyexecutedonthecommandline(theconsole)fromthetargetdirectory(theonesetattheCMakeconfigurationstep).Forexample,inWindows,thecompilationshouldbelaunchedfromthecommandlineasfollows:

OPENCV_BUILD>mingw32-make

ThiscommandlaunchesabuildprocessusingthemakefilesgeneratedbyCMake.Thewholecompilationtypicallytakesseveralminutes.Ifthecompilationendswithouterrors,theinstallationcontinueswiththeexecutionofthefollowingcommand:

OPENCV_BUILD>mingw32-makeinstall

ThiscommandcopiestheOpenCVbinariestotheOPENCV_BUILD\installdirectory.

Ifsomethingwentwrongduringthecompilation,weshouldrunCMakeagaintochangetheoptionsselectedduringtheconfiguration.Then,weshouldregeneratethemakefiles.

Theinstallationendsbyaddingthelocationofthelibrarybinaries(forexample,inWindows,theresultingDLLfilesarelocatedatOPENCV_BUILD\install\x64\mingw\bin)tothePathenvironmentvariable.WithoutthisdirectoryinthePathfield,theexecutionofeveryOpenCVexecutablewillgiveanerrorasthelibrarybinarieswon’tbefound.

Tocheckthesuccessoftheinstallationprocess,itispossibletorunsomeoftheexamplescompiledalongwiththelibrary(iftheBUILD_EXAMPLESoptionwassetusingCMake).Thecodesamples(writteninC++)canbefoundatOPENCV_BUILD\install\x64\mingw\samples\cpp.

NoteTheshortinstructionsgiventoinstallOpenCVapplytoWindows.AdetaileddescriptionwiththeprerequisitesforLinuxcanbereadathttp://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html.AlthoughthetutorialappliestoOpenCV2.0,almostalltheinformationisstillvalidforVersion3.0.

ThestructureofOpenCVOnceOpenCVisinstalled,theOPENCV_BUILD\installdirectorywillbepopulatedwiththreetypesoffiles:

Headerfiles:ThesearelocatedintheOPENCV_BUILD\install\includesubdirectoryandareusedtodevelopnewprojectswithOpenCV.Librarybinaries:Thesearestaticordynamiclibraries(dependingontheoptionselectedwithCMake)withthefunctionalityofeachoftheOpenCVmodules.Theyarelocatedinthebinsubdirectory(forexample,x64\mingw\binwhentheGNUcompilerisused).Samplebinaries:Theseareexecutableswithexamplesthatusethelibraries.Thesourcesforthesesamplescanbefoundinthesourcepackage(forexample,OPENCV_SRC\sources\samples).

OpenCVhasamodularstructure,whichmeansthatthepackageincludesastaticordynamic(DLL)libraryforeachmodule.Theofficialdocumentationforeachmodulecanbefoundathttp://docs.opencv.org/master/.Themainmodulesincludedinthepackageare:

core:ThisdefinesthebasicfunctionsusedbyalltheothermodulesandthefundamentaldatastructuresincludingtheimportantmultidimensionalarrayMat.highgui:Thisprovidessimpleuserinterface(UI)capabilities.BuildingthelibrarywithQtsupport(theWITH_QTCMakeoption)allowsUIcompatibilitywithsuchaframework.imgproc:Theseareimageprocessingfunctionsthatincludefiltering(linearandnonlinear),geometrictransformations,colorspaceconversion,histograms,andsoon.imgcodecs:Thisisaneasy-to-useinterfacetoreadandwriteimages.

NotePayattentiontothechangesinmodulessinceOpenCV3.0assomefunctionalityhasbeenmovedtoanewmodule(forexample,readingandwritingimagesfunctionsweremovedfromhighguitoimgcodecs).

photo:ThisincludesComputationalPhotographyincludinginpainting,denoising,HighDynamicRange(HDR)imaging,andsomeothers.stitching:Thisisusedforimagestitching.videoio:Thisisaneasy-to-useinterfaceforvideocaptureandvideocodecs.video:Thissuppliesthefunctionalityofvideoanalysis(motionestimation,backgroundextraction,andobjecttracking).features2d:Thesearefunctionsforfeaturedetection(cornersandplanarobjects),featuredescription,featurematching,andsoon.objdetect:Thesearefunctionsforobjectdetectionandinstancesofpredefineddetectors(suchasfaces,eyes,smile,people,cars,andsoon).

Someothermodulesarecalib3d(cameracalibration),flann(clusteringandsearch),ml(machinelearning),shape(shapedistanceandmatching),superres(superresolution),video(videoanalysis),andvideostab(videostabilization).

NoteAsofVersion3.0beta,thenewcontributedmodulesaredistributedinaseparatepackage(opencv_contrib-master.zip)thatcanbedownloadedfromhttps://github.com/itseez/opencv_contrib.Thesemodulesprovideextrafeaturesthatshouldbefullyunderstoodbeforeusingthem.ForaquickoverviewofthenewfunctionalityinthenewreleaseofOpenCV(Version3.0),refertothedocumentathttp://opencv.org/opencv-3-0-beta.html.

CreatinguserprojectswithOpenCVInthisbook,weassumethatC++isthemainlanguageforprogrammingimageprocessingapplications,althoughinterfacesandwrappersforotherprogramminglanguagesareactuallyprovided(forinstance,Python,Java,MATLAB/Octave,andsomemore).

Inthissection,weexplainhowtodevelopapplicationswithOpenCV’sC++APIusinganeasy-to-usecross-platformframework.

GeneralusageofthelibraryTodevelopanOpenCVapplicationwithC++,werequireourcodeto:

IncludetheOpenCVheaderfileswithdefinitionsLinktheOpenCVlibraries(binaries)togetthefinalexecutable

TheOpenCVheaderfilesarelocatedintheOPENCV_BUILD\install\include\opencv2directorywherethereisafile(*.hpp)foreachofthemodules.Theinclusionoftheheaderfileisdonewiththe#includedirective,asshownhere:

#include<opencv2/<module_name>/<module_name>.hpp>

//Includingtheheaderfileforeachmoduleusedinthecode

Withthisdirective,itispossibletoincludeeveryheaderfileneededbytheuserprogram.Ontheotherhand,iftheopencv.hppheaderfileisincluded,alltheheaderfileswillbeautomaticallyincludedasfollows:

#include<opencv2/opencv.hpp>

//IncludingalltheOpenCV'sheaderfilesinthecode

NoteRememberthatallthemodulesinstalledlocallyaredefinedintheOPENCV_BUILD\install\include\opencv2\opencv_modules.hppheaderfile,whichisgeneratedautomaticallyduringthebuildingprocessofOpenCV.

Theuseofthe#includedirectiveisnotalwaysaguaranteeforthecorrectinclusionoftheheaderfiles,becauseitisnecessarytotellthecompilerwheretofindtheincludefiles.Thisisachievedbypassingaspecialargumentwiththelocationofthefiles(suchasI\<location>forGNUcompilers).

Thelinkingprocessrequiresyoutoprovidethelinkerwiththelibraries(dynamicorstatic)wheretherequiredOpenCVfunctionalitycanbefound.Thisisusuallydonewithtwotypesofargumentsforthelinker:thelocationofthelibrary(suchas‑L<location>forGNUcompilers)andthenameofthelibrary(suchas-l<module_name>).

NoteYoucanfindacompletelistofavailableonlinedocumentationforGNUGCCandMakeathttps://gcc.gnu.org/onlinedocs/andhttps://www.gnu.org/software/make/manual/.

ToolstodevelopnewprojectsThemainprerequisitestodevelopourownOpenCVC++applicationsare:

OpenCVheaderfilesandlibrarybinaries:OfcourseweneedtocompileOpenCV,andtheauxiliarylibrariesareprerequisitesforsuchacompilation.Thepackageshouldbecompiledwiththesamecompilerusedtogeneratetheuserapplication.AC++compiler:Someassociatetoolsareconvenientasthecodeeditor,debugger,projectmanager,buildprocessmanager(forinstanceCMake),revisioncontrolsystem(suchasGit,Mercurial,SVN,andsoon),andclassinspector,amongothers.Usually,thesetoolsaredeployedtogetherinaso-calledIntegratedDevelopmentEnvironment(IDE).Anyotherauxiliarylibraries:Optionally,anyotherauxiliarylibrariesneededtoprogramthefinalapplication,suchasgraphical,statistical,andsoonwillberequired.

ThemostpopularavailablecompilerkitstoprogramOpenCVC++applicationsare:

MicrosoftVisualC(MSVC):ThisisonlysupportedonWindowsanditisverywellintegratedwiththeIDEVisualStudio,althoughitcanbealsointegratedwithothercross-platformIDEs,suchasQtCreatororEclipse.VersionsofMSVCthatcurrentlycompatiblewiththelatestOpenCVreleaseareVC10,VC11,andVC12(VisualStudio2010,2012,and2013).GNUCompilerCollectionGNUGCC:Thisisacross‑platformcompilersystemdevelopedbytheGNUproject.ForWindows,thiskitisknownasMinGW(MinimalGNUGCC).TheversioncompatiblewiththecurrentOpenCVreleaseisGNUGCC4.8.ThiskitmaybeusedwithseveralIDEs,suchasQtCreator,Code::Blocks,Eclipse,amongothers.

Fortheexamplespresentedinthisbook,weusedtheMinGW4.8compilerkitforWindowsplustheQt5.2.1libraryandtheQtCreatorIDE(3.0.1).Thecross-platformQtlibraryisrequiredtocompileOpenCVwiththenewUIcapabilitiesprovidedbysuchalibrary.

NoteForWindows,itispossibletodownloadaQtbundle(includingQtlibrary,QtCreator,andtheMinGWkit)fromhttp://qt-project.org/.Thebundleisapproximately700MB.

QtCreatorisacross-platformIDEforC++thatintegratesthetoolsweneedtocodeapplications.InWindows,itmaybeusedwithMinGWorMSVC.ThefollowingscreenshotshowsyoutheQtCreatormainwindowwiththedifferentpanelsandviewsforanOpenCVC++project:

ThemainwindowofQtCreatorwithsomeviewsfromanOpenCVC++project

CreatinganOpenCVC++programwithQtCreatorNext,weexplainhowtocreateacodeprojectwiththeQtCreatorIDE.Inparticular,weapplythisdescriptiontoanOpenCVexample.

WecancreateaprojectforanyOpenCVapplicationusingQtCreatorbynavigatingtoFile|NewFileorFile|Project…andthennavigatingtoNon-QtProject|PlainC++Project.Then,wehavetochooseaprojectnameandthelocationatwhichitwillbestored.Thenextstepistopickakit(thatis,thecompiler)fortheproject(inourcase,DesktopQt5.2.1MinGW32bit)andthelocationforthebinariesgenerated.Usually,twopossiblebuildconfigurations(profiles)areused:debugandrelease.Theseprofilessettheappropriateflagstobuildandrunthebinaries.

WhenaprojectiscreatedusingQtCreator,twospecialfiles(with.proand.pro.userextensions)aregeneratedtoconfigurethebuildandrunprocesses.Thebuildprocessisdeterminedbythekitchosenduringthecreationoftheproject.WiththeDesktopQt5.2.1MinGW32bitkit,thisprocessreliesontheqmakeandmingw32‑maketools.Usingthe*.profileastheinput,qmakegeneratesthemakefilethatdrivesthebuildprocessforeachprofile(thatis,releaseanddebug).TheqmaketoolisusedfromtheQtCreatorIDEasanalternativetoCMaketosimplifythebuildprocessofsoftwareprojects.Itautomatesthegenerationofmakefilesfromafewlinesofinformation.

Thefollowinglinesrepresentanexampleofa*.profile(forexample,showImage.pro):

TARGET:showImage

TEMPLATE=app

CONFIG+=console

CONFIG-=app_bundle

CONFIG-=qt

SOURCES+=\

showImage.cpp

INCLUDEPATH+=C:/opencv300-buildQt/install/include

LIBS+=-LC:/opencv300-buildQt/install/x64/mingw/lib\

-lopencv_core300.dll\

-lopencv_imgcodecs300.dll\

-lopencv_highgui300.dll\

-lopencv_imgproc300.dll

Theprecedingfileillustratestheoptionsthatqmakeneedstogeneratetheappropriatemakefilestobuildthebinariesforourproject.Eachlinestartswithatagindicatinganoption(TARGET,CONFIG,SOURCES,INCLUDEPATH,andLIBS)followedwithamarktoadd(+=)orremove(-=)thevalueoftheoption.Inthissampleproject,weusethenon-Qtconsoleapplication.TheexecutablefileisshowImage.exe(TARGET)andthesourcefileisshowImage.cpp(SOURCES).AsthisprojectisanOpenCV-basedapplication,thetwolasttagsindicatethelocationoftheheaderfiles(INCLUDEPATH)andtheOpenCVlibraries(LIBS)usedbythisparticularproject(core,imgcodecs,highgui,andimgproc).Notethatabackslashattheendofthelinedenotescontinuationinthenextline.

NoteForadetaileddescriptionofthetools(includingQtCreatorandqmake)developedwithintheQtproject,visithttp://doc.qt.io/.

ReadingandwritingimagefilesImageprocessingreliesongettinganimage(forinstance,aphotographoravideofame)and“playing”withitbyapplyingsignalprocessingtechniquesonittogetthedesiredresults.Inthissection,weshowyouhowtoreadimagesfromfilesusingthefunctionssuppliedbyOpenCV.

ThebasicAPIconceptsTheMatclassisthemaindatastructurethatstoresandmanipulatesimagesinOpenCV.Thisclassisdefinedinthecoremodule.OpenCVhasimplementedmechanismstoallocateandreleasememoryautomaticallyforthesedatastructures.However,theprogrammershouldstilltakespecialcarewhendatastructuressharethesamebuffermemory.Forinstance,theassignmentoperatordoesnotcopythememorycontentfromanobject(MatA)toanother(MatB);itonlycopiesthereference(thememoryaddressofthecontent).Then,achangeinoneobject(AorB)affectsbothobjects.ToduplicatethememorycontentofaMatobject,theMat::clone()memberfunctionshouldbeused.

NoteManyfunctionsinOpenCVprocessdensesingleormultichannelarrays,usuallyusingtheMatclass.However,insomecases,adifferentdatatypemaybeconvenient,suchasstd::vector<>,Matx<>,Vec<>,orScalar.Forthispurpose,OpenCVprovidestheproxyclassesInputArrayandOutputArray,whichallowanyoftheprevioustypestobeusedasparametersforfunctions.

TheMatclassisusedfordensen-dimensionalsingleormultichannelarrays.Itcanactuallystorerealorcomplex-valuedvectorsandmatrices,coloredorgrayscaleimages,histograms,pointclouds,andsoon.

TherearemanydifferentwaystocreateaMatobject,themostpopularbeingtheconstructorwherethesizeandtypeofthearrayarespecifiedasfollows:

Mat(nrows,ncols,type,fillValue)

TheinitialvalueforthearrayelementsmightbesetbytheScalarclassasatypicalfour-elementvector(foreachRGBandtransparencycomponentoftheimagestoredinthearray).Next,weshowyouausageexampleofMatasfollows:

Matimg_A(4,4,CV_8U,Scalar(255));

//Whiteimage:

//4x4single-channelarraywith8bitsofunsignedintegers

//(upto255values,validforagrayscaleimage,forexample,

//255=white)

TheDataTypeclassdefinestheprimitivedatatypesforOpenCV.Theprimitivedatatypescanbebool,unsignedchar,signedchar,unsignedshort,signedshort,int,float,double,oratupleofvaluesofoneoftheseprimitivetypes.Anyprimitivetypecanbedefinedbyanidentifierinthefollowingform:

CV_<bitdepth>{U|S|F}C(<numberofchannels>)

IntheprecedingcodeU,S,andFstandforunsigned,signed,andfloat,respectively.Forthesinglechannelarrays,thefollowingenumerationisapplied,describingthedatatypes:

enum{CV_8U=0,CV_8S=1,CV_16U=2,CV_16S=3,CV_32S=4,CV_32F=5,CV_64F=6};

Note

Here,itshouldbenotedthatthesethreedeclarationsareequivalent:CV_8U,CV_8UC1,andCV_8UC(1).Thesingle-channeldeclarationfitswellforintegerarraysdevotedtograyscaleimages,whereasthethreechanneldeclarationofanarrayismoreappropriateforimageswiththreecomponents(forexample,RGB,BRG,HSV,andsoon).Forlinearalgebraoperations,thearraysoftypefloat(F)mightbeused.

Wecandefinealloftheprecedingdatatypesformultichannelarrays(upto512channels).Thefollowingscreenshotsillustrateanimage’sinternalrepresentationwithonesinglechannel(CV_8U,grayscale)andthesameimagerepresentedwiththreechannels(CV_8UC3,RGB).ThesescreenshotsaretakenbyzoominginonanimagedisplayedinthewindowofanOpenCVexecutable(theshowImageexample):

An8-bitrepresentationofanimageinRGBcolorandgrayscale

NoteItisimportanttonoticethattoproperlysaveaRGBimagewithOpenCVfunctions,theimagemustbestoredinmemorywithitschannelsorderedasBGR.Inthesameway,whenanRGBimageisreadfromafile,itisstoredinmemorywithitschannelsinaBGRorder.Moreover,itneedsasupplementaryfourthchannel(alpha)tomanipulateimageswiththreechannels,RGB,plusatransparency.ForRGBimages,alargerintegervaluemeansabrighterpixelormoretransparencyforthealphachannel.

AllOpenCVclassesandfunctionsareinthecvnamespace,andconsequently,wewillhavethefollowingtwooptionsinoursourcecode:

Addtheusingnamespacecvdeclarationafterincludingtheheaderfiles(thisistheoptionusedinallthecodeexamplesinthisbook).Appendthecv::prefixtoalltheOpenCVclasses,functions,anddatastructuresthat

weuse.ThisoptionisrecommendediftheexternalnamesprovidedbyOpenCVconflictwiththeoften-usedstandardtemplatelibrary(STL)orotherlibraries.

Imagefile-supportedformatsOpenCVsupportsthemostcommonimageformats.However,someofthemneed(freelyavailable)third-partylibraries.ThemainformatssupportedbyOpenCVare:

Windowsbitmaps(*.bmp,*dib)Portableimageformats(*.pbm,*.pgm,*.ppm)Sunrasters(*.sr,*.ras)

Theformatsthatneedauxiliarylibrariesare:

JPEG(*.jpeg,*.jpg,*.jpe)JPEG2000(*.jp2)PortableNetworkGraphics(*.png)TIFF(*.tiff,*.tif)WebP(*.webp).

Inadditiontotheprecedinglistedformats,withtheOpenCV3.0version,itincludesadriverfortheformats(NITF,DTED,SRTM,andothers)supportedbytheGeographicDataAbstractionLibrary(GDAL)setwiththeCMakeoption,WITH_GDAL.NoticethattheGDALsupporthasnotbeenextensivelytestedonWindowsOSesyet.InWindowsandOSX,codecsshippedwithOpenCVareusedbydefault(libjpeg,libjasper,libpng,andlibtiff).Then,intheseOSes,itispossibletoreadtheJPEG,PNG,andTIFFformats.Linux(andotherUnix-likeopensourceOSes)looksforcodecsinstalledinthesystem.ThecodecscanbeinstalledbeforeOpenCVorelsethelibrariescanbebuiltfromtheOpenCVpackagebysettingtheproperoptionsinCMake(forexample,BUILD_JASPER,BUILD_JPEG,BUILD_PNG,andBUILD_TIFF).

TheexamplecodeToillustratehowtoreadandwriteimagefileswithOpenCV,wewillnowdescribetheshowImageexample.Theexampleisexecutedfromthecommandlinewiththecorrespondingoutputwindowsasfollows:

<bin_dir>\showImage.exefruits.jpgfruits_bw.jpg

TheoutputwindowfortheshowImageexample

Inthisexample,twofilenamesaregivenasarguments.Thefirstoneistheinputimagefiletoberead.Thesecondoneistheimagefiletobewrittenwithagrayscalecopyoftheinputimage.Next,weshowyouthesourcecodeanditsexplanation:

#include<opencv2/opencv.hpp>

#include<iostream>

usingnamespacestd;

usingnamespacecv;

intmain(int,char*argv[])

{

Matin_image,out_image;

//Usage:<cmd><file_in><file_out>

//Readoriginalimage

in_image=imread(argv[1],IMREAD_UNCHANGED);

if(in_image.empty()){

//Checkwhethertheimageisreadornot

cout<<"Error!Inputimagecannotberead…\n";

return-1;

}

//Createstwowindowswiththenamesoftheimages

namedWindow(argv[1],WINDOW_AUTOSIZE);

namedWindow(argv[2],WINDOW_AUTOSIZE);

//Showstheimageintothepreviouslycreatedwindow

imshow(argv[1],in_image);

cvtColor(in_image,out_image,COLOR_BGR2GRAY);

imshow(argv[2],in_image);

cout<<"Pressanykeytoexit…\n";

waitKey();//Waitforkeypress

//Writingimage

imwrite(argv[2],in_image);

return0;

}

Here,weusethe#includedirectivewiththeopencv.hppheaderfilethat,infact,includesalltheOpenCVheaderfiles.Byincludingthissinglefile,nomorefilesneedtobeincluded.Afterdeclaringtheuseofcvnamespace,allthevariablesandfunctionsinsidethisnamespacedon’tneedthecv::prefix.Thefirstthingtodointhemainfunctionistocheckthenumberofargumentspassedinthecommandline.Then,ahelpmessageisdisplayedifanerroroccurs.

ReadingimagefilesIfthenumberofargumentsiscorrect,theimagefileisreadintotheMatin_imageobjectwiththeimread(argv[1],IMREAD_UNCHANGED)function,wherethefirstparameteristhefirstargument(argv[1])passedinthecommandlineandthesecondparameterisaflag(IMREAD_UNCHANGED),whichmeansthattheimagestoredintothememoryobjectshouldbeunchanged.Theimreadfunctiondeterminesthetypeofimage(codec)fromthefilecontentratherthanfromthefileextension.

Theprototypefortheimreadfunctionisasfollows:

Matimread(constString&filename,

intflags=IMREAD_COLOR)

Theflagspecifiesthecoloroftheimagereadandtheyaredefinedandexplainedbythefollowingenumerationintheimgcodecs.hppheaderfile:

enum{IMREAD_UNCHANGED=-1,//8bit,colorornot

IMREAD_GRAYSCALE=0,//8bit,gray

IMREAD_COLOR=1,//unchangeddepth,color

IMREAD_ANYDEPTH=2,//anydepth,unchangedcolor

IMREAD_ANYCOLOR=4,//unchangeddepth,anycolor

IMREAD_LOAD_GDAL=8//Usegdaldriver

};

NoteAsofVersion3.0ofOpenCV,theimreadfunctionisintheimgcodecsmoduleandnotinhighguilikeinOpenCV2.x.

Tip

AsseveralfunctionsanddeclarationsaremovedintoOpenCV3.0,itispossibletogetsomecompilationerrorsasoneormoredeclarations(symbolsand/orfunctions)arenotfoundbythelinker.Tofigureoutwhere(*.hpp)asymbolisdefinedandwhichlibrarytolink,werecommendthefollowingtrickusingtheQtCreatorIDE:

Addthe#include<opencv2/opencv.hpp>declarationtothecode.PresstheF2functionkeywiththemousecursoroverthesymbolorfunction;thisopensthe*.hppfilewherethesymbolorfunctionisdeclared.

Aftertheinputimagefileisread,checktoseewhethertheoperationsucceeded.Thischeckisachievedwiththein_image.empty()memberfunction.Iftheimagefileisreadwithouterrors,twowindowsarecreatedtodisplaytheinputandoutputimages,respectively.Thecreationofwindowsiscarriedoutwiththefollowingfunction:

voidnamedWindow(constString&winname,intflags=WINDOW_AUTOSIZE)

OpenCVwindowsareidentifiedbyaunivocalnameintheprogram.Theflags’definitionandtheirexplanationaregivenbythefollowingenumerationinthehighgui.hppheaderfile:

enum{WINDOW_NORMAL=0x00000000,

//theusercanresizethewindow(noconstraint)

//alsousetoswitchafullscreenwindowtoanormalsize

WINDOW_AUTOSIZE=0x00000001,

//theusercannotresizethewindow,

//thesizeisconstrainedbytheimagedisplayed

WINDOW_OPENGL=0x00001000,//windowwithopenglsupport

WINDOW_FULLSCREEN=1,

WINDOW_FREERATIO=0x00000100,

//theimageexpendsasmuchasitcan(noratioconstraint)

WINDOW_KEEPRATIO=0x00000000

//theratiooftheimageisrespected

};

Thecreationofawindowdoesnotshowanythingonscreen.Thefunction(belongingtothehighguimodule)todisplayanimageinawindowis:

voidimshow(constString&winname,InputArraymat)

Theimage(mat)isshownwithitsoriginalsizeifthewindow(winname)wascreatedwiththeWINDOW_AUTOSIZEflag.

IntheshowImageexample,thesecondwindowshowsagrayscalecopyoftheinputimage.Toconvertacolorimagetograyscale,thecvtColorfunctionfromtheimgprocmoduleisused.Thisfunctioncanactuallybeusedtochangetheimagecolorspace.

Anywindowcreatedinaprogramcanberesizedandmovedfromitsdefaultsettings.Whenanywindowisnolongerrequired,itshouldbedestroyedinordertoreleaseitsresources.Thisresourceliberationisdoneimplicitlyattheendofaprogram,likeintheexample.

Eventhandlingintotheintrinsicloop

Ifwedonothingmoreaftershowinganimageonawindow,surprisingly,theimagewillnotbeshownatall.Aftershowinganimageonawindow,weshouldstartalooptofetchandhandleeventsrelatedtouserinteractionwiththewindow.Suchataskiscarriedoutbythefollowingfunction(fromthehighguimodule):

intwaitKey(intdelay=0)

Thisfunctionwaitsforakeypressedduringanumberofmilliseconds(delay>0)returningthecodeofthekeyor-1ifthedelayendswithoutakeypressed.Ifdelayis0ornegative,thefunctionwaitsforeveruntilakeyispressed.

NoteRememberthatthewaitKeyfunctiononlyworksifthereisacreatedandactivewindowatleast.

WritingimagefilesAnotherimportantfunctionintheimgcodecsmoduleis:

boolimwrite(constString&filename,InputArrayimg,constvector<int>&

params=vector<int>())

Thisfunctionsavestheimage(img)intoafile(filename),beingthethirdoptionalargumentavectorofproperty-valuepairsspecifyingtheparametersofthecodec(leaveitemptytousethedefaultvalues).Thecodecisdeterminedbytheextensionofthefile.

NoteForadetailedlistofcodecproperties,takealookattheimgcodecs.hppheaderfileandtheOpenCVAPIreferenceathttp://docs.opencv.org/master/modules/refman.html.

ReadingandwritingvideofilesRatherthanstillimages,avideodealswithmovingimages.Thesourcesofvideocanbeadedicatedcamera,awebcam,avideofile,orasequenceofimagefiles.InOpenCV,theVideoCaptureandVideoWriterclassesprovideaneasy-to-useC++APIforthetaskofcapturingandrecordinginvolvedinvideoprocessing.

TheexamplecodeTherecVideoexampleisashortsnippetofcodewhereyoucanseehowtouseadefaultcameraasacapturedevicetograbframes,processthemforedgedetection,andsavethisnewconvertedframetoafile.Also,twowindowsarecreatedtosimultaneouslyshowyoutheoriginalframeandtheprocessedone.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<iostream>

usingnamespacestd;

usingnamespacecv;

intmain(int,char**)

{

Matin_frame,out_frame;

constcharwin1[]="Grabbing…",win2[]="Recording…";

doublefps=30;//Framespersecond

charfile_out[]="recorded.avi";

VideoCaptureinVid(0);//Opendefaultcamera

if(!inVid.isOpened()){//Checkerror

cout<<"Error!Cameranotready…\n";

return-1;

}

//Getsthewidthandheightoftheinputvideo

intwidth=(int)inVid.get(CAP_PROP_FRAME_WIDTH);

intheight=(int)inVid.get(CAP_PROP_FRAME_HEIGHT);

VideoWriterrecVid(file_out,

VideoWriter::fourcc('M','S','V','C'),

fps,Size(width,height));

if(!recVid.isOpened()){

cout<<"Error!Videofilenotopened…\n";

return-1;

}

//Createtwowindowsfororig.andfinalvideo

namedWindow(win1);

namedWindow(win2);

while(true){

//Readframefromcamera(grabbinganddecoding)

inVid>>in_frame;

//Converttheframetograyscale

cvtColor(in_frame,out_frame,COLOR_BGR2GRAY);

//Writeframetovideofile(encodingandsaving)

recVid<<out_frame;

imshow(win1,in_frame);//Showframeinwindow

imshow(win2,out_frame);//Showframeinwindow

if(waitKey(1000/fps)>=0)

break;

}

inVid.release();//Closecamera

return0;

}

Inthisexample,thefollowingfunctionsdeserveaquickreview:

doubleVideoCapture::get(intpropId):ThisreturnsthevalueofthespecifiedpropertyforaVideoCaptureobject.AcompletelistofpropertiesbasedonDC1394(IEEE1394DigitalCameraSpecifications)isincludedwiththevideoio.hppheaderfile.staticintVideoWriter::fourcc(charc1,charc2,charc3,charc4):Thisconcatenatesfourcharacterstoafourcccode.Intheexample,MSVCstandsforMicrosoftVideo(onlyavailableforWindows).Thelistofvalidfourcccodesispublishedathttp://www.fourcc.org/codecs.php.boolVideoWriter::isOpened():Thisreturnstrueiftheobjectforwritingthevideowassuccessfullyinitialized.Forinstance,usinganimpropercodecproducesanerror.

TipBecautious;thevalidfourcccodesinasystemdependonthelocallyinstalledcodecs.Toknowtheinstalledfourcccodecsavailableinthelocalsystem,werecommendtheopensourcetoolMediaInfo,availableformanyplatformsathttp://mediaarea.net/en/MediaInfo.

VideoCapture&VideoCapture::operator>>(Mat&image):Thisgrabs,decodes,andreturnsthenextframe.ThismethodhastheequivalentboolVideoCapture::read(OutputArrayimage)function.ItcanbeusedratherthanusingtheVideoCapture::grab()function,followedbyVideoCapture::retrieve().VideoWriter&VideoWriter::operator<<(constMat&image):Thiswritesthenextframe.ThismethodhastheequivalentvoidVideoWriter::write(constMat&image)function.

Inthisexample,thereisareading/writingloopwherethewindoweventsarefetchedandhandledaswell.ThewaitKey(1000/fps)functioncallisinchargeofthat;however,inthiscase,1000/fpsindicatesthenumberofmillisecondstowaitbeforereturningtotheexternalloop.Althoughnotexact,anapproximatemeasureofframespersecondisobtainedfortherecordedvideo.

voidVideoCapture::release():Thisreleasesthevideofileorcapturingdevice.Althoughnotexplicitlynecessaryinthisexample,weincludeittoillustrateitsuse.

User-interactionstoolsIntheprevioussections,weexplainedhowtocreate(namedWindow)awindowtodisplay(imshow)animageandfetch/handleevents(waitKey).TheexamplesweprovideshowyouaveryeasymethodforuserinteractionwithOpenCVapplicationsthroughthekeyboard.ThewaitKeyfunctionreturnsthecodeofakeypressedbeforeatimeoutexpires.

Fortunately,OpenCVprovidesmoreflexiblewaysforuserinteraction,suchastrackbarsandmouseinteraction,whichcanbecombinedwithsomedrawingfunctionstoprovidearicheruserexperience.Moreover,ifOpenCVislocallycompiledwithQtsupport(theWITH_QToptionofCMake),asetofnewfunctionsareavailabletoprogramanevenbetterUI.

Inthissection,weprovideaquickreviewoftheavailablefunctionalitytoprogramuserinterfacesinanOpenCVprojectwithQtsupport.WeillustratethisreviewonOpenCVUIsupportwiththenextexamplenamedshowUI.

Theexampleshowsyouacolorimageinawindow,illustratinghowtousesomebasicelementstoenrichtheuserinteraction.ThefollowingscreenshotdisplaystheUIelementscreatedintheexample:

TheoutputwindowfortheshowUIexample

ThesourcecodeoftheshowUIexample(withoutthecallbackfunctions)isasfollows:

#include<opencv2/opencv.hpp>

#include<iostream>

usingnamespacestd;

usingnamespacecv;

//Callbackfunctionsdeclarations

voidcbMouse(intevent,intx,inty,intflags,void*);

voidtb1_Callback(intvalue,void*);

voidtb2_Callback(intvalue,void*);

voidcheckboxCallBack(intstate,void*);

voidradioboxCallBack(intstate,void*id);

voidpushbuttonCallBack(int,void*font);

//Globaldefinitionsandvariables

Matorig_img,tmp_img;

constcharmain_win[]="main_win";

charmsg[50];

intmain(int,char*argv[]){

constchartrack1[]="TrackBar1";

constchartrack2[]="TrackBar2";

constcharcheckbox[]="CheckBox";

constcharradiobox1[]="RadioBox1";

constcharradiobox2[]="RadioBox2";

constcharpushbutton[]="PushButton";

inttb1_value=50;//Initialvalueoftrackbar1

inttb2_value=25;//Initialvalueoftrackbar1

orig_img=imread(argv[1]);//Openandreadtheimage

if(orig_img.empty()){

cout<<"Error!!!Imagecannotbeloaded…"<<endl;

return-1;

}

namedWindow(main_win);//Createsmainwindow

//Createsafontforaddingtexttotheimage

QtFontfont=fontQt("Arial",20,Scalar(255,0,0,0),

QT_FONT_BLACK,QT_STYLE_NORMAL);

//CreationofCallBackfunctions

setMouseCallback(main_win,cbMouse,NULL);

createTrackbar(track1,main_win,&tb1_value,

100,tb1_Callback);

createButton(checkbox,checkboxCallBack,0,

QT_CHECKBOX);

//Passingvalues(font)totheCallBack

createButton(pushbutton,pushbuttonCallBack,

(void*)&font,QT_PUSH_BUTTON);

createTrackbar(track2,NULL,&tb2_value,

50,tb2_Callback);

//PassingvaluestotheCallBack

createButton(radiobox1,radioboxCallBack,

(void*)radiobox1,QT_RADIOBOX);

createButton(radiobox2,radioboxCallBack,

(void*)radiobox2,QT_RADIOBOX);

imshow(main_win,orig_img);//Showsoriginalimage

cout<<"Pressanykeytoexit…"<<endl;

waitKey();//Infiniteloopwithhandleforevents

return0;

}

WhenOpenCVisbuiltwithQtsupport,everycreatedwindow—throughthehighguimodule—showsadefaulttoolbar(seetheprecedingfigure)withoptions(fromlefttoright)forpanning,zooming,saving,andopeningthepropertieswindow.

Additionaltothementionedtoolbar(onlyavailablewithQt),inthenextsubsections,wecommentthedifferentUIelementscreatedintheexampleandthecodetoimplementthem.

TrackbarsTrackbarsarecreatedwiththecreateTrackbar(constString&trackbarname,constString&winname,int*value,intcount,TrackbarCallbackonChange=0,void*

userdata=0)functioninthespecifiedwindow(winname),withalinkedintegervalue(value),amaximumvalue(count),anoptionalcallbackfunction(onChange)tobecalledonchangesoftheslider,andanargument(userdata)tothecallbackfunction.Thecallbackfunctionitselfgetstwoarguments:value(selectedbytheslider)andapointertouserdata(optional).WithQtsupport,ifnowindowisspecified,thetrackbariscreatedinthepropertieswindow.IntheshowUIexample,wecreatetwotrackbars:thefirstinthemainwindowandthesecondoneinthepropertieswindow.Thecodeforthetrackbarcallbacksis:

voidtb1_Callback(intvalue,void*){

sprintf(msg,"Trackbar1changed.Newvalue=%d",value);

displayOverlay(main_win,msg);

return;

}

voidtb2_Callback(intvalue,void*){

sprintf(msg,"Trackbar2changed.Newvalue=%d",value);

displayStatusBar(main_win,msg,1000);

return;

}

MouseinteractionMouseeventsarealwaysgeneratedsothattheuserinteractswiththemouse(movingandclicking).Bysettingtheproperhandlerorcallbackfunctions,itispossibletoimplementactionssuchasselect,draganddrop,andsoon.Thecallbackfunction(onMouse)isenabledwiththesetMouseCallback(constString&winname,MouseCallbackonMouse,void*userdata=0)functioninthespecifiedwindow(winname)andoptionalargument(userdata).

Thesourcecodeforthecallbackfunctionthathandlesthemouseeventis:

voidcbMouse(intevent,intx,inty,intflags,void*){

//Staticvarsholdvaluesbetweencalls

staticPointp1,p2;

staticboolp2set=false;

//Leftmousebuttonpressed

if(event==EVENT_LBUTTONDOWN){

p1=Point(x,y);//Setorig.point

p2set=false;

}elseif(event==EVENT_MOUSEMOVE&&

flags==EVENT_FLAG_LBUTTON){

//Checkmovingmouseandleftbuttondown

//Checkoutbounds

if(x>orig_img.size().width)

x=orig_img.size().width;

elseif(x<0)

x=0;

//Checkoutbounds

if(y>orig_img.size().height)

y=orig_img.size().height;

elseif(y<0)

y=0;

p2=Point(x,y);//Setfinalpoint

p2set=true;

//Copyorig.totemp.image

orig_img.copyTo(tmp_img);

//Drawsrectangle

rectangle(tmp_img,p1,p2,Scalar(0,0,255));

//Drawtemporalimagewithrect.

imshow(main_win,tmp_img);

}elseif(event==EVENT_LBUTTONUP

&&p2set){

//Checkifleftbuttonisreleased

//andselectedanarea

//Setsubarrayonorig.image

//withselectedrectangle

Matsubmat=orig_img(Rect(p1,p2));

//Heresomeprocessingforthesubmatrix

//...

//Marktheboundariesofselectedrectangle

rectangle(orig_img,p1,p2,Scalar(0,0,255),2);

imshow("main_win",orig_img);

}

return;

}

IntheshowUIexample,themouseeventsareusedtocontrolthroughacallbackfunction(cbMouse),theselectionofarectangularregionbydrawingarectanglearoundit.Intheexample,thisfunctionisdeclaredasvoidcbMouse(intevent,intx,inty,intflags,void*),theargumentsbeingthepositionofthepointer(x,y)wheretheeventoccurs,theconditionwhentheeventoccurs(flags),andoptionally,userdata.

NoteTheavailableevents,flags,andtheircorrespondingdefinitionsymbolscanbefoundinthehighgui.hppheaderfile.

ButtonsOpenCV(onlywithQtsupport)allowsyoutocreatethreetypesofbuttons:checkbox(QT_CHECKBOX),radiobox(QT_RADIOBOX),andpushbutton(QT_PUSH_BUTTON).Thesetypesofbuttoncanbeusedrespectivelytosetoptions,setexclusiveoptions,andtakeactionsonpush.ThethreearecreatedwiththecreateButton(constString&button_name,ButtonCallbackon_change,void*userdata=0,int

type=QT_PUSH_BUTTON,boolinit_state=false)functioninthepropertieswindowarrangedinabuttonbarafterthelasttrackbarcreatedinthiswindow.Theargumentsforthebuttonareitsname(button_name),thecallbackfunctioncalledonthestatuschange(on_change),andoptionally,anargument(userdate)tothecallback,thetypeofbutton(type),andtheinitialstateofthebutton(init_state).

Next,weshowyouthesourcecodeforthecallbackfunctionscorrespondingtobuttonsintheexample:

voidcheckboxCallBack(intstate,void*){

sprintf(msg,"Checkboxchanged.Newstate=%d",state);

displayStatusBar(main_win,msg);

return;

}

voidradioboxCallBack(intstate,void*id){

//IdoftheradioboxpassedtothecallBack

sprintf(msg,"%schanged.Newstate=%d",

(char*)id,state);

displayStatusBar(main_win,msg);

return;

}

voidpushbuttonCallBack(int,void*font){

//Addtexttotheimage

addText(orig_img,"Pushbuttonclicked",

Point(50,50),*((QtFont*)font));

imshow(main_win,orig_img);//Showsoriginalimage

return;

}

Thecallbackfunctionforabuttongetstwoarguments:itsstatusand,optionally,apointertouserdata.IntheshowUIexample,weshowyouhowtopassaninteger(radioboxCallBack(intstate,void*id))toidentifythebuttonandamorecomplexobject(pushbuttonCallBack(int,void*font)).

DrawinganddisplayingtextAveryefficientwaytocommunicatetheresultsofsomeimageprocessingtotheuserisbydrawingshapesor/anddisplayingtextoverthefigurebeingprocessed.Throughtheimgprocmodule,OpenCVprovidessomeconvenientfunctionstoachievesuchtasksasputtingtext,drawinglines,circles,ellipses,rectangles,polygons,andsoon.TheshowUIexampleillustrateshowtoselectarectangularregionoveranimageanddrawarectangletomarktheselectedarea.Thefollowingfunctiondraws(img)arectangledefinedbytwopoints(p1,p2)overanimagewiththespecifiedcolorandotheroptionalparametersasthickness(negativeforafillshape)andthetypeoflines:

voidrectangle(InputOutputArrayimg,Pointpt1,Pointpt2,constScalar&

color,intthickness=1,intlineType=LINE_8,intshift=0)

Additionaltoshapes’drawingsupport,theimgprocmoduleprovidesafunctiontoputtextoveranimagewiththefunction:

voidputText(InputOutputArrayimg,constString&text,Pointorg,int

fontFace,doublefontScale,Scalarcolor,intthickness=1,int

lineType=LINE_8,boolbottomLeftOrigin=false)

NoteTheavailablefontfacesforthetextcanbeinspectedinthecore.hppheaderfile.

Qtsupport,inthehighguimodule,addssomeadditionalwaystoshowtextonthemainwindowofanOpenCVapplication:

Textovertheimage:WegetthisresultusingtheaddText(constMat&img,constString&text,Pointorg,constQtFont&font)function.ThisfunctionallowsyoutoselecttheoriginpointforthedisplayedtextwithafontpreviouslycreatedwiththefontQt(constString&nameFont,intpointSize=-1,Scalarcolor=Scalar::all(0),intweight=QT_FONT_NORMAL,int

style=QT_STYLE_NORMAL,intspacing=0)function.IntheshowUIexample,thisfunctionisusedtoputtextovertheimagewhenthepushbuttonisclickedon,callingtheaddTextfunctioninsidethecallbackfunction.Textonthestatusbar:UsingthedisplayStatusBar(constString&winname,constString&text,intdelayms=0)function,wedisplaytextinthestatusbarforanumberofmillisecondsgivenbythelastargument(delayms).IntheshowUIexample,thisfunctionisused(inthecallbackfunctions)todisplayaninformativetextwhenthebuttonsandtrackbarofthepropertieswindowchangetheirstate.Textoverlaidontheimage:UsingthedisplayOverlay(constString&winname,constString&text,intdelayms=0)function,wedisplaytextoverlaidontheimageforanumberofmillisecondsgivenbythelastargument.IntheshowUIexample,thisfunctionisused(inthecallbackfunction)todisplayinformativetextwhenthemainwindowtrackbarchangesitsvalue.

SummaryInthischapter,yougotaquickreviewofthemainpurposeoftheOpenCVlibraryanditsmodules.Youlearnedthefoundationsofhowtocompile,install,andusethelibraryinyourlocalsystemtodevelopC++OpenCVapplicationswithQtsupport.Todevelopyourownsoftware,weexplainedhowtostartwiththefreeQtCreatorIDEandtheGNUcompilerkit.

Tostartwith,fullcodeexampleswereprovidedinthechapter.Theseexamplesshowedyouhowtoreadandwriteimagesandvideo.Finally,thechaptergaveyouanexampleofdisplayingsomeeasy-to-implementuserinterfacecapabilitiesinOpenCVprograms,suchastrackbars,buttons,puttingtextonimages,drawingshapes,andsoon.

Thenextchapterwillbedevotedtoestablishingthemainimageprocessingtoolsandtasksthatwillsetthebasisfortheremainingchapters.

Chapter2.EstablishingImageProcessingToolsThischapterdescribesthemaindatastructuresandbasicproceduresthatwillbeusedinsubsequentchapters:

ImagetypesPixelaccessBasicoperationswithimagesHistograms

Thesearesomeofthemostfrequentoperationsthatwewillhavetoperformonimages.Mostofthefunctionalitycoveredhereisinthecoremoduleofthelibrary.

BasicdatatypesThefundamentaldatatypeinOpenCVisMat,asitisusedtostoreimages.Basically,animageisstoredasaheaderplusamemoryzonecontainingthepixeldata.Imageshaveanumberofchannels.Grayscaleimageshaveasinglechannel,whilecolorimagestypicallyhavethreeforthered,green,andbluecomponents(althoughOpenCVstorestheminareverseorder,thatisblue,green,andred).Afourthtransparency(alpha)channelcanalsobeused.Thenumberofchannelsforanimgimagecanberetrievedwithimg.channels().

Eachpixelinanimageisstoredusinganumberofbits.Thisiscalledtheimagedepth.Forgrayscaleimages,pixelsarecommonlystoredin8bits,thusallowing256graylevels(integervalues0to255).Forcolorimages,eachpixelisstoredinthreebytes,onepercolorchannel.Insomeoperations,itwillbenecessarytostorepixelsinafloating-pointformat.Theimagedepthcanbeobtainedwithimg.depth(),andthevaluesreturnedare:

CV_8U,8-bitunsignedintegers(0..255)CV_8S,8-bitsignedintegers(-128..127)CV_16U,16-bitunsignedintegers(0..65,535)CV_16S,16-bitsignedintegers(-32,768..32,767)CV_32S,32-bitsignedintegers(-2,147,483,648..2,147,483,647)CV_32F,32-bitfloating-pointnumbersCV_64F,64-bitfloating-pointnumbers

NotethatthemostcommonimagedepthwillbeCV_8Uforbothgrayscaleandcolorimages.ItispossibletoconvertfromonedepthtoanotherusingtheconvertTomethod:

Matimg=imread("lena.png",IMREAD_GRAYSCALE);

Matfp;

img.convertTo(fp,CV_32F);

Itiscommontoperformanoperationonfloating-pointimages(thatis,pixelvaluesaretheresultofamathematicaloperation).Ifweuseimshow()todisplaythisimage,wewillnotseeanythingmeaningful.Inthiscase,wehavetoconvertpixelstotheintegerrange0..255.TheconvertTofunctionimplementsalineartransformationandhastwoadditionalparameters,alphaandbeta,whichrepresentascalefactorandadeltavaluetoadd,respectively.Thismeansthateachpixelpisconvertedwith:

newp=alpha*p+beta

Thiscanbeusedtodisplayfloatingpointimagesproperly.AssumingthattheimgimagehasmandMminimumandmaximumvalues(refertothefollowingcodetoseehowtoobtainthesevalues),wewouldusethis:

Matm1=Mat(100,100,CV_32FC1);

randu(m1,0,1e6);//randomvaluesbetween0and1e6

imshow("original",m1);

doubleminRange,MaxRange;

PointmLoc,MLoc;

minMaxLoc(m1,&minRange,&MaxRange,&mLoc,&MLoc);

Matimg1;

m1.convertTo(img1,CV_8U,255.0/(MaxRange-minRange),-255.0/minRange);

imshow("result",img1);

Thiscodemapstherangeoftheresultimagevaluestotherange0-255.Thefollowingimageshowsyoutheresultofrunningthecode:

TheresultofconvertTo(notethattheimageontheleft-handsideisdisplayedaswhite)

Theimagesizecanbeobtainedwiththerowsandcolsattributes.Thereisalsoasizeattributethatretrievesboth:

MatSizes=img.size;

intr=l[0];

intc=l[1];

Apartfromtheimageitself,otherdatatypesareverycommon;refertothefollowingtable:

Type Typekeyword Example

(Small)vector VecAB,whereAcanbe2,3,4,5or6,Bcanbeb,s,i,f,ord Vec3brgb;

rgb[0]=255;

(Upto4)scalars Scalar

Scalara;

a[0]=0;

a[1]=0;

Point PointAB,whereAcanbe2or3andBcanbei,f,ordPoint3dp;

p.x=0;

p.y=0;

p.z=0;

Size Size

Sizes;

s.width=30;

s.height=40;

Rectangle Rect

Rectr;

r.x=r.y=0;

r.width=r.height=100;

Someofthesetypeshaveadditionaloperations.Forexample,wecancheckwhetherapointliesinsidearectangle:

p.inside(r)

Thepandrargumentsare(two-dimensional)pointandrectangle,respectively.Notethatinanycase,theprecedingtableisnotexhaustive;OpenCVprovidesmanymoresupport

structureswithassociatedmethods.

Pixel-levelaccessToprocessimages,wehavetoknowhowtoaccesseachpixelindependently.OpenCVprovidesanumberofwaystodothis.Inthissection,wecovertwomethods;thefirstoneiseasyfortheprogrammer,whilethesecondoneismoreefficient.

Thefirstmethodusestheat<>templatefunction.Inordertouseit,wehavetospecifythetypeofmatrixcells,suchasinthisshortexample:

Matsrc1=imread("lena.jpg",IMREAD_GRAYSCALE);

ucharpixel1=src1.at<uchar>(0,0);

cout<<"Valueofpixel(0,0):"<<(unsignedint)pixel1<<endl;

Matsrc2=imread("lena.jpg",IMREAD_COLOR);

Vec3bpixel2=src2.at<Vec3b>(0,0);

cout<<"Bcomponentofpixel(0,0):"<<(unsignedint)pixel2[0]<<endl;

Theexamplereadsanimageinbothgrayscaleandcolorandaccessesthefirstpixelat(0,0).Inthefirstcase,thepixeltypeisunsignedchar(thatis,uchar).Inthesecondcase,whentheimageisreadinfullcolor,wehavetousetheVec3btype,whichreferstoatripletofunsignedchars.Ofcourse,theat<>functioncanalsoappearontheleft-handsideofanassignment,thatis,tochangethevalueofapixel.

Thefollowingisanothershortexampleinwhichafloating-pointmatrixisinitializedtothePivalueusingthismethod:

MatM(200,200,CV_64F);

for(inti=0;i<M.rows;i++)

for(intj=0;j<M.cols;j++)

M.at<double>(i,j)=CV_PI;

Notethattheat<>methodisnotveryefficientasithastocalculatetheexactmemorypositionfromthepixelrowandcolumn.Thiscanbeverytimeconsumingwhenweprocessthewholeimagepixelbypixel.Thesecondmethodusestheptrfunction,whichreturnsapointertoaspecificimagerow.Thefollowingsnippetobtainsthepixelvalueofeachpixelinacolorimage:

ucharR,G,B;

for(inti=0;i<src2.rows;i++)

{

Vec3b*pixrow=src2.ptr<Vec3b>(i);

for(intj=0;j<src2.cols;j++)

{

B=pixrow[j][0];

G=pixrow[j][1];

R=pixrow[j][2];

}

}

Intheexampleabove,ptrisusedtogetapointertothefirstpixelineachrow.Usingthispointer,wecannowaccesseachcolumnintheinnermostloop.

MeasuringthetimeProcessingimagestakestime(comparablymuchmorethanthetimeittakestoprocess1Ddata).Often,processingtimeisthecrucialfactorthatdecideswhetherasolutionispracticalornot.OpenCVprovidestwofunctionstomeasuretheelapsedtime:getTickCount()andgetTickFrequency().You’llusethemlikethis:

doublet0=(double)getTickCount();

//yourstuffhere…

elapsed=((double)getTickCount()–t0)/getTickFrequency();

Here,elapsedisinseconds.

CommonoperationswithimagesThefollowingtablesummarizesthemosttypicaloperationswithimages:

Operation Codeexamples

Setmatrixvalues img.setTo(0);//for1-channelimg

img.setTo(Scalar(B,G,R);//3-channelimg

MATLAB-stylematrixinitializationMatm1=Mat::eye(100,100,CV_64F);

Matm3=Mat::zeros(100,100,CV_8UC1);

Matm2=Mat::ones(100,100,CV_8UC1)*255;

Randominitialization Matm1=Mat(100,100,CV_8UC1);

randu(m1,0,255);

Createacopyofthematrix Matimg1=img.clone();

Createacopyofthematrix(withthemask)

img.copy(img1,mask);

Referenceasubmatrix(thedataisnotcopied)

Matimg1=img(Range(r1,r2),Range(c1,c2));

Imagecrop Rectroi(r1,c2,width,height);

Matimg1=img(roi).clone();//datacopied

Resizeimage resize(img,imag1,Size(),0.5,0.5);//decimatebyafactor

of2

Flipimageflip(imgsrc,imgdst,code);

//code=0=>verticalflipping

//code>0=>horizontalflipping

//code<0=>vertical&horizontalflipping

SplitchannelsMatchannel[3];

split(img,channel);

imshow("B",channel[0]);//showblue

Mergechannels merge(channel,img);

Countnonzeropixels intnz=countNonZero(img);

Minimumandmaximum doublem,M;

PointmLoc,MLoc;minMaxLoc(img,&m,&M,&mLoc,&MLoc);

ThemeanpixelvalueScalarm,stdd;

meanStdDev(img,m,stdd);

uintmean_pxl=mean.val[0];

Checkwhethertheimagedataisnull If(img.empty())

cout<<"couldn'tloadimage";

ArithmeticoperationsArithmeticoperatorsareoverloaded.ThismeansthatwecanoperateonMatimageslikewecaninthisexample:

imgblend=0.2*img1+0.8*img2;

InOpenCV,theresultvalueofanoperationissubjecttotheso-calledsaturationarithmetic.Thismeansthatthefinalvalueisactuallythenearestintegerinthe0..255range.

Bitwiseoperationsbitwise_and(),bitwise_or(),bitwise_xor(),andbitwise_not()canbeveryusefulwhenworkingwithmasks.Masksarebinaryimagesthatindicatethepixelsinwhichanoperationistobeperformed(insteadofthewholeimage).Thefollowingbitwise_andexampleshowsyouhowtousetheANDoperationtocroppartofanimage:

#include<opencv2/opencv.hpp>

usingnamespacecv;

usingnamespacestd;

intmain()

{

Matimg1=imread("lena.png",IMREAD_GRAYSCALE);

if(img1.empty())

{

cout<<"Cannotloadimage!"<<endl;

return-1;

}

imshow("Original",img1);//Original

//Createmaskimage

Matmask(img1.rows,img1.cols,CV_8UC1,Scalar(0,0,0));

circle(mask,Point(img1.rows/2,img1.cols/2),150,255,-1);

imshow("Mask",mask);

//performAND

Matr;

bitwise_and(img1,mask,r);

//filloutsidewithwhite

constucharwhite=255;

for(inti=0;i<r.rows;i++)

for(intj=0;j<r.cols;j++)

if(!mask.at<uchar>(i,j))

r.at<uchar>(i,j)=white;

imshow("Result",r);

waitKey(0);

return0;

}

Afterreadinganddisplayingtheinputimage,wecreateamaskbydrawingafilledwhitecircle.IntheANDoperation,thismaskisused.Thelogicaloperationisonlyappliedinthosepixelsinwhichthemaskvalueisnotzero;otherpixelsarenotaffected.Finally,inthisexample,wefilltheouterpartoftheresultimage(thatis,outsidethecircle)withwhite.Thisisdoneusingoneofthepixelaccessmethodsexplainedpreviously.Seetheresultingimagesinthefollowingscreenshot:

Theresultofthebitwise_andexample

Next,anothercoolexampleisshowninwhichweestimatethevalueofPi.Let’sconsiderasquareanditsenclosedcircle:

Theirareasaregivenby:

Fromthis,wehave:

Let’sassumethatwehaveasquareimageofunknownsidelengthandanenclosedcircle.Wecanestimatetheareaoftheenclosedcirclebypaintingmanypixelsinrandompositionswithintheimageandcountingthosethatfallinsidetheenclosedcircle.Ontheotherhand,theareaofthesquareisestimatedasthetotalnumberofpixelspainted.ThiswouldallowyoutoestimatethevalueofPiusingthepreviousequation.

Thefollowingalgorithmsimulatesthis:

1. Onablacksquareimage,paintasolidwhiteenclosedcircle.2. Onanotherblacksquareimage(samedimensions),paintalargenumberofpixelsat

randompositions.3. PerformanANDoperationbetweenthetwoimagesandcountnonzeropixelsinthe

resultingimage.4. EstimatePiusingtheequation.

ThefollowingisthecodefortheestimatePiexample:

#include<opencv2/opencv.hpp>

usingnamespacecv;

usingnamespacestd;

intmain()

{

constintside=100;

constintnpixels=8000;

inti,j;

Mats1=Mat::zeros(side,side,CV_8UC1);

Mats2=s1.clone();

circle(s1,Point(side/2,side/2),side/2,255,-1);

imshow("s1",s1);

for(intk=0;k<npixels;k++)

{

i=rand()%side;

j=rand()%side;

s2.at<uchar>(i,j)=255;

}

Matr;

bitwise_and(s1,s2,r);

imshow("s2",s2);

imshow("r",r);

intAcircle=countNonZero(r);

intAsquare=countNonZero(s2);

floatPi=4*(float)Acircle/Asquare;

cout<<"EstimatedvalueofPi:"<<Pi<<endl;

waitKey();

return0;

}

Theprogramfollowstheprecedingalgorithmexactly.NotethatweusethecountNonZerofunctiontocountnonzero(white,inthiscase)pixels.Fornpixels=8000,theestimatewas3.125.Thelargerthevalueofnpixels,thebettertheestimation.

TheoutputoftheestimatePiexample

DatapersistenceApartfromthespecificfunctionstoreadandwriteimagesandvideo,inOpenCV,thereisamoregenericwaytosave/loadthedata.Thisisreferredtoasdatapersistence:thevalueofobjectsandvariablesintheprogramcanberecorded(serialized)onthedisk.Thiscanbeveryusefultosaveresultsandloadtheconfigurationdata.ThemainclassistheaptlynamedFileStorage,whichrepresentsafileonadisk.DataisactuallystoredinXMLorYAMLformats.

Thesearethestepsinvolvedwhenwritingdata:

1. CalltheFileStorageconstructor,passingafilenameandaflagwiththeFileStorage::WRITEvalue.Thedataformatisdefinedbythefileextension(thatis,.xml,.yml,or.yaml).

2. Usethe<<operatortowritedatatothefile.Dataistypicallywrittenasstring-valuepairs.

3. Closethefileusingthereleasemethod.

Readingdatarequiresthatyoufollowthesesteps:

1. CalltheFileStorageconstructor,passingafilenameandaflagwiththeFileStorage::READvalue.

2. Usethe[]or>>operatortoreaddatafromthefile.3. Closethefileusingthereleasemethod.

Thefollowingexampleusesdatapersistencetosaveandloadtrackbarvalues.

#include<opencv2/opencv.hpp>

usingnamespacecv;

usingnamespacestd;

Matimg1;

voidtb1_Callback(intvalue,void*)

{

Mattemp=img1+value;

imshow("main_win",temp);

}

intmain()

{

img1=imread("lena.png",IMREAD_GRAYSCALE);

if(img1.empty())

{

cout<<"Cannotloadimage!"<<endl;

return-1;

}

inttb1_value=0;

//loadtrackbarvalue

FileStoragefs1("config.xml",FileStorage::READ);

tb1_value=fs1["tb1_value"];//method1

fs1["tb1_value"]>>tb1_value;//method2

fs1.release();

//createtrackbar

namedWindow("main_win");

createTrackbar("brightness","main_win",&tb1_value,

255,tb1_Callback);

tb1_Callback(tb1_value,NULL);

waitKey();

//savetrackbarvalueuponexiting

FileStoragefs2("config.xml",FileStorage::WRITE);

fs2<<"tb1_value"<<tb1_value;

fs2.release();

return0;

}

TipWhenOpenCVhasbeencompiledwithQtsupport,windowpropertiescanbesaved,includingtrackbarvalues,withthesaveWindowParameters()function.

Oncethetrackbarisusedtocontrolanintegervalue,itissimplyaddedtotheoriginalimage,makingitbrighter.Thisvalueisreadwhentheprogramstarts(thevaluewillbe0thefirsttime)andsavedwhentheprogramexitsnormally.Notethattwoequivalentmethodsareshowntoreadthevalueofthetb1_valuevariable.Thecontentsoftheconfig.xmlfileare:

<?xmlversion="1.0"?>

<opencv_storage>

<tb1_value>112</tb1_value>

</opencv_storage>

HistogramsOncetheimagehasbeendefinedwithadatatypeandweareabletoaccessitsgraylevelvalues,thatis,thepixels,wemaywanttoobtainaprobabilitydensityfunctionofthedifferentgraylevels,whichiscalledthehistogram.Theimagehistogramrepresentsthefrequencyofoccurrenceofthevariousgraylevelsintheimage.Thehistogramcanbemodeledsothattheimagemaychangeitscontrastlevels.Thisisknownashistogramequalization.Histogrammodelingisapowerfultechniqueforimageenhancementbymeansofcontrastvariation.Theequalizationallowsforimageareasoflowercontrasttogainahighercontrast.Thefollowingimageshowsyouanexampleofanequalizedimageanditshistogram:

Anexampleofanequalizedimagehistogram

InOpenCV,theimagehistogramcanbecalculatedwiththevoidcalcHistfunctionandhistogramequalizationisperformedwiththevoidequalizeHistfunction.

Theimagehistogramcalculationisdefinedwithtenparameters:voidcalcHist(constMat*images,intnimages,constint*channels,InputArraymask,OutputArrayhist,intdims,constint*histSize,constfloat**ranges,booluniform=true,andboolaccumulate=false).

constMat*images:Thefirstparameteristheaddressofthefirstimagefromacollection.Thiscanbeusedtoprocessabatchofimages.intnimages:Thesecondparameteristhenumberofsourceimages.constint*channels:Thethirdinputparameteristhelistofthechannelsusedtocomputethehistogram.Thenumberofchannelsgoesfrom0to2.InputArraymask:Thisisanoptionalmasktoindicatetheimagepixelscountedinthehistogram.OutputArrayhist:Thefifthparameteristheoutputhistogram.intdims:Thisparameterallowsyoutoindicatethedimensionofthehistogram.constint*histSize:Thisparameteristhearrayofhistogramsizesineachdimension.constfloat**ranges:Thisparameteristhearrayofthedimsarraysofthehistogrambinboundariesineachdimension.booluniform=true:Bydefault,theBooleanvalueistrue.Itindicatesthatthehistogramisuniform.boolaccumulate=false:Bydefault,theBooleanvalueisfalse.Itindicatesthatthe

histogramisnonaccumulative.

Thehistogramequalizationrequiresonlytwoparameters,voidequalizeHist(InputArraysrc,OutputArraydst).Thefirstparameteristheinputimageandthesecondoneistheoutputimagewiththehistogramequalized.

Itispossibletocalculatethehistogramofmorethanoneinputimage.Thisallowsyoutocompareimagehistogramsandcalculatethejointhistogramofseveralimages.Thecomparisonoftwoimagehistograms,histImage1andhistImage2,canbeperformedwiththevoidcompareHist(InputArrayhistImage1,InputArrayhistImage2,method)function.TheMethodmetricistheusedtocomputethematchingbetweenbothhistograms.TherearefourmetricsimplementedinOpenCV,thatis,correlation(CV_COMP_CORREL),chi-square(CV_COMP_CHISQR),intersectionorminimumdistance(CV_COMP_INTERSECT),andBhattacharyyadistance(CV_COMP_BHATTACHARYYA).

Itispossibletocalculatethehistogramofmorethanonechannelofthesamecolorimage.Thisispossiblethankstothethirdparameter.

Thefollowingsectionsshowyoutwoexamplecodesforcolorhistogramcalculation(ColourImageEqualizeHist)andcomparisonColourImageComparison.InColourImageEqualizeHist,itisalsoshownhowtocalculatethehistogramequalizationaswellasthe2Dhistogramfortwochannels,thatis,hue(H)andsaturation(S),intheColourImageComparisonexample.

TheexamplecodeThefollowingColourImageEqualizeHistexampleshowsyouhowtoequalizeacolorimageanddisplaythehistogramofeachchannelatthesametime.ThehistogramcalculationofeachcolorchannelintheRGBimageisdonewiththehistogramcalculation(InputArrayImagesrc,OutputArrayhistoImage)function.Tothisend,thecolorimageissplitintothechannels:R,G,andB.Thehistogramequalizationisalsoappliedtoeachchannelthatisthenmergedtoformtheequalizedcolorimage:

#include"opencv2/highgui/highgui.hpp"

#include"opencv2/imgproc/imgproc.hpp"

#include<iostream>

#include<stdio.h>

usingnamespacecv;

usingnamespacestd;

voidhistogramcalculation(constMat&Image,Mat&histoImage)

{

inthistSize=255;

//Settheranges(forB,G,R))

floatrange[]={0,256};

constfloat*histRange={range};

booluniform=true;boolaccumulate=false;

Matb_hist,g_hist,r_hist;

vector<Mat>bgr_planes;

split(Image,bgr_planes);

//Computethehistograms:

calcHist(&bgr_planes[0],1,0,Mat(),b_hist,1,&histSize,&histRange,

uniform,accumulate);

calcHist(&bgr_planes[1],1,0,Mat(),g_hist,1,&histSize,&histRange,

uniform,accumulate);

calcHist(&bgr_planes[2],1,0,Mat(),r_hist,1,&histSize,&histRange,

uniform,accumulate);

//DrawthehistogramsforB,GandR

inthist_w=512;inthist_h=400;

intbin_w=cvRound((double)hist_w/histSize);

MathistImage(hist_h,hist_w,CV_8UC3,Scalar(0,0,0));

//Normalizetheresultto[0,histImage.rows]

normalize(b_hist,b_hist,0,histImage.rows,NORM_MINMAX,-1,Mat());

normalize(g_hist,g_hist,0,histImage.rows,NORM_MINMAX,-1,Mat());

normalize(r_hist,r_hist,0,histImage.rows,NORM_MINMAX,-1,Mat());

//Drawforeachchannel

for(inti=1;i<histSize;i++){

line(histImage,Point(bin_w*(i-1),hist_h-cvRound(b_hist.at<float>(i-

1))),Point(bin_w*(i),hist_h-cvRound(b_hist.at<float>(i))),Scalar(

255,0,0),2,8,0);

line(histImage,Point(bin_w*(i-1),hist_h-cvRound(g_hist.at<float>(i-

1))),Point(bin_w*(i),hist_h-cvRound(g_hist.at<float>(i))),Scalar(

0,255,0),2,8,0);

line(histImage,Point(bin_w*(i-1),hist_h-cvRound(r_hist.at<float>(i-

1))),Point(bin_w*(i),hist_h-cvRound(r_hist.at<float>(i))),Scalar(

0,0,255),2,8,0);

}

histoImage=histImage;

}

intmain(int,char*argv[])

{

Matsrc,imageq;

MathistImage;

//Readoriginalimage

src=imread("fruits.jpg");

if(!src.data)

{printf("Errorimagen\n");exit(1);}

//Separatetheimagein3places(B,GandR)

vector<Mat>bgr_planes;

split(src,bgr_planes);

//Displayresults

imshow("Sourceimage",src);

//Calculatethehistogramtoeachchannelofthesourceimage

histogramcalculation(src,histImage);

//Displaythehistogramforeachcolourchannel

imshow("ColourImageHistogram",histImage);

//EqualizedImage

//ApplyHistogramEqualizationtoeachchannel

equalizeHist(bgr_planes[0],bgr_planes[0]);

equalizeHist(bgr_planes[1],bgr_planes[1]);

equalizeHist(bgr_planes[2],bgr_planes[2]);

//Mergetheequalizedimagechannelsintotheequalizedimage

merge(bgr_planes,imageq);

//DisplayEqualizedImage

imshow("EqualizedImage",imageq);

//Calculatethehistogramtoeachchanneloftheequalizedimage

histogramcalculation(imageq,histImage);

//DisplaytheHistogramoftheEqualizedImage

imshow("EqualizedColourImageHistogram",histImage);

//Waituntiluserexitstheprogram

waitKey();

return0;

}

Theexamplecreatesfourwindowswith:

Thesourceimage:Thisisshowninthefollowingfigureintheupper-leftcorner.Theequalizedcolorimage:Thisisshowninthefollowingfigureintheupper-rightcorner.Thehistogramofthreechannels:Here,R=Read,G=GreenandB=Blue,forthesourceimage.Thisisshowninthefollowingfigureinthelower-leftcorner.ThehistogramofRGBchannelfortheequalizedimage:Thisisshowninnextfigureinthelower-rightcorner.ThefigureshowsyouhowthemostfrequentintensityvaluesforR,G,andBhavebeenstretchedoutduetotheequalizationprocess.

Thefollowingfigureshowsyoutheresultsofthealgorithm:

TheexamplecodeThefollowingColourImageComparisonexampleshowsyouhowtocalculatea2Dhistogramcomposedoftwochannelsfromthesamecolorimage.Theexamplecodealsoperformsacomparisonbetweentheoriginalimageandtheequalizedimagebymeansofhistogrammatching.Themetricsusedforthematchingarethefourmetricsthathavebeenmentionedpreviously,thatis,Correlation,Chi-Square,Minimumdistance,andBhattacharyyadistance.The2DhistogramcalculationoftheHandScolorchannelisdonewiththehistogram2Dcalculation(InputArrayImagesrc,OutputArrayhisto2D)function.Toperformthehistogramcomparison,thenormalized1DhistogramhasbeencalculatedfortheRGBimage.Inordertocomparethehistogram,theyhavebeennormalized.ThisisdoneinhistogramRGcalculation(InputArrayImagesrc,OutputArrayhisto):

voidhistogram2Dcalculation(constMat&src,Mat&histo2D)

{

Mathsv;

cvtColor(src,hsv,CV_BGR2HSV);

//Quantizethehueto30-255levels

//andthesaturationto32-255levels

inthbins=255,sbins=255;

inthistSize[]={hbins,sbins};

//huevariesfrom0to179,seecvtColor

floathranges[]={0,180};

//saturationvariesfrom0(black-gray-white)to

//255(purespectrumcolor)

floatsranges[]={0,256};

constfloat*ranges[]={hranges,sranges};

MatNDhist,hist2;

//wecomputethehistogramfromthe0-thand1-stchannels

intchannels[]={0,1};

calcHist(&hsv,1,channels,Mat(),hist,1,histSize,ranges,true,

false);

doublemaxVal=0;

minMaxLoc(hist,0,&maxVal,0,0);

intscale=1;

MathistImg=Mat::zeros(sbins*scale,hbins*scale,CV_8UC3);

for(inth=0;h<hbins;h++)

for(ints=0;s<sbins;s++)

{

floatbinVal=hist.at<float>(h,s);

intintensity=cvRound(binVal*255/maxVal);

rectangle(histImg,Point(h*scale,s*scale),

Point((h+1)*scale-1,(s+1)*scale-1),

Scalar::all(intensity),

CV_FILLED);

}

histo2D=histImg;

}

voidhistogramRGcalculation(constMat&src,Mat&histoRG)

{

//Using50binsforredand60forgreen

intr_bins=50;intg_bins=60;

inthistSize[]={r_bins,g_bins};

//redvariesfrom0to255,greenfrom0to255

floatr_ranges[]={0,255};

floatg_ranges[]={0,255};

constfloat*ranges[]={r_ranges,g_ranges};

//Usetheo-thand1-stchannels

intchannels[]={0,1};

//Histograms

MatNDhist_base;

//CalculatethehistogramsfortheHSVimages

calcHist(&src,1,channels,Mat(),hist_base,2,histSize,ranges,

true,false);

normalize(hist_base,hist_base,0,1,NORM_MINMAX,-1,Mat());

histoRG=hist_base;

}

intmain(intargc,char*argv[])

{

Matsrc,imageq;

MathistImg,histImgeq;

MathistHSorg,histHSeq;

//Readoriginalimage

src=imread("fruits.jpg");

if(!src.data)

{printf("Errorimagen\n");exit(1);}

//Separatetheimagein3places(B,GandR)

vector<Mat>bgr_planes;

split(src,bgr_planes);

//Displayresults

namedWindow("Sourceimage",0);

imshow("Sourceimage",src);

//Calculatethehistogramofthesourceimage

histogram2Dcalculation(src,histImg);

//Displaythehistogramforeachcolourchannel

imshow("H-SHistogram",histImg);

//EqualizedImage

//ApplyHistogramEqualizationtoeachchannel

equalizeHist(bgr_planes[0],bgr_planes[0]);

equalizeHist(bgr_planes[1],bgr_planes[1]);

equalizeHist(bgr_planes[2],bgr_planes[2]);

//Mergetheequalizedimagechannelsintotheequalizedimage

merge(bgr_planes,imageq);

//DisplayEqualizedImage

namedWindow("EqualizedImage",0);

imshow("EqualizedImage",imageq);

//Calculatethe2DhistogramforHandSchannels

histogram2Dcalculation(imageq,histImgeq);

//Displaythe2DHistogram

imshow("H-SHistogramEqualized",histImgeq);

histogramRGcalculation(src,histHSorg);

histogramRGcalculation(imageq,histHSeq);

///Applythehistogramcomparisonmethods

for(inti=0;i<4;i++)

{

intcompare_method=i;

doubleorig_orig=compareHist(histHSorg,histHSorg,compare_method

);

doubleorig_equ=compareHist(histHSorg,histHSeq,compare_method);

printf("Method[%d]Original-Original,Original-Equalized:%f,%f

\n",i,orig_orig,orig_equ);

}

printf("Done\n");

waitKey();

}

Theexamplecreatesfourwindowswiththesourceimage,theequalizedcolorimageandthe2DhistogramforHandSchannelsforbothimagestheoriginal,andtheequalizedimage.ThealgorithmalsodisplaysthefournumericalmatchingparametersobtainedfromthecomparisonoftheoriginalRGBimagehistogramwithitselfandwiththeequalizedRGBimage.Forthecorrelationandintersectionmethods,thehigherthemetric,themoreaccuratethematch.Forthechi-squareandBhattacharyyadistance,thelesstheresult,thebetterthematch.ThefollowingfigureshowsyoutheoutputoftheColourImageComparisonalgorithm:

Finally,youcanrefertoChapter3,CorrectingandEnhancingImages,aswellastheexampleswithintocoveressentialaspectsofthisbroadtopic,suchasimageenhancementbymeansofhistogrammodeling.

NoteFormoreinformation,refertoOpenCVEssentials,DenizO.,FernándezM.M.,VállezN.,BuenoG.,SerranoI.,Patón.A.,SalidoJ.byPacktPublishing,https://www.packtpub.com/application-development/opencv-essentials.

SummaryThischaptercoveredandestablishedthebasisofapplyingimageprocessingmethodsusedincomputervision.Imageprocessingisoftenthefirststeptofurthercomputervisionapplications,andtherefore,ithasbeencoveredhere:basicdatatypes,pixellevelaccess,commonoperationswithimages,arithmeticoperations,datapersistence,andhistograms.

YoucanalsorefertoChapter3,CorrectingandEnhancingImages,ofOpenCVEssentialsbyPacktPublishingtocoverfurtheressentialaspectsofthisbroadtopic,suchasimageenhancement,imagerestorationbymeansoffiltering,andgeometricalcorrection.

Thenextchapterwillcoverfurtheraspectsofimageprocessingtocorrectandenhanceimagesbymeansofsmoothing,sharpening,imageresolutionanalysis,morphologicalandgeometricaltransforms,inpainting,anddenoising.

Chapter3.CorrectingandEnhancingImagesThischapterpresentsmethodsforimageenhancementandcorrection.Sometimes,itisnecessarytoreducethenoiseinanimageoremphasizeorsuppresscertaindetailsinit.Theseproceduresareusuallycarriedoutbymodifyingpixelvalues,performingsomeoperationsonthem,orontheirlocalneighborhoodaswell.Bydefinition,image-enhancementoperationsareusedtoimproveimportantimagedetails.Enhancementoperationsincludenoisereduction,smoothing,andedgeenhancement.Ontheotherhand,imagecorrectionattemptstorestoreadamagedimage.InOpenCV,theimgprocmodulecontainsfunctionsforimageprocessing.

Inthischapter,wewillcover:

Imagefiltering.Thisincludesimagesmoothing,imagesharpening,andworkingwithimagepyramids.Applyingmorphologicaloperations,suchasdilation,erosion,opening,orclosing.Geometricaltransformations(affineandperspectivetransformations).Inpainting,whichisusedtoreconstructdamagedpartsofimages.Denoising,whichisnecessarytoreducetheimagenoiseproducedbytheimage-capturedevice.

ImagefilteringImagefilteringisaprocesstomodifyorenhanceimages.Emphasizingcertainfeaturesorremovingothersinanimageareexamplesofimagefiltering.Filteringisaneighborhoodoperation.Theneighborhoodisasetofpixelsaroundaselectedone.Imagefilteringdeterminestheoutputvalueofacertainpixellocatedataposition(x,y)byperformingsomeoperationswiththevaluesofthepixelsinitsneighborhood.

OpenCVprovidesseveralfilteringfunctionsforcommonimage-processingoperations,suchassmoothingorsharpening.

SmoothingSmoothing,alsocalledblurring,isanimage-processingoperationthatisfrequentlyusedtoreducenoise,amongotherpurposes.Asmoothingoperationisperformedbyapplyinglinearfilterstotheimage.Then,thepixelvaluesoftheoutputatpositions(xi,yj)arecomputedasaweightedsumoftheinputpixelvaluesatpositions(xi,yj)andtheirneighborhoods.Theweightsforthepixelsinthelinearoperationareusuallystoredinamatrixcalledkernel.Therefore,afiltercouldberepresentedasaslidingwindowofcoefficients.

Therepresentationofthepixelneighborhood

LetKbethekernelandIandOtheinputandoutputimages,respectively.Then,eachoutputpixelvalueat(i,j)iscalculatedasfollows:

Median,Gaussian,andbilateralarethemostusedOpenCVsmoothingfilters.Medianfilteringisverygoodtogetridofsalt-and-pepperorspecklenoise,whileGaussianisamuchbetterpreprocessingstepforedgedetection.Ontheotherhand,bilateralfilteringisagoodtechniquetosmoothanimagewhilerespectingstrongedges.

ThefunctionsincludedinOpenCVforthispurposeare:

voidboxFilter(InputArraysrc,OutputArraydst,intddepth,Sizeksize,

Pointanchor=Point(-1,-1),boolnormalize=true,intborderType=

BORDER_DEFAULT):Thisisaboxfilterwhosekernelcoefficientsareequal.Withnormalize=true,eachoutputpixelvalueisthemeanofitskernelneighborswithallcoefficientsequalto1/n,wheren=thenumberofelements.Withnormalize=false,allcoefficientsareequalto1.Thesrcargumentistheinputimage,whilethefilteredimageisstoredindst.Theddepthparameterindicatestheoutputimagedepththatis-1tousethesamedepthastheinputimage.Thekernelsizeisindicatedinksize.Theanchorpointindicatesthepositionoftheso-calledanchorpixel.The(-1,-1)defaultvaluemeansthattheanchorisatthecenterofthekernel.Finally,theborder-typetreatmentisindicatedintheborderTypeparameter.voidGaussianBlur(InputArraysrc,OutputArraydst,Sizeksize,double

sigmaX,doublesigmaY=0,intborderType=BORDER_DEFAULT):ThisisdonebyconvolvingeachpointinthesrcinputarraywithaGaussiankerneltoproducethe

dstoutput.ThesigmaXandsigmaYparametersindicatetheGaussiankernelstandarddeviationinXandYdirections.IfsigmaYiszero,itissettobeequaltosigmaX,andifbothareequaltozero,theyarecomputedusingthewidthandheightgiveninksize.

NoteConvolutionisdefinedastheintegraloftheproductoftwofunctionsinwhichoneofthemispreviouslyreversedandshifted.

voidmedianBlur(InputArraysrc,OutputArraydst,intksize):Thisrunsthrougheachelementoftheimageandreplaceseachpixelwiththemedianofitsneighboringpixels.voidbilateralFilter(InputArraysrc,OutputArraydst,intd,double

sigmaColor,doublesigmaSpace,intborderType=BORDER_DEFAULT):ThisisanalogoustotheGaussianfilterconsideringtheneighboringpixelswithweightsassignedtoeachofthembuthavingtwocomponentsoneachweight,whichisthesameusedbytheGaussianfilter,andanotheronethattakesintoaccountthedifferenceinintensitybetweentheneighboringandevaluatedpixels.Thisfunctionneedsthediameterofthepixelneighborhoodasparameterd,andsigmaColorsigmaSpacevalues.AlargervalueofthesigmaColorparametermeansthatfarthercolorswithinthepixelneighborhoodwillbemixedtogether,generatinglargerareasofsemi-equalcolors,whereasalargervalueofthesigmaSpaceparametermeansthatfartherpixelswillinfluenceeachotheraslongastheircolorsarecloseenough.voidblur(InputArraysrc,OutputArraydst,Sizeksize,Point

anchor=Point(-1,-1),intborderType=BORDER_DEFAULT):Thisblursanimageusingthenormalizedboxfilter.ItisequivalenttousingboxFilterwithnormalize=true.Thekernelusedinthisfunctionis:

NoteThegetGaussianKernelandgetGaborKernelfunctionscanbeusedinOpenCVtogeneratecustomkernels,whichcanthenbepassedontofilter2D.

Inallcases,itisnecessarytoextrapolatethevaluesofthenon-existentpixelsoutsidetheimageboundary.OpenCVenablesthespecificationoftheextrapolationmethodinmostofthefilterfunctions.Thesemethodsare:

BORDER_REPLICATE:Thisrepeatsthelastknownpixelvalue:aaaaaa|abcdefgh|hhhhhhhBORDER_REFLECT:Thisreflectstheimageborder:fedcba|abcdefgh|hgfedcb

BORDER_REFLECT_101:Thisreflectstheimageborderwithoutduplicatingthelastpixeloftheborder:gfedcb|abcdefgh|gfedcbaBORDER_WRAP:Thisappendsthevalueoftheoppositeborder:cdefgh|abcdefgh|abcdefgBORDER_CONSTANT:Thisestablishesaconstantoverthenewborder:kkkkkk|abcdefgh|kkkkkk

TheexamplecodeThefollowingSmoothexampleshowsyouhowtoloadanimageandapplyGaussianandmedianblurringtoitthroughGaussianBlurandmedianBlurfunctions:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applythefilters

Matdst,dst2;

GaussianBlur(src,dst,Size(9,9),0,0);

medianBlur(src,dst2,9);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("GAUSSIANBLUR",WINDOW_AUTOSIZE);

imshow("GAUSSIANBLUR",dst);

namedWindow("MEDIANBLUR",WINDOW_AUTOSIZE);

imshow("MEDIANBLUR",dst2);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

OriginalandblurredimagesfromGaussianandMedianblurringtransformations

SharpeningSharpeningfiltersareusedtohighlightbordersandotherfinedetailswithinimages.Theyarebasedonfirst-andsecond-orderderivatives.Thefirstderivativeofanimagecomputesanapproximationoftheimageintensitygradient,whereasthesecondderivativeisdefinedasthedivergenceofthisgradient.Sincedigitalimageprocessingdealswithdiscretequantities(pixelvalues),thediscreteversionsofthefirstandsecondderivativesareusedforsharpening.

First-orderderivativesproducethickerimageedgesandarewidelyusedforedge-extractionpurposes.However,second-orderderivativesareusedforimageenhancementduetotheirbetterresponsetofinedetails.TwopopularoperatorsusedtoobtainderivativesaretheSobelandtheLaplacian.

TheSobeloperatorcomputesthefirstimagederivativeofanimage,I,through:

TheSobelgradientmagnitudecanbeobtainedbycombiningthegradientapproximationsinthetwodirections,asfollows:

Ontheotherhand,thediscreteLaplacianofanimagecanbegivenasaconvolutionwiththefollowingkernel:

ThefunctionsincludedinOpenCVforthispurposeare:

voidSobel(InputArraysrc,OutputArraydst,intddepth,intdx,intdy,

intksize=3,doublescale=1,doubledelta=0,intborderType=

BORDER_DEFAULT):Thiscalculatesthefirst,second,third,ormixed-imagederivativeswiththeSobeloperatorfromanimageinsrc.Theddepthparameterindicatestheoutputimagedepth,thatis,-1tousethesamedepthastheinputimage.Thekernelsizeisindicatedinksizeandthedesiredderivativeordersindxanddy.Ascalefactorforthecomputedderivativevalescanbeestablishedwithscale.Finally,theborder-typetreatmentisindicatedintheborderTypeparameterandadeltavaluecanbeaddedtotheresultsbeforestoringthemindst.voidScharr(InputArraysrc,OutputArraydst,intddepth,intdx,int

dy,doublescale=1,doubledelta=0,intborderType=BORDER_DEFAULT

):Thiscalculatesamoreaccuratederivativeforakernelofsize3x3.Scharr(src,dst,ddepth,dx,dy,scale,delta,borderType)isequivalenttoSobel(src,dst,ddepth,dx,dy,CV_SCHARR,scale,delta,borderType).voidLaplacian(InputArraysrc,OutputArraydst,intddepth,intksize=

1,doublescale=1,doubledelta=0,intborderType=BORDER_DEFAULT):ThiscalculatestheLaplacianofanimage.AlltheparametersareequivalenttotheonesfromtheSobelandScharrfunctionsexceptforksize.Whenksize>1,itcalculatestheLaplacianoftheimageinsrcbyaddingupthesecondxandyderivativescalculatedusingSobel.Whenksize=1,theLaplacianiscalculatedbyfilteringtheimagewitha3x3kernelthatcontains-4forthecenter,0forthecorners,and1fortherestofthecoefficients.

NotegetDerivKernelscanbeusedinOpenCVtogeneratecustomderivativekernels,whichcanthenbepassedontosepFilter2D.

TheexamplecodeThefollowingSharpenexampleshowsyouhowtocomputeSobelandLaplacianderivativesfromanimagethroughSobelandLaplacianfunctions.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//ApplySobelandLaplacian

Matdst,dst2;

Sobel(src,dst,-1,1,1);

Laplacian(src,dst2,-1);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("SOBEL",WINDOW_AUTOSIZE);

imshow("SOBEL",dst);

namedWindow("LAPLACIAN",WINDOW_AUTOSIZE);

imshow("LAPLACIAN",dst2);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

ContoursobtainedbySobelandLaplacianderivatives

WorkingwithimagepyramidsOnsomeoccasions,workingwithafixedimagesizeisnotpossible,andwewillneedtheoriginalimageatdifferentresolutions.Forexample,inobject-detectionproblems,examiningthewholeimagetryingtofindtheobjecttakestoomuchtime.Inthiscase,searchingforobjectsbystartingatsmallerresolutionsismoreefficient.Thistypeofimagesetiscalledpyramidormipmapduetothesimilaritywiththepyramidstructuretypeifimagesareorganizedfromthelargesttothesmallestfromthebottomtothetop.

TheGaussianpyramid

Therearetwokindsofimagepyramids:GaussianpyramidsandLaplacianpyramids.

GaussianpyramidsGaussianpyramidsarecreatedbyalternatelyremovingrowsandcolumnsinthelowerlevelandthenobtainingthevalueofthehigher-levelpixelbyapplyingaGaussianfilterusingtheneighborhoodfromtheunderlyinglevel.Aftereachpyramidstep,theimagereducesitswidthandheightbyhalfanditsareaisaquarterofthepreviouslevelimagearea.InOpenCV,GaussianpyramidscanbecomputedusingthepyrDown,pyrUp,andbuildPyramidfunctions:

voidpyrDown(InputArraysrc,OutputArraydst,constSize&dstsize=

Size(),intborderType=BORDER_DEFAULT):Thissubsamplesandblursansrcimage,savingtheresultindst.ThesizeoftheoutputimageiscomputedasSize((src.cols+1)/2,(src.rows+1)/2)whenitisnotsetwiththedstsizeparameter.voidpyrUp(InputArraysrc,OutputArraydst,constSize&dstsize=

Size(),intborderType=BORDER_DEFAULT):ThiscomputestheoppositeprocessofpyrDown.voidbuildPyramid(InputArraysrc,OutputArrayOfArraysdst,int

maxlevel,intborderType=BORDER_DEFAULT):ThisbuildsaGaussianpyramidforanimagestoredinsrc,obtainingmaxlevelnewimagesandstoringtheminthedstarrayfollowingtheoriginalimagethatisstoredindst[0].Thus,dststoresmaxlevel+1imagesasaresult.

Pyramidsarealsousedforsegmentation.OpenCVprovidesafunctiontocomputemean-shiftpyramidsbasedonthefirststepofthemean-shiftsegmentationalgorithm:

voidpyrMeanShiftFiltering(InputArraysrc,OutputArraydst,doublesp,

doublesr,intmaxLevel=1,TermCriteriatermcrit=TermCriteria

(TermCriteria::MAX_ITER+TermCriteria::EPS,5,1)):Thisimplementsthefilteringstageofthemean-shiftsegmentation,obtaininganimage,dst,withcolorgradientsandfine-graintextureflattened.Thespandsrparametersindicatethespatialwindowandthecolorwindowradii.

NoteMoreinformationaboutthemean-shiftsegmentationcanbefoundathttp://docs.opencv.org/trunk/doc/py_tutorials/py_video/py_meanshift/py_meanshift.html?highlight=meanshift.

LaplacianpyramidsLaplacianpyramidsdonothaveaspecificfunctionimplementationinOpenCV,buttheyareformedfromtheGaussianpyramids.Laplacianpyramidscanbeseenasborderimageswheremostofitselementsarezeros.TheithlevelintheLaplacianpyramidisthedifferencebetweentheithlevelintheGaussianpyramidandtheexpandedversionoftheith+1levelintheGaussianpyramid.

TheexamplecodeThefollowingPyramidsexampleshowsyouhowtoobtaintwolevelsfromaGaussianpyramidthroughthepyrDownfunctionandtheoppositeoperationthroughpyrUp.NoticethattheoriginalimagecannotbeobtainedafterusingpyrUp:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//ApplytwotimespyrDown

Matdst,dst2;

pyrDown(src,dst);

pyrDown(dst,dst2);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("1stPYRDOWN",WINDOW_AUTOSIZE);

imshow("1stPYRDOWN",dst);

namedWindow("2stPYRDOWN",WINDOW_AUTOSIZE);

imshow("2stPYRDOWN",dst2);

//ApplytwotimespyrUp

pyrUp(dst2,dst);

pyrUp(dst,src);

//Showtheresults

namedWindow("NEWORIGINAL",WINDOW_AUTOSIZE);

imshow("NEWORIGINAL",dst2);

namedWindow("1stPYRUP",WINDOW_AUTOSIZE);

imshow("1stPYRUP",dst);

namedWindow("2stPYRUP",WINDOW_AUTOSIZE);

imshow("2stPYRUP",src);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalandtwolevelsoftheGaussianpyramid

MorphologicaloperationsMorphologicaloperationsprocessimagesaccordingtoshapes.Theyapplyadefined“structuringelement”toanimage,obtaininganewimagewherethepixelsatpositions(xi,yj)arecomputedbycomparingtheinputpixelvaluesatpositions(xi,yj)andtheirneighborhoods.Dependingonthestructuringelementselected,amorphologicaloperationismoresensitivetoonespecificshapeortheother.

Thetwobasicmorphologicaloperationsaredilationanderosion.Dilationaddspixelsfromthebackgroundtotheboundariesoftheobjectsinanimage,whileerosionremovespixels.Hereiswherethestructuringelementistakenintoaccounttoselectthepixelsthataretobeaddedordeleted.Indilation,thevalueoftheoutputpixelisthemaximumofallthepixelsintheneighborhood.Usingerosion,thevalueoftheoutputpixelistheminimumvalueofallthepixelsintheneighborhood.

Anexampleofdilationanderosion

Otherimage-processingoperationscanbedefinedbycombiningdilationanderosion,suchastheopeningandclosingoperations,andthemorphologicalgradient.Theopeningoperationisdefinedaserosion,followedbydilation,whileclosingisitsreverseoperation—dilationfollowedbyerosion.Therefore,openingremovessmallobjectsfromanimagewhilepreservingthelargeronesandclosingisusedtoremovesmallholeswhilepreservingthelargeronesinamannersimilartoopening.Themorphologicalgradientisdefinedasthedifferencebetweenthedilationandtheerosionofanimage.Furthermore,twomoreoperationsaredefinedusingopeningandclosing:top-hatandblack-hatoperations.Theyaredefinedasthedifferencebetweenthesourceimageanditsopeninginthecaseoftophatandthedifferencebetweentheclosingofanimageandthesourceimageinthecaseofblackhat.Alltheoperationsareappliedwiththesamestructuringelement.

InOpenCV,itispossibletoapplydilation,erosion,opening,andclosingthroughthefollowingfunctions:

voiddilate(InputArraysrc,OutputArraydst,InputArraykernel,Point

anchor=Point(-1,-1),intiterations=1,intborderType=

BORDER_CONSTANT,constScalar&borderValue=

morphologyDefaultBorderValue()):Thisdilatesanimagestoredinsrcusinga

specificstructuringelement,savingtheresultindst.Thekernelparameteristhestructuringelementused.Theanchorpointindicatesthepositionofanchorpixel.The(-1,-1)valuemeansthattheanchorisatthecenter.Theoperationcanbeappliedseveraltimesusingiterations.Theborder-typetreatmentisindicatedintheborderTypeparameterandisthesameasinotherfiltersfromprevioussections.Finally,aconstantisindicatedinborderValueiftheBORDER_CONSTANTbordertypeisused.voiderode(InputArraysrc,OutputArraydst,InputArraykernel,Point

anchor=Point(-1,-1),intiterations=1,intborderType=

BORDER_CONSTANT,constScalar&borderValue=

morphologyDefaultBorderValue()):Thiserodesanimageusingaspecificstructuringelement.Itsparametersarethesameasthatindilate.voidmorphologyEx(InputArraysrc,OutputArraydst,intop,InputArray

kernel,Pointanchor=Point(-1,-1),intiterations=1,intborderType

=BORDER_CONSTANT,constScalar&borderValue=

morphologyDefaultBorderValue()):Thisperformsadvancedmorphologicaloperationsdefinedusingtheopparameter.PossibleopvaluesareMORPH_OPEN,MORPH_CLOSE,MORPH_GRADIENT,MORPH_TOPHAT,andMORPH_BLACKHAT.MatgetStructuringElement(intshape,Sizeksize,Pointanchor=

Point(-1,-1)):Thisreturnsastructuringelementofthespecifiedsizeandshapeformorphologicaloperations.SupportedtypesareMORPH_RECT,MORPH_ELLIPSE,andMORPH_CROSS.

TheexamplecodeThefollowingMorphologicalexampleshowsyouhowtosegmentredcheckersinacheckerboard,applyingabinarythreshold(theinRangefunction)andthenrefiningtheresultswithdilationanderosionoperations(throughdilateanderodefunctions).Thestructureusedisacircleof15x15pixels.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

usingnamespacestd;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applythefilters

Matdst,dst2,dst3;

inRange(src,Scalar(0,0,100),Scalar(40,30,255),dst);

Matelement=getStructuringElement(MORPH_ELLIPSE,Size(15,15));

dilate(dst,dst2,element);

erode(dst2,dst3,element);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("SEGMENTED",WINDOW_AUTOSIZE);

imshow("SEGMENTED",dst);

namedWindow("DILATION",WINDOW_AUTOSIZE);

imshow("DILATION",dst2);

namedWindow("EROSION",WINDOW_AUTOSIZE);

imshow("EROSION",dst3);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Original,redcolorsegmentation,dilation,anderosion

LUTsLook-uptables(LUTs)areverycommonincustomfiltersinwhichtwopixelswiththesamevalueintheinputinvolvesthesamevalueintheoutputtoo.AnLUTtransformationassignsanewpixelvaluetoeachpixelintheinputimageaccordingtothevaluesgivenbyatable.Inthistable,theindexrepresentstheinputintensityvalueandthecontentofthecellgivenbytheindexrepresentsthecorrespondingoutputvalue.Asthetransformationisactuallycomputedforeachpossibleintensityvalue,thisresultsinareductioninthetimeneededtoapplythetransformationoveranimage(imagestypicallyhavemorepixelsthanthenumberofintensityvalues).

TheLUT(InputArraysrc,InputArraylut,OutputArraydst,intinterpolation=0)OpenCVfunctionappliesalook-uptabletransformationoveran8-bitsignedoransrcunsignedimage.Thus,thetablegiveninthelutparametercontains256elements.Thenumberofchannelsinlutiseither1orsrc.channels.Ifsrchasmorethanonechannelbutluthasasingleone,thesamelutchannelisappliedtoalltheimagechannels.

TheexamplecodeThefollowingLUTexampleshowsyouhowtodivide(bytwo)theintensityofthepixelsfromanimageusingalook-uptable.TheLUTneedstobeinitializedbeforeusingitwiththiscode:

uchar*M=(uchar*)malloc(256*sizeof(uchar));

for(inti=0;i<256;i++){

M[i]=i*0.5;//Theresultisroundedtoanintegervalue

}

Matlut(1,256,CV_8UC1,M);

AMatobjectiscreatedwhereeachcellcontainsthenewvalue.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//CreatetheLUT

uchar*M=(uchar*)malloc(256*sizeof(uchar));

for(inti=0;i<256;i++){

M[i]=i*0.5;

}

Matlut(1,256,CV_8UC1,M);

//ApplytheLUT

Matdst;

LUT(src,lut,dst);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("LUT",WINDOW_AUTOSIZE);

imshow("LUT",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalandLUT-transformedimages

GeometricaltransformationsGeometricaltransformationsdonotchangetheimagecontentbutinsteaddeformtheimagebydeformingtheirgrid.Inthiscase,outputimagepixelvaluesarecomputedbyfirstobtainingthecoordinatesoftheappropriateinputpixelsbyapplyingthecorrespondingmappingfunctionsandcopyingtheoriginalpixelvaluesfromtheobtainedpositionstothenewones:

Thistypeofoperationhastwoproblems:

Extrapolation:fx(x,y)andfy(x,y)couldobtainvaluesthatindicateapixeloutsidetheimageboundary.TheextrapolationmethodsusedingeometricaltransformationsarethesameastheonesusedinimagefilteringplusanotheronecalledBORDER_TRANSPARENT.Interpolation:fx(x,y)andfy(x,y)areusuallyfloating-pointnumbers.InOpenCV,itispossibletoselectbetweennearest-neighborandpolynomialinterpolationmethods.Nearest-neighborinterpolationconsistsofroundingthefloating-pointcoordinatetothenearestinteger.Thesupportedinterpolationmethodsare:

INTER_NEAREST:Thisisthenearest-neighborinterpolationexplainedpreviously.INTER_LINEAR:Thisisabilinearinterpolationmethod.Itisusedbydefault.INTER_AREA:Thisresamplesusingpixelarearelation.INTER_CUBIC:Thisisbicubicinterpolationmethodovera4x4pixelneighborhood.INTER_LANCZOS4:ThisistheLanczosinterpolationmethodoveran8x8pixelneighborhood.

ThegeometricaltransformationssupportedinOpenCVincludeaffine(scaling,translation,rotation,andsoon)andperspectivetransformations.

AffinetransformationAnaffinetransformationisageometrictransformationthatpreservesallpointsfromaninitiallineonalineafterapplyingit.Furthermore,distanceratiosfromeachofthesepointstotheendsofthelinesarealsopreserved.Ontheotherhand,affinetransformationsdon’tnecessarilypreserveanglesandlengths.

Geometrictransformationssuchasscaling,translation,rotation,skewing,andreflectionareallaffinetransformations.

ScalingScalinganimageisresizingitbyshrinkingorzooming.ThefunctioninOpenCVforthispurposeisvoidresize(InputArraysrc,OutputArraydst,Sizedsize,doublefx=0,doublefy=0,intinterpolation=INTER_LINEAR).Apartfromsrcanddst,theinputandoutputimages,ithassomeparameterstospecifythesizetowhichtheimageistoberescaled.Ifthenewimagesizeisspecifiedbysettingdsizetoavaluedifferentfrom0,thescaledfactorparameters,fxandfy,areboth0andfxandfyarecalculatedfromdsizeandtheoriginalsizeoftheinputimage.Iffxandfyaredifferentfrom0anddsizeequals0,dsizeiscalculatedfromtheotherparameters.Ascaleoperationcouldberepresentedbyitstransformationmatrix:

Here,sxandsyarethescalefactorsinthexandyaxis.

Theexamplecode

ThefollowingScaleexampleshowsyouhowtoscaleanimagethroughtheresizefunction.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applythescale

Matdst;

resize(src,dst,Size(0,0),0.5,0.5);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("SCALED",WINDOW_AUTOSIZE);

imshow("SCALED",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Originalandscaledimages;fxandfyareboth0.5

TranslationTranslationissimplymovingtheimagealongaspecificdirectionanddistance.Thus,atranslationcouldberepresentedbymeansofavector,(tx,ty),oritstransformationmatrix:

InOpenCV,itispossibletoapplytranslationsusingthevoidwarpAffine(InputArraysrc,OutputArraydst,InputArrayM,Sizedsize,intflags=INTER_LINEAR,

intborderMode=BORDER_CONSTANT,constScalar&borderValue=Scalar())

function.TheMparameteristhetransformationmatrixthatconvertssrcintodst.Theinterpolationmethodisspecifiedusingtheflagsparameter,whichalsosupportstheWARP_INVERSE_MAPvalue,whichmeansthatMistheinversetransformation.TheborderModeparameteristheextrapolationmethod,andborderValueisusedwhenborderModeisBORDER_CONSTANT.

Theexamplecode

TheTranslationexampleshowsyouhowtousethewarpAffinefunctiontotranslateanimage.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applytranslation

Matdst;

MatM=(Mat_<double>(2,3)<<1,0,200,0,1,150);

warpAffine(src,dst,M,src.size());

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("TRANSLATED",WINDOW_AUTOSIZE);

imshow("TRANSLATED",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Originalanddisplacedimages.Thehorizontaldisplacementis200andverticaldisplacementis150.

ImagerotationImagerotationinvolvesaspecificangle,θ.OpenCVsupportsscaledrotationsinaspecificlocationusingatransformationmatrixdefinedasfollows:

Here,xandyarethecoordinatesoftherotationpointandsfisthescalefactor.

RotationsareappliedliketranslationsbymeansofthewarpAffinefunctionbutusingtheMatgetRotationMatrix2D(Point2fcenter,doubleangle,doublescale)functiontocreatetherotationtransformationmatrix.TheMparameteristhetransformationmatrixthatconvertssrcintodst.Asthenamesoftheparametersindicate,centeristhecenterpointoftherotation,angleistherotationangle(inacounter-clockwisedirection),andscaleisthescalefactor.

Theexamplecode

ThefollowingRotateexampleshowsyouhowtousethewarpAffinefunctiontorotateanimage.A45-degreecenteredrotationmatrixisfirstlyobtainedbygetRotationMatrix2D(Point2f(src.cols/2,src.rows/2),45,1).Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applytherotation

Matdst;

MatM=getRotationMatrix2D(Point2f(src.cols/2,src.rows/2),45,1);

warpAffine(src,dst,M,src.size());

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("ROTATED",WINDOW_AUTOSIZE);

imshow("ROTATED",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Theoriginalimageandtheimageaftera45-degreecenteredrotationisapplied

SkewingAskewingtransformationdisplaceseachpointinafixeddirectionbyanamountproportionaltoitssigneddistancefromalinethatisparalleltothatdirection.Therefore,itwillusuallydistorttheshapeofageometricfigure,forexample,turningsquaresintonon-squareparallelogramsandcirclesintoellipses.However,askewingpreservestheareaofgeometricfigures,thealignment,andrelativedistancesofcollinearpoints.Askewingmappingisthemaindifferencebetweenuprightandslanted(oritalic)stylesofletters.

Skewingcanalsobedefinedbyitsangle,θ.

Theoriginalanditsrotated45degreesfromcenterimage

Usingtheskewingangle,thetransformationmatricesforhorizontalandverticalskewing

are:

Duetothesimilaritieswithprevioustransformations,thefunctionusedtoapplyskewingiswarpAffine.

TipOnmostoccasions,itwillbenecessarytoaddsomesizetotheoutputimageand/orapplytranslation(changingthelastcolumnonthesheartransformationmatrix)inordertodisplaytheoutputimagecompletelyandinacenteredmanner.

Theexamplecode

ThefollowingSkewexampleshowsyouhowtousethewarpAffinefunctiontoskewθ=π/3horizontallyinanimage.Theexamplecodeis:

#include"opencv2/opencv.hpp"

#include<math.h>

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applyskew

Matdst;

doublem=1/tan(M_PI/3);

MatM=(Mat_<double>(2,3)<<1,m,0,0,1,0);

warpAffine(src,dst,M,Size(src.cols+0.5*src.cols,src.rows));

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("SKEWED",WINDOW_AUTOSIZE);

imshow("SKEWED",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Theoriginalimageandtheimagewhenskewedhorizontally

ReflectionAsreflectionisdoneoverthexandyaxesbydefault,itisnecessarytoapplytranslation(thelastcolumnofthetransformationmatrix).Then,thereflectionmatrixis:

Here,txisthenumberofimagecolumnsandtyisthenumberofimagerows.

Aswithprevioustransformations,thefunctionusedtoapplyreflectioniswarpAffine.

NoteOtheraffinetransformationscanbeappliedusingthewarpAffinefunctionwiththeircorrespondingtransformationmatrices.

Theexamplecode

ThefollowingReflectexampleshowsyouanexampleofhorizontal,vertical,andcombinedreflectionofanimageusingthewarpAffinefunction.Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Applythereflections

Matdsth,dstv,dst;

MatMh=(Mat_<double>(2,3)<<-1,0,src.cols,0,1,0

MatMv=(Mat_<double>(2,3)<<1,0,0,0,-1,src.rows);

MatM=(Mat_<double>(2,3)<<-1,0,src.cols,0,-1,src.rows);

warpAffine(src,dsth,Mh,src.size());

warpAffine(src,dstv,Mv,src.size());

warpAffine(src,dst,M,src.size());

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("H-REFLECTION",WINDOW_AUTOSIZE);

imshow("H-REFLECTION",dsth);

namedWindow("V-REFLECTION",WINDOW_AUTOSIZE);

imshow("V-REFLECTION",dstv);

namedWindow("REFLECTION",WINDOW_AUTOSIZE);

imshow("REFLECTION",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputforthecode:

TheoriginalandrotatedimagesinX,Y,andbothaxes

PerspectivetransformationForperspectivetransformation,a3x3transformationmatrixisneeded,althoughtheworkisperformedovertwo-dimensionalimages.Straightlinesremainstraightintheoutputimage,butinthiscase,proportionschange.Findingthetransformationmatrixismorecomplexthanwithaffinetransformations.Whenworkingwithperspective,thecoordinatesoffourpointsoftheinputimagematrixandtheircorrespondingcoordinatesontheoutputimagematrixareusedtoperformthisoperation.

WiththesepointsandthegetPerspectiveTransformOpenCVfunction,itispossibletofindtheperspectivetransformationmatrix.Afterobtainingthematrix,warpPerspectiveisappliedtoobtaintheoutputoftheperspectivetransformation.Thetwofunctionsareexplainedindetailhere:

MatgetPerspectiveTransform(InputArraysrc,InputArraydst)andMatgetPerspectiveTransform(constPoint2fsrc[],constPoint2fdst[]):Thisreturnstheperspectivetransformationmatrixcalculatedfromsrcanddst.voidwarpPerspective(InputArraysrc,OutputArraydst,InputArrayM,

Sizedsize,intflags=INTER_LINEAR,intborderMode=BORDER_CONSTANT,

constScalar&borderValue=Scalar()):ThisappliesanMaffinetransformationtoansrcimage,obtainingthenewdstimage.Therestoftheparametersarethesameasinothergeometricaltransformationsdiscussed.

TheexamplecodeThefollowingPerspectiveexampleshowsyouanexampleofhowtochangetheperspectiveofanimageusingthewarpPerspectivefunction.Inthiscase,itisnecessarytoindicatethecoordinatesoffourpointsfromthefirstimageandanotherfourfromtheoutputtocalculatetheperspectivetransformationmatrixthroughgetPerspectiveTransform.Theselectedpointsare:

Point2fsrc_verts[4];

src_verts[2]=Point(195,140);

src_verts[3]=Point(410,120);

src_verts[1]=Point(220,750);

src_verts[0]=Point(400,750);

Point2fdst_verts[4];

dst_verts[2]=Point(160,100);

dst_verts[3]=Point(530,120);

dst_verts[1]=Point(220,750);

dst_verts[0]=Point(400,750);

Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

Matdst;

Point2fsrc_verts[4];

src_verts[2]=Point(195,140);

src_verts[3]=Point(410,120);

src_verts[1]=Point(220,750);

src_verts[0]=Point(400,750);

Point2fdst_verts[4];

dst_verts[2]=Point(160,100);

dst_verts[3]=Point(530,120);

dst_verts[1]=Point(220,750);

dst_verts[0]=Point(400,750);

//ObtainandApplytheperspectivetransformation

MatM=getPerspectiveTransform(src_verts,dst_verts);

warpPerspective(src,dst,M,src.size());

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("PERSPECTIVE",WINDOW_AUTOSIZE);

imshow("PERSPECTIVE",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Perspectiveresultswiththepointsmarkedintheoriginalimage

InpaintingInpaintingistheprocessofreconstructingdamagedpartsofimagesandvideos.Thisprocessisalsoknownasimageorvideointerpolation.Thebasicideaistosimulatetheprocessdonebyrestorerswithantiques.Nowadays,withthewideuseofdigitalcameras,inpaintinghasbecomeanautomaticprocessthatisusednotonlyforimagerestorationbydeletingscratches,butalsoforothertasks,suchasobjectortextremoval.

OpenCVsupportsaninpaintingalgorithmasofVersion2.4.Thefunctionforthispurposeis:

voidinpaint(InputArraysrc,InputArrayinpaintMask,OutputArraydst,

doubleinpaintRadius,intflags):Thisrestorestheareasindicatedwithnon-zerovaluesbytheinpaintMaskparameterinthesource(src)image.TheinpaintRadiusparameterindicatestheneighborhoodtobeusedbythealgorithmspecifiedbyflags.TwomethodscouldbeusedinOpenCV:

INPAINT_NS:ThisistheNavier-Stokes-basedmethodINPAINT_TELEA:ThisisthemethodproposedbyAlexandruTelea

Finally,therestoredimageisstoredindst.

NoteMoredetailsabouttheinpaintingalgorithmsusedinOpenCVcanbefoundathttp://www.ifp.illinois.edu/~yuhuang/inpainting.html

TipForvideoinpainting,considerthevideoasasequenceofimagesandapplythealgorithmoverallofthem.

TheexamplecodeThefollowinginpaintingexampleshowsyouhowtousetheinpaintfunctiontoinpainttheareasofanimagespecifiedinanimagemask.

Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Readthemaskfile

Matmask;

mask=imread(argv[2]);

cvtColor(mask,mask,COLOR_RGB2GRAY);

//Applytheinpaintingalgorithms

Matdst,dst2;

inpaint(src,mask,dst,10,INPAINT_TELEA);

inpaint(src,mask,dst2,10,INPAINT_NS);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("MASK",WINDOW_AUTOSIZE);

imshow("MASK",mask);

namedWindow("INPAINT_TELEA",WINDOW_AUTOSIZE);

imshow("INPAINT_TELEA",dst);

namedWindow("INPAINT_NS",WINDOW_AUTOSIZE);

imshow("INPAINT_NS",dst2);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Resultsofapplyinginpainting

NoteThefirstrowcontainstheoriginalimageandthemaskused.ThesecondrowcontainstheresultsfromtheinpaintingproposedbyTeleaontheleft-handsideandtheNavier-Stokes-basedmethodontheright-handside.

Gettingtheinpaintingmaskisnotaneasytask.Theinpainting2examplecodeshowsyouanexampleofhowwecanobtainthemaskfromthesourceimageusingbinarythresholdingthroughthreshold(mask,mask,235,255,THRESH_BINARY):

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Createthemask

Matmask;

cvtColor(src,mask,COLOR_RGB2GRAY);

threshold(mask,mask,235,255,THRESH_BINARY);

//Applytheinpaintingalgorithms

Matdst,dst2;

inpaint(src,mask,dst,10,INPAINT_TELEA);

inpaint(src,mask,dst2,10,INPAINT_NS);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("MASK",WINDOW_AUTOSIZE);

imshow("MASK",mask);

namedWindow("INPAINT_TELEA",WINDOW_AUTOSIZE);

imshow("INPAINT_TELEA",dst);

namedWindow("INPAINT_NS",WINDOW_AUTOSIZE);

imshow("INPAINT_NS",dst2);

waitKey();

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Resultsofapplyinginpaintingalgorithmswithoutknowingthemask

NoteThefirstrowcontainstheoriginalimageandtheextractedmask.ThesecondrowcontainstheresultsfromtheinpaintingproposedbyTeleaontheleft-handsideandtheNavier-Stokes-basedmethodontheright-handside.

Theresultsfromthisexampleshowyouthatobtainingaperfectmaskisnotalwayspossible.Someotherpartsoftheimage,suchasthebackgroundornoise,areincludedsometimes.However,theinpaintingresultsremainacceptableastheresultingimagesareclosetotheonesobtainedintheothercase.

DenoisingDenoisingornoisereductionistheprocessofremovingnoisefromsignalsobtainedfromanalogordigitaldevices.Thissectionfocusesitsattentiononreducingnoisefromdigitalimagesandvideos.

Althoughsmoothingandmedianfilteringaregoodoptionstodenoiseanimage,OpenCVprovidesotheralgorithmstoperformthistask.ThesearethenonlocalmeansandtheTVL1(TotalVariationL1)algorithms.Thebasicideaofthenonlocalmeansalgorithmistoreplacethecolorofapixelwithanaverageofthecolorsfromseveralimagesub-windowsthataresimilartotheonethatcomprisesthepixelneighborhood.Ontheotherhand,theTVL1variationaldenoisingmodel,whichisimplementedwiththeprimal-dualoptimizationalgorithm,considerstheimage-denoisingprocessavariationalproblem.

NoteMoreinformationaboutthenonlocalmeansandtheTVL1denoisingalgorithmscanbefoundathttp://www.ipol.im/pub/art/2011/bcm_nlmandhttp://znah.net/rof-and-tv-l1-denoising-with-primal-dual-algorithm.html,respectively.

OpenCVprovidesfourfunctionstodenoisecolorandgrayscaleimagesfollowingthenonlocalmeansapproach.FortheTVL1model,onefunctionisprovided.Thesefunctionsare:

voidfastNlMeansDenoising(InputArraysrc,OutputArraydst,floath=3,

inttemplateWindowSize=7,intsearchWindowSize=21):Thisdenoisesasinglegrayscaleimageloadedinsrc.ThetemplateWindowSizeandsearchWindowSizeparametersarethesizesinpixelsofthetemplatepatchthatisusedtocomputeweightsandthewindowthatisusedtocomputetheweightedaverageforthegivenpixel.Theseshouldbeoddandtheirrecommendedvaluesare7and21pixels,respectively.Thehparameterregulatestheeffectofthealgorithm.Largerhvaluesremovemorenoisedefectsbutwiththedrawbackofremovingmoreimagedetails.Theoutputisstoredindst.voidfastNlMeansDenoisingColored(InputArraysrc,OutputArraydst,float

h=3,floathForColorComponents=3,inttemplateWindowSize=7,int

searchWindowSize=21):Thisisamodificationofthepreviousfunctionforcoloredimages.ItconvertsthesrcimagetotheCIELABcolorspaceandthenseparatelydenoisestheLandABcomponentswiththefastNlMeansDenoisingfunction.voidfastNlMeansDenoisingMulti(InputArrayOfArrayssrcImgs,OutputArray

dst,intimgToDenoiseIndex,inttemporalWindowSize,floath=3,int

templateWindowSize=7,intsearchWindowSize=21):Thisusesanimagesequencetoobtainadenoisedimage.Twomoreparametersareneededinthiscase:imgToDenoiseIndexandtemporalWindowSize.ThevalueofimgToDenoiseIndexisthetargetimageindexinsrcImgstobedenoised.Finally,temporalWindowSizeisusedtoestablishthenumberofsurroundingimagestobeusedfordenoising.Thisshouldbeodd.

voidfastNlMeansDenoisingColoredMulti(InputArrayOfArrayssrcImgs,

OutputArraydst,intimgToDenoiseIndex,inttemporalWindowSize,floath

=3,floathForColorComponents=3,inttemplateWindowSize=7,int

searchWindowSize=21):ThisisbasedonthefastNlMeansDenoisingColoredandfastNlMeansDenoisingMultifunctions.Theparametersareexplainedintherestofthefunctions.voiddenoise_TVL1(conststd::vector<Mat>&observations,Mat&result,

doublelambda,intniters):Thisobtainsadenoisedimageinresultfromoneormorenoisyimagesstoredinobservations.Thelambdaandnitersparameterscontrolthestrengthandthenumberofiterationsofthealgorithm.

TheexamplecodeThefollowingdenoisingexampleshowsyouhowtouseoneofthedenoisingfunctionsfornoisereductionoveracoloredimage(fastNlMeansDenoisingColored).Astheexampleusesanimagewithoutnoise,somethingneedstobeadded.Forthispurpose,thefollowinglinesofcodeareused:

Matnoisy=src.clone();

Matnoise(src.size(),src.type());

randn(noise,0,50);

noisy+=noise;

AMatelementiscreatedwiththesamesizeandtypeoftheoriginalimagetostorenoisegeneratedbytherandnfunctiononit.Finally,thenoiseisaddedtotheclonedimagetoobtainthenoisyimage.

Theexamplecodeis:

#include"opencv2/opencv.hpp"

usingnamespacecv;

intmain(intargc,char**argv)

{

//Readthesourcefile

Matsrc;

src=imread(argv[1]);

//Addsomenoise

Matnoisy=src.clone();

Matnoise(src.size(),src.type());

randn(noise,0,50);

noisy+=noise;

//Applythedenoisingalgorithm

Matdst;

fastNlMeansDenoisingColored(noisy,dst,30,30,7,21);

//Showtheresults

namedWindow("ORIGINAL",WINDOW_AUTOSIZE);

imshow("ORIGINAL",src);

namedWindow("ORIGINALWITHNOISE",WINDOW_AUTOSIZE);

imshow("ORIGINALWITHNOISE",noisy);

namedWindow("DENOISED",WINDOW_AUTOSIZE);

imshow("DENOISED",dst);

waitKey();

return0;

}

Thefollowingfigureshowsyounoisyanddenoisedimagesfromexecutingthepreviouscode:

Resultsfromapplyingdenoising

SummaryInthischapter,weexplainedmethodsforimageenhancementandcorrection,includingnoisereduction,edgeenhancement,morphologicaloperations,geometricaltransformations,andtherestorationofdamagedimages.DifferentoptionshavebeenpresentedineachcasetoprovidethereaderwithalltheoptionsthatcanbeusedinOpenCV.

Thenextchapterwillcovercolorspacesandhowtoconvertthem.Inaddition,color-space-basedsegmentationandcolor-transfermethodswillbeexplained.

Chapter4.ProcessingColorColorisaperceptualresultcreatedinresponsetotheexcitationofourvisualsystembylightincidentupontheretinainthevisibleregionofthespectrum.Thecolorofanimagemaycontainagreatdealofinformation,whichcanbeusedforsimplifyingimageanalysis,objectidentification,andextractionbasedoncolor.Theseproceduresareusuallycarriedoutconsideringthepixelvaluesinthecolorspaceinwhichitisdefined.Inthischapter,thefollowingtopicswillbecovered:

ThecolorspacesusedinOpenCVandhowtoconvertanimagefromonecolormodeltoanotherAnexampleofhowtosegmentapictureconsideringthecolorspaceinwhichitisdefinedHowtotransfertheappearanceofanimagetoanotherusingthecolortransfermethod

ColorspacesThehumanvisualsystemisabletodistinguishhundredsofthousandsofcolors.Toobtainthisinformation,thehumanretinahasthreetypesofcolorphotoreceptorconecells,whichrespondtoincidentradiation.Becauseofthis,mosthumancolorperceptionscanbegeneratedwiththreenumericalcomponentscalledprimaries.

Tospecifyacolorintermsofthreeormoreparticularcharacteristics,thereareanumberofmethodscalledcolorspacesorcolormodels.Selectingbetweenthemtorepresentanimagedependsontheoperationstobeperformed,becausesomearemoreappropriateaccordingtotherequiredapplication.Forexample,insomecolorspacessuchasRGB,thebrightnessaffectsthethreechannels,afactthatcouldbeunfavorableforsomeimage-processingoperations.ThenextsectionexplainscolorspacesusedinOpenCVandhowtoconvertapicturefromonecolormodeltoanother.

Conversionbetweencolorspaces(cvtColor)Therearemorethan150color-spaceconversionmethodsavailableinOpenCV.ThefunctionprovidedbyOpenCVintheimgprocmoduleisvoidcvtColor(InputArraysrc,OutputArraydst,intcode,intdstCn=0).Theargumentsofthisfunctionare:

src:Thisisaninputimage8-bitunsigned,16-bitunsigned(CV_16UC),orsingle-precisionfloating-point.dst:Thisistheoutputimageofthesamesizeanddepthassrc.code:Thisisthecolorspaceconversioncode.ThestructureofthisparameterisCOLOR_SPACEsrc2SPACEdst.SomeexamplevaluesareCOLOR_BGR2GRAYandCOLOR_YCrCb2BGR.dstCn:Thisisthenumberofchannelsinthedestinationimage.Ifthisparameteris0oromitted,thenumberofthechannelsisderivedautomaticallyfromsrcandcode.

Examplesofthisfunctionwillbedescribedintheupcomingsections.

TipThecvtColorfunctioncanonlyconvertfromRGBtoanothercolorspaceorfromanothercolorspacetoRGB,soifthereaderwantstoconvertbetweentwocolorspacesotherthanRGB,afirstconversiontoRGBmustbedone.

VariouscolorspacesinOpenCVarediscussedintheupcomingsections.

RGBRGBisanadditivemodelinwhichanimageconsistsofthreeindependentimageplanesorchannels:red,green,andblue(andoptionally,afourthchannelforthetransparency,sometimescalledalphachannel).Tospecifyaparticularcolor,eachvalueindicatestheamountofeachofthecomponentspresentoneachpixel,withhighervaluescorrespondingtobrighterpixels.Thiscolorspaceiswidelyusedbecauseitcorrespondstothethreephotoreceptorsofthehumaneye.

NoteThedefaultcolorformatinOpenCVisoftenreferredtoasRGBbutitisactuallystoredasBGR(thechannelsarereversed).

Theexamplecode

ThefollowingBGRsplitexampleshowsyouhowtoloadanRGBimage,splittingandshowingeachparticularchannelingrayandinacolor.Thefirstpartofthecodeisusedtoloadandshowthepicture:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

vector<Mat>showSeparatedChannels(vector<Mat>channels);

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("BGR.png");

imshow("Picture",image);

Thenextpartofthecodesplitsthepictureintoeachchannelandshowsit:

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Bluechannel(gray)",WINDOW_AUTOSIZE);

imshow("Bluechannel(gray)",channels[0]);

namedWindow("Greenchannel(gray)",WINDOW_AUTOSIZE);

imshow("Greenchannel(gray)",channels[1]);

namedWindow("Redchannel(gray)",WINDOW_AUTOSIZE);

imshow("Redchannel(gray)",channels[2]);

//showchannelsinBGR

vector<Mat>separatedChannels=showSeparatedChannels(channels);

namedWindow("Bluechannel",WINDOW_AUTOSIZE);

imshow("Bluechannel",separatedChannels[0]);

namedWindow("Greenchannel",WINDOW_AUTOSIZE);

imshow("Greenchannel",separatedChannels[1]);

namedWindow("Redchannel",WINDOW_AUTOSIZE);

imshow("Redchannel",separatedChannels[2]);

waitKey(0);

return0;

}

Itisworthnotingtheuseofthevoidsplit(InputArraym,OutputArrayOfArraysmv)OpenCVfunctiontosplittheimageminitsthreechannelsandsaveitinavectorofMatcalledmv.Onthecontrary,thevoidmerge(InputArrayOfArraysmv,OutputArraydst)functionisusedtomergeallthemvchannelsinonedstimage.Furthermore,afunctiondenominatedasshowSeparatedChannelsisusedtocreatethreecolorimagesrepresentingeachofthechannels.Foreachchannel,thefunctiongeneratesvector<Mat>auxcomposedbythechannelitselfandtwoauxiliarychannelsorderedwithalltheirvaluessetto0,whichrepresenttheothertwochannelsofthecolormodel.Finally,theauxpictureismerged,generatinganimagewithonlyonechannelfulfilled.Thisfunctioncode,whichwillalsobeusedinotherexamplesofthischapter,isasfollows:

vector<Mat>showSeparatedChannels(vector<Mat>channels){

vector<Mat>separatedChannels;

//createeachimageforeachchannel

for(inti=0;i<3;i++){

Matzer=Mat::zeros(channels[0].rows,channels[0].cols,

channels[0].type());

vector<Mat>aux;

for(intj=0;j<3;j++){

if(j==i)

aux.push_back(channels[i]);

else

aux.push_back(zer);

}

Matchann;

merge(aux,chann);

separatedChannels.push_back(chann);

}

returnseparatedChannels;

}

Thefollowingfigureshowsyoutheoutputoftheexample:

TheoriginalRGBimageandchannelsplitting

GrayscaleIngrayscale,thevalueofeachpixelisrepresentedasasinglevaluecarryingonlytheintensityinformation,composinganimageexclusivelyformedfromdifferentshadesofgray.ThecolorspaceconversioncodetoconvertbetweenRGBandgrayscale(Y)inOpenCVusingcvtColorisCOLOR_BGR2GRAY,COLOR_RGB2GRAY,COLOR_GRAY2BGR,andCOLOR_GRAY2RGB.Thesetransformationsareinternallycomputedasfollows:

Note

Notefromtheprecedingformulathatitisnotpossibletoretrievecolorsdirectlyfromagrayscaleimage.

Examplecode

ThefollowingGrayexampleshowsyouhowtoconvertanRGBimagetograyscale,showingthetwopictures.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

namedWindow("Picture",WINDOW_AUTOSIZE);

imshow("Picture",image);

MatimageGray;

cvtColor(image,imageGray,COLOR_BGR2GRAY);

namedWindow("Graypicture",WINDOW_AUTOSIZE);

imshow("Graypicture",imageGray);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimageandthegrayscaleconversion

NoteThismethodtoconvertfromRGBtograyscalehasthedisadvantageoflosingthecontrastoftheoriginalimage.Chapter6,ComputationalPhotography,ofthisbookdescribesthedecolorizationprocess,whichmakesthissameconversionwhileovercomingthisissue.

CIEXYZTheCIEXYZsystemdescribescolorwithaluminancecomponentY,whichisrelatedtothebrightnesssensitivityofhumanvisionandtwoadditionalchannels,XandZ,standardizedbytheCommissionInternationaledeL’Éclairage(CIE)usingstatisticsfromexperimentswithseveralhumanobservers.Thiscolorspaceisusedtoreportcolorfrommeasuringinstruments,suchasacolorimeteroraspectrophotometer,anditisusefulwhenaconsistentcolorrepresentationacrossdifferentdevicesisneeded.Themainproblemofthiscolorspaceisthatthecolorsarescaledinanon-uniformmanner.ThisfactcausedtheCIEtoadopttheCIEL*a*b*andCIEL*u*v*colormodels.

ThecolorspaceconversioncodetoconvertbetweenRGBandCIEXYZinOpenCVusingcvtColorisCOLOR_BGR2XYZ,COLOR_RGB2XYZ,COLOR_XYZ2BGR,andCOLOR_XYZ2RGB.Thesetransformationsarecomputedasfollows:

Theexamplecode

ThefollowingCIExyzexampleshowsyouhowtoconvertanRGBimagetotheCIEXYZcolorspace,splittingandshowingeachparticularchannelingrayandinacolor.Thefirstpartofthecodeisusedtoloadandconvertthepicture:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

vector<Mat>showSeparatedChannels(vector<Mat>channels);

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoCIEXYZ

cvtColor(image,image,COLOR_BGR2XYZ);

ThenextpartofthecodesplitsthepictureineachoftheCIEXYZchannelsandshowsthem:

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Xchannel(gray)",WINDOW_AUTOSIZE);

imshow("Xchannel(gray)",channels[0]);

namedWindow("Ychannel(gray)",WINDOW_AUTOSIZE);

imshow("Ychannel(gray)",channels[1]);

namedWindow("Zchannel(gray)",WINDOW_AUTOSIZE);

imshow("Zchannel(gray)",channels[2]);

//showchannelsinBGR

vector<Mat>separatedChannels=showSeparatedChannels(channels);

for(inti=0;i<3;i++){

cvtColor(separatedChannels[i],separatedChannels[i],COLOR_XYZ2BGR);

}

namedWindow("Xchannel",WINDOW_AUTOSIZE);

imshow("Xchannel",separatedChannels[0]);

namedWindow("Ychannel",WINDOW_AUTOSIZE);

imshow("Ychannel",separatedChannels[1]);

namedWindow("Zchannel",WINDOW_AUTOSIZE);

imshow("Zchannel",separatedChannels[2]);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimageandtheCIEXYZchannelsplitting

YCrCbThiscolorspaceiswidelyusedinvideo-andimage-compressionschemes,anditisnotanabsolutecolorspacebecauseitisawaytoencodetheRGBcolorspace.TheYchannelrepresentsluminance,whileCrandCbrepresentred-difference(thedifferencebetweentheRchannelintheRGBcolorspaceandY)andblue-difference(thedifferencebetweentheBchannelintheRGBcolorspaceandY)chromacomponents,respectively.Itisusedwidelyinvideo-andimage-compressionschemes,suchasMPEGandJPEG.

ThecolorspaceconversioncodetoconvertbetweenRGBandYCrCbinOpenCVusingcvtColorisCOLOR_BGR2YCrCb,COLOR_RGB2YCrCb,COLOR_YCrCb2BGR,andCOLOR_YCrCb2RGB.Thesetransformationsarecomputedasfollows:

Then,takealookatthefollowing:

Theexamplecode

ThefollowingYCrCbcolorexampleshowsyouhowtoconvertanRGBimagetotheYCrCbcolorspace,splittingandshowingeachparticularchannelingrayandinacolor.Thefirstpartofthecodeisusedtoloadandconvertthepicture:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

vector<Mat>showSeparatedChannels(vector<Mat>channels);

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoYCrCb

cvtColor(image,image,COLOR_BGR2YCrCb);

ThenextpartofthecodesplitsthepictureintoeachoftheYCrCbchannelsandshowsthem:

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Ychannel(gray)",WINDOW_AUTOSIZE);

imshow("Ychannel(gray)",channels[0]);

namedWindow("Crchannel(gray)",WINDOW_AUTOSIZE);

imshow("Crchannel(gray)",channels[1]);

namedWindow("Cbchannel(gray)",WINDOW_AUTOSIZE);

imshow("Cbchannel(gray)",channels[2]);

//showchannelsinBGR

vector<Mat>separatedChannels=showSeparatedChannels(channels);

for(inti=0;i<3;i++){

cvtColor(separatedChannels[i],separatedChannels[i],COLOR_YCrCb2BGR);

}

namedWindow("Ychannel",WINDOW_AUTOSIZE);

imshow("Ychannel",separatedChannels[0]);

namedWindow("Crchannel",WINDOW_AUTOSIZE);

imshow("Crchannel",separatedChannels[1]);

namedWindow("Cbchannel",WINDOW_AUTOSIZE);

imshow("Cbchannel",separatedChannels[2]);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimageandtheYCrCbchannelsplitting

HSVTheHSVcolorspacebelongstothegroupoftheso-calledhue-orientedcolor-coordinatesystems.Thistypeofcolormodelcloselyemulatesmodelsofhumancolorperception.Whileinothercolormodels,suchasRGB,animageistreatedasanadditiveresultofthreebasecolors,thethreechannelsofHSVrepresenthue(Hgivesameasureofthespectralcompositionofacolor),saturation(Sgivestheproportionofpurelightofthedominantwavelength,whichindicateshowfaracolorisfromagrayofequalbrightness),andvalue(Vgivesthebrightnessrelativetothebrightnessofasimilarlyilluminatedwhitecolor)correspondingtotheintuitiveappealoftint,shade,andtone.HSViswidelyusedtomakeacomparisonofcolorsbecauseHisalmostindependentlightvariations.Thefollowingfigureshowsyouthiscolormodelrepresentingeachofthechannelsasapartofacylinder:

ThecolorspaceconversioncodetoconvertbetweenRGBandHSVinOpenCVusingcvtColorisCOLOR_BGR2HSV,COLOR_RGB2HSV,COLOR_HSV2BGR,andCOLOR_HSV2RGB.Inthiscase,itisworthnotingthatifthesrcimageformatis8-bitor16-bit,cvtColorfirstconvertsittoafloating-pointformat,scalingthevaluesbetween0and1.Afterthat,thetransformationsarecomputedasfollows:

IfH<0,thenH=H+360.Finally,thevaluesarereconvertedtothedestinationdatatype.

Theexamplecode

ThefollowingHSVcolorexampleshowsyouhowtoconvertanRGBimagetotheHSVcolorspace,splittingandshowingeachparticularchannelingrayscaleandtheHSVimage.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoHSV

cvtColor(image,image,COLOR_BGR2HSV);

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Hchannel(gray)",WINDOW_AUTOSIZE);

imshow("Hchannel(gray)",channels[0]);

namedWindow("Schannel(gray)",WINDOW_AUTOSIZE);

imshow("Schannel(gray)",channels[1]);

namedWindow("Vchannel(gray)",WINDOW_AUTOSIZE);

imshow("Vchannel(gray)",channels[2]);

namedWindow("HSVimage(allchannels)",WINDOW_AUTOSIZE);

imshow("HSVimage(allchannels)",image);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimage,HSVconversion,andchannelsplitting

NoteTheimshowfunctionofOpenCVassumesthatthecoloroftheimagetobeshownisRGB,

soitdisplaysitincorrectly.Ifyouhaveanimageinanothercolorspaceandyouwanttodisplayitcorrectly,youfirsthavetoconvertitbacktoRGB.

HLSTheHLScolorspacebelongstothegroupofhue-orientedcolor-coordinatesystems,suchastheHSVcolormodelexplainedpreviously.Thismodelwasdevelopedtospecifythevaluesofhue,lightness,andsaturationofacolorineachchannel.ThedifferencewithrespecttotheHSVcolormodelisthatthelightnessofapurecolordefinedbyHLSisequaltothelightnessofamediumgray,whilethebrightnessofapurecolordefinedbyHSVisequaltothebrightnessofwhite.

ThecolorspaceconversioncodetoconvertbetweenRGBandHLSinOpenCVusingcvtColorisCOLOR_BGR2HLS,COLOR_RGB2HLS,COLOR_HLS2BGR,andCOLOR_HLS2RGB.Inthiscase,aswithHSV,ifthesrcimageformatis8-bitor16-bit,cvtColorfirstconvertsittoafloating-pointformat,scalingthevaluesbetween0and1.Afterthat,thetransformationsarecomputedasfollows:

IfH<0,thenH=H+360.Finally,thevaluesarereconvertedtothedestinationdatatype.

Theexamplecode

ThefollowingHLScolorexampleshowsyouhowtoconvertanRGBimagetoHLScolor

space,splittingandshowingeachparticularchannelingrayscaleandtheHLSimage.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoHSV

cvtColor(image,image,COLOR_BGR2HLS);

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Hchannel(gray)",WINDOW_AUTOSIZE);

imshow("Hchannel(gray)",channels[0]);

namedWindow("Lchannel(gray)",WINDOW_AUTOSIZE);

imshow("Lchannel(gray)",channels[1]);

namedWindow("Schannel(gray)",WINDOW_AUTOSIZE);

imshow("Schannel(gray)",channels[2]);

namedWindow("HLSimage(allchannels)",WINDOW_AUTOSIZE);

imshow("HLSimage(allchannels)",image);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimage,HLSconversion,andchannelsplitting

CIEL*a*b*TheCIEL*a*b*colorspaceistheseconduniformcolorspacestandardizedbyCIEafterCIEL*u*v*,whichisderivedbasedontheCIEXYZspaceandwhitereferencepoint.Actually,itisthemostcompletecolorspacespecifiedbyCIEandwascreatedtobedevice-independent,liketheCYEXYZmodel,andtobeusedasareference.Itisabletodescribethecolorsvisibletothehumaneye.Thethreechannelsrepresentthelightnessofthecolor(L*),itspositionbetweenmagentaandgreen(a*),anditspositionbetweenyellowandblue(b*).

ThecolorspaceconversioncodetoconvertbetweenRGBandCIEL*a*b*inOpenCVusingcvtColorisCOLOR_BGR2Lab,COLOR_RGB2Lab,COLOR_Lab2BGR,andCOLOR_Lab2RGB.Theprocedureusedtocomputethesetransformationsisexplainedathttp://docs-hoffmann.de/cielab03022003.pdf.

Theexamplecode

ThefollowingCIElabexampleshowsyouhowtoconvertanRGBimagetotheCIEL*a*b*colorspace,splittingandshowingeachparticularchannelingrayscaleandtheCIEL*a*b*image.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoCIELab

cvtColor(image,image,COLOR_BGR2Lab);

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Lchannel(gray)",WINDOW_AUTOSIZE);

imshow("Lchannel(gray)",channels[0]);

namedWindow("achannel(gray)",WINDOW_AUTOSIZE);

imshow("achannel(gray)",channels[1]);

namedWindow("bchannel(gray)",WINDOW_AUTOSIZE);

imshow("bchannel(gray)",channels[2]);

namedWindow("CIELabimage(allchannels)",WINDOW_AUTOSIZE);

imshow("CIELabimage(allchannels)",image);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheoriginalRGBimage,CIEL*a*b*conversion,andchannelsplitting

CIEL*u*v*TheCIEL*u*v*colorspaceisthefirstuniformcolorspacestandardizedbyCIE.Itisasimple-to-computetransformationoftheCIEXYZspaceandwhitereferencepoint,whichattemptsperceptualuniformity.LiketheCIEL*a*b*colorspace,itwascreatedtobedevice-independent.Thethreechannelsrepresentthelightnessofthecolor(L*)anditspositionbetweengreenandred(u*),andthelastonerepresentsmostlyblueandpurpletypecolors(v*).Thiscolormodelisusefulforadditivemixturesoflightsduetoitslinearadditionproperties.

ThecolorspaceconversioncodetoconvertbetweenRGBandCIEL*u*v*inOpenCVusingcvtColorisCOLOR_BGR2Luv,COLOR_RGB2Luv,COLOR_Luv2BGR,andCOLOR_Luv2RGB.Theprocedureusedtocomputethesetransformationscanbeseenathttp://docs.opencv.org/trunk/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor

Theexamplecode

ThefollowingCIELuvcolorexampleshowsyouhowtoconvertanRGBimagetotheCIEL*u*v*colorspace,splittingandshowingeachparticularchannelingrayscaleandtheCIEL*u*v*image.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimage

Matimage=imread("Lovebird.jpg");

imshow("Picture",image);

//transformtoCIELuv

cvtColor(image,image,COLOR_BGR2Luv);

vector<Mat>channels;

split(image,channels);

//showchannelsingrayscale

namedWindow("Lchannel(gray)",WINDOW_AUTOSIZE);

imshow("Lchannel(gray)",channels[0]);

namedWindow("uchannel(gray)",WINDOW_AUTOSIZE);

imshow("uchannel(gray)",channels[1]);

namedWindow("vchannel(gray)",WINDOW_AUTOSIZE);

imshow("vchannel(gray)",channels[2]);

namedWindow("CIELuvimage(allchannels)",WINDOW_AUTOSIZE);

imshow("CIELuvimage(allchannels)",image);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

OriginalRGBimage,CIEL*u*v*conversion,andchannelsplitting

BayerTheBayerpixel-spacecompositioniswidelyusedindigitalcameraswithonlyoneimagesensor.Unlikecameraswiththreesensors(oneperRGBchannel,whichisabletoobtainalltheinformationofaparticularcomponent),inonesensorcamera,everypixeliscoveredbyadifferentcolorfilter,soeachpixelisonlymeasuredinthiscolor.ThemissingcolorinformationisextrapolatedfromitsneighborsusingtheBayermethod.Itallowsyoutogetcompletecolorpicturesfromasingleplanewherethepixelsareinterleavedasfollows:

ABayerpatternexample

NoteNotethattheBayerpatternisrepresentedbymoreGpixelsthanRandBbecausethehumaneyeismoresensitivetogreenfrequencies.

Thereareseveralmodificationsoftheshownpatternobtainedbyshiftingthepatternbyonepixelinanydirection.ThecolorspaceconversioncodetoconvertfromBayertoRGBinOpenCVisdefinedconsideringthecomponentsofthesecondandthirdcolumnsofthesecondrow(XandY,respectively)asCOLOR_BayerXY2BGR.Forexample,thepatternofthepreviouspicturehasa“BG”type,soitsconversioncodeisCOLOR_BayerBG2BGR.

Theexamplecode

ThefollowingBayerexampleshowsyouhowtoconvertapicturedefinedbyanRGBayerpatternobtainedfromanimagesensortoanRGBimage.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Showbayeredimageincolor

Matbayer_color=imread("Lovebird_bayer_color.jpg");

namedWindow("Bayerpictureincolor",WINDOW_AUTOSIZE);

imshow("Bayerpictureincolor",bayer_color);

//Loadbayeredimage

Matbayer=imread("Lovebird_bayer.jpg",CV_8UC3);

namedWindow("Bayerpicture",WINDOW_AUTOSIZE);

imshow("Bayerpicture",bayer);

MatimageColor;

cvtColor(bayer,imageColor,COLOR_BayerRG2BGR);

namedWindow("Colorpicture",WINDOW_AUTOSIZE);

imshow("Colorpicture",imageColor);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

TheBayerpatternimageandRGBconversion

Color-space-basedsegmentationEachcolorspacerepresentsanimageindicatingthenumericvalueofthespecificcharacteristicmeasuredbyeachchanneloneachpixel.Consideringthesecharacteristics,itispossibletopartitionthecolorspaceusinglinearboundaries(forexample,planesinthree-dimensionalspacesandonespaceperchannel),allowingyoutoclassifyeachpixelaccordingtothepartitionitliesin,thereforeallowingyoutoselectasetofpixelswithpredefinedcharacteristics.Thisideacanbeusedtosegmentobjectsofanimageweareinterestedin.

OpenCVprovidesthevoidinRange(InputArraysrc,InputArraylowerb,InputArrayupperb,OutputArraydst)functiontocheckwhetheranarrayofelementsliebetweentheelementsoftwootherarrays.Withrespecttocolor-space-basedsegmentation,thisfunctionallowsyoutoobtainthesetofpixelsofansrcimage,thevaluesofwhosechannelsliebetweenthelowerblowerboundariesandupperbupperboundaries,obtainingthedstimage.

NoteThelowerbandupperbboundariesareusuallydefinedasScalar(x,y,z),wherex,y,andzarethenumericalvaluesofeachchanneldefinedaslowerorupperboundaries.

Thefollowingexamplesshowyouhowtodetectpixelsthatcanbeconsideredtobeskin.Ithasbeenobservedthatskincolordiffersmoreinintensitythanchrominance,sonormally,theluminancecomponentisnotconsideredforskindetection.ThisfactmakesitdifficulttodetectskininapicturerepresentedinRGBbecauseofthedependenceofthiscolorspaceonluminance,soHSVandYCrCbcolormodelsareused.Itisworthnotingthatforthistypeofsegmentation,itisnecessarytoknoworobtainthevaluesoftheboundariesperchannel.

HSVsegmentationAsstatedpreviously,HSViswidelyusedtomakeacomparisonofcolorsbecauseHisalmostindependentoflightvariations,soitisusefulinskindetection.Inthisexample,thelowerboundaries(0,10,60)andtheupperboundaries(20,150,255)areselectedtodetecttheskinineachpixel.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain()

{

//Loadtheimage

Matimage=imread("hand.jpg");

namedWindow("Picture",WINDOW_AUTOSIZE);

imshow("Picture",image);

Mathsv;

cvtColor(image,hsv,COLOR_BGR2HSV);

//selectpixels

Matbw;

inRange(hsv,Scalar(0,10,60),Scalar(20,150,255),bw);

namedWindow("Selectedpixels",WINDOW_AUTOSIZE);

imshow("Selectedpixels",bw);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

SkindetectionusingtheHSVcolorspace

YCrCbsegmentationTheYCrCbcolorspacereducestheredundancyofRGBcolorchannelsandrepresentsthecolorwithindependentcomponents.Consideringthattheluminanceandchrominancecomponentsareseparated,thisspaceisagoodchoiceforskindetection.

ThefollowingexampleusestheYCrCbcolorspaceforskindetectionusingthelowerboundaries(0,133,77)andtheupperboundaries(255,173,177)ineachpixel.Theexamplecodeis:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain()

{

//Loadtheimage

Matimage=imread("hand.jpg");

namedWindow("Picture",WINDOW_AUTOSIZE);

imshow("Picture",image);

Matycrcb;

cvtColor(image,ycrcb,COLOR_BGR2HSV);

//selectpixels

Matbw;

inRange(ycrcb,Scalar(0,133,77),Scalar(255,173,177),bw);

namedWindow("Selectedpixels",WINDOW_AUTOSIZE);

imshow("Selectedpixels",bw);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

SkindetectionusingtheYCrCbcolorspace

NoteFormoreimage-segmentationmethods,refertoChapter4ofOpenCVEssentialsbyPacktPublishing.

ColortransferAnothertaskcommonlycarriedoutinimageprocessingistomodifythecolorofanimage,specificallyincaseswhereitisnecessarytoremoveadominantorundesirablecolorcast.Oneofthesemethodsiscalledcolortransfer,whichcarriesoutasetofcolorcorrectionsthatborrowonesourceimage’scolorcharacteristics,andtransfertheappearanceofthesourceimagetothetargetimage.

TheexamplecodeThefollowingcolorTransferexampleshowsyouhowtotransferthecolorfromasourcetotargetimage.ThismethodfirstconvertstheimagecolorspacetoCIEL*a*b*.Next,itsplitsthechannelsforsourceandtargetimages.Afterthat,itfitsthechanneldistributionfromoneimagetoanotherusingthemeanandthestandarddeviation.Finally,thechannelsaremergedbacktogetherandconvertedtoRGB.

NoteForfulltheoreticaldetailsofthetransformationusedintheexample,refertoColorTransferbetweenImagesathttp://www.cs.tau.ac.il/~turkel/imagepapers/ColorTransfer.pdf.

ThefirstpartofthecodeconvertstheimagetotheCIEL*a*b*colorspace,whilealsochangingthetypeoftheimagetoCV_32FC1:

#include<opencv2/opencv.hpp>

#include<opencv2/imgproc.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,constchar**argv)

{

//Loadtheimages

Matsrc=imread("clock_tower.jpg");

Mattar=imread("big_ben.jpg");

//ConverttoLabspaceandCV_32F1

Matsrc_lab,tar_lab;

cvtColor(src,src_lab,COLOR_BGR2Lab);

cvtColor(tar,tar_lab,COLOR_BGR2Lab);

src_lab.convertTo(src_lab,CV_32FC1);

tar_lab.convertTo(tar_lab,CV_32FC1);

Thenextpartofthecodeperformsthecolortransferasstatedpreviously:

//Findmeanandstdofeachchannelforeachimage

Matmean_src,mean_tar,stdd_src,stdd_tar;

meanStdDev(src_lab,mean_src,stdd_src);

meanStdDev(tar_lab,mean_tar,stdd_src);

//Splitintoindividualchannels

vector<Mat>src_chan,tar_chan;

split(src_lab,src_chan);

split(tar_lab,tar_chan);

//Foreachchannelcalculatethecolordistribution

for(inti=0;i<3;i++){

tar_chan[i]-=mean_tar.at<double>(i);

tar_chan[i]*=(stdd_src.at<double>(i)/stdd_src.at<double>(i));

tar_chan[i]+=mean_src.at<double>(i);

}

//Mergethechannels,converttoCV_8UC1eachchannelandconvertto

BGR

Matoutput;

merge(tar_chan,output);

output.convertTo(output,CV_8UC1);

cvtColor(output,output,COLOR_Lab2BGR);

//showpictures

namedWindow("Sourceimage",WINDOW_AUTOSIZE);

imshow("Sourceimage",src);

namedWindow("Targetimage",WINDOW_AUTOSIZE);

imshow("Targetimage",tar);

namedWindow("Resultimage",WINDOW_AUTOSIZE);

imshow("Resultimage",output);

waitKey(0);

return0;

}

Thefollowingfigureshowsyoutheoutputofthecode:

Anightappearancecolor-transferexample

SummaryInthischapter,weprovidedadeeperviewofthecolorspacesusedinOpenCVandshowedyouhowtoconvertbetweenthemusingthecvtColorfunction.Furthermore,thepossibilitiesofimageprocessingusingdifferentcolormodelsandtheimportanceofselectingthecorrectcolorspaceconsideringtheoperationsweneedtomakewashighlighted.Tothisend,color-space-basedsegmentationandcolor-transfermethodswereimplemented.

Thenextchapterwillcoverimage-processingtechniquesusedforvideoorasequenceofimages.Wewillseehowtoimplementvideostabilization,superresolution,andstitchingalgorithmswithOpenCV.

Chapter5.ImageProcessingforVideoThischaptershowsyoudifferenttechniquesrelatedtoimageprocessingforvideo.Whilemostclassicalimageprocessingdealswithstaticimages,video-basedprocessingisbecomingpopularandaffordable.

Thischaptercoversthefollowingtopics:

VideostabilizationThevideosuperresolutionprocessImagestitching

Inthischapter,wewillworkwithavideosequenceoralivecameradirectly.Theoutputofimageprocessingmaybeeitherasetofmodifiedimagesorusefulhigh-levelinformation.Mostimage-processingtechniquesconsiderimagesasatwo-dimensionaldigitalsignalandapplydifferenttechniquestoit.Inthischapter,asequenceofimagesfromavideoorlivecamerawillbeusedtomakeorimproveanewenhancedsequencewithdifferenthigh-leveltechniques.Thus,moreusefulinformationisobtained,thatis,athirdtimedimensionisincorporated.

VideostabilizationVideostabilizationreferstoafamilyofmethodsusedtoreducetheblurringassociatedwiththemotionofthecamera.Inotherwords,itcompensatesforanyangularmovement,equivalenttoyaw,pitch,roll,andxandytranslationsofthecamera.Thefirstimagestabilizersappearedintheearly60s.Thesesystemswereabletoslightlycompensateforcamerashakesandinvoluntarymovements.Theywerecontrolledbygyroscopesandaccelerometersbasedonmechanismsthatcouldcancelorreduceunwantedmovementbychangingthepositionofalens.Currently,thesemethodsarewidelyusedinbinoculars,videocameras,andtelescopes.

Therearevariousmethodsforimageorvideostabilization,andthischapterfocusesonthemostextendedfamiliesofmethods:

Mechanicalstabilizationsystems:Thesesystemsuseamechanicalsystemonthecameralenssothatwhenthecameraismoved,motionisdetectedbyaccelerometersandgyroscopes,andthesystemgeneratesamovementonthelens.Thesesystemswillnotbeconsideredhere.Digitalstabilizationsystems:Thesearenormallyusedinvideoandtheyactdirectlyontheimageobtainedfromthecamera.Inthesesystems,thesurfaceofthestabilizedimageisslightlysmallerthanthesourceimage’ssurface.Whenthecameraismoved,thecapturedimageismovedtocompensatethismovement.Althoughthesetechniqueseffectivelyworktocancelmovementbyreducingtheusableareaofmovementsensor,resolutionandimageclarityaresacrificed.

Video-stabilizationalgorithmsusuallyencompassthefollowingsteps:

GeneralstepsofVideo-Stabilizationalgorithms

ThischapterfocusesonthevideostabmoduleinOpenCV3.0Alpha,whichcontainsasetoffunctionsandclassesthatcanbeusedtosolvethevideo-stabilizationproblem.

Let’sexplorethegeneralprocessinmoredetail.Videostabilizationisachievedbyafirstestimationoftheinter-framemotionbetweenconsecutiveframesusingtheRANSACmethod.Attheendofthisstep,anarrayof3x3matricesisobtained,andeachofthemdescribesthemotionofthetwopairsofconsecutiveframes.Globalmotionestimationisveryimportantforthisstepanditaffectstheaccuracyofthestabilizedfinalsequence.

Note

YoucanfindmoredetailedinformationabouttheRANSACmethodathttp://en.wikipedia.org/wiki/RANSAC.

Thesecondstepgeneratesanewsequenceofframesbasedontheestimatedmotion.Additionalprocessingisperformed,suchassmoothing,deblurring,borderextrapolation,andsoon,toimprovethequalityofstabilization.

Thethirdstepremovestheannoyingirregularperturbations—refertothefollowingfigure.Thereareapproachesthatassumeacamera-motionmodel,whichworkwellwhensomeassumptionscanbemadeabouttheactualcameramotion.

Removingtheirregularperturbations

IntheOpenCVexamples([opencv_source_code]/samples/cpp/videostab.cpp),avideo-stabilizationprogramexamplecanbefound.ForthefollowingvideoStabilizerexample,thevideoStabilizer.proprojectneedstheselibraries:lopencv_core300,lopencv_highgui300,lopencv_features2d300,lopencv_videoio300,andlopencv_videostab300.

ThefollowingvideoStabilizerexamplehasbeencreatedusingthevideostabmoduleofOpenCV3.0Alpha:

#include<string>

#include<iostream>

#include<opencv2/opencv.hpp>

#include<opencv2/videostab.hpp>

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::videostab;

voidprocessing(Ptr<IFrameSource>stabilizedFrames,stringoutputPath);

intmain(intargc,constchar**argv)

{

Ptr<IFrameSource>stabilizedFrames;

try

{

//1-Preparetheinputvideoandcheckit

stringinputPath;

stringoutputPath;

if(argc>1)

inputPath=argv[1];

else

inputPath=".\\cube4.avi";

if(argc>2)

outputPath=argv[2];

else

outputPath=".\\cube4_stabilized.avi";

Ptr<VideoFileSource>source=makePtr<VideoFileSource>(inputPath);

cout<<"framecount(rough):"<<source->count()<<endl;

//2-Preparethemotionestimator

//first,preparethemotiontheestimationbuilder,RANSACL2

doublemin_inlier_ratio=0.1;

Ptr<MotionEstimatorRansacL2>est=makePtr<MotionEstimatorRansacL2>

(MM_AFFINE);

RansacParamsransac=est->ransacParams();

ransac.size=3;

ransac.thresh=5;

ransac.eps=0.5;

est->setRansacParams(ransac);

est->setMinInlierRatio(min_inlier_ratio);

//second,createafeaturedetector

intnkps=1000;

Ptr<GoodFeaturesToTrackDetector>feature_detector=

makePtr<GoodFeaturesToTrackDetector>(nkps);

//third,createthemotionestimator

Ptr<KeypointBasedMotionEstimator>motionEstBuilder=

makePtr<KeypointBasedMotionEstimator>(est);

motionEstBuilder->setDetector(feature_detector);

Ptr<IOutlierRejector>outlierRejector=makePtr<NullOutlierRejector>();

motionEstBuilder->setOutlierRejector(outlierRejector);

//3-Preparethestabilizer

StabilizerBase*stabilizer=0;

//first,preparetheoneortwopassstabilizer

boolisTwoPass=1;

intradius_pass=15;

if(isTwoPass)

{

//withatwopassstabilizer

boolest_trim=true;

TwoPassStabilizer*twoPassStabilizer=newTwoPassStabilizer();

twoPassStabilizer->setEstimateTrimRatio(est_trim);

twoPassStabilizer-

>setMotionStabilizer(makePtr<GaussianMotionFilter>(radius_pass));

stabilizer=twoPassStabilizer;

}

else

{

//withanonepassstabilizer

OnePassStabilizer*onePassStabilizer=newOnePassStabilizer();

onePassStabilizer-

>setMotionFilter(makePtr<GaussianMotionFilter>(radius_pass));

stabilizer=onePassStabilizer;

}

//second,setuptheparameters

intradius=15;

doubletrim_ratio=0.1;

boolincl_constr=false;

stabilizer->setFrameSource(source);

stabilizer->setMotionEstimator(motionEstBuilder);

stabilizer->setRadius(radius);

stabilizer->setTrimRatio(trim_ratio);

stabilizer->setCorrectionForInclusion(incl_constr);

stabilizer->setBorderMode(BORDER_REPLICATE);

//caststabilizertosimpleframesourceinterfacetoread

stabilizedframes

stabilizedFrames.reset(dynamic_cast<IFrameSource*>(stabilizer));

//4-Processingthestabilizedframes.Theresultsareshowedandsaved.

processing(stabilizedFrames,outputPath);

}

catch(constexception&e)

{

cout<<"error:"<<e.what()<<endl;

stabilizedFrames.release();

return-1;

}

stabilizedFrames.release();

return0;

}

voidprocessing(Ptr<IFrameSource>stabilizedFrames,stringoutputPath)

{

VideoWriterwriter;

MatstabilizedFrame;

intnframes=0;

doubleoutputFps=25;

//foreachstabilizedframe

while(!(stabilizedFrame=stabilizedFrames->nextFrame()).empty())

{

nframes++;

//initwriter(once)andsavestabilizedframe

if(!outputPath.empty())

{

if(!writer.isOpened())

writer.open(outputPath,VideoWriter::fourcc('X','V','I','D'),

outputFps,stabilizedFrame.size());

writer<<stabilizedFrame;

}

imshow("stabilizedFrame",stabilizedFrame);

charkey=static_cast<char>(waitKey(3));

if(key==27){cout<<endl;break;}

}

cout<<"processedframes:"<<nframes<<endl;

cout<<"finished"<<endl;

}

Thisexampleacceptsthenameofaninputvideofileasadefaultvideofilename(.\cube4.avi).Theresultingvideowillbedisplayedandthensavedas.\cube4_stabilized.avi.Notehowthevideostab.hppheaderisincludedandthecv::videostabnamespaceisused.Theexampletakesfourimportantsteps.Thefirststeppreparestheinputvideopath,andthisexampleusesthestandardcommand-lineinputarguments(inputPath=argv[1])toselectthevideofile.Ifitdoesnothaveaninputvideofile,thenitusesthedefaultvideofile(.\cube4.avi).

Thesecondstepbuildsamotionestimator.ArobustRANSAC-basedglobal2Dmethodiscreatedforthemotionestimatorusingasmartpointer(Ptr<object>)ofOpenCV(Ptr<MotionEstimatorRansacL2>est=makePtr<MotionEstimatorRansacL2>(MM_AFFINE)).Therearedifferentmotionmodelstostabilizethevideo:

MM_TRANSLATION=0MM_TRANSLATION_AND_SCALE=1MM_ROTATIO=2MM_RIGID=3MM_SIMILARITY=4MM_AFFINE=5MM_HOMOGRAPHY=6MM_UNKNOWN=7

Thereisatrade-offbetweenaccuracytostabilizethevideoandcomputationaltime.Themorebasicmotionmodelshaveworseaccuracyandbettercomputationaltime;however,themorecomplexmodelshavebetteraccuracyandworsecomputationaltime.

TheRANSACobjectisnowcreated(RansacParamsransac=est->ransacParams())andtheirparametersareset(ransac.size,ransac.threshandransac.eps).Afeaturedetectorisalsoneededtoestimatethemovementbetweeneachconsecutiveframethatwillbeusedbythestabilizer.ThisexampleusestheGoodFeaturesToTrackDetectormethodtodetect(nkps=1000)salientfeaturesineachframe.Then,itusestherobustRANSACandfeature-detectormethodstocreatethemotionestimatorusingthePtr<KeypointBasedMotionEstimator>motionEstBuilder=

makePtr<KeypointBasedMotionEstimator>(est)classandsettingthefeaturedetector

withmotionEstBuilder->setDetector(feature_detector).

RANSACparameters

Size Thesubsetsize

Thresh Themaximumerrortoclassifyasinliers

Eps Themaximumoutliersratio

Prob Theprobabilityofsuccess

Thethirdstepcreatesastabilizerthatneedsthepreviousmotionestimator.Youcanselect(isTwoPass=1)aone-ortwo-passstabilizer.Ifyouusethetwo-passstabilizer(TwoPassStabilizer*twoPassStabilizer=newTwoPassStabilizer()),theresultsareusuallybetter.However,inthisexample,thisiscomputationallyslower.Ifyouusetheotheroption,one-passstabilizer(OnePassStabilizer*onePassStabilizer=newOnePassStabilizer()),theresultsareworsebuttheresponseisfaster.Thestabilizerneedstosetotheroptionstoworkcorrectly,suchasthesourcevideofile(stabilizer->setFrameSource(source))andthemotionestimator(stabilizer->setMotionEstimator(motionEstBuilder)).Italsoneedstocastthestabilizertosimpleframesourcevideotoreadstabilizedframes(stabilizedFrames.reset(dynamic_cast<IFrameSource*>(stabilizer))).

Thelaststepstabilizesthevideousingthecreatedstabilizer.Theprocessing(Ptr<IFrameSource>stabilizedFrames)functioniscreatedtoprocessandstabilizeeachframe.Thisfunctionneedstointroduceapathtosavetheresultingvideo(stringoutputPath=".//stabilizedVideo.avi")andsettheplaybackspeed(doubleoutputFps=25).Afterwards,thisfunctioncalculateseachstabilizedframeuntiltherearenomoreframes(stabilizedFrame=stabilizedFrames->nextFrame().empty()).Internally,thestabilizerfirstestimatesthemotionofeveryframe.Thisfunctioncreatesavideowriter(writer.open(outputPath,VideoWriter::fourcc('X','V','I','D'),outputFps,stabilizedFrame.size()))tostoreeachframeintheXVIDformat.Finally,itsavesandshowseachstabilizedframeuntiltheuserpressestheEsckey.

TodemonstratehowtostabilizeavideowithOpenCV,thepreviousvideoStabilizerexampleisused.Theexampleisexecutedfromthecommandlineasfollows:

<bin_dir>\videoStabilizer.exe.\cube4.avi.\cube4_stabilized.avi

NoteThiscube4.avivideocanbefoundintheOpenCVsamplesfolder.Italsohasagreatdealofcameramovement,whichisperfectforthisexample.

Toshowthestabilizationresults,first,seefourframesofcube4.aviinthefollowingfigure.Thefigurefollowingtheseframesshowsthefirst10framesofcube4.aviandcube4_stabilized.avisuperimposedwithout(left-handsideofthefigure)andwith(right-handsideofthefigure)stabilization.

Thefourconsecutiveframesofcube4.avivideothatarecameramovements

10superimposedframesofcube4.aviandcube4_stabilizatedvideoswithoutandwithstabilization

Lookingattheprecedingfigure,ontheright-handside,youcanseethatthevibrationsproducedbythecameramovementhavebeenreducedduetothestabilization.

SuperresolutionSuperresolutionreferstothetechniquesoralgorithmsdesignedtoincreasethespatialresolutionofanimageorvideo,usuallyfromasequenceofimagesoflowerresolution.Itdiffersfromthetraditionaltechniquesofimagescaling,whichuseasingleimagetoincreasetheresolution,keepingthesharpedges.Incontrast,superresolutionmergesinformationfrommultipleimagestakenfromthesamescenetorepresentdetailsthatwerenotinitiallycapturedintheoriginalimages.

Theprocessofcapturinganimageorvideofromareal-lifescenerequiresthefollowingsteps:

Sampling:ThisisthetransformationofthecontinuoussystemfromthesceneofanidealdiscretesystemwithoutaliasingGeometrictransformation:Thisreferstoapplyingasetoftransformations,suchastranslationorrotation,duetothecamerapositionandlenssystemtoinfer,ideally,detailsofthescenethatarriveateachsensorBlur:ThishappensduetothelenssystemortheexistingmotioninthesceneduringtheintegrationtimeSubsampling:Withthis,thesensoronlyintegratesthenumberofpixelsatitsdisposal(photosites)

Youcanseethisprocessofimagecapturinginthefollowingfigure:

Theprocessofcapturinganimagefromarealscene

Duringthiscaptureprocess,thedetailsofthesceneareintegratedbydifferentsensorssothateachpixelineachcaptureincludesdifferentinformation.Therefore,superresolutionisbasedontryingtofindtherelationshipbetweendifferentcapturesthathaveobtaineddifferentdetailsofthesceneinordertocreateanewimagewithmoreinformation.Superresolutionis,therefore,usedtoregenerateadiscretizedscenewithahigherresolution.

Superresolutioncanbeobtainedbyvarioustechniques,rangingfromthemostintuitivein

thespatialdomaintotechniquesbasedonanalyzingthefrequencyspectrum.Techniquesarebasicallydividedintooptical(usinglenses,zoom,andsoon)orimage-processing-basedtechniques.Thischapterfocusesonimage-processing-basedsuperresolution.Thesemethodsuseotherpartsofthelower-resolutionimages,orotherunrelatedimages,toinferwhatthehigh-resolutionimageshouldlooklike.Thesealgorithmscanbealsodividedintothefrequencyorspatialdomain.Originally,superresolutionmethodsonlyworkedwellongrayscaleimages,butnewmethodshavebeendevelopedtoadaptthemtocolorimages.

Ingeneral,superresolutioniscomputationallydemanding,bothspatiallyandtemporally,becausethesizeoflow-resolutionandhigh-resolutionimagesishighandhundredsofsecondsmaybeneededtogenerateanimage.Totrytoreducethecomputationaltime,preconditionersareusedcurrentlyforoptimizersthatareresponsibleforminimizingthesefunctions.AnotheralternativeistouseGPUprocessingtoimprovethecomputationaltimebecausethesuperresolutionprocessisinherentlyparallelizable.

ThischapterfocusesonthesuperresmoduleinOpenCV3.0Alpha,whichcontainsasetoffunctionsandclassesthatcanbeusedtosolvetheproblemofresolutionenhancement.Themoduleimplementsanumberofmethodsbasedonimage-processingsuperresolution.ThischapterspecificallyfocusesontheBilateralTV-L1(BTVL1)superresolutionmethodimplemented.Amajordifficultyofthesuperresolutionprocessistoestimatethewarpingfunctiontobuildthesuperresolutionimage.TheBilateralTV-L1usesopticalflowtoestimatethewarpingfunction.

NoteYoucanfindmoredetailedinformationabouttheBilateralTV-Lmethodathttp://www.ipol.im/pub/art/2013/26/andopticalflowathttp://en.wikipedia.org/wiki/Optical_flow.

IntheOpenCVexamples([opencv_source_code]/samples/gpu/super_resolution.cpp),abasicexampleofsuperresolutioncanbefound.

NoteYoucanalsodownloadthisexamplefromtheOpenCVGitHubrepositoryathttps://github.com/Itseez/opencv/blob/master/samples/gpu/super_resolution.cpp.

Forthefollowingsuperresolutionexampleproject,thesuperresolution.proprojectfileshallincludetheselibraries:lopencv_core300,lopencv_imgproc300,lopencv_highgui300,lopencv_features2d300,lopencv_videoio300,andlopencv_superres300toworkcorrectly:

#include<iostream>

#include<iomanip>

#include<string>

#include<opencv2/core.hpp>

#include<opencv2/core/utility.hpp>

#include<opencv2/highgui.hpp>

#include<opencv2/imgproc.hpp>

#include<opencv2/superres.hpp>

#include<opencv2/superres/optical_flow.hpp>

#include<opencv2/opencv_modules.hpp>

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::superres;

staticPtr<DenseOpticalFlowExt>createOptFlow(stringname);

intmain(intargc,char*argv[])

{

//1-Initializetheinitialparameters

//Inputandoutputvideo

stringinputVideoName;

stringoutputVideoName;

if(argc>1)

inputVideoName=argv[1];

else

inputVideoName=".\\tree.avi";

if(argc>2)

outputVideoName=argv[2];

else

outputVideoName=".\\tree_superresolution.avi";

constintscale=4;//Scalefactor

constintiterations=180;//Iterationscount

constinttemporalAreaRadius=4;//Radiusofthetemporalsearcharea

stringoptFlow="farneback";//Opticalflowalgorithm

//optFlow="farneback";

//optFlow="tvl1";

//optFlow="brox";

//optFlow="pyrlk";

doubleoutputFps=25.0;//Playbackspeedoutput

//2-Createanopticalflowmethod

Ptr<DenseOpticalFlowExt>optical_flow=createOptFlow(optFlow);

if(optical_flow.empty())return-1;

//3-Createthesuperresolutionmethodandsetitsparameters

Ptr<SuperResolution>superRes;

superRes=createSuperResolution_BTVL1();

superRes->set("opticalFlow",optical_flow);

superRes->set("scale",scale);

superRes->set("iterations",iterations);

superRes->set("temporalAreaRadius",temporalAreaRadius);

Ptr<FrameSource>frameSource;

frameSource=createFrameSource_Video(inputVideoName);

superRes->setInput(frameSource);

//Notusethefirstframe

Matframe;

frameSource->nextFrame(frame);

//4-Processingtheinputvideowiththesuperresolution

//Showtheinitialoptions

cout<<"Input:"<<inputVideoName<<""<<

frame.size()<<endl;

cout<<"Output:"<<outputVideoName<<endl;

cout<<"Playbackspeedoutput:"<<outputFps<<endl;

cout<<"Scalefactor:"<<scale<<endl;

cout<<"Iterations:"<<iterations<<endl;

cout<<"Temporalradius:"<<temporalAreaRadius<<endl;

cout<<"OpticalFlow:"<<optFlow<<endl;

cout<<endl;

VideoWriterwriter;

doublestart_time,finish_time;

for(inti=0;;++i)

{

cout<<'['<<setw(3)<<i<<"]:";

Matresult;

//Calculatetheprocessingtime

start_time=getTickCount();

superRes->nextFrame(result);

finish_time=getTickCount();

cout<<(finish_time-start_time)/getTickFrequency()<<"secs,

Size:"<<result.size()<<endl;

if(result.empty())break;

//Showtheresult

imshow("SuperResolution",result);

if(waitKey(1000)>0)break;

//Savetheresultonoutputfile

if(!outputVideoName.empty())

{

if(!writer.isOpened())

writer.open(outputVideoName,VideoWriter::fourcc('X','V',

'I','D'),outputFps,result.size());

writer<<result;

}

}

writer.release();

return0;

}

staticPtr<DenseOpticalFlowExt>createOptFlow(stringname)

{

if(name=="farneback")

returncreateOptFlow_Farneback();

elseif(name=="tvl1")

returncreateOptFlow_DualTVL1();

elseif(name=="brox")

returncreateOptFlow_Brox_CUDA();

elseif(name=="pyrlk")

returncreateOptFlow_PyrLK_CUDA();

else

cerr<<"IncorrectOpticalFlowalgorithm-"<<name<<endl;

returnPtr<DenseOpticalFlowExt>();

}

Thisexamplecreatesaprogram(superresolution)toobtainvideoswithsuperresolution.Ittakesthepathofaninputvideoorusesadefaultvideopath(.\tree.avi).Theresultingvideoisdisplayedandsavedas.\tree_superresolution.avi.Inthefirstplace,thesuperres.hppandsuperres/optical_flow.hppheadersareincludedandthecv::superresnamespaceisused.Theexamplefollowsfourimportantsteps.

Thefirststepsetstheinitialparameters.Itistheinputvideopaththatusesthestandardinput(inputVideoName=argv[1])toselectthevideofile,andifitdoesnothaveaninputvideofile,thenitusesadefaultvideofile.Theoutputvideopathalsousestheinputstandard(outputVideoName=argv[2])toselecttheoutputvideofile,andifitdoesnothaveanoutputvideofile,thenitusesthedefaultoutputvideofile(.\tree_superresolution)andtheoutputplaybackspeedisalsoset(doubleoutputFps=25.0).Otherimportantparametersofthesuperresolutionmethodarethescalefactor(constintscale=4),theiterationcount(constintiterations=100),theradiusofthetemporalsearcharea(constinttemporalAreaRadius=4),andtheoptical-flowalgorithm(stringoptFlow="farneback").

Thesecondstepcreatesanoptical-flowmethodtodetectsalientfeaturesandtrackthemforeachvideoframe.Anewmethod(staticPtr<DenseOpticalFlowExt>createOptFlow(stringname))hasbeencreatedtoselectbetweenthedifferentoptical-flowmethods.Youcanselectbetweenfarneback,tvl1,brox,andpyrlkoptical-flowmethods.Anewmethod(staticPtr<DenseOpticalFlowExt>createOptFlow(stringname))iswrittentocreateanoptical;-flowmethodtotrackfeatures.ThetwomostimportantmethodsareFarneback(createOptFlow_Farneback())andTV-L1(createOptFlow_DualTVL1()).ThefirstmethodisbasedonGunnerFarneback’salgorithmthatcomputestheopticalflowforallpointsintheframe.ThesecondmethodcalculatestheopticalflowbetweentwoimageframesthatarebasedondualformulationontheTVenergyandemploysanefficientpoint-wisethresholdingstep.Thissecondmethodiscomputationallymoreefficient.

Comparisonbetweenthedifferentopticalflowmethods

Method Complexity Parallelizable

Farneback Quadratic No

TV-L1 Lineal Yes

Brox Lineal Yes

PyrLK Lineal No

NoteYoucanalsolearnmoreabouttheFarnebackoptical-flowmethodusedathttp://www.diva-portal.org/smash/get/diva2:273847/FULLTEXT01.pdf.

Thethirdstepcreatesandsetsthesuperresolutionmethod.Aninstanceofthismethodiscreated(Ptr<SuperResolution>superRes),whichusestheBilateralTV-L1algorithm(superRes=createSuperResolution_BTVL1()).Thismethodhasthefollowingparametersforthealgorithm:

scale:Thisisthescalefactoriterations:Thisistheiterationcounttau:Thisisanasymptoticvalueofthesteepestdescentmethodlambda:Thisistheweightparametertobalancethedatatermandsmoothnesstermalpha:ThisisaparameterofspatialdistributioninBilateral-TVbtvKernelSize:ThisisthekernelsizeoftheBilateral-TVfilterblurKernelSize:ThisistheGaussianblurkernelsizeblurSigma:ThisistheGaussianblursigmatemporalAreaRadius:ThisistheradiusofthetemporalsearchareaopticalFlow:Thisisadenseoptical-flowalgorithm

Theseparametersaresetasfollows:

superRes->set("parameter",value);

Onlythefollowingparametersareset;theotherparametersusetheirdefaultvalues:

superRes->set("opticalFlow",optical_flow);

superRes->set("scale",scale);

superRes->set("iterations",iterations);

superRes->set("temporalAreaRadius",temporalAreaRadius);

Afterwards,theinputvideoframeisselected(superRes->setInput(frameSource)).

Thelaststepprocessestheinputvideotocomputethesuperresolution.Foreachvideoframe,thesuperresolutioniscalculated(superRes->nextFrame(result));thiscalculationiscomputationallyveryslow,thustheprocessingtimeisestimatedtoshowprogress.Finally,eachresultframeisshown(imshow("SuperResolution",result))andsaved(writer<<result).

Toshowthesuperresolutionresults,asmallpartofthefirstframeofthetree.aviandtree_superresolution.avivideosarecomparedwithandwithoutsuperresolution:

Partofthefirstframeoftree.aviandtree_superresolution.avivideoswithoutandwiththesuperresolutionprocess

Intheright-hand-sidesectionoftheprecedingfigure,youcanobservemoredetailsintheleavesandbranchesofthetreeduetothesuperresolutionprocess.

StitchingImagestitching,orphotostitching,candiscoverthecorrespondencerelationshipbetweenimageswithsomedegreeofoverlap.Thisprocesscombinesasetofimageswithoverlappingfieldsofviewtoproduceapanoramaorhigher-resolutionimage.Mostofthetechniquesforimagestitchingneednearlyexactoverlapsbetweentheimagestoproduceseamlessresults.Somedigitalcamerascaninternallystitchasetofimagestobuildapanoramaimage.Anexampleisshowninthefollowingfigure:

Apanoramaimagecreatedwithstitching

NoteTheprecedingimageexampleandmoreinformationaboutimagestitchingcanbefoundathttp://en.wikipedia.org/wiki/Image_stitching.

Stitchingcannormallybedividedintothreeimportantsteps:

Registration(images)impliesmatchingfeaturesinasetofimagestosearchforadisplacementthatminimizesthesumofabsolutevaluesinthedifferencesbetweenoverlappingpixels.Direct-alignmentmethodscouldbeusedtogetbetterresults.Theusercouldalsoaddaroughmodelofthepanoramatohelpthefeaturematchingstage,inwhichcase,theresultsaretypicallymoreaccurateandcomputationallyfaster.Calibration(images)focusesonminimizingdifferencesbetweenanidealmodelandthecamera-lenssystem:differentcamerapositionsandopticaldefectssuchasdistortions,exposure,chromaticaberrations,andsoon.Compositing(images)usestheresultsofthepreviousstep,calibration,combinedwiththeremappingoftheimagestoanoutputprojection.Colorsarealsoadjustedbetweenimagestocompensateforexposuredifferences.Imagesareblendedtogetherandseam-lineadjustmentisdonetominimizethevisibilityofseamsbetweenimages.

Whenthereareimagesegmentsthathavebeentakenfromthesamepointinspace,stitchingcanbeperformedusingoneofvariousmapprojections.Themostimportantmapprojectionsareshownasfollows:

Rectilinearprojection:Here,thestitchingimageisviewedonatwo-dimensionalplaneintersectingthepanoramasphereinasinglepoint.Linesthatarestraightin

realityareshownsimilarregardlessoftheirdirectionontheimage.Whentherearewideviews(around120degrees),imagesaredistortedneartheborders.Cylindricalprojection:Here,thestitchingimageshowsa360-degreehorizontalfieldofviewandalimitedverticalfieldofview.Thisprojectionismeanttobeviewedasthoughtheimageiswrappedintoacylinderandviewedfromwithin.Whenviewedona2Dplane,horizontallinesappearcurved,whileverticallinesremainstraight.Sphericalprojection:Here,thestitchingimageshowsa360-degreehorizontalfieldofviewand180-degreeverticalfieldofview,thatis,thewholesphere.Panoramaimageswiththisprojectionaremeanttobeviewedasthoughtheimageiswrappedintoasphereandviewedfromwithin.Whenviewedona2Dplane,horizontallinesappearcurvedasinacylindricalprojection,whileverticallinesremainvertical.Stereographicprojectionorfisheyeprojection:Thiscanbeusedtoformalittleplanetpanoramabypointingthevirtualcamerastraightdownandsettingthefieldofviewlargeenoughtoshowthewholegroundandsomeoftheareasaboveit;pointingthevirtualcameraupwardscreatesatunneleffect.Paniniprojection:Thishasspecializedprojectionsthatmayhavemoreaestheticallypleasingadvantagesovernormalcartographyprojections.Thisprojectioncombinesdifferentprojectionsinthesameimagetofine-tunethefinallookoftheoutputpanoramaimage.

ThischapterfocusesonthestitchingmoduleandthedetailsubmoduleinOpenCV3.0Alpha,whichcontainsasetoffunctionsandclassesthatimplementastitcher.Usingthesemodules,itispossibletoconfigureorskipsomesteps.Theimplementedstitchingexamplehasthefollowinggeneraldiagram:

IntheOpenCVexamples,therearetwobasicexamplesofstitching,whichcanbefoundat([opencv_source_code]/samples/cpp/stitching.cpp])and([opencv_source_code]/samples/cpp/stitching_detailed.cpp]).

Forthefollowing,moreadvancedstitchingAdvancedexample,thestitchingAdvanced.proprojectfilemustincludethefollowinglibrariestoworkcorrectly:lopencv_core300,lopencv_imgproc300,lopencv_highgui300,lopencv_features2d300,lopencv_videoio300,lopencv_imgcodecs300,andlopencv_stitching300:

#include<iostream>

#include<string>

#include<opencv2/opencv_modules.hpp>

#include<opencv2/core/utility.hpp>

#include<opencv2/imgcodecs.hpp>

#include<opencv2/highgui.hpp>

#include<opencv2/features2d.hpp>

#include<opencv2/stitching/detail/blenders.hpp>

#include<opencv2/stitching/detail/camera.hpp>

#include<opencv2/stitching/detail/exposure_compensate.hpp>

#include<opencv2/stitching/detail/matchers.hpp>

#include<opencv2/stitching/detail/motion_estimators.hpp>

#include<opencv2/stitching/detail/seam_finders.hpp>

#include<opencv2/stitching/detail/util.hpp>

#include<opencv2/stitching/detail/warpers.hpp>

#include<opencv2/stitching/warpers.hpp>

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::detail;

intmain(intargc,char*argv[])

{

//Defaultparameters

vector<String>img_names;

doublescale=1;

stringfeatures_type="orb";//"surf"or"orb"featurestype

floatmatch_conf=0.3f;

floatconf_thresh=1.f;

stringadjuster_method="ray";//"reproj"or"ray"adjustermethod

booldo_wave_correct=true;

WaveCorrectKindwave_correct_type=WAVE_CORRECT_HORIZ;

stringwarp_type="spherical";

intexpos_comp_type=ExposureCompensator::GAIN_BLOCKS;

stringseam_find_type="gc_color";

floatblend_strength=5;

intblend_type=Blender::MULTI_BAND;

stringresult_name="panorama_result.jpg";

doublestart_time=getTickCount();

//1-Inputimages

if(argc>1)

{

for(inti=1;i<argc;i++)

img_names.push_back(argv[i]);

}

else

{

img_names.push_back("./panorama_image1.jpg");

img_names.push_back("./panorama_image2.jpg");

}

//Checkifhaveenoughimages

intnum_images=static_cast<int>(img_names.size());

if(num_images<2){cout<<"Needmoreimages"<<endl;return-1;}

//2-Resizeimagesandfindfeaturessteps

cout<<"Findingfeatures…"<<endl;

doublet=getTickCount();

Ptr<FeaturesFinder>finder;

if(features_type=="surf")

finder=makePtr<SurfFeaturesFinder>();

elseif(features_type=="orb")

finder=makePtr<OrbFeaturesFinder>();

else{cout<<"Unknown2Dfeaturestype:'"<<features_type<<endl;

return-1;}

Matfull_img,img;

vector<ImageFeatures>features(num_images);

vector<Mat>images(num_images);

vector<Size>full_img_sizes(num_images);

for(inti=0;i<num_images;++i)

{

full_img=imread(img_names[i]);

full_img_sizes[i]=full_img.size();

if(full_img.empty()){cout<<"Can'topenimage"<<img_names[i]

<<endl;return-1;}

resize(full_img,img,Size(),scale,scale);

images[i]=img.clone();

(*finder)(img,features[i]);

features[i].img_idx=i;

cout<<"Featuresinimage#"<<i+1<<"are:"<<

features[i].keypoints.size()<<endl;

}

finder->collectGarbage();

full_img.release();

img.release();

cout<<"Findingfeatures,time:"<<((getTickCount()-t)/

getTickFrequency())<<"sec"<<endl;

//3-Matchfeatures

cout<<"Pairwisematching"<<endl;

t=getTickCount();

vector<MatchesInfo>pairwise_matches;

BestOf2NearestMatchermatcher(false,match_conf);

matcher(features,pairwise_matches);

matcher.collectGarbage();

cout<<"Pairwisematching,time:"<<((getTickCount()-t)/

getTickFrequency())<<"sec"<<endl;

//4-Selectimagesandmatchessubsettobuildpanorama

vector<int>indices=leaveBiggestComponent(features,pairwise_matches,

conf_thresh);

vector<Mat>img_subset;

vector<String>img_names_subset;

vector<Size>full_img_sizes_subset;

for(size_ti=0;i<indices.size();++i)

{

img_names_subset.push_back(img_names[indices[i]]);

img_subset.push_back(images[indices[i]]);

full_img_sizes_subset.push_back(full_img_sizes[indices[i]]);

}

images=img_subset;

img_names=img_names_subset;

full_img_sizes=full_img_sizes_subset;

//Estimatecameraparametersrough

HomographyBasedEstimatorestimator;

vector<CameraParams>cameras;

if(!estimator(features,pairwise_matches,cameras)){cout<<

"Homographyestimationfailed."<<endl;return-1;}

for(size_ti=0;i<cameras.size();++i)

{

MatR;

cameras[i].R.convertTo(R,CV_32F);

cameras[i].R=R;

cout<<"Initialintrinsic#"<<indices[i]+1<<":\n"<<

cameras[i].K()<<endl;

}

//5-Refinecameraparametersglobally

Ptr<BundleAdjusterBase>adjuster;

if(adjuster_method=="reproj")

//"reproj"method

adjuster=makePtr<BundleAdjusterReproj>();

else//"ray"method

adjuster=makePtr<BundleAdjusterRay>();

adjuster->setConfThresh(conf_thresh);

if(!(*adjuster)(features,pairwise_matches,cameras)){cout<<"Camera

parametersadjustingfailed."<<endl;return-1;}

//Findmedianfocallength

vector<double>focals;

for(size_ti=0;i<cameras.size();++i)

{

cout<<"Camera#"<<indices[i]+1<<":\n"<<cameras[i].K()<<

endl;

focals.push_back(cameras[i].focal);

}

sort(focals.begin(),focals.end());

floatwarped_image_scale;

if(focals.size()%2==1)

warped_image_scale=static_cast<float>(focals[focals.size()/2]);

else

warped_image_scale=static_cast<float>(focals[focals.size()/2-

1]+focals[focals.size()/2])*0.5f;

//6-Wavecorrelation(optional)

if(do_wave_correct)

{

vector<Mat>rmats;

for(size_ti=0;i<cameras.size();++i)

rmats.push_back(cameras[i].R.clone());

waveCorrect(rmats,wave_correct_type);

for(size_ti=0;i<cameras.size();++i)

cameras[i].R=rmats[i];

}

//7-Warpimages

cout<<"Warpingimages(auxiliary)..."<<endl;

t=getTickCount();

vector<Point>corners(num_images);

vector<UMat>masks_warped(num_images);

vector<UMat>images_warped(num_images);

vector<Size>sizes(num_images);

vector<UMat>masks(num_images);

//Prepareimagesmasks

for(inti=0;i<num_images;++i)

{

masks[i].create(images[i].size(),CV_8U);

masks[i].setTo(Scalar::all(255));

}

//Mapprojections

Ptr<WarperCreator>warper_creator;

if(warp_type=="rectilinear")

warper_creator=makePtr<cv::CompressedRectilinearWarper>(2.0f,1.0f);

elseif(warp_type=="cylindrical")

warper_creator=makePtr<cv::CylindricalWarper>();

elseif(warp_type=="spherical")

warper_creator=makePtr<cv::SphericalWarper>();

elseif(warp_type=="stereographic")

warper_creator=makePtr<cv::StereographicWarper>();

elseif(warp_type=="panini")

warper_creator=makePtr<cv::PaniniWarper>(2.0f,1.0f);

if(!warper_creator){cout<<"Can'tcreatethefollowingwarper'"<<

warp_type<<endl;return1;}

Ptr<RotationWarper>warper=warper_creator->create(static_cast<float>

(warped_image_scale*scale));

for(inti=0;i<num_images;++i)

{

Mat_<float>K;

cameras[i].K().convertTo(K,CV_32F);

floatswa=(float)scale;

K(0,0)*=swa;K(0,2)*=swa;

K(1,1)*=swa;K(1,2)*=swa;

corners[i]=warper->warp(images[i],K,cameras[i].R,INTER_LINEAR,

BORDER_REFLECT,images_warped[i]);

sizes[i]=images_warped[i].size();

warper->warp(masks[i],K,cameras[i].R,INTER_NEAREST,BORDER_CONSTANT,

masks_warped[i]);

}

vector<UMat>images_warped_f(num_images);

for(inti=0;i<num_images;++i)

images_warped[i].convertTo(images_warped_f[i],CV_32F);

cout<<"Warpingimages,time:"<<((getTickCount()-t)/

getTickFrequency())<<"sec"<<endl;

//8-Compensateexposureerrors

Ptr<ExposureCompensator>compensator=

ExposureCompensator::createDefault(expos_comp_type);

compensator->feed(corners,images_warped,masks_warped);

//9-Findseammasks

Ptr<SeamFinder>seam_finder;

if(seam_find_type=="no")

seam_finder=makePtr<NoSeamFinder>();

elseif(seam_find_type=="voronoi")

seam_finder=makePtr<VoronoiSeamFinder>();

elseif(seam_find_type=="gc_color")

seam_finder=makePtr<GraphCutSeamFinder>

(GraphCutSeamFinderBase::COST_COLOR);

elseif(seam_find_type=="gc_colorgrad")

seam_finder=makePtr<GraphCutSeamFinder>

(GraphCutSeamFinderBase::COST_COLOR_GRAD);

elseif(seam_find_type=="dp_color")

seam_finder=makePtr<DpSeamFinder>(DpSeamFinder::COLOR);

elseif(seam_find_type=="dp_colorgrad")

seam_finder=makePtr<DpSeamFinder>(DpSeamFinder::COLOR_GRAD);

if(!seam_finder){cout<<"Can'tcreatethefollowingseamfinder'"<<

seam_find_type<<endl;return1;}

seam_finder->find(images_warped_f,corners,masks_warped);

//Releaseunusedmemory

images.clear();

images_warped.clear();

images_warped_f.clear();

masks.clear();

//10-Createablender

Ptr<Blender>blender=Blender::createDefault(blend_type,false);

Sizedst_sz=resultRoi(corners,sizes).size();

floatblend_width=sqrt(static_cast<float>(dst_sz.area()))*

blend_strength/100.f;

if(blend_width<1.f)

blender=Blender::createDefault(Blender::NO,false);

elseif(blend_type==Blender::MULTI_BAND)

{

MultiBandBlender*mb=dynamic_cast<MultiBandBlender*>(blender.get());

mb->setNumBands(static_cast<int>(ceil(log(blend_width)/log(2.))-

1.));

cout<<"Multi-bandblender,numberofbands:"<<mb->numBands()

<<endl;

}

elseif(blend_type==Blender::FEATHER)

{

FeatherBlender*fb=dynamic_cast<FeatherBlender*>(blender.get());

fb->setSharpness(1.f/blend_width);

cout<<"Featherblender,sharpness:"<<fb->sharpness()<<endl;

}

blender->prepare(corners,sizes);

//11-Compositingstep

cout<<"Compositing…"<<endl;

t=getTickCount();

Matimg_warped,img_warped_s;

Matdilated_mask,seam_mask,mask,mask_warped;

for(intimg_idx=0;img_idx<num_images;++img_idx)

{

cout<<"Compositingimage#"<<indices[img_idx]+1<<endl;

//11.1-Readimageandresizeitifnecessary

full_img=imread(img_names[img_idx]);

if(abs(scale-1)>1e-1)

resize(full_img,img,Size(),scale,scale);

else

img=full_img;

full_img.release();

Sizeimg_size=img.size();

MatK;

cameras[img_idx].K().convertTo(K,CV_32F);

//11.2-Warpthecurrentimage

warper->warp(img,K,cameras[img_idx].R,INTER_LINEAR,BORDER_REFLECT,

img_warped);

//Warpthecurrentimagemask

mask.create(img_size,CV_8U);

mask.setTo(Scalar::all(255));

warper->warp(mask,K,cameras[img_idx].R,INTER_NEAREST,

BORDER_CONSTANT,mask_warped);

//11.3-Compensateexposureerrorstep

compensator->apply(img_idx,corners[img_idx],img_warped,mask_warped);

img_warped.convertTo(img_warped_s,CV_16S);

img_warped.release();

img.release();

mask.release();

dilate(masks_warped[img_idx],dilated_mask,Mat());

resize(dilated_mask,seam_mask,mask_warped.size());

mask_warped=seam_mask&mask_warped;

//11.4-Blendingimagesstep

blender->feed(img_warped_s,mask_warped,corners[img_idx]);

}

Matresult,result_mask;

blender->blend(result,result_mask);

cout<<"Compositing,time:"<<((getTickCount()-t)/

getTickFrequency())<<"sec"<<endl;

imwrite(result_name,result);

cout<<"Finished,totaltime:"<<((getTickCount()-start_time)/

getTickFrequency())<<"sec"<<endl;

return0;

}

ThisexamplecreatesaprogramtostitchimagesusingOpenCVsteps.Ittakesaninputpathtoselectthedifferentinputimagesorusesdefaultinputimages(.\panorama_image1.jpgandpanorama_image2.jpg),whichareshownlater.Finally,theresultingimageisshownandsavedas.\panorama_result.jpg.Inthefirstplace,thestitching.hppanddetailheadersareincludedandthecv::detailnamespaceisused.Themoreimportantparametersarealsoset,andyoucanconfigurethestitchingprocesswiththeseparameters.Ifyouneedtouseacustomconfiguration,itisveryusefultounderstandthegeneraldiagramofthestitchingprocess(thepreviousfigure).Thisadvancedexamplehas11importantsteps.Thefirststepreadsandcheckstheinputimages.Thisexampleneedstwoormoreimagestowork.

Thesecondstepresizestheinputimagesusingthedoublescale=1parameterandfindsthefeaturesoneachimage;youcanselectbetweentheSurf(finder=makePtr<SurfFeaturesFinder>())orOrb(finder=makePtr<OrbFeaturesFinder>())featurefindersusingthestringfeatures_type="orb"parameter.Afterwards,thisstepresizestheinputimages(resize(full_img,img,Size(),scale,scale))andfindsthefeatures((*finder)(img,features[i])).

NoteFormoreinformationaboutSURFandORBdescriptors,refertoChapter5ofOpenCVEssentialsbyPacktPublishing.

Thethirdstepmatchesthefeaturesthathavebeenfoundpreviously.Amatcheriscreated(BestOf2NearestMatchermatcher(false,match_conf))withthefloatmatch_conf=0.3fparameter.

Thefourthstepselectsimagesandmatchessubsetstobuildthepanorama.Then,thebestfeaturesareselectedandmatchedusingthevector<int>indices=leaveBiggestComponent(features,pairwise_matches,conf_thresh)function.Withthesefeatures,anewsubsetiscreatedtobeused.

Thefifthsteprefinesparametersgloballyusingbundleadjustmenttobuildanadjuster(Ptr<BundleAdjusterBase>adjuster).Givenasetofimagesdepictinganumberof2Dor3Dpointsfromdifferentviewpoints,bundleadjustmentcanbedefinedastheproblemofsimultaneouslyrefiningthe2Dor3Dcoordinates,describingthescenegeometryaswellastheparametersoftherelativemotionandopticalcharacteristicsofthecamerasemployedtoacquiretheimagesaccordingtoanoptimalitycriterioninvolvingthecorrespondingimageprojectionsofallpoints.Therearetwomethodstocalculatethisbundleadjustment,reproj(adjuster=makePtr<BundleAdjusterReproj>())orray(adjuster=makePtr<BundleAdjusterRay>()),whichareselectedwiththestringadjuster_method="ray"parameter.Finallythisbundleadjustmentisusedas(*adjuster)(features,pairwise_matches,cameras).

Thesixthstepisanoptionalstep(booldo_wave_correct=true)thatcalculatesthewavecorrelationtoimprovethecamerasetting.ThetypeofwavecorrelationisselectedwiththeWaveCorrectKindwave_correct_type=WAVE_CORRECT_HORIZparameterandcalculatedaswaveCorrect(rmats,wave_correct_type).

Theseventhstepcreatesawarperimagethatneedsamapprojection.Themapprojectionshavebeendescribedpreviously,andtheycanberectilinear,cylindrical,spherical,stereographic,orpanini.ThereareactuallymoremapprojectionsimplementedinOpenCV.Themapprojectionscanbeselectedwiththestringwarp_type="spherical"parameter.Afterwards,awarperiscreated(Ptr<RotationWarper>warper=warper_creator->create(static_cast<float>(warped_image_scale*scale)))andeachimageiswarped(warper->warp(masks[i],K,cameras[i].R,INTER_NEAREST,BORDER_CONSTANT,masks_warped[i])).

Theeighthstepcompensatesexposureerrorsbycreatingacompensator(Ptr<ExposureCompensator>compensator=

ExposureCompensator::createDefault(expos_comp_type))anditisappliedtoeachwarpedimage(compensator->feed(corners,images_warped,masks_warped)).

Theninthstepfindsseammasks.Thisprocesssearchesforthebestareasofattachmentforeachpanoramaimage.TherearesomemethodsimplementedinOpenCVtoperformthistask,andthisexampleusesthestringseam_find_type="gc_color"parametertoselectthem.ThesemethodsareNoSeamFinder(there’snouseofthismethod),VoronoiSeamFinder,GraphCutSeamFinderBase::COST_COLOR,GraphCutSeamFinderBase::COST_COLOR_GRAD,DpSeamFinder::COLOR,andDpSeamFinder::COLOR_GRAD.

Thetenthstepcreatesablendertocombineeachimagetobuildthepanorama.TherearetwotypesofblendersimplementedinOpenCV,MultiBandBlender*mb=dynamic_cast<MultiBandBlender*>(blender.get())andFeatherBlender*fb=dynamic_cast<FeatherBlender*>(blender.get()),whichcanbeselectedwiththeintblend_type=Blender::MULTI_BANDparameter.Finally,theblenderisprepared(blender->prepare(corners,sizes)).

Thelaststepcompositesthefinalpanorama.Thisstepneedswhatthepreviousstepshavedonetoconfigurethestitching.Foursub-stepsareperformedtocalculatethefinalpanorama.First,eachinputimageisread(full_img=imread(img_names[img_idx]))and,ifnecessary,resized(resize(full_img,img,Size(),scale,scale)).Second,theseimagesarewarpedwiththecreatedwarper(warper->warp(img,K,cameras[img_idx].R,INTER_LINEAR,BORDER_REFLECT,img_warped)).Third,theseimagesarecompensatedforexposureerrorswiththecreatedcompensator(compensator->apply(img_idx,corners[img_idx],img_warped,mask_warped)).Finally,theseimagesareblendedusingthecreatedblender.Thefinalresultpanoramaisnowsavedinthestringresult_name="panorama_result.jpg"file.

ToshowyouthestitchingAdvancedresults,twoinputimagesarestitchedandtheresultingpanoramaisshownasfollows:

SummaryInthischapter,youlearnedhowtousethreeimportantmodulesofOpenCVthathandleimageprocessinginvideo.Thesemodulesarevideostabilization,superresolution,andstitching.Someofthetheoreticalunderpinningshavealsobeenexplainedforeachmodule.

Ineachsectionofthischapter,acompleteexample,developedinC++,isexplained.Animageresultwasalsoshownforeachmodule,showingthemaineffect.

Thenextchapterintroduceshigh-dynamic-rangeimagesandshowsyouhowtohandlethemwithOpenCV.High-dynamic-rangeimagingistypicallyconsideredwithinwhatisnowcalledcomputationalphotography.Roughlyspeaking,computationalphotographyreferstotechniquesthatallowyoutoextendthetypicalcapabilitiesofdigitalphotography.Thismayincludehardwareadd-onsormodifications,butitmostlyreferstosoftware-basedtechniques.Thesetechniquesmayproduceoutputimagesthatcannotbeobtainedwitha“traditional”digitalcamera.

Chapter6.ComputationalPhotographyComputationalphotographyreferstotechniquesthatallowyoutoextendthetypicalcapabilitiesofdigitalphotography.Thismayincludehardwareadd-onsormodifications,butitmostlyreferstosoftware-basedtechniques.Thesetechniquesmayproduceoutputimagesthatcannotbeobtainedwitha“traditional”digitalcamera.Thischapterintroducessomeofthelesser-knowntechniquesavailableinOpenCVforcomputationalphotography:high-dynamic-rangeimaging,seamlesscloning,decolorization,andnon-photorealisticrendering.Thesethreeareinsidethephotomoduleofthelibrary.Notethatothertechniquesinsidethismodule(inpaintinganddenoising)havebeenalreadyconsideredinpreviouschapters.

High-dynamic-rangeimagesThetypicalimagesweprocesshave8bitsperpixel(bpp).Colorimagesalsouse8bitstorepresentthevalueofeachchannel,thatis,red,green,andblue.Thismeansthatonly256differentintensityvaluesareused.This8bpplimithasprevailedthroughoutthehistoryofdigitalimaging.However,itisobviousthatlightinnaturedoesnothaveonly256differentlevels.Weshould,therefore,considerwhetherthisdiscretizationisdesirableorevensufficient.Thehumaneye,forexample,isknowntocaptureamuchhigherdynamicrange(thenumberoflightlevelsbetweenthedimmestandbrightestlevels),estimatedatbetween1and100millionlightlevels.Withonly256lightlevels,therearecaseswherebrightlightsappearoverexposedorsaturated,whiledarkscenesaresimplycapturedasblack.

Therearecamerasthatcancapturemorethan8bpp.However,themostcommonwaytocreatehigh-dynamic-rangeimagesistousean8bppcameraandtakeimageswithdifferentexposurevalues.Whenwedothis,problemsofalimiteddynamicrangeareevident.Consider,forexample,thefollowingfigure:

Ascenecapturedwithsixdifferentexposurevalues

NoteThetop-leftimageismostlyblack,butwindowdetailsarevisible.Conversely,thebottom-rightimageshowsdetailsoftheroom,butthewindowdetailsarebarelyvisible.

Wecantakepictureswithdifferentexposurelevelsusingmodernsmartphonecameras.WithiPhoneandiPads,forexample,asofiOS8,itisveryeasytochangetheexposurewiththenativecameraapp.Bytouchingthescreen,ayellowboxappearswithasmallsunonitsside.Swipingupordowncanthenchangetheexposure(seethefollowingscreenshot).

Note

Therangeofexposurelevelsisquitelarge,sowemayhavetorepeattheswipinggestureanumberoftimes.

IfyouusepreviousversionsofiOS,youcandownloadcameraappssuchasCamera+thatallowyoutofocusonaspecificpointandchangeexposure.

ForAndroid,tonsofcameraappsareavailableonGooglePlaythatcanadjusttheexposure.OneexampleisCameraFV-5,whichhasbothfreeandpaidversions.

TipIfyouuseahandhelddevicetocapturetheimages,makesurethedeviceisstatic.Infact,youmaywelluseatripod.Otherwise,imageswithdifferentexposureswillnotbealigned.Also,movingsubjectswillinevitablyproduceghostartifacts.Threeimagesaresufficientformostcases,withlow,medium,andhighexposurelevels.

TheexposurecontrolusingthenativecameraappinaniPhone5S

Smartphonesandtablesarehandytocaptureanumberofimageswithdifferentexposures.TocreateHDRimages,weneedtoknowtheexposure(orshutter)timeforeachcapturedimage(seethefollowingsectionforthereason).Notallappsallowyoutocontrol(orevensee)thismanually(theiOS8nativeappdoesn’t).Atthetimeofwritingthis,atleasttwofreeappsallowthisforiOS:ManuallyandManualShot!InAndroid,thefreeCameraFV-5allowsyoutocontrolandseeexposuretimes.NotethatF/StopandISOaretwootherparametersthatcontroltheexposure.

ImagesthatarecapturedcanbetransferredtothedevelopmentcomputerandusedtocreatetheHDRimage.

Note

AsofiOS7,thenativecameraapphasanHDRmodethatautomaticallycapturesthreeimagesinarapidsequence,eachwithdifferentexposure.Theseimagesarealsoautomaticallycombinedintoasingle(sometimesbetter)image.

CreatingHDRimagesHowdowecombinemultiple(three,forexample)exposureimagesintoanHDRimage?Ifweconsideronlyoneofthechannelsandagivenpixel,thethreepixelvalues(oneforeachexposurelevel)mustbemappedtoasinglevalueinthelargeroutputrange(say,16bpp).Thismappingisnoteasy.Firstofall,wehavetoconsiderthatpixelintensitiesarea(rough)measureofsensorirradiance(theamountoflightincidentonthecamerasensor).Digitalcamerasmeasureirradiancebutinanonlinearway.Camerashaveanonlinearresponsefunctionthattranslatesirradiancetopixelintensityvaluesintherangeof0to255].Inordertomapthesevaluestoalargersetofdiscretevalues,wemustestimatethecameraresponsefunction(thatis,theresponsewithinthe0to255range).

Howdoweestimatethecameraresponsefunction?Wedothatfromthepixelsthemselves!TheresponsefunctionisanS-shapedcurveforeachcolorchannel,anditcanbeestimatedfromthepixels(withthreeexposuresofapixel,wehavethreepointsonthecurveforeachcolorchannel).Asthisisverytimeconsuming,usually,asetofrandompixelsischosen.

There’sonlyonethingleft.Wepreviouslytalkedaboutestimatingtherelationshipbetweenirradianceandpixelintensity.Howdoweknowirradiance?Sensorirradianceisdirectlyproportionaltotheexposuretime(orequivalently,theshutterspeed).Thisisthereasonwhyweneedexposuretime!

Finally,theHDRimageiscomputedasaweightedsumoftherecoveredirradiancevaluesfromthepixelsofeachexposure.Notethatthisimagecannotbedisplayedonconventionalscreens,whichalsohavealimitedrange.

NoteAgoodbookonhigh-dynamic-rangeimagingisHighDynamicRangeImaging:Acquisition,Display,andImage-BasedLightingbyReinhardetal,MorganKaufmannPub.ThebookisaccompaniedbyaDVDcontainingimagesindifferentHDRformats.

ExampleOpenCV(asof3.0only)providesfunctionstocreateHDRimagesfromasetofimagestakenwithdifferentexposures.There’sevenatutorialexamplecalledhdr_imaging,whichreadsalistofimagefilesandexposuretimes(fromatextfile)andcreatestheHDRimage.

NoteInordertorunthehdr_imagingtutorial,youwillneedtodownloadtherequiredimagefilesandtextfileswiththelist.Youcandownloadthemfromhttps://github.com/Itseez/opencv_extra/tree/master/testdata/cv/hdr.

TheCalibrateDebevecandMergeDebevecclassesimplementDebevec’smethodtoestimatethecameraresponsefunctionandmergetheexposuresintoanHDRimage,respectively.ThefollowingcreateHDRexampleshowsyouhowtousebothclasses:

#include<opencv2/photo.hpp>

#include<opencv2/highgui.hpp>

#include<iostream>

usingnamespacecv;

usingnamespacestd;

intmain(int,char**argv)

{

vector<Mat>images;

vector<float>times;

//Loadimagesandexposures…

Matimg1=imread("1div66.jpg");

if(img1.empty())

{

cout<<"Error!Inputimagecannotberead…\n";

return-1;

}

Matimg2=imread("1div32.jpg");

Matimg3=imread("1div12.jpg");

images.push_back(img1);

images.push_back(img2);

images.push_back(img3);

times.push_back((float)1/66);

times.push_back((float)1/32);

times.push_back((float)1/12);

//Estimatecameraresponse…

Matresponse;

Ptr<CalibrateDebevec>calibrate=createCalibrateDebevec();

calibrate->process(images,response,times);

//Showtheestimatedcameraresponsefunction…

cout<<response;

//CreateandwritetheHDRimage…

Mathdr;

Ptr<MergeDebevec>merge_debevec=createMergeDebevec();

merge_debevec->process(images,hdr,times,response);

imwrite("hdr.hdr",hdr);

cout<<"\nDone.Pressanykeytoexit…\n";

waitKey();//Waitforkeypress

return0;

}

Theexampleusesthreeimagesofacup(theimagesareavailablealongwiththecodeaccompanyingthisbook).TheimagesweretakenwiththeManualShot!appmentionedpreviously,usingexposuresof1/66,1/32,and1/12seconds;refertothefollowingfigure:

Thethreeimagesusedintheexampleasinputs

NotethatthecreateCalibrateDebevecmethodexpectstheimagesandexposuretimesinanSTLvector(STLisakindoflibraryofusefulcommonfunctionsanddatastructuresavailableinstandardC++).Thecameraresponsefunctionisgivenasa256real-valuedvector.Thisrepresentsthemappingbetweenthepixelvalueandirradiance.Actually,itisa256x3matrix(onecolumnpereachofthethreecolorchannels).Thefollowingfigureshowsyoutheresponsegivenbytheexample:

TheestimatedRGBcameraresponsefunctions

TipThecoutpartofcodedisplaysthematrixintheformatusedbyMATLABandOctave,twowidelyusedpackagesfornumericalcomputation.ItisstraightforwardtocopythematrixintheoutputandpasteitinMATLAB/Octaveinordertodisplayit.

TheresultingHDRimageisstoredinthelosslessRGBEformat.Thisimageformatusesonebytepercolorchannelplusonebyteasasharedexponent.Theformatusesthesameprincipleastheoneusedinthefloating-pointnumberrepresentation:thesharedexponentallowsyoutorepresentamuchwiderrangeofvalues.RGBEimagesusethe.hdrextension.Notethatasitisalosslessimageformat,.hdrfilesarerelativelylarge.Inthisexample,theRGBinputimagesare1224x1632each(100to200KBeach),whiletheoutput.hdrfileoccupies5.9MB.

TheexampleusesDebevecandMalik’smethod,butOpenCValsoprovidesanothercalibrationfunctionbasedonRobertson’smethod.Bothcalibrationandmergefunctionsareavailable,thatis,createCalibrateRobertsonandMergeRobertson.

NoteFormoreinformationontheotherfunctionsandthetheorybehindthem,refertohttp://docs.opencv.org/trunk/modules/photo/doc/hdr_imaging.html.

Finally,notethattheexampledoesnotdisplaytheresultingimage.TheHDRimagecannotbedisplayedinconventionalscreens,soweneedtoperformanotherstepcalledtonemapping.

TonemappingWhenhigh-dynamic-rangeimagesaretobedisplayed,informationcanbelost.Thisisduetothefactthatcomputerscreensalsohavealimitedcontrastratio,andprintedmaterialisalsotypicallylimitedto256tones.Whenwehaveahigh-dynamic-rangeimage,itisnecessarytomaptheintensitiestoalimitedsetofvalues.Thisiscalledtonemapping.

SimplyscalingtheHDRimagevaluestothereducedrangeofthedisplaydeviceisnotsufficientinordertoprovidearealisticoutput.Scalingtypicallyproducesimagesthatappearaslackingdetail(contrast),eliminatingtheoriginalscenecontent.Ultimately,tone-mappingalgorithmsaimatprovidingoutputsthatappearvisuallysimilartotheoriginalscene(thatis,similartowhatahumanwouldseewhenviewingthescene).Varioustone-mappingalgorithmshavebeenproposedanditisstillamatterofextensiveresearch.ThefollowinglinesofcodecanapplytonemappingtotheHDRimageobtainedinthepreviousexample:

Matldr;

Ptr<TonemapDurand>tonemap=createTonemapDurand(2.2f);

tonemap->process(hdr,ldr);//ldrisafloatingpointimagewith

ldr=ldr*255;//valuesininterval[0..1]

imshow("LDR",ldr);

ThemethodwasproposedbyDurandandDorseyin2002.Theconstructoractuallyacceptsanumberofparametersthataffecttheoutput.Thefollowingfigureshowsyoutheoutput.Notehowthisimageisnotnecessarilybetterthananyofthethreeoriginalimages:

Thetone-mappedoutput

Threeothertone-mappingalgorithmsareavailableinOpenCV:createTonemapDrago,

createTonemapReinhard,andcreateTonemapMantiuk.

AnHDRimage(theRGBEformat,thatis,fileswiththe.hdrextension)canbedisplayedusingMATLAB.Allittakesisthreelinesofcode:

hdr=hdrread('hdr.hdr');

rgb=tonemap(hdr);

imshow(rgb);

Notepfstoolsisanopensourcesuiteofcommand-linetoolstoread,write,andrenderHDRimages.Thesuite,whichcanread.hdrandotherformats,includesanumberofcameracalibrationandtone-mappingalgorithms.LuminanceHDRisfreeGUIsoftwarebasedonpfstools.

AlignmentThescenethatwillbecapturedwithmultipleexposureimagesmustbestatic.Thecameramustalsobestatic.Evenifthetwoconditionsmet,itisadvisabletoperformanalignmentprocedure.

OpenCVprovidesanalgorithmforimagealignmentproposedbyG.Wardin2003.Themainfunction,createAlignMTB,takesaninputparameterthatdefinesthemaximumshift(actually,alogarithmthebasetwoofthemaximumshiftineachdimension).Thefollowinglinesshouldbeinsertedrightbeforeestimatingthecameraresponsefunctioninthepreviousexample:

vector<Mat>images_(images);

Ptr<AlignMTB>align=createAlignMTB(4);//4=max16pixelshift

align->process(images_,images);

ExposurefusionWecanalsocombineimageswithmultipleexposureswithneithercameraresponsecalibration(thatis,exposuretimes)norintermediateHDRimage.Thisiscalledexposurefusion.ThemethodwasproposedbyMertensetalin2007.Thefollowinglinesperformexposurefusion(imagesistheSTLvectorofinputimages;refertothepreviousexample):

Matfusion;

Ptr<MergeMertens>merge_mertens=createMergeMertens();

merge_mertens->process(images,fusion);//fusionisa

fusion=fusion*255;//float.pointimagew.valuesin[0..1]

imwrite("fusion.png",fusion);

Thefollowingfigureshowsyoutheresult:

Exposurefusion

SeamlesscloningInphotomontages,wetypicallywanttocutanobject/personinasourceimageandinsertitintoatargetimage.Ofcourse,thiscanbedoneinastraightforwardwaybysimplypastingtheobject.However,thiswouldnotproducearealisticeffect.See,forexample,thefollowingfigure,inwhichwewantedtoinserttheboatinthetophalfoftheimageintotheseaatthebottomhalfoftheimage:

Cloning

AsofOpenCV3,thereareseamlesscloningfunctionsavailableinwhichtheresultismorerealistic.ThisfunctioniscalledseamlessCloneanditusesamethodproposedbyPerezandGangnetin2003.ThefollowingseamlessCloningexampleshowsyouhowitcanbeused:

#include<opencv2/photo.hpp>

#include<opencv2/highgui.hpp>

#include<iostream>

usingnamespacecv;

usingnamespacestd;

intmain(int,char**argv)

{

//Loadandshowimages…

Matsource=imread("source1.png",IMREAD_COLOR);

Matdestination=imread("destination1.png",IMREAD_COLOR);

Matmask=imread("mask.png",IMREAD_COLOR);

imshow("source",source);

imshow("mask",mask);

imshow("destination",destination);

Matresult;

Pointp;//pwillbeneartoprightcorner

p.x=(float)2*destination.size().width/3;

p.y=(float)destination.size().height/4;

seamlessClone(source,destination,mask,p,result,NORMAL_CLONE);

imshow("result",result);

cout<<"\nDone.Pressanykeytoexit…\n";

waitKey();//Waitforkeypress

return0;

}

Theexampleisstraightforward.TheseamlessClonefunctiontakesthesource,destination,andmaskimagesandapointinthedestinationimageinwhichthecroppedobjectwillbeinserted(thesethreeimagescanbedownloadedfromhttps://github.com/Itseez/opencv_extra/tree/master/testdata/cv/cloning/Normal_Cloning).Seetheresultinthefollowingfigure:

Seamlesscloning

ThelastparameterofseamlessClonerepresentstheexactmethodtobeused(therearethreemethodsavailablethatproduceadifferentfinaleffect).Ontheotherhand,thelibraryprovidesthefollowingrelatedfunctions:

Function Effect

colorChangeMultiplieseachofthethreecolorchannelsofthesourceimagebyafactor,applyingthemultiplicationonlyintheregiongivenbythemask

illuminationChange Changesilluminationofthesourceimage,onlyintheregiongivenbythemask

textureFlattening Washesouttexturesinthesourceimage,onlyintheregiongivenbythemask

AsopposedtoseamlessClone,thesethreefunctionsonlyacceptsourceandmaskimages.

DecolorizationDecolorizationistheprocessofconvertingacolorimagetograyscale.Giventhisdefinition,thereadermaywellask,don’twealreadyhavegrayscaleconversion?Yes,grayscaleconversionisabasicroutineinOpenCVandanyimage-processinglibrary.ThestandardconversionisbasedonalinearcombinationoftheR,G,andBchannels.Theproblemisthatsuchaconversionmayproduceimagesinwhichcontrastintheoriginalimageislost.Thereasonisthattwodifferentcolors(whichareperceivedascontrastsintheoriginalimage)mayendupbeingmappedtothesamegrayscalevalue.Considertheconversionoftwocolors,AandB,tograyscale.Let’ssupposethatBisavariationofAintheRandGchannels:

A=(R,G,B)=>G=(R+G+B)/3

B=(R-x,G+x,B)=>G=(R-x+G+x+B)/3=(R+G+B)/3

Eventhoughtheyareperceivedasdistinct,thetwocolorsAandBaremappedtothesamegrayscalevalue!Theimagesfromthefollowingdecolorizationexampleshowthis:

#include<opencv2/photo.hpp>

#include<opencv2/highgui.hpp>

#include<iostream>

usingnamespacecv;

usingnamespacestd;

intmain(int,char**argv)

{

//Loadandshowimages…

Matsource=imread("color_image_3.png",IMREAD_COLOR);

imshow("source",source);

//firstcomputeandshowstandardgrayscaleconversion…

Matgrayscale=Mat(source.size(),CV_8UC1);

cvtColor(source,grayscale,COLOR_BGR2GRAY);

imshow("grayscale",grayscale);

//nowcomputeandshowdecolorization…

Matdecolorized=Mat(source.size(),CV_8UC1);

Matdummy=Mat(source.size(),CV_8UC3);

decolor(source,decolorized,dummy);

imshow("decolorized",decolorized);

cout<<"\nDone.Pressanykeytoexit…\n";

waitKey();//Waitforkeypress

return0;

}

Decolorizationexampleoutput

Theexampleisstraightforward.Afterreadingtheimageandshowingtheresultofastandardgrayscaleconversion,itusesthedecolorfunctiontoperformthedecolorization.Theimageused(thecolor_image_3.pngfile)isincludedintheopencv_extrarepositoryathttps://github.com/Itseez/opencv_extra/tree/master/testdata/cv/decolor.

NoteTheimageusedintheexampleisactuallyanextremecase.Itscolorshavebeenchosensothatthestandardgrayscaleoutputisfairlyhomogeneous.

Non-photorealisticrenderingAspartofthephotomodule,fourfunctionsareavailablethattransformaninputimageinawaythatproducesanon-realisticbutstillartisticoutput.ThefunctionsareveryeasytouseandaniceexampleisincludedwithOpenCV(npr_demo).Forillustrativepurposes,hereweshowyouatablethatallowsyoutograsptheeffectofeachfunction.Takealookatthefollowingfruits.jpginputimage,includedwithOpenCV:

Theinputreferenceimage

Theeffectsare:

Function Effect

edgePreservingFilterSmoothingisahandyandfrequentlyusedfilter.Thisfunctionperformssmoothingwhilepreservingobjectedgedetails.

detailEnhance Enhancesdetailsintheimage

pencilSketch Apencil-likelinedrawingversionoftheinputimage

stylization Watercoloreffect

SummaryInthischapter,youlearnedwhatcomputationalphotographyisandtherelatedfunctionsavailableinOpenCV3.Weexplainedthemostimportantfunctionswithinthephotomodule,butnotethatotherfunctionsofthismodule(inpaintingandnoisereduction)werealsoconsideredinpreviouschapters.Computationalphotographyisarapidlyexpandingfield,withstrongtiestocomputergraphics.Therefore,thismoduleofOpenCVisexpectedtogrowinfutureversions.

Thenextchapterwillbedevotedtoanimportantaspectthatwehavenotyetconsidered:time.Manyofthefunctionsexplainedtakeasignificanttimetocomputetheresults.Thenextchapterwillshowyouhowtodealwiththatusingmodernhardware.

Chapter7.AcceleratingImageProcessingThischapterdealswiththeaccelerationofimageprocessingtasksusingGeneralPurposeGraphicsProcessingUnits(GPGPUs)or,inshort,GPUswithparallelprocessing.AGPUisessentiallyacoprocessordedicatedtographicsprocessingorfloatingpointoperations,aimedatimprovingperformanceonapplicationssuchasvideogamesandinteractive3Dgraphics.WhilethegraphicsprocessingisexecutedintheGPU,theCPUcanbededicatedtoothercalculations(suchastheartificialintelligencepartingames).EveryGPUisequippedwithhundredsofsimpleprocessingcoresthatallowmassiveparallelexecutiononhundredsof“simple”mathematicaloperationson(normally)floatingpointnumbers.

CPUsseemtohavereachedtheirspeedandthermalpowerlimits.BuildingacomputerwithseveralCPUshasbecomeacomplexproblem.ThisiswhereGPUscomeintoplay.GPUprocessingisanewcomputingparadigmthatusestheGPUtoimprovethecomputationalperformance.GPUsinitiallyimplementedcertainparalleloperationscalledgraphicsprimitivesthatwereoptimizedforgraphicsprocessing.Oneofthemostcommonprimitivesfor3Dgraphicsprocessingisantialiasing,whichmakestheedgesofthefigureshaveamorerealisticappearance.Otherprimitivesaredrawingsofrectangles,triangles,circles,andarcs.GPUscurrentlyincludehundredsofgeneral-purposeprocessingfunctionsthatcandomuchmorethanrenderinggraphics.Particularly,theyareveryvaluableintasksthatcanbeparallelized,whichisthecaseformanycomputervisionalgorithms.

OpenCVlibrariesincludesupportfortheOpenCLandCUDAGPUarchitectures.CUDAimplementsmanyalgorithms;however,itonlyworkswithNVIDIAgraphiccards.CUDAisaparallelcomputingplatformandprogrammingmodelcreatedbyNVIDIAandimplementedbytheGPUsthattheyproduce.ThischapterfocusesontheOpenCLarchitecture,asitissupportedbymoredevicesandisevenincludedinsomeNVIDIAgraphiccards.

TheOpenComputingLanguage(OpenCL)isaframeworkthatwritesprogramsthatcanbeexecutedonCPUsorGPUsattachedtoahostprocessor(aCPU).ItdefinesaC-likelanguagetowritefunctions,calledkernels,whichareexecutedonthecomputingdevices.UsingOpenCL,kernelscanberunonallormanyoftheindividualprocessingelements(PEs)inparalleltotheCPUsorGPUs.

Inaddition,OpenCLdefinesanApplicationProgrammingInterface(API)thatallowsprogramsrunningonthehost(theCPU)tolaunchkernelsonthecomputerdevicesandmanagetheirdevicememories,whichare(atleastconceptually)separatedfromthehostmemory.OpenCLprogramsareintendedtobecompiledatruntimesothatapplicationsthatuseOpenCLareportablebetweenimplementationsofvarioushostcomputerdevices.OpenCLisalsoanopenstandardmaintainedbythenonprofittechnologyconsortiumKhronosGroup(https://www.khronos.org/).

OpenCVcontainsasetofclassesandfunctionsthatimplementandacceleratetheOpenCVfunctionalityusingOpenCL.OpenCVcurrentlyprovidesatransparentAPIthatenablestheunificationofitsoriginalAPIwithOpenCL-acceleratedprogramming.

Therefore,youonlyneedtowriteyourcodeonce.Thereisanewunifieddatastructure(UMat)thathandlesdatatransferstotheGPUswhenitisneededandpossible.

SupportforOpenCLinOpenCVhasbeendesignedforeaseofuseanddoesnotrequireanyknowledgeofOpenCL.Ataminimumlevel,itcanbeviewedasasetofaccelerations,whichcantakeadvantageofthehighcomputingpowerwhenusingmodernCPUandGPUdevices.

TocorrectlyrunOpenCLprograms,theOpenCLruntimeshouldbeprovidedbythedevicevendor,typicallyintheformofadevicedriver.Also,touseOpenCVwithOpenCL,acompatibleSDKisneeded.Currently,therearefiveavailableOpenCLSDKs:

AMDAPPSDK:ThisSDKsupportsOpenCLonCPUsandGPUs,suchasX86+SSE2(orhigher)CPUsandAMDFusion,AMDRadeon,AMDMobility,andATIFireProGPUs.IntelSDK:ThisSDKsupportsOpenCLonIntelCoreprocessorsandIntelHDGPUs,suchasIntel+SSE4.1,SSE4.2orAVX,IntelCorei7,i5andi3(1st,2nd,and3rdGeneration),IntelHDGraphics,IntelCore2Solo(DuoQuadandExtreme),andIntelXeonCPUs.IBMOpenCLDevelopmentKit:ThisSDKsupportsOpenCLonAMDserverssuchasIBMPower,IBMPERCS,andIBMBladeCenter.IBMOpenCLCommonRuntime:ThisSDKsupportsOpenCVonCPUsandGPUs,suchasX86+SSE2(orhigher)CPUsandAMDFusionandRaedon,NVIDIAIon,NVIDIAGeForce,andNVIDIAQuadroGPUs.NvidiaOpenCLDriverandTools:ThisSDKsupportsOpenCLonsomeNvidiagraphicdevicessuchasNVIDIATesla,NVIDIAGeForce,NVIDIAIon,andNVIDIAQuadroGPUs.

OpenCVwiththeOpenCLinstallationTheinstallationstepsalreadypresentedinChapter1,HandlingImageandVideoFiles,needsomeadditionalstepstoincludeOpenCL.Thenewlyrequiredsoftwareisexplainedinthefollowingsection.

TherearenewrequirementstocompileandinstallOpenCVwithOpenCLonWindows:

OpenCL-capableGPUorCPU:Thisisthemostimportantrequirement.NotethatOpenCLsupportsmanycomputingdevicesbutnotall.YoucancheckwhetheryourgraphiccardsorprocessorsarecompatiblewithOpenCL.ThischapterusestheAMDAPPSDKforanAMDFireProW5000GPUtoexecutetheexamples.

NoteThereisalistwiththesupportedcomputerdevicesforthisSDKathttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/system-requirements-driver-compatibility/.There,youcanalsoconsulttheminimumSDKversionthatyouneed.

Compilers:OpenCVwithOpenCLiscompatiblewithMicrosoftandMinGWcompilers.ItispossibletoinstallthefreeVisualStudioExpressedition.However,ifyouchooseMicrosofttocompileOpenCV,atleastVisualStudio2012isrecommended.However,theMinGWcompilerisusedinthischapter.AMDAPPSDK:ThisSDKisasetofadvancedsoftwaretechnologiesthatenableustousecompatiblecomputingdevicestoexecuteandacceleratemanyapplicationsbeyondjustgraphics.ThisSDKisavailableathttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/.ThischapterusesVersion2.9oftheSDK(for64bitWindows);youcanseetheinstallationprogressinthefollowingscreenshot.

NoteIfthisstepfails,maybeyoumightneedtoupdatethecontrollerofyourgraphiccard.TheAMDcontrollersareavailableathttp://www.amd.com/en-us/innovations/software-technologies.

InstallingtheAMDAPPSDK

OpenCLBLAS:BasicLinearAlgebraSubroutines(BLAS)isasetofopensourcemathlibrariesforparallelprocessingonAMDdevices.Itcanbedownloadedfromhttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-math-libraries/.Thischapterusesthe1.1BLASversionforWindows32/64bits,andyoucanseetheinstallationprogressinthefollowingscreenshot(theleft-handside).OpenCLFFT:FastFourierTransform(FFT)isaveryusefulfunctionthatmanyalgorithmsofimageprocessingneed.Therefore,thisfunctionisimplementedforparallelprocessingonAMDdevices.ItcanbedownloadedfromthesameURLasgivenpreviously.Thischapterusesthe1.1FFTversionforWindows32/64bits,andyoucanseetheinstallationprogressinthefollowingscreenshot(theright-handside):

InstallingBLASandFFTforOpenCL

QtlibrariesforC++compiler:Inthischapter,theMinGWbinariesofQtlibrariesareusedtocompileOpenCVwithOpenCL.TheotheralternativeistoinstallthelatestversionofQtandusetheVisualC++compiler.YoucanchoosetheQtversionandusedcompiler.Thepackagemanager,bymeansoftheMaintenanceTool.exeapplicationlocatedatC:\Qt\Qt5.3.1,canbeusedtodownloadotherQtversions.ThischapterusesQt(5.3.1)andMinGW(4.8.2)32bitstocompileOpenCVwithOpenCL.

Whenthepreviousrequirementsaremet,youcangenerateanewbuildconfigurationwithCMake.Thisprocessdiffersinsomepointsfromthetypicalinstallationthatwasexplainedinthefirstchapter.Thedifferencesareexplainedinthislist:

Whenselectingthegeneratorfortheproject,youcanchoosethecompilerversioncorrespondingwiththeinstalledenvironmentinyourmachine.ThischapterusesMinGWtocompileOpenCVwithOpenCL,andthentheMinGWMakefilesoptionisselected,specifyingthenativecompilers.Thefollowingscreenshotshowsthisselection:

CMakeselectingthegeneratorproject

TheoptionsshowninthefollowingscreenshotareneededtobuildtheOpenCVwithOpenCLproject.TheWITH_OPENCL,WITH_OPENCLAMDBLAS,andWITH_OPENCLAMDFFToptionsmustbeenabled.TheBLASandFFTpathsmustbeintroducedonCLAMDBLAS_INCLUDE_DIR,CLAMDBLAS_ROOT_DIR,CLAMDFFT_INCLUDE_DIR,andCLAMDFFT_ROOT_DIR.Inaddition,asshowninChapter1,HandlingImageandVideoFiles,youwillneedtoenableWITH_QTanddisabletheWITH_IPPoptionaswell.ItisalsoadvisabletoenableBUILD_EXAMPLES.Thefollowingscreenshotshowsyouthemainoptionsselectedinthebuildconfiguration:

CMakeselectingthemainoptions

Finally,tobuildtheOpenCVwithOpenCLproject,theCMakeprojectthatwaspreviouslygeneratedmustbecompiled.TheprojectwasgeneratedforMinGW,andtherefore,theMinGWcompilerisneededtobuildthisproject.Firstly,the[opencv_build]/folderisselectedwithWindowsConsole,andweexecutethis:

./mingw32-make.exe-j4install

The-j4parameteristhenumberofthecoreCPUsofthesystemthatwewanttousefor

theparallelizationofthecompilation.

NowtheOpenCVwithOpenCLprojectisreadytobeused.Thepathofthenewbinariesfilesmustbeaddedtothesystempath,inthiscase,[opencv_build]/install/x64/mingw/bin.

NoteDonotforgettoremovetheoldbinaryfilesofOpenCVfromthepathenvironmentvariable.

AquickrecipetoinstallOpenCVwithOpenCLTheinstallationprocesscanbesummarizedinthefollowingsteps:

1. DownloadandinstallAMDAPPSDK,whichisavailableathttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk.

2. DownloadandinstallBLASandFFTAMD,whichareavailableathttp://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-math-libraries.

3. ConfiguretheOpenCVbuildwithCMake.EnabletheWITH_OPENCL,WITH_OPENCLAMDBLAS,WITH_QT,andBuild_EXAMPLESWITH_OPENCLAMDFFToptions.DisabletheWITH_IPPoption.Finally,introducetheBLASandFFTpathsonCLAMDBLAS_INCLUDE_DIR,CLAMDBLAS_ROOT_DIR,CLAMDFFT_INCLUDE_DIR,andCLAMDFFT_ROOT_DIR.

4. CompiletheOpenCVprojectwithmingw32-make.exe.5. Finally,modifythepathenvironmentvariabletoupdatetheOpenCVbindirectory

(forexample,[opencv_build]/install/x64/mingw/bin).

ChecktheGPUusageWhentheGPUisbeingusedonaWindowsplatform,thereisnoapplicationtomeasureitsusage.TheGPUusageisusefulfortworeasons:

ItispossibletoknowwhetheryouareusingtheGPUcorrectlyYoucanmonitortheGPUusagepercentage

Therearesomeapplicationsinthemarketforthispurpose.ThischapterusesAMDSystemMonitortochecktheGPUusage.ThisapplicationmonitorstheCPU,memoryRAM,andGPUusage.Refertothefollowingscreenshotforthis:

AMDSystemMonitormonitorstheCPU,GPU,andMemoryRAMusages

Note

TheAMDSystemMonitorcanbedownloadedfromhttp://support.amd.com/es-xl/kb-articles/Pages/AMDSystemMonitor.aspxforMicrosoftWindows(32or64bits).

AcceleratingyourownfunctionsInthissection,therearethreeexamplesofusingOpenCVwithOpenCL.ThefirstexampleallowsyoutocheckwhethertheinstalledSDKisavailableandobtainusefulinformationaboutthecomputingdevicesthatsupportOpenCL.ThesecondexampleshowsyoutwoversionsofthesameprogramusingCPUandGPUprogramming,respectively.Thelastexampleisacompleteprogramtodetectandmarkfaces.Inaddition,acomputationalcomparativeisperformed.

CheckingyourOpenCLThefollowingisasimpleprogramthatisshowntocheckyourSDKandtheavailablecomputingdevices.ThisexampleiscalledcheckOpenCL.ItallowsyoutodisplaythecomputerdevicesusingtheOCLmoduleofOpenCV:

#include<opencv2/opencv.hpp>

#include<opencv2/core/ocl.hpp>

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::ocl;

intmain()

{

vector<ocl::PlatformInfo>info;

getPlatfomsInfo(info);

PlatformInfosdk=info.at(0);

if(sdk.deviceNumber()<1)

return-1;

cout<<"******SDK*******"<<endl;

cout<<"Name:"<<sdk.name()<<endl;

cout<<"Vendor:"<<sdk.vendor()<<endl;

cout<<"Version:"<<sdk.version()<<endl;

cout<<"Numberofdevices:"<<sdk.deviceNumber()<<endl;

for(inti=0;i<sdk.deviceNumber();i++){

Devicedevice;

sdk.getDevice(device,i);

cout<<"\n\n*********************\nDevice"<<i+1<<endl;

cout<<"VendorID:"<<device.vendorID()<<endl;

cout<<"Vendorname:"<<device.vendorName()<<endl;

cout<<"Name:"<<device.name()<<endl;

cout<<"Driverversion:"<<device.driverVersion()<<endl;

if(device.isAMD())cout<<"IsanAMDdevice"<<endl;

if(device.isIntel())cout<<"IsaInteldevice"<<endl;

cout<<"GlobalMemorysize:"<<device.globalMemSize()<<endl;

cout<<"Memorycachesize:"<<device.globalMemCacheSize()<<endl;

cout<<"Memorycachetype:"<<device.globalMemCacheType()<<endl;

cout<<"LocalMemorysize:"<<device.localMemSize()<<endl;

cout<<"LocalMemorytype:"<<device.localMemType()<<endl;

cout<<"MaxClockfrequency:"<<device.maxClockFrequency()<<

endl;

}

return0;

}

ThecodeexplanationThisexampledisplaystheinstalledSDKandtheavailablecomputingdevicesthatarecompatiblewithOpenCL.Firstly,thecore/ocl.hppheaderisincludedandthecv::ocl

namespaceisdeclared.

TheinformationabouttheavailableSDKinyourcomputerisobtainedusingthegetPlatfomsInfo(info)method.Thisinformationisstoredinthevector<ocl::PlatformInfo>infovectorandselectedwithPlatformInfosdk=info.at(0).Afterwards,themaininformationaboutyourSDKisshown,suchasthename,vendor,SDKversion,andthenumberofcomputingdevicescompatiblewithOpenCL.

Finally,foreachcompatibledevice,itsinformationisobtainedwiththesdk.getDevice(device,i)method.Nowdifferentinformationabouteachcomputingdevicecanbeshown,suchasthevendorID,vendorname,driverversion,globalmemorysize,memorycachesize,andsoon.

Thefollowingscreenshotshowsyoutheresultsofthisexampleforthecomputerused:

InformationabouttheSDKusedandcompatiblecomputingdevices

YourfirstGPU-basedprogramInthefollowingcode,twoversionsofthesameprogramareshown:oneonlyusestheCPU(native)toperformthecomputationsandtheotherusestheGPU(withOpenCL).ThesetwoexamplesarecalledcalculateEdgesCPUandcalculateEdgesGPU,respectively,andallowyoutoobservethedifferencesbetweenCPUandGPUversions.

ThecalculateEdgesCPUexampleisshowninthefirstplace:

#include<opencv2/opencv.hpp>

usingnamespacestd;

usingnamespacecv;

intmain(intargc,char*argv[])

{

if(argc<2)

{

cout<<"./calculateEdgesCPU<image>"<<endl;

return-1;

}

MatcpuFrame=imread(argv[1]);

MatcpuBW,cpuBlur,cpuEdges;

namedWindow("CannyEdgesCPU",1);

cvtColor(cpuFrame,cpuBW,COLOR_BGR2GRAY);

GaussianBlur(cpuBW,cpuBlur,Size(1,1),1.5,1.5);

Canny(cpuBlur,cpuEdges,50,100,3);

imshow("CannyEdgesCPU",cpuEdges);

waitKey();

return0;

}

Now,thecalculateEdgesGPUexampleisshown:

#include"opencv2/opencv.hpp"

#include"opencv2/core/ocl.hpp"

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::ocl;

intmain(intargc,char*argv[])

{

if(argc<2)

{

cout<<"./calculateEdgesGPU<image>"<<endl;

return-1;

}

setUseOpenCL(true);

MatcpuFrame=imread(argv[1]);

UMatgpuFrame,gpuBW,gpuBlur,gpuEdges;

cpuFrame.copyTo(gpuFrame);

namedWindow("CannyEdgesGPU",1);

cvtColor(gpuFrame,gpuBW,COLOR_BGR2GRAY);

GaussianBlur(gpuBW,gpuBlur,Size(1,1),1.5,1.5);

Canny(gpuBlur,gpuEdges,50,100,3);

imshow("CannyEdgesGPU",gpuEdges);

waitKey();

return0;

}

ThecodeexplanationThesetwoexamplesobtainthesameresult,asshowninthefollowingscreenshot.Theyreadanimagefromthestandardcommand-lineinputarguments.Afterwards,theimageisconvertedtograyscaleandtheGaussianBlurandtheCannyfilterfunctionsareapplied.

Inthesecondexample,therearesomedifferencesthatarerequiredtoworkwiththeGPU.First,OpenCLmustbeactivatedwiththesetUseOpenCL(true)method.Second,UnifiedMats(UMat)areusedtoallocatememoryintheGPU(UMatgpuFrame,gpuBW,gpuBlur,gpuEdges).Third,theinputimageiscopiedfromtheRAMtoGPUmemorywiththecpuFrame.copyTo(gpuFrame)method.Now,whenthefunctionsareused,iftheyhaveanOpenCLimplementation,thenthesefunctionswillbeexecutedontheGPU.IfsomeofthesefunctionsdonothaveanOpenCLimplementation,thenormalfunctionwillbeexecutedontheCPU.Inthisexample,thetimeelapseusingtheGPUprogramming(secondexample)is10timesbetter:

Resultsoftheprecedingtwoexamples

GoingrealtimeOneofthemainadvantagesofGPUprocessingistoperformcomputationsinamuchfasterway.Thisincreaseinspeedallowsyoutoexecuteheavycomputationalalgorithmsinreal-timeapplications,suchasstereovision,pedestriandetection,opticalflow,orfacedetection.ThefollowingdetectFacesexampleshowsyouanapplicationtodetectfacesfromavideocamera.ThisexamplealsoallowsyoutoselectbetweentheCPUorGPUprocessinginordertocomparethecomputationaltime.

IntheOpenCVexamples([opencv_source_code]/samples/cpp/facedetect.cpp),arelatedfacedetectorexamplecanbefound.ForthefollowingdetectFacesexample,thedetectFace.proprojectneedstheselibraries:-lopencv_core300,-opencv_imgproc300,-lopencv_highgui300,-lopencv_videoio300,andlopencv_objdetct300.

ThedetectFacesexampleusestheoclmoduleofOpenCV:

#include<opencv2/core/core.hpp>

#include<opencv2/core/ocl.hpp>

#include<opencv2/objdetect.hpp>

#include<opencv2/videoio.hpp>

#include<opencv2/highgui.hpp>

#include<opencv2/imgproc.hpp>

#include<iostream>

#include<stdio.h>

usingnamespacestd;

usingnamespacecv;

usingnamespacecv::ocl;

intmain(intargc,char*argv[])

{

//1-Settheinitialparameters

//Vectortostorethefaces

vector<Rect>faces;

CascadeClassifierface_cascade;

Stringface_cascade_name=argv[2];

intface_size=30;

doublescale_factor=1.1;

intmin_neighbours=2;

VideoCapturecap(0);

UMatframe,frameGray;

boolfinish=false;

//2-Loadthefilexmltousetheclassifier

if(!face_cascade.load(face_cascade_name))

{

cout<<"Cannotloadthefacexml!"<<endl;

return-1;

}

namedWindow("VideoCapture");

//3-SelectbetweentheCPUorGPUprocessing

if(argc<2)

{

cout<<"./detectFaces[CPU/GPU|C/G]"<<endl;

cout<<"TryingtouseGPU…"<<endl;

setUseOpenCL(true);

}

else

{

cout<<"./detectFacestryingtouse"<<argv[1]<<endl;

if(argv[1][0]=='C')

//TryingtousetheCPUprocessing

setUseOpenCL(false);

else

//TryingtousetheGPUprocessing

setUseOpenCL(true);

}

Rectr;

doublestart_time,finish_time,start_total_time,finish_total_time;

intcounter=0;

//4-Detectthefacesforeachimagecapture

start_total_time=getTickCount();

while(!finish)

{

start_time=getTickCount();

cap>>frame;

if(frame.empty())

{

cout<<"Nocaptureframe-->finish"<<endl;

break;

}

cvtColor(frame,frameGray,COLOR_BGR2GRAY);

equalizeHist(frameGray,frameGray);

//Detectthefaces

face_cascade.detectMultiScale(frameGray,faces,scale_factor,

min_neighbours,0|CASCADE_SCALE_IMAGE,Size(face_size,face_size));

//Foreachdetectedface

for(intf=0;f<faces.size();f++)

{

r=faces[f];

//Drawarectangleovertheface

rectangle(frame,Point(r.x,r.y),Point(r.x+r.width,r.y+r.height),

Scalar(0,255,0),3);

}

//Showtheresults

imshow("VideoCapture",frame);

//Calculatethetimeprocessing

finish_time=getTickCount();

cout<<"Timeperframe:"<<(finish_time-start_time)/getTickFrequency()

<<"secs"<<endl;

counter++;

//PressEsckeytofinish

if(waitKey(1)==27)finish=true;

}

finish_total_time=getTickCount();

cout<<"Averagetimeperframe:"<<((finish_total_time-

start_total_time)/getTickFrequency())/counter<<"secs"<<endl;

return0;

}

ThecodeexplanationThefirststepsetstheinitialparameters,suchasthexmlfile(Stringface_cascade_nameargv[2])thatusestheclassifiertodetectthefaces,theminimumsizeofeachdetectedface(face_size=30),thescalefactor(scale_factor=1.1),andtheminimumnumberofneighbors(min_neighbours=2)tofindatrade-offbetweentruepositiveandfalsepositivedetections.YoucanalsoseethemoreimportantdifferencesbetweentheCPUandGPUsourcecodes;youonlyneedtousetheUnifiedMat(UMatframe,frameGray).

NoteThereareotheravailablexmlfilesinthe[opencv_source_code]/data/haarcascades/foldertodetectdifferentbodypartssuchaseyes,lowerbodies,smiles,andsoon.

Thesecondstepcreatesadetectorusingtheprecedingxmlfiletodetectfaces.ThisdetectorisbasedonaHaarfeature-basedclassifierthatisaneffectiveobject-detectionmethodproposedbyPaulViolaandMichaelJones.Thisclassifierhasahighaccuracyofdetectingfaces.Thissteploadsthexmlfilewiththeface_cascade.load(face_cascade_name)method.

NoteYoucanfindmoredetailedinformationaboutPaulViolaandMichaelJonesmethodathttp://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework.

ThethirdstepallowsyoutoselectbetweentheCPUorGPUprocessing(setUseOpenCL(false)orsetUseOpenCL(true),respectively).Thisexampleusesthestandardcommand-lineinputarguments(argv[1])toselectthis.TheusercanexecutethefollowingfromtheWindowsconsoletoselectbetweenCPUorGPUprocessing,respectively,andtheclassifierpath:

<bin_dir>/detectFacesCPUpathClassifier

<bin_dir>/detectFacesGPUpathClassifier

Iftheuserdoesnotintroduceaninputargument,thenGPUprocessingisused.

Thefourthstepdetectsfacesforeachimagecapturedfromthevideocamera.Beforethat,

eachcapturedimageisconvertedtograyscale(cvtColor(frame,frameGray,COLOR_BGR2GRAY))anditshistogramisequalized(equalizeHist(frameGray,frameGray)).Afterwards,usingthecreatedfacedetector,thedifferentfacesaresearchedoverthecurrentframeusingtheface_cascade.detectMultiScale(frameGray,faces,scale_factor,min_neighbours,0|CASCADE_SCALE_IMAGE,

Size(face_size,face_size))multiscaledetectionmethod.Finally,agreenrectangleisdrawnovereachdetectedface,andthenitisdisplayed.Ascreenshotofthisexamplerunningisshowninthefollowingscreenshot:

Theprecedingexampledetectingaface

TheperformanceIntheprecedingexample,thecomputationaltimeiscalculatedtocomparetheCPUandGPUprocessing.Theaverageprocessingtimeperframeisobtained.

AbigadvantageofchoosingGPUprogrammingisperformance.Therefore,thepreviousexamplecalculatesthetimemeasurementstocomparethespeedupsobtainedwithrespecttotheCPUversion.ThetimeisstoredatthebeginningoftheprogramusingthegetTickCount()method.Afterwards,attheendoftheprogram,thesamefunctiontoestimatethetimeisusedagain.Acounterisstoredtoknowthenumberofiterationsaswell.Finally,theaverageprocessingtimeperframeiscalculated.Theprecedingexamplehasanaverageprocessingtimeperframeof0.057secondsperframe(or17.5FPS)usingtheGPU,whereasthesameexampleusingtheCPUhasanaverageprocessingtimeperframeof0.335secondsperframe(or2.9FPS).Inconclusion,thereisaspeedincrementof6x.Thisincrementissignificant,especiallywhenyouonlyneedtochangeafewlines

ofthecode.However,itispossibletoachieveamuchhigherrateofspeedincrements,whichisrelatedtotheproblemandevenwithhowthekernelsaredesigned.

SummaryInthischapter,youlearnedhowtoinstallOpenCVwithOpenCLonyourcomputeranddevelopapplicationsusingyourcomputerdevices,whicharecompatiblewithOpenCL,withthelastversionofOpenCV.

ThefirstsectionexplainswhatOpenCLisandtheavailableSDKs.Rememberthatdependingyourcomputingdevices,youwillneedaspecificSDKtoworkcorrectlywithOpenCL.Inthesecondsection,theinstallationprocesstoinstallOpenCVwithOpenCLisexplained,andtheAMDAPPSDKhasbeenused.Inthelastsection,therearethreeexamplesusingGPUprogramming(thesecondexamplealsohasaCPUversioninordertocomparethem).Inaddition,inthelastsection,thereisacomputationalcomparativebetweenCPUandGPUprocessing,wheretheGPUisshowntobesixtimesfasterthantheCPUversion.

IndexA

affinetransformationabout/Affinetransformationscaling/Scalingtranslation/Translationimagerotation/Imagerotationskewing/Skewingreflection/Reflection

alignment,HDRimages/AlignmentAMDAPPSDK

URL/OpenCVwiththeOpenCLinstallation,AquickrecipetoinstallOpenCVwithOpenCL

AMDcontrollersURL/OpenCVwiththeOpenCLinstallation

AMDSystemMonitorURL,fordownloading/ChecktheGPUusage

arithmeticoperationsabout/Arithmeticoperations

BbasicAPIconcepts

about/ThebasicAPIconceptsbasicdatatypes

about/BasicdatatypesBayer

about/Bayerexamplecode/Theexamplecode

BilateralTV-L1algorithmabout/Superresolutionparameters/Superresolution

BilateralTV-Lmethodabout/SuperresolutionURL/Superresolution

bitsperpixel(bpp)/High-dynamic-rangeimagesBLAS

URL/OpenCVwiththeOpenCLinstallation,AquickrecipetoinstallOpenCVwithOpenCL

BSDlicenseabout/AnintroductiontoOpenCV

bundleadjustment/Stitchingbuttons

about/Buttons

CC++

OpenCVapplication,creatingwith/GeneralusageofthelibrarycalculateEdgesCPUexample

about/YourfirstGPU-basedprogramcode/Thecodeexplanation

calculateEdgesGPUexampleabout/YourfirstGPU-basedprogramcode/Thecodeexplanation

checkOpenCLexampleabout/CheckingyourOpenCLcode/Thecodeexplanation

CIEL*a*b*about/CIEL*a*b*examplecode/Theexamplecode

CIEL*u*v*about/CIEL*u*v*examplecode/Theexamplecode

CIEXYZabout/CIEXYZexamplecode/Theexamplecode

CMakesetting/GettingacompilerandsettingCMakeabout/GettingacompilerandsettingCMakeURL/GettingacompilerandsettingCMakeOpenCV,configuringwith/ConfiguringOpenCVwithCMakelibrary,installing/Compilingandinstallingthelibrarylibrary,compiling/Compilingandinstallingthelibrary

color-space-basedsegmentationabout/Color-space-basedsegmentationHSVsegmentation/HSVsegmentationYCrCbsegmentation/YCrCbsegmentation

colorspacesabout/Colorspacesconversion,withcvtColormethod/Conversionbetweencolorspaces(cvtColor)RGB/RGBgrayscale/GrayscaleCIEXYZ/CIEXYZYCrCb/YCrCbHSV/HSVHLS/HLSCIEL*a*b*/CIEL*a*b*CIEL*u*v*/CIEL*u*v*

Bayer/Bayercolortransfer

about/Colortransferexamplecode/Theexamplecode

ColourImageComparisonexample,histogramsabout/Theexamplecode

ColourImageEqualizeHistexample,histogramsabout/Theexamplecodesourceimagewindow/Theexamplecodeequalizedcolorimagewindow/Theexamplecodehistogramofthreechannelswindow/TheexamplecodehistogramofRGBchannelfortheequalizedimagewindow/Theexamplecode

CommissionInternationaledeL’Éclairage(CIE)about/CIEXYZ

compilerkits,OpenCVC++applicationsMicrosoftVisualC(MSVC)/ToolstodevelopnewprojectsGNUGCC(GNUCompilerCollection)/Toolstodevelopnewprojects

ComputeUnifiedDeviceArchitecture(CUDA)about/AnintroductiontoOpenCV

cvtColormethodused,forcolorspacesconversion/Conversionbetweencolorspaces(cvtColor)srcargument/Conversionbetweencolorspaces(cvtColor)dstargument/Conversionbetweencolorspaces(cvtColor)codeargument/Conversionbetweencolorspaces(cvtColor)dstCnargument/Conversionbetweencolorspaces(cvtColor)

cylindricalprojection/Stitching

Ddatapersistence

about/Datapersistencedecolorization

about/Decolorizationexample/Decolorization

denoisingabout/Denoisingreferencelink/Denoisingfunctions/Denoisingexamplecode/Theexamplecode

detectFacesexampleabout/Goingrealtimecode/Goingrealtime,Thecodeexplanation

digitalstabilizationsystemsabout/Videostabilization

EestimatePiexample

about/Arithmeticoperationsexposurefusion,HDRimages/Exposurefusionextrapolationmethods

about/Geometricaltransformations

FFarnebackopticalflowmethod

URL/SuperresolutionFFTAMD

URL/AquickrecipetoinstallOpenCVwithOpenCLfilestructure,OpenCV

headerfiles/ThestructureofOpenCVlibrarybinaries/ThestructureofOpenCVsamplebinaries/ThestructureofOpenCV

fourcccodeURL/Theexamplecode

functions,acceleratingabout/AcceleratingyourownfunctionscheckOpenCLexample/CheckingyourOpenCLcalculateEdgesCPUexample/YourfirstGPU-basedprogramcalculateEdgesGPUexample/YourfirstGPU-basedprogramdetectFacesexample/GoingrealtimeGPUprogramming,performance/Theperformance

functions,seamlessCloningexamplecolorChange/SeamlesscloningilluminationChange/SeamlesscloningtextureFlattening/Seamlesscloning

GGaussianpyramids

about/Gaussianpyramidsfunctions/Gaussianpyramids

GDAL(GeographicDataAbstractionLibrary)about/Imagefile-supportedformats

geometricaltransformationsabout/Geometricaltransformationsextrapolationmethods/Geometricaltransformationsinterpolationmethods/Geometricaltransformationsaffinetransformation/Affinetransformationperspectivetransformation/Perspectivetransformation

GNUGCConlinedocumentation,URL/Generalusageofthelibrary

GNUtoolkit/GettingacompilerandsettingCMakeGPUusage

checking/ChecktheGPUusageGraphicProcessingUnit(GPU)

about/AnintroductiontoOpenCVgrayscale

about/Grayscaleexamplecode/Examplecode

GUI(GraphicalUserInterface)/ConfiguringOpenCVwithCMake

HHaarfeature-basedclassifier/ThecodeexplanationHDRimages

about/High-dynamic-rangeimagescreating/CreatingHDRimagescreateHDRexample/Exampletonemapping/Tonemappingalignment/Alignmentexposurefusion/Exposurefusion

hdr_imagingtutorialURL,forfileprerequisites/Example

HighDynamicRange(HDR)/ThestructureofOpenCVhighdynamicrange(HDR)

about/AnintroductiontoOpenCVhistogramequalization

about/Histogramshistograms

about/HistogramsColourImageEqualizeHistexample/TheexamplecodeColourImageComparisonexample/Theexamplecode

HLSabout/HLSexamplecode/Theexamplecode

HSVabout/HSVhue/HSVsaturation/HSVvalue/HSVexamplecode/Theexamplecode

HSVsegmentationabout/HSVsegmentation

IIDE(IntegratedDevelopmentEnvironment)/GettingacompilerandsettingCMakeimagecapturingprocess

about/Superresolutionsampling/Superresolutiongeometrictransformation/Superresolutionblur/Superresolutionsubsampling/Superresolution

imagefile-supportedformatsabout/Imagefile-supportedformats

imagefilesreading/Readingandwritingimagefiles,Readingimagefileswriting/Readingandwritingimagefiles,Writingimagefilesexamplecode/Theexamplecodeeventhandling,intointrinsicloop/Eventhandlingintotheintrinsicloop

imagefilteringabout/Imagefilteringsmoothing/Smoothingsharpening/Sharpeningimagepyramids/Workingwithimagepyramids

imageprocessingtimemeasuring/Measuringthetime

imagepyramidsabout/WorkingwithimagepyramidsLaplacianpyramids/Workingwithimagepyramids,LaplacianpyramidsGaussianpyramids/Gaussianpyramidsexamplecode/Theexamplecode

imagerotationabout/Imagerotationexamplecode/Theexamplecode

imagestitchingabout/StitchingURL/Stitchingregistration/Stitchingcalibration/Stitchingcompositing/StitchingstitchingAdvancedexample/Stitching

imshowfunctionabout/Theexamplecode

inpaintingabout/Inpaintingfunctions/Inpaintingreferencelink/Inpainting

examplecode/TheexamplecodeIntegratedDevelopmentEnvironment(IDE/ToolstodevelopnewprojectsIntegratedPerformancePrimitives(IPP)

about/AnintroductiontoOpenCVIntelIntegratedPerformancePrimitives(IPP)/ConfiguringOpenCVwithCMakeinterpolationmethods

about/Geometricaltransformations

Kkernel

about/Smoothing

LLaplacianpyramids

about/Workingwithimagepyramids,LaplacianpyramidsLinux/CompilingandinstallingthelibraryLuminanceHDR/TonemappingLUTs

about/LUTsexamplecode/Theexamplecode

MMake

onlinedocumentation,URL/Generalusageofthelibrarymapprojections

about/Stitchingrectilinear/Stitchingcylindrical/Stitchingspherical/Stitchingstereographic/Stitchingpanini/Stitching

mean-siftsegmentationreferencelink/Gaussianpyramids

mechanicalstabilizationsystemsabout/Videostabilization

MinGW(MinimalGNUGCC)/Toolstodevelopnewprojectsmipmap

about/Workingwithimagepyramidsmodules,OpenCV

core/ThestructureofOpenCVhighgui/ThestructureofOpenCVimgproc/ThestructureofOpenCVimgcodecs/ThestructureofOpenCVphoto/ThestructureofOpenCVstitching/ThestructureofOpenCVvideoio/ThestructureofOpenCVvideo/ThestructureofOpenCVfeatures2d/ThestructureofOpenCVobjdetect/ThestructureofOpenCV

morphologicaloperationsabout/Morphologicaloperationsfunctions/Morphologicaloperationsexamplecode/Theexamplecode

mouseinteractionabout/Mouseinteraction

Nnon-photorealisticrendering

about/Non-photorealisticrenderingedgePreservingFiltereffect/Non-photorealisticrenderingdetailEnhanceeffect/Non-photorealisticrenderingstylizationeffect/Non-photorealisticrendering

OOpenComputingLanguage(OpenCL)

about/AnintroductiontoOpenCVOpenCV

about/AnintroductiontoOpenCVURL,fordownloading/DownloadingandinstallingOpenCVdownloading/DownloadingandinstallingOpenCVinstalling/DownloadingandinstallingOpenCVURL,formainrepository/DownloadingandinstallingOpenCVURL,fortestdatarepository/DownloadingandinstallingOpenCVURL,forcontributionsrepository/DownloadingandinstallingOpenCVURL,fordocumentationsite/DownloadingandinstallingOpenCVURL,fordevelopmentrepository/DownloadingandinstallingOpenCVURL,fortutorials/DownloadingandinstallingOpenCVcompiler,obtaining/GettingacompilerandsettingCMakeconfiguring,withCMake/ConfiguringOpenCVwithCMakestructure/ThestructureofOpenCVreferencelink,formodules/ThestructureofOpenCVuserprojects,creatingwith/CreatinguserprojectswithOpenCV

OpenCV,withOpenCLinstallationprocess/AquickrecipetoinstallOpenCVwithOpenCL

OpenCVAPIURL/Writingimagefiles

OpenCVapplicationdeveloping,withC++/Generalusageofthelibrary

OpenCVC++applicationsprerequisites/Toolstodevelopnewprojects

OpenCVC++programcreating,withQtCreator/CreatinganOpenCVC++programwithQtCreator

operations,withimagesabout/Commonoperationswithimages

opticalflowmethodsbrox/Superresolutionpyrlk/Superresolutionfarneback/Superresolutiontvl1/Superresolutioncomparing/Superresolution

Ppaniniprojection/Stitchingparameters,BilateralTV-L1algorithm

scale/Superresolutioniterations/Superresolutiontau/Superresolutionlamba/Superresolutionalpha/SuperresolutionbtvKernelSize/SuperresolutionblurKernelSize/SuperresolutionblurSigma/SuperresolutiontemporalAreaRadius/SuperresolutionopticalFlow/Superresolution

perspectivetransformationabout/Perspectivetransformationfunctions/Perspectivetransformationexamplecode/Theexamplecode

pfstools/Tonemappingpixel-levelaccess

about/Pixel-levelaccessprerequisites,OpenCVC++applications

OpenCVheaderfilesandlibrarybinaries/ToolstodevelopnewprojectsC++compiler/Toolstodevelopnewprojectsauxiliarylibraries/Toolstodevelopnewprojects

pyramidabout/Workingwithimagepyramids

QQtbundle

URL,fordownloading/ToolstodevelopnewprojectsQtCreator

about/ToolstodevelopnewprojectsOpenCVC++program,creatingwith/CreatinganOpenCVC++programwithQtCreator

Qtframeworkabout/GettingacompilerandsettingCMakeURL/GettingacompilerandsettingCMake

QtprojectURL/CreatinganOpenCVC++programwithQtCreator

RRANSACmethod

URL/VideostabilizationRapidEnvironmentEditortool

URL/GettingacompilerandsettingCMakerectilinearprojection/Stitchingreflection

about/Reflectionexamplecode/Theexamplecode

requisites,forinstallingOpenCVwithOpenCLabout/OpenCVwiththeOpenCLinstallationOpenCL-capableGPUorCPU/OpenCVwiththeOpenCLinstallationcompilers/OpenCVwiththeOpenCLinstallationAMDAPPSDK/OpenCVwiththeOpenCLinstallationOpenCLBLAS(BasicLinearAlgebraSubroutines)/OpenCVwiththeOpenCLinstallationOpenCLFFT(FastFourierTransform)/OpenCVwiththeOpenCLinstallationQtlibraries,forC++compiler/OpenCVwiththeOpenCLinstallation

RGBabout/RGBexamplecode/Theexamplecode

Sscaling

about/Scalingexamplecode/Theexamplecode

seamlesscloningabout/Seamlesscloning

seamlessCloningexampleabout/Seamlesscloningfunctions/Seamlesscloning

sharpeningabout/Sharpeningfunctions/Sharpeningexamplecode/Theexamplecode

skewingabout/Skewingexamplecode/Theexamplecode

smoothingabout/Smoothingfunctions/Smoothingexamplecode/Theexamplecode

sphericalprojection/Stitchingstandardtemplatelibrary(STL)

about/ThebasicAPIconceptsstereographicprojection/Stitchingsticher/Stitchingsuperresmodule/Superresolutionsuperresolution

about/SuperresolutionURL,forexample/Superresolutionexample/Superresolution

TThreadingBuildingBlocks(TBB)

about/AnintroductiontoOpenCVtonemapping,HDRimages/Tonemappingtrackbars

about/Trackbarstranslation

about/Translationexamplecode/Theexamplecode

TVL1(TotalVariationL1)about/Denoising

UUnifiedMats(UMat)/Thecodeexplanationuser-interactionstools

about/User-interactionstoolstrackbars/Trackbarsmouseinteraction/Mouseinteractionbuttons/Buttonstext,drawing/Drawinganddisplayingtexttext,displaying/Drawinganddisplayingtext

userinterface(UI)/ThestructureofOpenCVuserprojects

creating,withOpenCV/CreatinguserprojectswithOpenCV

Vvideofiles

writing/Readingandwritingvideofilesreading/ReadingandwritingvideofilesrecVideoexample/Theexamplecode

videostabilizationabout/Videostabilizationmechanicalstabilizationsystems/Videostabilizationdigitalstabilizationsystems/VideostabilizationvideoStabilizerexample/Videostabilization

videostabilizationalgorithmssteps/Videostabilization

WWindows/Compilingandinstallingthelibrary

YYCrCb

about/YCrCbexamplecode/Theexamplecode

YCrCbsegmentationabout/YCrCbsegmentation