Masthead LogoFordham University

DigitalResearch@Fordham

Faculty Publications Robotics and Computer Vision Laboratory

2017

Establishing A-Priori Performance Guarantees forRobot Missions that include Localization SoftwareDamian LyonsFordham University, dlyons@fordham.edu

Ron ArkinGeorgia Institute of Technology

Shu Jiang

Matt O'Brien

Feng Tang

See next page for additional authors

Follow this and additional works at: https://fordham.bepress.com/frcv_facultypubs

Part of the Artificial Intelligence and Robotics Commons, and the Robotics Commons

This Article is brought to you for free and open access by the Robotics and Computer Vision Laboratory at DigitalResearch@Fordham. It has beenaccepted for inclusion in Faculty Publications by an authorized administrator of DigitalResearch@Fordham. For more information, please contactconsidine@fordham.edu.

Recommended CitationDamian Lyons, Ron Arkin, Shu Jiang, Matthew O'Brien, Feng Tang and Peng Tang, "Establishing A-Priori Performance Guarantees forRobot Missions that include Localization Software" International Journal of Monitoring and Surveillance Technologies Research(IJMSTR) Volume 5, Issue 1 2017.

AuthorsDamian Lyons, Ron Arkin, Shu Jiang, Matt O'Brien, Feng Tang, and Peng Tang

This article is available at DigitalResearch@Fordham: https://fordham.bepress.com/frcv_facultypubs/58

DOI: 10.4018/IJMSTR.2017010103

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

Copyright©2017,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.

Establishing A-Priori Performance Guarantees for Robot Missions that Include Localization SoftwareDamian Lyons, Department of Computer and Information Science, Fordham University, New York City, NY, USA

Ronald C Arkin, Georgia Institute of Technology, Atlanta, GA, USA

Shu Jiang, Georgia Institute of Technology, Atlanta, GA, USA

Matthew J O’Brien, Georgia Institute of Technology, Atlanta, GA, USA

Feng Tang, Fordham University, New York City, NY, USA

Peng Tang, Fordham University, New York City, NY, USA

ABSTRACT

Oneapproachtodeterminingwhetheranautomatedsystemisperformingcorrectlyistomonitoritsperformance,signalingwhentheperformanceisnotacceptable;anotherapproachistoautomaticallyanalyzethepossiblebehaviorsofthesystema-priorianddetermineperformanceguarantees.Theaauthorshaveappliedthissecondapproachtoautomaticallyderiveperformanceguaranteesforbehavior-based, multi-robot critical mission software using an innovative approach to formal verificationforroboticsoftware.Localizationandmappingalgorithmscanallowarobottonavigatewellinanunknownenvironment.However,whethersuchalgorithmsenhanceanyspecificrobotmissioniscurrentlyamatterforempiricalvalidation.Severalapproachestoincorporatingpre-existingsoftwareintotheauthors’probabilisticverificationframeworkarepresented,andoneusedtoincludeMonte-Carlobasedlocalizationsoftware.Verificationandexperimentalvalidationresultsarediscussedforreallocalizationmissionswiththissoftware,showingthattheproposedapproachaccuratelypredictsperformance.

KEywoRdSBehavior-Based, Formal Verification, Localization, Robot Software, Uncertainty

1. INTRodUCTIoN

Forsystemsthatneedtofunctionincriticalsituations,suchasinhealthcareapplications,searchandrescuerobotics,andautomatedcounterweaponsofmassdestructions(CWMD)missions,itiscruciallyimportantthatthesystemfunctionasspecifiedortheresultmightbelossoflife,orpropertydamageorboth.Oneapproachtothisproblemistomonitorthesysteminoperation(Leucker&Schallhart,2009)andtosignalanalerttoasupervisorofthesystemwhentheperformanceisgoing,orpredictedtogo,outsidethenecessaryperformanceenvelope.Thismonitoringapproachcanbevery

49

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

50

effectiveincaseswhereasupervisorcanstepin,orthesystemcanbedisabledwithoutconsequence,whenaperformanceproblemisobserved.However,inthecasethatthesystemisautonomous,orthesupervisorcannotinteractsufficientlyquicklyandthesystemcannotsimplybedisabled,thenanalternateapproachisnecessary.InpreviousworkfortheDefenseThreatReductionAgency(Lyonsetal.,2015)wehavedevelopedanefficientapproachtotheautomatic,a-priorideterminationofperformanceguaranteesforbehavior-basedrobotmissionsoperatinginuncertainenvironments.

Thisworkisrelatedtoformalsoftwareverification(Jhala&Majumdar,2009)(DeMoura&Bjorner,2012):adesigntooltodeterminewhetherapieceofsoftwarewillfunctionproperlywithouthavingtoexecutethesoftware.Thefieldhasprogressedstronglyinrecentyearswithdevelopmentsin model-checking (Jhala & Majumdar, 2009) and satisfiability (SMT) engines (DeMoura &Bjorner,2012).However,allsuchmethodscanatbestapproximaterealrobotperformancebecauseoftheundecidabilityoftheunderlyingverificationproblem.Designingaverificationapproachformissioncriticalrobotsoftwarerequiresunderstandingwhataspectsoftherobotsoftwareareofmostimportancetotheproblem.Behavior-basedrobotprogramming(Arkin,1998)isanimportanttoolinautonomousroboticsthatyieldsrobotprogramsthatarerobusttouncertaintyaboutexactlywhatenvironmenttherobotswillfaceduringexecution.Forthisreason,inrecentwork(Lyonsetal.,2015),weaddressedtheproblemofautomaticallyverifyingbehavior-basedrobotprogramsbyleveragingthe structure of such programs. The approach employs a unique combination of static analysistechniquesandprobabilisticreasoningtoprovideperformanceguaranteesforbehavior-basedrobotprogramsoperatinginphysicalenvironmentswithuncertainknowledgeaboutobstaclestomotion.Ratherthanaddressingcomputationalverificationproblemssuchasabsenceofdeadlockorabsenceofrun-timeerrors(Trojanek&Eder,2014)(Walter,Taubig,&Luth,2010),orverifyingsoftwaregeneratedcontrolsignalswithoutconsiderationofthephysicalplatform(Kim,Kang,&Lee,2005),ourworkfocusesonestablishingperformanceguaranteesforthemissionsoftwarewithamodelofanuncertainly-knownphysicalenvironment.

A key robotics development in recent years has been the use of probabilistic mapping andlocalization algorithms that allow a robot to operate robustly in previously unseen areas byautomaticallybuildingamapandcontinuallylocalizingtherobotwithrespecttothemap(Thrun,Burgard,&Fox,2005).Manysuchalgorithmshavebeenprogrammedandtestedandmadeavailablein software libraries (Dellaert, Fox, Burgard, & Thrun, 1999). Our prior work in verifying theperformance of behavior-based robot missions involved manual and autotranslation of missions(O’BrienM.,Arkin,Harrington,Lyons,&Jiang,2014)specifiedusingGeorgiaTech’sMissionLab(MacKenzie,Arkin,&Cameron, 1997) robotmissiondevelopment toolkit.Other approaches totheverificationofbehavior-basedsystemshavealsobeenbasedonspecificprogrammingtoolkits(Kiekbusch,Armbrust,&Berns,2015).However,since therealreadyexistsoftware librariesforprobabilisticmappingandlocalization,itwouldbeadvantageoustosimplyincludeoneoftheseinarobotmissionifitprovidedthenecessaryperformance.Inthispaper,weaddresstheproblemofincorporatingexistingsoftwareintoournovelapproachtoperformanceverificationofbehavior-basedsystems.Buildingonourworkin(LyonsD.,etal.,2016),wepresentsomegeneraltheoreticalresultsrelatingtothischallenge,andthenweaddressthespecificproblemofincorporatingprobabilisticlocalization(ROSAMCL)(Dellaert,Fox,Burgard,&Thrun,1999)(Jiang&Arkin,2015)intoaC-WMDrobotmission.

Theremainderofthepaperislaidoutasfollows.InSection2,wereviewtherelevantliterature.InSection3,wepresentasbackgroundanoverviewofourapproachtoverification.InSection4,wepresentthemainchallenge:incorporationofexistingsoftwareintotheverificationprocess,andwedevelopatheoreticalbasisforaddressingthis.Section5presentsanapplicationofthetheorytothespecificexampleofaC-WMDmissionusingtheROSAMCLlocalizationmodule.Section6presentsourconclusions.

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

51

2. LITERATURE REVIEw

Approachestotheproblemofensuringthatacriticalsystemwillperformaccordingtoana-prioriformalperformanceguaranteeincludebothrun-timemonitoring(Leucker&Schallhart,2009)andformalverification(Jhala&Majumdar,2009).Run-timemonitoringhastheadvantageoftypicallybeinglightweight,however,ithasthedisadvantagethatitcanonlybeappliedifasupervisorcanbealerted,orthesystemdisabled,intheeventofarun-timemonitorperformancealert.Certaincriticalsystems, suchashealthmonitoringsystemswith life-critical interventionevents,orautonomoussystemshandlingexplosivesdisposalwithsignificantdangeroflifeandpropertydamage,donothavethisproperty.Insuchcases,itisnecessarytoestablishthatthesystemwillalwaysabidebyspecified performance guarantees in advance. Our work addresses Counter-Weapons for MassDestruction(C-WMD)robotmissionsoftware:softwareformultiple-robotteamssearchingforandlocatingWMDs.Thisfallsintothecategoryofsoftwarerequiringa-prioriperformanceguarantees.

Approachestosoftwareverificationincludemodelchecking(Jhala&Majumdar,2009),staticanalysis(Venet,2008)andtheoremproving(Shankar,2009).Themodelcheckingapproach,whichhasprovedwidelysuccessful,involvesacomprehensiveexplorationofreachablestatesoftheprogramtobeverified.Inthecaseofrobotmissions,itisveryimportanttomodeltherobotsoftware,thepropertiesof the robot actuators and sensors, and thebehaviorof the environment inwhich therobotistooperate.Theadditionofanenvironmentmodelisrarelyseeningeneralpurposesoftwareverification,butiscrucialforanysoftwarethatinteractswithacomplexphysicalsystem.Modelcheckingapproachesaresensitivetothesizeoftheproblemstatespace.Thishasbeenrecognizedasaprobleminapplyingthesetechniquestorobotics(Wongpiromsarn&Murray,2008)becauseoftherequirementtomodelsensors,actuatorsandenvironmentbehaviorinadditiontothesoftware.

Onapproachtosidesteptheadditionalstatecomplexity(Fisher,Dennis,&Webster,2013)isbyverifyingtherobot’sbeliefratherthantheactualphysicalbehavior.However,thisbegsthequestionofhowbeliefwillcorrespondtorealityonanyspecificmission.Wehavefollowedanalternateapproach(LyonsD.,etal.,2013)(LyonsD.,Arkin,Jiang,Harrington,&Liu,2014)–avoidingmakingthestate-spaceoftheproblemexplicit.Wedevelopedastaticanalysismethodtoextractasetofflow-functionsfrombehavior-basedrobotsoftware.Thefunctionsdescribehowthevariablesoftheprogram,robot,sensorandenvironmentmodelsrecurrentlyinteractduringexecution.Theperformanceguaranteeismappedtoboundaryconstraintsonthesefunctions.TheflowfunctionswhichcharacterizetherobotmissionsoftwareareusedtoconstructaDynamicBayesianNetwork(DBN),andverificationofthesoftwareconsistsoffilteringtheDBN(asopposedtoinspectingsampleexecutionpathse.g.,(Younes&Simmons,2002)).RandomvariablesarerepresentedasMixturesofGaussians(MoG)(LyonsD.,Arkin,Liu,Jiang,&Nirmal,2013),essentiallyamultiplehypothesisrepresentationforthevariablevalue.PropagationofparametricdistributionsthroughaBayesiannetworkisveryfast;manyoftheverificationstakelessthanafewminutesdespiteextensivelogginganddebuggingcommands.

There are two challenges in automatically verifying existing probabilistic robot localizationsoftware.Thefirstisthatitrequiresanenvironmentmodel,separatefrom,andinteractingwith,therobotsoftware.Themodelhastoincludethephysicallocationoftherobot,thegeometryofthemap,andtherelationshipbetweentheseandthesensormeasurements.Uncertaintyinphysicallocation(attheleast)needstobemodeled.Thesecondisthatexistingsoftwaremustbedirectlyincludedintotheverification,ratherthanrecodedandthenincluded.

Thereislittledirectlyrelatedworkinapplyingautomatedverificationtocriticalmissionsinreal-worldenvironments.Proetzschetal.(Proetzsch,Berns,Schuele,&Schneider,2007)documentverificationofabehavior-basedarchitectureofanoutdoormobilerobotRAVON.Theyrewritethesoftwareintotheformal,synchronouslanguageQuartzandcarryoutmodel-checkingusingtheAverestmodelcheckertoensurethattherobotslowsdownandstopsbasedonitsproximitysensors.Sincetheyhavenoenvironmentmodel,whattheyverifyisthatthesoftwareissuesappropriatevelocitycommands–inourworkweverifythestateoftherobotitself.Walteretal.(Walter,Taubig,&Luth,

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

52

2010)alsoverifycollisionavoidancesoftwareforamobilerobot.Theydoincludeanenvironmentmodel(theirDomainmodel)andusetheIsabelletheoremprovingsystem.TheymanuallyannotatethecodeintheirownformallanguageanduseIsabelleasanassistantforhuman-driven,ratherthanautomated,verificationoftheprogram.

Verificationofsoftwarethatincludesprobabilisticalgorithmshasbeenaddressedbytheformalverificationcommunity(Baeir&Katoen,2008)(Katoen,2010),andprobabilisticmodel-checkingtoolssuchasPRISM(Kwiatkowska,Norman,&Parker,2004)haveexistedforsometime.However,suchtoolsrequireausertocodetheiralgorithmspecificallyforverification,whereasourobjectiveis to have automatic verification, and to include existing probabilistic localization code into theverificationprocess.Therecodingapproachhasthegeneralweaknessthatare-implementationmaynotrepresenttheactualcode(itmightbeabetterorworseimplementation).Evenpublishedalgorithmdescriptionsforwidelyknownalgorithmshavebeenshowntocontainerrors(Zaks&Joshi,2008).Recodingalsoinherentlyrequiresalargeinvestmentofexpertiseandmanpower(Kim,Kang,&Lee,2005).However,themainchallengeinincludingpreexistingcodeisthatsuchcodeisdesignedtoexecuteasingleinstancewhereasverificationreasonsaboutallexecutionsthatarepossiblegiventhea-priorienvironmentmodelinformation.Includingexistingcodeispossibleinsomemodel-checkingtools–forexampleSPIN4.0andlaterallowsCcodetobeembeddedinaPromelamodelandthecodeisexecutedasanatomictransitionwithinSPIN.Ourapproachistoconsidertheembeddedcodecantransformasamplefromarandomvariable,andwedefineaframeworkforsamplingandreconstructingvariabledistributions.

3. VERIFICATIoN oF BEHAVIoR-BASEd RoBoT MISSIoNS

Asbackgroundforourapproachtoaddressingverificationofexistinglocalizationcode,wepresentanoverviewofourapproach.ThefirstsubsectionbrieflyintroducesandmotivatesourtheoreticalapproachusingPARS(ProcessAlgebraforRobotSchemas).ThesecondsubsectionintroducestheimplementationinMissionLabandVIPARS(VerificationinPARS).

3.1. Theoretical Framework for Automatic Verification of Behavior-Based SystemsAswehavediscussed,state-spaceexplosionisakeychallengeforautomaticsoftwareverification.Thisproblemisexacerbatedinourcasebytheneedtoincludenotjusttherobotsoftware,butalsoamodeloftheenvironmentinwhichtherobotprogramwilloperate.Ontopofthis,boththerobotsoftwareandtheenvironmentmodelwillneedtosupporttherepresentationofuncertainty,asine qua noninreal-worldrobotics.Toaddresstheseseriouscomputationalcomplexityproblems,weelectedto1)notfocusonstate-basedrepresentationsbutratherprocess-basedrepresentations,and2)toleveragethestructureofbehavior-basedprogramming(Arkin,1998)–asuccessful,robustrobotprogrammingparadigm.

Abehavior-basedprogramanditsenvironmentaremodeledinPARSasasetofinterconnected,recurrentprocesses,whereaprocessPiswrittenas:

Pu u i i o o v vn j k m1 1 1 1

, , , , , , , ,… …( ) …( ) … (3.1)

whereu1,…,unaretheinitialvaluesfortheprocess variables,i1,…,ijando1,…,okareinput and output port connections,andv1,…,vmare final result valuesoftheprocessvariables.Processescomputeresultvaluesfrominitialvalues,andthiscomputationmaybeinfluencedbyanycommunicationsthatoccuroverportconnections.Byfocusingonprocessesratherthanstates,weshiftthecomputationalcomplexityfromstatecombinatoricstothegeneratorofthosestates–theprocessdescription.So,ofcourse,weneedagoodlanguageforthatdescription.

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

53

Processescanbedefinedascombinationsofotherprocessesusingthefollowingcompositionoperators:parallel(‘|’),disabling(‘#’)andsequential(‘;’).Boundedrecursioniscapturedusingtail-recursiveprocessdefinitions,e.g.:

Px Qxy Py� �;= (3.2)

HereprocessPfirstactivatesprocessQwithinputvaluex.Qdeliversoutputvaluey,whichisthenusedtorecurP.AvariableflowfunctionfPrelatesthevaluesofvariablesatthestartofeachrecursivestepofPtothoseattheend.Theflow-functionforatomicprocessesarespecifieda-priori;thoseforcompositeprocess,thosedefinedascompositionsofotherprocesses,e.g.,(3.2),arecomposedfromtheflowfunctionsofthecomponentprocesses.Thiscanbeautomatedinastandardstaticanalysis(theapproachusedbycompilerstocheckoroptimizesoftware)approachtogenerateflowfunctionsgivenasetofprocesses(Lyons,Arkin,Jiang,Liu,&Nirmal,2015)withcomplexitylinearinthenumberofprocesses.SinceanyexecutionofEquation(3.2)ismodeledbyfP

n(x0)forn≥1andinitialparametervaluex0,wehaveastraight-forwardverificationmethod.However,notallprocessesaredefinedinthisform.

Thesystemtobeverifiedisexpressedastheparallel,communicatingcompositionSysofarobotcontroller,e.g.,Ctrwithvariabler1,andanenvironmentmodel,e.g.,Envwithvariabler2,written:

Sys r r Ctr r a b Env r b a1 2 1 2, ( )( ) ( )( )= (3.3)

= ( )Sys r r Sys f r rSys

' , ; ,1 2 1 2

(3.4)

f r r f r r f r rSys Sys r Sys r1 2 1 1 2 2 1 2

, , , ,, ,( ) = ( ) ( )( ) (3.5)

InEquation(3.3),theinputofCtrisconnectedtotheoutputofEnv,(a),andtheoutputofEnvisconnectedtotheinputofCtr,(b).If(3.3)wereasequentialcompositionlike(3.2)thenwecouldextractflowfunctionsforthecombinedinteractionofcontrollerandenvironmentandconductanefficientverification.Therefore, in(Lyons,Arkin,Jiang,Liu,&Nirmal,2015)wedevelopedaninterleaving theorem1 for behavior-based systems to convert Equation (3.3) to a sequential formEquation(3.4).Theintuitionisthatabehavior-basedsystemhasbehavioral‘states’eachwithanassociatedsetofsensory-triggeredresponses.AstaticanalysisalgorithmSysgenwasdevelopedtoidentifythesetofprocessesforthesestatesandrewriteparallelcompositionsoftheform(3.3)intoasequentialcomposition(3.4)whereSys’istheautomaticallyidentifiedbehavioralstateprocess,referred to as the systemperiod.OnceSysgenanalysis is complete, a system flow functioncanbeextractedfromSys’intheusualway.InthesmallexampleofEquations(3.3),(3.4)above,thefunctionextractedisshowninEquation(3.5).Thisisarecurrentfunctionthatevaluatesthevaluesforr1andr2ascomputedbytheinteractionsbetweenCtrandEnvineachexecutionofthesystemperiodSys’.Theresultofthisisthataparallelcompositionofcontrollersoftwareandenvironmentmodel,suchasinEquation(3.3),canbeautomaticallyverifiedbycheckingfSys

n(x0)forn≥1andinitialparametervalue(s)x0.

Whilethisaddressesthestatecombinatoricsproblemmentionedinthefirstparagraphof3.1,it doesnot address the challengeof representinguncertaintymentioned in that sameparagraph.Processvariables,e.g.,r1, r2inEquation(3.3),canberandomordeterministic.Randomvariablesare represented as multivariate mixtures of Gaussians, and operations on random variables areautomatically translatedintooperationsondistributions(LyonsD.,Arkin,Liu,Jiang,&Nirmal,2013);forexample,asumofrandomvariablesistranslatedtoaconvolutionoftheirdistributions.Alltheseoperationscanbewritten(asapproximationsinsomecases)asoperationsontheparametersoftheGaussianmixtures.FlowfunctionsrelatevariablevaluesatrecursionsteptofSys’tothoseatt+1,andcanbewrittenasconditionalprobabilities,e.g.:

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

54

fSys,r1(r1,t,r2,t)=P(r1,t+1|r1,t,r2,t) (3.6)

Inthefinalphaseofverification,extractedflowfunctionsareconvertedtoconditionalprobabilitiese.g.,(3.6).TheseareusedtobuildaDynamicBayesianNetwork(Russel&Norvig,2010)tocarryoutaforwardpropagationofprobabilitydistributions–aprobabilisticversionoffSys

n(x0)–todeterminewhether the combination of controller and environment will meet a performance specification.Although (Lyons, Arkin, Jiang, Liu, & Nirmal, 2015) discusses more complicated performanceguarantees,webasically restrictour attention to theguarantee that amissionwill achieve somecriteriononenvironmentvariables(usuallyaspatialaccuracyforawaypointgoaland/oratemporalrequirementforachievingthemission)withprobabilitygreaterthanathresholdbeforeatime-limithasexpired.Wehavedemonstratedthatthisapproachisfastandaccuratewhenvalidatedagainstphysicalexecutions(mostrecently(Lyonsetal.,2015)).

3.2. Implementation of Verification in MissionLab and VIPARSThe result of our research effort is a software verification tool VIPARS, which provides theperformanceguaranteeforarobotmissionbasedonhowwellthespecifiedperformancecriteriaaresatisfiedbythecontrolprogram,robot,andtheenvironmentmodelsforthemission.

VIPARS(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015)isbuiltuponMissionLab(MacKenzie,Arkin,&Cameron,1997),abehavior-based robotmissionspecificationenvironment (Figure1).MissionLabprovidesausability-testedgraphicalprogramminginterface,wheretherobot’sprogramis specified in the formofa finite stateautomaton (FSA)assembled froma libraryofprimitivebehaviors.MissionLabmissionsareautotranslated(O’Brien,Arkin,Harrington,Lyons,&Jiang,2014) to theprocess-algebranotationPARS foranalysis.Environmentmodelsarealsoprocessesinthisnotationandwehaveproposedthatastandardizedsetofenvironmentmodelscouldbeused

Figure 1. System architecture (reproduced from (Lyons et al., 2015))

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

55

tocapturedifferentclassesofenvironment(e.g.,motionuncertainty(Lyons,Arkin,Jiang,Liu,&Nirmal,2015);obstacleuncertainty(Lyonsetal.,2015)).

ThefirstphaseofVIPARSverificationisastaticanalysisoftheconcurrent,communicatingmissionsoftwareandenvironmentmodeltogenerateasetofrecurrentflow-functionscapturinghowthetwointeract.ThesecondphaseofVIPARSusesaBayesiannetworktopredicttheperformanceofthemissionfromthesefunctions.

TheoutputofVIPARSistheperformanceguarantee,currentlyquantifiedasasetofprobabilitydistributions,thatdescribesthelikelihoodofmissionsuccess.Ratherthanasingleyes/noanswerforwhethertherobotsoftwarebeinganalyzedmeetstheperformancerequirements,VIPARScanoutputrandomvariabledistributions(e.g.,thelocationoftherobotattheendofthemission)andasetofperformancegraphs(e.g.,theprobabilitythatamissionwillbesuccessfulversustime,orthespatialaccuracyofthefinalpositionasagraphofprobabilityversusspatialerror).Thisrichoutputfuelsafeedbackloopthatallowsthemissionoperatortoimprovetheperformanceofthemissionsoftware.

4. VERIFyING EXISTING LoCALIZATIoN SoFTwARE

Therearetwokeychallengestoextendingautomaticverificationtohandletheverificationofpreviouslyexistinglocalizationsoftwareincorporatedintoarobotmission.Thefirstchallengeistoprovideanenvironmentmodelforverificationthatincludesthegeometricandpositionuncertaintyneededtoverifyanyprobabilisticlocalizationapproach.Thatwillbeaddressedfirst,insubsection4.1.Thesecondchallengeistoprovideaframeworkforincludingtheautomaticanalysisofpreviouslyexistingsoftwareintoprobabilisticverificationwithoutrequiringthatthesoftwareberewritten,annotated,ormodifiedinanyway.Thatwillbeaddressedinsubsection4.2.

4.1. Environment Model for LocalizationSection3includedanintroductiontotheVIPARSverificationmoduleandtoPARS,theprocessalgebraframeworkunderlyingit.ThesoftwareforarobotmissionistranslatedfromtheMissionLabGUIEditortoasetofPARSprocessdefinitions.Werefertothetop-levelprocessasMissionandweverifythisprocessbyanalyzingitsinteractionsthroughthesensorandactuatorsignalsitgenerateswithamodeloftheenvironmentinwhichitistoexecute.ThisenvironmentmodelisalsodefinedasasetofPARSprocesses.Theconcurrent,communicatingcombinationoftherobotcontrolsoftwareMissionandtheenvironmentmodeliscalledthesystemprocess,Sys.

Inpriorwork,wehavedefinedenvironmentmodelstocapturethemotionandsensinguncertaintyoftherobot(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015),andenvironmentobstacleswithuncertainlocationandsize(LyonsD.,etal.,2015).Toverifylocalizationsoftware,theenvironmentmodelneedstocontaininformationabouttheuncertaingeometryoftheenvironment.ThesystemprocessSysforthelocalizationmissionisshowninEquation(4.1)andillustratedasaprocessnetworkinFigure2:

Sys Mission clp clh cl cv Localization D cp co ch cl cm= ( ( )( ) ( ), , , , , ,0 cclp clh

Map sysmap cm MB Laser ms mo lo cm cp ch

,

() _ , , , ,

( )( ) (( )( )( )( ))

cl

Robot P H cv cp ch co 0 0, , ,

(4.1)

RecallfromSection3,thatthenamesinparenthesisaftertheprocessnameareportconnectionnames.Theinitialvaluesfortheprocessvariablesareshowbetweenanglebrackets.TheMissionsoftware,aswe’llseeinalatersection,programsawaypointmissionandisfundamentallysimilartoallpriorwaypointmissionswehaveverifiedandvalidated.Ithasinputsclp(position),clh(heading)andcl(laserreadings);andoutputcv(velocity).Robotistheenvironmentmodel,capturingmotionandodometryerroraswellastherobotinteractionswithobstacles.Itisalsofundamentallysimilar

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

56

tothemodelsusedinourpriorwork.PO, HOaretheinitialrobotpositionandheading,andtheprocessportsareinputscv(velocity)andoutputscp, ch(odometrypositionandheading)andco(realpositiondistribution,i.e.,withoutsensingnoise–usedforperformanceguaranteeevaluationonly).

Inthebehavior-basedlocalizationapproach(Jiang&Arkin,2015),theobstacleavoidancesensor(MB_Laser)getsitsinformationfromthemap,ratherthandirectlyfrommeasuringsensoryinput.Mapmakesmapinformationavailableonitsoutputcm;MB_Laserusesthemaptogeneratemap-basedlaserdataonitsoutputcl.Localizationimplementsalocalizationmethodusingthemapcm,lasercl,androbotcp, co, chinputs,whereD0istheinitialpositionuncertainty.TheoutputofLocalization,clp,isthelocalizedposition(andheadingclh)usedbytheMissionprocess.HowtheLocalizationprocessencapsulatesanexistinglocalizationalgorithmwillbeaddressedinsubsection4.2.

Akeydifferencebetweenthislocalizationmissionandpriormissionstowhichwehaveappliedourverificationapproach(LyonsD.,Arkin,Jiang,Liu,&Nirmal,2015)(LyonsD.,etal.,2015)(O’BrienM.,Arkin,Harrington,Lyons,&Jiang,2014)isthemapandtheroleitplaysinboththeobstacle avoidance behavior and in localization. The Map process in Equation (4.1) contains amap(sysmap)whichrepresentsthegeometryofthephysicalenvironmentinwhichthemissionisexecuted.Ourassumptionisthatthismaphasbeengenerateda-prioriforanenvironmentofinterest.Ourconcernhereishowthismapcanberepresentedforthepurposesofprobabilisticverification.

RandomvariablesarerepresentedinVIPARSasMixturesofGaussiansdistributions(MG).Ifa ~ MG(CM),forCM={(μi, Σi, wi) | i ∈ 1…m}thesetofthemixtureparameters(means,variances,weights),thenaireferstomixturememberN(μi, Σi,),andw(ai)=wiarethemixtureweights,where

i

m

iw

=∑ =

1

1, � and MG(x; CM)=i

m

i i iw N x

=∑ ( )

1

; ,µ Σ . The mixture size is written | a | = m. Map

information–thelocationsandgeometryofobstacles,wallsandotherphysicalaspectsofthemissionenvironment–canbedirectlyrepresentedusingthismixturemodelasweexplainbelow.Theadvantageofthisapproachtorepresentingphysicalgeometryisthatthereisnorestrictiononthespatiallocationorextentofobstacles,andfinerprecisionofmodelingcanbeobtainedatthecostofaddingmoremixturemembers(Figure3).

Figure 2. The system process Sys for the localization mission

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

57

Definition:AnindexedmixtureofGaussiansisamixtureofGaussiansdistributiona ~ MG(CM)togetherwithanindexsetI.Themixtureisrestrictedasfollows:◦ a[x]≡aiwhereμ(ai)=x∈I,i∈1…m;◦ μ(ai)∈I,foralli∈1…m;aonly contains members indexed byI;◦ For anyx∈I,|{a[x]}|≤1;ahas at most one member for each index.

Wedefinew[x]andΣ[x]similarlytoa[x]tolabelmemberweightsandcovariances.Amapisdefinedasan indexedbivariatemixtureofGaussianswhere I=[0…X]×[0..Y]andwhereeachmemberisaGaussiankernelwithcovarianceΣ[x,y]=σm

2I,whereσmreflectsthemapresolution.Thiscorrespondssomewhatintuitivelywithanoccupancygridrepresentation,wherew[x,y]isrelatedtoprobabilityofoccupancyforthelocation(x,y).

Duringverification,thelocationrandomvariable(theconnectioncpinEquation(4.1))representsthelocationoftherobotforallpossibleexecutions.It’srelevanttocomparethiswiththerepresentationofrobotlocationinalocalizationalgorithm:therepresentationtheremaybealsobearandomvariable,buttheinterpretationisdifferent.Inanysingleexecution,therobotcanonlybeatasinglephysicallocation;thelocalizationdistributionisanestimateofthis.Inverification,theobjectiveisnottofindthesinglemostlikelylocation,buttopropagatetheeffectsofbeingatalllocations.Ratherthanusingaraytracealgorithmtodeterminehoweachlocationissupportedbysensorreadingsandrefiningthepositionestimatebasedonthat,theraytracealgorithmisusedbytheMB_Laserprocesstogatherallpossiblesensorreadingsthatcanariseduetotherobotlocationdistribution.

Webeginbydefiningarayscanonamaprepresentedasanindexedmixture:

Figure 3. Example of VIPARS Gaussian mixture model map representation

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

58

Definition:Scanline((xs,ys), θ) ⊆ R2 = { (x,y): x=xs + t cos θ, y = ys + t sin θ, t ∈ R+ }

ThesetofmapmixturemembersencounteredbyScanlineiseasilycalculatedasScanline ∩ IandforconveniencewealsodefinethemembersencounteredbyScanlineontheindexedmixtureMapas:

Definition: Scanmix((xs,ys), θ) ⊆ Map ={ (a[x,y], w[x,y]/Σw[x,y]): a[x,y]∈Map ∧ (x,y) ∈Scanline((xs,ys),θ)∩Y}

Thereturnfromasensorwillthenbeanindexedunivariatemixturecalculatedasfollows.Letbifori=1…mbethemembersofScanmix,thenwecalculateeachmembersiofthesensorreturnSensor((xs,ys), θ)fromthemembersofScanmix.EachmembermeanisthedistancefromthesensororigintothemapindexandeachmembervarianceisthemapaccuracyvarianceconvolvedwithasensorspecificnoisedistributionSN:

� , * ���� ��� , ,���s s N S where x y bi i i i N i s s i i= = ( ) = ( ) ( )µ µ σ µ µ σ == σ

m (4.2)

However,calculatingthemixtureweightsisalittlemoreinvolvedduetothepotentialocclusionofmixtures (representingobjects)byothermixtures (representingobjects) that arecloser to thesensor.TheweightsofthemembersofScanmixneedtobemodifiedforthecorrespondingmembersofSensorinorderofvisibility–thatis,inorderofμiasfollows,wherewareweightsinScanmixandw’thenew,correspondingweightsinSensor:

� [ ]�

′ =w w0 0

µ

′ = −

=∑w w w

i ij

i

jµ � ’1

0

(4.3)

Sensor((xs,ys), θ)definesthesensorreturntohaveanuncertaintyrelatedtotheaccuracyofthemap.However,itdoesnotincludeanyuncertaintyassociatedwiththerobotposition.Toincorporatetheuncertaintyinrobotpositionintothesensorreturn,weextendthedefinitionasfollows:

Definition:Sensor(N((xs,ys),Σ), θ) = {(si, w’i): i=1…m }asinEquation(4.2)butwhereΣisadiagonal

covarianceΣ =

σ

σx

y

2

2

0

0andσ σ σ θ σ θ

i m x y= + +( )� cos cos � .

However,thisdefinitionassumesadiagonalvarianceΣfortherobotmotionuncertainty.Thisisunrealisticandweneedtorelaxtheassumptionifwearetohandlerealisticrobotmotion.AnycovariancematrixcanbeinterpretedgeometricallyastheproductofrotationRθandscalingSmatrices:

Σ =

= ( )( ) =

−σ σ σ ρ

σ σ ρ σ

φφ φ θ

x x y

x y y

TR S R S where R

2

2 ��cos sinφφφ φsin cos

�

=

andSs

sx

y

0

0 (4.4)

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

59

Usingthisrelationship,theangle(ϕ)anddiagonalvariances(sx2,sy

2)ofanon-diagonalcovariancematrixcanbeidentifiedandthescananglerotatedaccordingly.

Definition: Sensor N xs ys x

y

, , ,( )

σ

σ

2

2

0

0

= ( )

� , , �Sensor N xs yss

sx

y

2

2

0

0 −

,�� φ

Havingensuredthatenvironments,mapsandsensorreadingscanallbereasonedaboutusingtherandomvariableframeworkinPARS–necessaryforbothobstacleavoidanceandlocalization–wenextturnourattentiontolocalizationitself.

4.2. Including Existing Software in Probabilistic VerificationThemaindifficultywiththeincorporationofexistinglocalizationcodedirectlyintotheVIPARSverificationalgorithmisthatlocalizationcodeisdesignedtoexecutejustasingleinstanceofarobotmission,whereasVIPARSisprobabilisticallyreasoningaboutallexecutionsthatarepossiblegiventhea-priorienvironmentmodelinformation.ConsiderthattheC++programwewanttoaddtoamissionisP.APARSprocesswrapperforPisbuilt,sothecodebehaveslikea‘blackbox’processP⟨x⟩⟨y⟩.However,whenPiscalled,itwillmaponeinputvaluextoanoutput,y;onlyonepossibleexecutionofP,whereasverificationhastocheckallpossibleexecutions.

Ourapproachtothischallengeconsidersbuildingaprocesswrapperfortheembeddedcodethatiscapableoftransformingasample(whichrepresentsasingleinstanceofamission)fromaPARSrandomvariable,andforthispurposewedefineaframeworkforsamplingandreconstructingvariabledistributions.Thisapproachhastheadvantageofusingtheactualcodethatwillgetexecutedbytherobotatrun-timeforthemission.Ithasthedisadvantageofpotentiallylengtheningverificationtimes,sincemultiplesamplesneedtobeevaluatedforarepresentativeresult.

Wedefine an extension to the flow function fP for theprocess/programP (representing theactualembeddedcode):themixtureextendedflowfunctionFPtakesarandomvariablexasinputandproducesarandomvariableyasoutput.ItsamplestheinputdistributionxandcallsfPonthesamples,andreconstructstheoutputdistributionmixturep(y | x)= FP(x)fromtheresult.Letx,y~MG(CM);ourobjectiveistoevaluatey=Π(x)(=fPabove)whereΠisthetransformationcarriedoutbytheembeddedcodeonthevariablex.Wewillpresentseveralwaysthiscanbedone.

Definition:TheMGmean-extendedrealfunction(MER)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ,wherey=Π(x),forx, y∈ℜ,then;◦ TheMGmeanextendedrealfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG(andMGisthesetofallMGs);and◦ Wherewedefiney=x;◦ Exceptμ(yi)=Π(μ(xi))forallxiinx.

So,theMERfunctionpreservesnumberofmembersandvariances.Onlythemeanistransformed,andanormaldistributionistransformedtoanothernormaldistributionwiththesamevariance.ThislimitsourmodellingofwhatthesoftwareinΠcando:

• Itcannotforexampleaddorreduceuncertaintysincevarianceisunchanged.Theentiredistributionismovedtocenteroveranewmean;

• Itcannotaddnewmemberstothemixturesincethenumberofmembersisunchanged.

Thisisclearlyverylimiting,thoughitmightbesufficientforafastestimateofperformance.ConsiderthatifwecallΠwithasinglenumberx,itreturnsanormaldistributionwiththeresultfor

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

60

xasthemeanandavariance:y=N(μ,σ)=Π(x).Now,iftheinputtocodeisaMGx,thenwecoulddefineaMGextensionofΠas:

Definition:TheMGmean extended normal function (MEN)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ×ℜ,where(μ,σ)=Π(x),forx,μ,σ∈ℜ,then;◦ TheMGnormalextendedfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG;and◦ Wherewedefiney=x;◦ Except(μ(yi),σ(yi))=Π(μ(xi))forallxiinx.

Notice however that the MEN function effectively ignores σ(xi); only the mean is used. Inprobabilisticterms,thisislikeonlyoperatingonthemostlikelyvalue.However,sinceavarianceisreturned,itmeansthecodedoesincludeanincreaseordecreaseintheuncertaintyassociatedwiththetransformedvariable.

Definition:ThemeanandvarianceofanMGisdefinedsimilarlytothemeanandvarianceofamixtureofGaussians.Forarandomvariablex~MG:

◦ μ(x)==∑i

m

i iw

1

�µ , thatis,theweightedsumofmeans;

◦ σ(x)= −( ) +

=∑i

m

i i iw

1

2 2� µ

Let’sconsidertheinputxitoactuallybeacollectionofsamples,wheretheweightofsamplesjisgivenbyN(sj;μI,σi)–thatis,memberievaluatedatsj.Thisdiscretizesandsamplesthea-prioridistribution.WecanevaluatetheMENfunctionforeachsample,andthenusetheresultstoreconstructthea-posteriori(posterior)distributionasfollow:

Definition:TheMGextended normal function(ENF)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ×ℜ,where(μ,σ)=Π(x),forx,μ,σ∈ℜ,then;◦ TheMGextendednormalfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG;and◦ Wherewedefiney=x;◦ Except(μ(yi),σ)=Π(μ(xi))forallxiinx;and◦ Whereσ(yi)is calculated as follows:

▪ (μ’j,σ’j)=Π(si)forsiasampleoftheinputxi;

▪ σ(yi)= ( ) ( )( ) −( )( ) +

=∑j

k

j i i j i jN s

1

22�; , ’x x � y ’µ

SoratherthanevaluatingtheprogramΠonasinglenumber,itisevaluatedonasetofnumbersofpredefinedsize(k)andtheresultscombinedtoformasinglenormaldistribution.

It’salsopossibletotakeasampleapproachtoarealfunctionΠ,andcalculateavarianceontheresultsample.

Definition:TheMG extended real function(ERF)Πisdefinedasfollows:◦ IfΠisdefinedasΠ:ℜ→ℜ,wherey=Π(x),forx,y∈ℜ,then;◦ TheMGmeanextendedrealfunctionΠisdefinedasΠ:MG→MG;◦ Wherey=Π(x),forx,y∈MG);and

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

61

◦ Wherewedefiney=x;◦ Exceptμ(yi)=Π(μ(xi))forallxiinx;and◦ Whereσ(yi)is calculated as follows:

▪ μ’j=Π(si)forsiasampleoftheinputxi;

▪ σ(yi)= ( ) ( )( ) −( )( )

=∑j

k

j i i j iN s

1

2; , 'x x yµ

ThissetofdefinitionsforMEN,MER,ENF,andERF,allowsforarangeofoptionsininterfacingembeddedcodewithverificationingeneral,andwithVIPARSinparticular.

5. VERIFICATIoN ANd VALIdATIoN oF A RoBoT MISSIoN wITH LoCALIZATIoN

To assess the effectiveness our approach in providing performance guarantees for probabilisticlocalizationmissions,wepresenttwowaypointmissionswheretherobotistaskedtonavigatethroughaseriesofwaypoints,avoidingobstaclesandtowardagoalwithbehaviorsthatarebasedonprobabilisticlocalization.Thegeneralassessmentprocessconsistsofthreesteps:1)verification–useVIPARStogenerateaperformanceguaranteeforthemissionwithrespecttosomespecifiedperformancecriteria,2)validation–conductexperimentaltrialsofthemissionwitharealrobot,3)evaluation–comparethepredictedperformancegeneratedbyVIPARSwiththeactualperformanceoftherobot.

ThewaypointmissionsareillustratedinFigure4.Themissionproceedswithrobotstartingat(2,2)andnavigatesbyfollowingaseriesofwaypointtothegoallocationsat(11.7,12.5)and(1.0,7.3)respectivelyforeachmission.ThebehavioroftherobotforMission-B(Figure4a)isshowninFigure5,whichwascreatedinMissionLabintheformofanFSA.TheFSAconsistsofaseriesofGoToGuardedandSpinbehaviors,whosetransitionsarepromptedbyAtGoalandHasTurnedtriggers.TheFSAforMission-AissimilartotheoneshowninFigure5butisomittedforbrevity.

Incontrasttothebehaviorswehadexaminedinourpriorwork,thebehaviorsherehaveleveragedaprobabilisticlocalizationalgorithmtoimprovemissionperformance.Specifically,theperceptualschemasofMoveToGuardedandAvoidObstacles,twooftheconstituentprimitivebehaviorsofthehigh-levelGoToGuardedbehavior,areaugmentedwithaSLAM-basedspatialmap(Jiang&Arkin,

Figure 4. Waypoint missions for verification and validation

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

62

2015).TheMoveToGuardedprimitivebehaviordrivestherobottoaspecifiedlocationwitharadiusofvelocitydropoffaroundthegoal.Insteadofusingodometryforlocalization,theperceptualschemaof MoveToGuarded is replaced with the adaptive Monte Carlo localization (AMCL) algorithm(Dellaert,Fox,Burgard,&Thrun,1999).Thisprobabilisticlocalizationalgorithmtakestherobotodometryandana-prioriacquiredmapasinputs,andoutputsanestimatedposeoftherobotalongwith a covariance matrix representing the uncertainty of the estimated pose. Furthermore, theAvoidObstacles behavior uses the spatial map, instead of using direct sensory reading from thelaserscanner,togeneraterepulsionvectors.TheperceptualschemaoftheAvoidObstaclesbehaviorismodifiedtoturnthespatialmapintopseudolaserscansoftheenvironmentthroughraytracingwithintheoccupancymap.Asaresult,theGoToGuardedbehaviorutilizesperceptualinformation(i.e.,robotposeandobstacles)generatedbyprobabilisticalgorithmstoproducemotioncommandswhilenavigatingthroughthewaypoints.

Performancecriteriaaremissionspecificationsthatmustbemet.FormissionsAandB,theseincludeconstraintson:

• Rmax:Maximumradiusofspatialdeviationallowedfromthegoal;• Tmax:Maximumallowablemissioncompletiontime.

Inthisexample,eachwaypointmissionisconsideredsuccessfulonlywhentwoconstraintsaremet:

Success=(r≤Rmax)and(t≤ Tmax) (5.1)

where r istherobot’srelativedistancetoitsgoallocationandtisthetimefortherobottofinishamission.TheobjectiveofVIPARSisthentoverifyhowwelltheseperformancecriteriaaresatisfiedbythecombinationoftherobot,itscontrolsoftware,andtheoperatingenvironment.

Figure 5. Behavioral FSA for Mission-B

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

63

5.1. VerificationThelocalizationalgorithmusedinthispaperwasAdaptiveMonteCarloSampling(AMCL)(Fox,2001)asimplementedinROS.Inthesamplingapproach,theDBNfilteringengineofVIPARSissuedrequeststoaROS-basedAMCLservertoevaluatetheERFfunctionfortheLocalizationprocess.TheinteractionisshowninFigure6:WhenevertheflowfunctionfortheLocalizationprocessneededtobeevaluatedonapositionrandomvariable,thepositionvariablewassentfromtheDBNfilteringengine(Top,Figure6)viaapipetoaconcurrentlyrunningROSIndigosystem(Bottom,Figure6).TheROSSTDRsimulatornodewasinstructedtomovetherobottotheappropriateposition,andlocalizationdatacollectedfromtheAMCLnode.Forsimplicity,theERFfunctionwasrestrictedtosinglemembermixtures,andratherthancalculatingthevariancebyevaluatingmultiplesamples,onlythemeanvaluewastransformedandthevariancecalculatedbyconvolvingthemeanwithazero-meandistributionN(0, σs).ThissimplifiedthehysteresisissuewithcallingAMCL.ThehysteresischallengeinfullyimplementingtheERFDefinitionforAMCLisdiscussedintheConclusion.

Theresultsofcarryingoutverificationonbothwaypointmissionswasasetofperformancegraphs(asdescribedSection3)showingthepredictedperformanceofthemissionswithrespecttotheperformancecriteria(5.1).

5.2. ValidationValidationexperimentsofthewaypointmissionswereconductedtoillustratethatVIPARS’predictedperformanceofthemissionisconsistentwiththerobot’sactualperformance.TherobotusedfortheexperimentaltrialsisthePioneer3-AT,afour-wheeledskid-steeredmobilerobot.Therobot

Figure 6. VIPARS-ROS architecture

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

64

isalsoequippedwithaforward-facingSICKlaserscanner.Thecompletevalidationexperimentconsistsof50trialrunsforeachwaypointmissionrespectively,whichresultedinatotalof100trialruns.SnapshotsofthewaypointmissionBareshowninFigure7.Missionsuccessisdefinedbyhowwelltheperformancecriteriain(5.1)aremet.Foreachtrial,thefollowingperformancevariablesweremeasured:

• t:Missioncompletiontime;• r:Robot’srelativedistancetoitsgoallocation.

5.3. Verification vs. Validation (V&V)Verificationandvalidationareconductedindependentlybyourtworesearchgroups,andtheresultsarenotshareduntilthefinalcomparisonstage.Figure8showstheresultsofverificationandvalidationof the waypoint missions. The performance guarantee is quantified as a probability distributionthatrepresentstherobotmission’slikelihoodforsuccess.Theseresultsalsoserveasthebasisforperformancefeedback;andhowthisinformationshouldultimatelybepresentedtothemissionoperatorwasinvestigatedinourrecenthuman-subjectsstudy(O’Brien&Arkin,2016).

Figure8comparesthevalidationandverificationresultsoftheperformanceguaranteesforthetwowaypointmissions.Figures8aand8cshowtheV&VresultsforthespatialcriterionP(r≤Rmax)of(5.1),theprobabilitythattherobotarriveswithinRmaxradiusofitsgoallocation.Figures8band8dshowthecomparisonsforthetimecriterionP t T≤( )max

,theprobabilitythatthewaypointmissionis completed under the time limit, Tmax. The results illustrate that the VIPARS verification ofperformanceguaranteesareconsistentwiththeoutcomesfromexperimentalvalidation.

TheV&Vresultscanbedividedintothreeregionsforfurtherinterpretation:HighConfidence(Unsuccessful), Uncertain, and High Confidence (Successful) regions. The High Confidence(Unsuccessful)istheregionofnearzeroverificationerrorandthemissionhasazeroprobabilityofsuccess.TheUncertainregionistheregionwhereverificationerrorissignificantlygreaterthanzero

Figure 7. Snapshots of validation for Mission-B

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

65

andtheprobabilityofmissionsuccessisbetween0and1.0.Asaresult,therobotisnotguaranteedtosucceedwiththemission.TheHighConfidence(Successful)istheregionofnearzeroverificationerrorandthemissionisguaranteedtosucceedwithprobabilityof1.0.Consequently,themissionoperator’sdecisionforrobotdeploymentcanbebasedonwhichregionthemissioncriteriafallinto.Forinstance,ifthespecifiedperformancecriterionfallswithintheUnsuccessfulregion(e.g.,Rmax=0.5m),theoperatorcaneitherabortthemissionormodifymissionparametersordesign.

Theoverallmissionsuccess(Equation5.1)isdefinedintermsofbothspatialandtimecriteria.Thus,weexaminedfurtherinFigures9and10theeffectsofvariouscombinationsofspatialandtimecriteria(RmaxandTmax)onthemissionsuccessandverificationerror.TheresultscanalsobeusedtoanswerqueriesregardingtheperformanceguaranteeforaspecificcombinationofTmaxandRmax.Figure9showstheeffectsofthetimecriterionTmaxontheV&VresultsofthespatialcriterionP(r≤Rmax)forMission A.WhiletheTmax’sinbothofitshighconfidenceregions(Figure8b)havenoeffectontheverificationerrorforP(r≤Rmax),Tmax’sthatareintheUncertainregion(e.g.,Tmax=415sec)incursignificantverificationerrors.Forinstance,forTmax=415sec,VIPARSpredictedasuccessprobabilityof0.18,whiletherobotwasactuallysuccessful76%ofthetimeinexperimentaltrials.

Figure 8. Results of VIPARS verification and experimental validation of spatial and time performance criteria for waypoint missions A and B

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

66

Figure 9. V&V of spatial criterion at various Tmax for Mission A

Figure 10. V&V of time criterion at various Rmax for Mission A

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

67

Figure10showstheeffectsofthespatialcriterionRmaxontheV&VresultsofthetimecriterionP(t≤Tmax).WhilesimilarobservationscanbemadehereasinFigure9,inthiscase,Rmax’shavemuchlessimpactontheverificationerrorofP(t≤Tmax)duetoVIPARS’saccuracyinpredictingthespatialperformanceofmissionevenintheuncertainregion(asshowninFigure8a).Nonetheless,missionswithperformancecriteriaintheUncertainregionsshouldgenerallybeavoided.

6. CoNCLUSIoN

Wehavepresentedanapproachtotheproblemofestablishinga-prioriperformanceguaranteesforrobotmissionsoftwareforC-WMDrobotmissionsinwhichfailurecouldmeansignificantlossoflifeorseverepropertydamage.Inparticular,wehaveaddressedtheissueofautomaticverificationofrobotmissionsoftwarethatincludessomepreexistingsoftwarelibrariesthathavetobeincludedinthemissionwithoutbeingrewrittenorannotated.

Twouniquetheoreticalresultswerepresented.Thefirsttheoreticalresultswasthedevelopmentofanenvironmentalmodelrepresentationforprobabilisticmaps,anindexedmixtureofGaussians,sufficient to support the reasoningnecessary for automatic verificationofmissions that includeprobabilisticlocalizationalgorithms.Thesecondtheoreticalresultpresentedacollectionoftechniquesfor ‘wrapping’ existing software in a random variable envelope so that it could be included inprobabilistic verification. A series of four wrapper functions – the mean-extended real functionMER,themean-extendednormalfunctionMEN,theextendednormalfunctionENF,andextendedrealfunctionERF–weredeveloped,eachwithdifferentconstraintsandrepresentationalpower.

Localization and mapping techniques intuitively offer advantages for robots navigating inunknownenvironments.ThispaperhasappliedMissionLab/VIPARSmissiondesignandverificationapproachtotheverificationofautonomousbehavior-basedrobotmissionsthatusetheROSAMCLlocalizationmodule.VerificationresultusingtheindexmixtureofGaussianmaprepresentationandERFwrapperfunctionforROSAMCLwerepresentedfortwodifferentlocalizationmissions.TheERFwrapperwaslimitedtoasinglememberandafixedvariancewasused.Tocompletelyimplementthemixtureextendedfunctionforthesampling-basedapproachinthispaper,thefullmotionhistoryforeachsamplerequestwouldneedtobesenttotheSTDRnodeandAMCLresetbetweensamples.Theabilitytocachethesemultiplesensoryhistorieswouldimprovecomputationtime,butatthecostofdirectlyinstrumentingAMCL–astepwewereavoidingforreasonsdiscussed.

Experimentalvalidationwasalsocarriedoutforthesetwomissions,andtheverificationandvalidationresultscomparedtodemonstratetheeffectivenessoftheapproach.Itisobservedhowever,thatVIPARSperformsmuchbetteronthespatialperformancecriterionthanonthetimecriterion.WhilethespatialrandomvariabletransformationsarecalculatedusingflowfunctionsinaBayesiannetwork,timeisrepresentedonlyasdiscreteiterationsoftheBayesiannetwork.Akeyavenueoffutureinvestigationistorepresenttimeasarandomvariableandhaveitalsocalculatedusingflow-functions,separatingitfromiterationsooftheBayesiannetwork.

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

68

REFERENCES

Arkin,R.C.(1998).Behavior-Based Robots.Cambridge,MA:MITPress.

Baeir,C.,&Katoen,J.-P.(2008).Introduction to Model Checking.Cambridge,MA:MITPress.

Bailey,T.,&Durrant-Whyte,H.(June,September2006).SimultaneousLocalizationandMapping(DLAM):PartyI,II.IEEE Robotics and Automation Magazine.

Blackman,S.(2004).MultipleHypothesisTrackingforMultipleTargetTracking.IEEE A&E Systems Magazine, 19(1).

Brahman,J.(2009).Verification and Analysis of Goal-Based Hybrid Control Systems[Thesis].P.D.CaliforniaInstituteofTechnology,Pasadena,CA.

Cowley,A.,&Taylor,C.(2011).TowardsLanguage-BasedVerificationofRobotBehaviors.Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

Dellaert,F.,Fox,D.,Burgard,W.,&Thrun,S.(1999).MonteCarlolocalizationformobilerobots.Proceedings of theIEEE Int. Conf. on Rob. & Aut.,Detroit.

DeMoura, L., & Bjorner, N. (2012). Satisfiability Modulo Theories: Introduction and applications.Communications of the ACM,54(9),54–67.

Fisher,M.,Dennis,L.,&Webster,M.(2013).VerifyingAutonomousSystems.Communications of the ACM,56(9),84–93.doi:10.1145/2500468.2494558

Fox,D.(2001).KLD–Sampling: Adaptive Particle Filters. Neural Information Processing Systems 14.Vancouver,Canada:NIPS.

Huang,J.,Erdogan,C.,Zhang,Y.,Moore,B.,Luo,Q.,Sundaresan,A.,&Rosu,G.(2014).ROSRV:RuntimeVerificationforRobots.Proceedings of the14th International Conference on Runtime Verification,Toronto.

Jhala, R., & Majumdar, R. (2009). Software Model Checking. ACM Computing Surveys, 41(4), 1–54.doi:10.1145/1592434.1592438

Jiang,S.,&Arkin,R.(2015).SLAM-BasedSpatialMemoryforBehavior-BasedRobots.Proceedings of the11th IFAC Symposium on Robot Control (SYROCO),Salvador,Brazil.

Katoen,J.-P.(2010).AdvancesinProbabilisticModelChecking.Proceedings of the11th International Conference VMCAI ‘10,MadridSpain.

Kiekbusch,L.,Armbrust,C.,&Berns,K.(2015).FormalVerificationofBehaviorNetworksincludingSensorFailures.Robotics and Autonomous Systems,74,331–339.doi:10.1016/j.robot.2015.08.002

Kim,M.,Kang,K.-C.,&Lee,H.(2005).FormalVerificationofRobotMovements-aCaseStudyonHomeServiceRobotSHR100.Proceedings of theIEEE Int. Conf. Robotics and Automation.

Klavins,E.(2004).ALanguageforModelingandProgrammingCooperativeControlSystems.Proceedings of the International Conference on Robotics and Automation,NewOrleans,LA.doi:10.1109/ROBOT.2004.1308780

Kress-Gazit,H.,Wongpiromsarn,T.,&Topcu,U.(2011).Correct,ReactiveRobotControlfromAbstractionandTemporalLogicSpecifications.IEEE Rob. & Aut. Mag., 18(3).

Kwiatkowska,M.,Norman,G.,&Parker,D.(2004).ProbabilisticsymbolicmodelcheckingwithPRISM:Ahybridapproach.International Journal of Software Tools and Technology Transfer,6(2),128–142.doi:10.1007/s10009-004-0140-2

Leucker,M.,&Schallhart,C.(2009).Abriefaccountofruntimeverification.Journal of Logic and Algebraic Programming,78(5),293–303.doi:10.1016/j.jlap.2008.08.004

Livingston,S.,Murray,R.,&Burdick,J.(2012).Backtrackingtemporallogicsynthesisforuncertainenvironments.Proceedings of theInternational Conference on Robotics and Automation.doi:10.1109/ICRA.2012.6225208

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

69

Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,&Liu,T.(2014).VerifyingandValidatingMultirobotMissions.Proceedings of theIEEE/RSJ Int. Conf. on Robots and Systems,Chicago.

Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,Tang,F.,&Tang,P.(2015).ProbabilisticVerificationofMulti-RobotMissionsinUncertainEnvironments.Proceedings of theIEEE Int. Conf. on Tools with AI.,VietrosulMare,Italy.doi:10.1109/ICTAI.2015.22

Lyons,D.,Arkin,R.,Jiang,S.,Harrington,D.,Tang,F.,&Tang,P.(2016).FormalPerformanceGuaranteesforBehavior-basedLocalizationMIssions.Proceedings of theIEEE Int. Conf. on Tools with AI,SanJoseCA.doi:10.1109/ICTAI.2016.0025

Lyons,D.,Arkin,R.,Jiang,S.,Liu,T.-L.,&Nirmal,P.(2015).PerformanceVerificationforBehavior-basedRobotMissions.IEEE Transactions on Robotics,31(3),619–636.doi:10.1109/TRO.2015.2418592

Lyons,D.,Arkin,R.,Liu,T.-L.,Jiang,S.,&Nirmal,P.(2013).VerifyingPerformanceforAutonomousRobotMissionswithUncertainty.Proceedings of the IFAC Intelligent Vehicle Symposium,GoldCoast,Australia.doi:10.3182/20130626-3-AU-2035.00034

Lyons,D.,Arkin,R.,Nirmal,P.,Jiang,S.,Liu,T.-L.,&Deeb,J.(2013).GettingitRighttheFirsttime:RobotMissionGuaranteesinthePresenceofUncertainty.Proceedings of theIEEE/RSJ Int. Conf. on Intelligent Robots and Systems,Tokyo,Japan.doi:10.1109/IROS.2013.6697122

MacKenzie,D.,Arkin,R.,&Cameron,R.(1997).MultiagentMissionSpecificationandExecution.Autonomous Robots,4(1),29–52.doi:10.1023/A:1008807102993

O’Brien,M.,&Arkin,R.(2016).AnAnalysisofDisplaysforProbabilisticRoboticMissionVerificationResults.Proceedings of the7th International Conference on Applied Human Factors and Ergonomics,LasVegasNV.

O’Brien,M.,Arkin,R.,Harrington,D.,Lyons,D.,&Jiang,S.(2014).AutomaticVerificationofAutonomousRobotMissions.Proceedings of the4th Int. Conf. on Simulation, Modelling and Prog. for Aut. Robots,Bergamo,Italy.

Piterman,N.,Pneuli,A.,&Sa’ar,Y.(2006).SynthesisofReactive(1)Designs.Proceedings of the7th International Conference VMCAI ‘06,CharlestownSC.

Proetzsch,M.,Berns,K.,Schuele,T.,&Schneider,K.(2007).FormalVerificationOfSafetyBehavioursOfTheOutdoorRobotRavon.Proceedings of the4th Int. Conf. on Informatics, Automation and Control,Dortmund,Germany.

Ropertz,T.,&Berns,R.(2014).Verificationofbehavior-basednetworks-usingsatisfiabilitymodulotheories.Proceedings of the41st International Symposium on RoboticsISR/Robotik ‘14.

Russel,S.,&Norvig,P.(2010).Artificial Intelligence.Prentice-Hall.

Shankar, N. (2009). Automated Deduction for Verification. ACM Computing Surveys, 41(4), 1–56.doi:10.1145/1592434.1592437

Thrun,S.,Burgard,W.,&Fox,D.(2005).Probabilistic Robotics.Cambridge,MA:MITPress.

Trojanek,P.,&Eder,K.(2014).Verificationandtestingofmobilerobotnavigationalgorithms.Proceedings of theIEEE/RSJ Int. Conf on Intelligent Robots and Systems (IROS),Chicago.

Venet,A.(2008).Apracticalapproachtoformalsoftwareverificationbystaticanalysis.ACM SIGAda Letters,27(1),92–95.

Walter, D., Taubig, H., & Luth, C. (2010). Experiences in Applying Formal Verification in Robotics.Proceedings of the29th International Conference on Computer Safety, Reliability and Security,ViennaAustria.doi:10.1007/978-3-642-15651-9_26

Watkins,O.,&Lygeros,J.(2003).Stochasticreachabilityfordiscretetimesystems:anapplicationtoaircraftcollision avoidance. Proceedings of the IEEE Conf. on Decision and Control, Maui, Hawaii. doi:10.1109/CDC.2003.1272482

International Journal of Monitoring and Surveillance Technologies ResearchVolume 5 • Issue 1 • January-March 2017

70

Damian M. Lyons is a Professor of Computer Science at Fordham University, and Director of Fordham’s Robotics & Computer Vision Laboratory. He has degrees in Math, Engineering, and Computer Science from Trinity College, University of Dublin, Ireland, and a doctorate in Computer Science from the University of Massachusetts. His research interests include formal approaches to plan and program analysis and robot team exploration strategies. He was a senior researcher and department head at Philips Corporate Research, NY. He has served as Chair of the IEEE RAS TC on Assembly and Task Planning and is an IEEE Senior Member.

Ronald Arkin is Regents’ Professor and Associate Dean in College of Computing at Georgia Tech. He served as STINT visiting Professor at KTH Stockholm, Sabbatical Chair at Sony IDL in Tokyo, and in the Robotics/AI Group at LAAS/CNRS in Toulouse. His research interests include behavior-based control and action-oriented perception, deliberative/reactive architectures, multiagent robotics, biorobotics, human-robot interaction, and robot ethics. He served on the Board of Governors of IEEE Society on Social Implications of Technology, IEEE Robotics and Automation AdCom, and founding co-chair of IEEE RAS TC on Robot Ethics. He is Distinguished Lecturer for the IEEE SSIT and IEEE Fellow.

Shu Jiang is a Robotics Ph.D. student in the Institute for Robotics and Intelligent Machines at Georgia Institute of Technology and a member of the Mobile Robot Laboratory. He received B.S. and M.S. degrees in Electrical Engineering from the University of Florida in 2009. His research interests include formal methods, human-robot team, educational and assistive robotics.

Matthew O’Brien is a Robotics Ph.D. student in the Institute for Robotics and Intelligent Machines at Georgia Institute of Technology and a member of the Mobile Robot Laboratory.

Feng Tang is a graduate student in the Robotics and Computer Vision Laboratory, Fordham University NY.

Peng Tang is a graduate student in the Robotics and Computer Vision Laboratory, Fordham University NY.

Wongpiromsarn,T.,&Murray,R.(2008).FormalVerificationofanAutonomousVehicleSystem.Proceedings of theConference on Decision and Control.

Younes,H.,&Simmons,R.(2002).Probabilisticverificationofdiscreteeventsystemsusingacceptancesampling.Proceedings of the14th Int. Conf. on Computer Aided Verification,CopenhagenDenmark.doi:10.1007/3-540-45657-0_17

Zaks, A., & Joshi, R. (2008). Verifying Multi-threaded C programs with SPIN. Proceedings of the 15th International SPIN Workshop,LosAngelesCA.

ENdNoTES

1 Inprocessalgebra,aninterleavingtheoremrelatesthesequentialandparallelcompositionoperations.