drones

Article

Energy-Efficient Inference on the Edge Exploiting TinyML Capabilities for UAVs

Wamiq Raza *,†, Anas Osman † , Francesco Ferrini and Francesco De Natale

DepartmentofInformationEngineeringandComputerScience—DISI,UniversityofTrento,38122Trento,Italy;anas.osman@studenti.unitn.it(A.O.);francesco.ferrini@studenti.unitn.it(F.F.);francesco.denatale@unitn.it (F.D.N.)* Correspondence:wamiq.raza@studenti.unitn.it† Theseauthorscontributedequallytothiswork.

check ror updates

Citation: Raza,W.;Osman,A.;

Ferrini,F.;Natale,F.D.

Energy-EfficientInferenceontheEdgeExploitingTinyMLCapabilities

forUAVs.Drones 2021,5,127.

https://doi.org/10.3390/drones5040127

AcademicEditors:Higinio

GonzálezJorge,LuisMiguelGonzálezdeSantosand

AbdessattarAbdelkefi

Received:17September2021

Accepted: 25October2021

Published: 29October2021

Publisher’s Note: MDPI stays neutral

with regard to jurisdictional claims inpublished maps and institutional affil-iations.

Copyright: © 2021 by the authors.

Licensee MDPI, Basel, Switzerland.

This article is anopenaccess article

distributed under the terms and

conditions of theCreativeCommons

Attribution(CCBY) license(https://

creativecommons.org/licenses/by/

4.0/).

Abstract: In recent years, the proliferation of unmanned aerial vehicles (UAVs) has increased dra-matically. UAVs can accomplish complex or dangerous tasks in a reliable and cost-effective waybut are still limited by power consumption problems, which pose serious constraints on the flightduration and completion of energy-demanding tasks. The possibility of providing UAVs with ad-vanced decision-making capabilities in an energy-effective way would be extremely beneficial. Inthispaper,weproposeapractical solution to thisproblem thatexploitsdeep learningon theedge.The developed system integrates an OpenMVmicrocontroller into a DJI TelloMicro Aerial Vehicle(MAV). Themicrocontroller hosts a set ofmachine learning-enabled inference tools that cooperateto control the navigation of the drone and complete a given mission objective. The goal of thisapproachisto leveragethenewopportunistic featuresofTinyMLthroughOpenMVincludingofflineinference,lowlatency,energyefficiency,anddatasecurity.Theapproachissuccessfullyvalidatedonapractical application consistingof theonboarddetectionofpeoplewearingprotectionmasks in acrowdedenvironment.

Keywords: UAVs;energyefficiency;TinyML;microcontrollers;machinelearning;deeplearning;edgecomputing

1. Introduction Drones,intheformofbothRemotelyPilotedAerialSystems(RPAS)andunmanned

aerialvehicles(UAV),areincreasinglybeingusedtorevolutionizemanyexistingapplica-tions.TheInternetofThings(IoT)isbecomingmoreubiquitouseveryday,thankstothewidespreadadoptionandintegrationofmobilerobotsintoIoTecosystems.Astheworldbecomesmoredependentontechnology,thereisagrowingneedforautonomoussystemsthatsupporttheactivitiesandmitigatetherisksforhumanoperators[1].Inthiscontext,UAVsarebecomingincreasinglypopularinarangeofcivilandmilitaryapplicationssuchassmartagriculture[2],defense[3],constructionsitemonitoring[4],andenvironmentalmonitoring[5].

Theseaerialvehiclesaresubjecttonumerouslimitationssuchassafety,energy,weight,andspacerequirements.ElectricallypoweredUAVs,whichrepresentthemajorityofmicroaerialvehicles,showaseverelimitationinthedurationofbatteries,whicharenecessarilysmallduetodesignconstraints. Thisproblemaffectsboththeflightdurationandthe

capabilityofperformingfastmaneuvers(e.g.,toavoidobstacles)duetotheslowpowerresponseofthebattery.Therefore,despitetheiruniquecapabilitiesandvirtuallyunlimitedopportunities,thepracticalapplicationofUAVsstillsuffersfromsignificantrestrictions[6].

RecentadvancesinembeddedsystemsthroughIoTdevicescouldopennewandinterestingpossibilitiesinthisdomain.EdgecomputingbringsnewinsightsintoexistingIoTenvironmentsbysolvingmanycriticalchallenges. Deeplearning(DL)attheedge

presentssignificantadvantageswithrespecttoitsdistributedcounterpart:itallowsthe

Drones 2021,5,127.https://doi.org/10.3390/drones5040127 https://www.mdpi.com/journal/drones

Drones 2021,5,127 2of10

performanceofcomplexinferencetaskswithouttheneedtoconnecttothecloud,resultinginasignificantlatencyreduction;itensuresdataprotectionbyeliminatingthevulnerabilityconnectedtotheconstantexchangeofdata;anditreducesenergyconsumptionbyavoidingthetransmissionofdatabetweenthedeviceandtheserver[7].

AnotherrecenttrendreferstothepossibilityofshiftingtheMLinferenceperipherallybyexploitingnewclassesofmicrocontrollers, thusgenerating thenotionofTinyMachineLearning (TinyML) [8]. TinyML aims to bringML inference into devices characterized bya very low power consumption. This enables intelligent functions on tiny and portabledeviceswithapowerconsumptionof less than1mW.AsTinyML targetsmicrocontrollerunit(MCU)classdevices,thetrainedanddevelopedmodelsmustconformtothehardwareand software constraints of MCUs. Therefore, many TinyML frameworks have beendeveloped to fully exploit MCU resources and ensure the optimization of the modelconverted from the originalmodel, e.g., the TensorFlowLite forMicrocontrollers (TFLM)framework. TFLMisoneofthemostwidelyusedTinyMLframeworks,whichiscompatiblewiththewell-knownMLlibrariesTensorFlow[9]andKeras[10].

Buildingupontheabovetechnologicaltrends,theintegrationofstate-of-the-artultra-lowpower embeddeddevices intoUAVs couldprovide energy-aware solutions to embedanincreasingamountofautonomyandintelligenceintothedrone,thuspavingthewayformanynovelandversatileapplications.Suchdevicescanrunonacoin-sizedbattery,areca-pableofprocessinglargeamountsofdatainreal-time,and,mostimportantly,canperforminferencesattheedgewithoutrequiringanexternalconnectiontothecloudortoaremoteprocessingunit[11],thusavoidingtheneedforenergy-eagerdatatransmissionprotocols.

Inthispaper,weproposeanovelapproachtoendowdroneswithdeeplearningcapa-bilitieswithoutcompromisingflighttimeorimposingfurtherconstraints.Wepresentthe

integrationofanOpenMVCamH7MCU,whichsupports theTFLM, intoaDJITellodroneto enable the development and deployment of TinyML applications on amicrocontrolleronboardthedrone. The inferenceengine implementedonthemicrocontrollertakescareof both the navigation and recognition tasks in an integrated way. The drone flight iscontrolledbytheonboardintelligenceandisstabilizedthroughaPIDcontrollertoreduceturbulenceandobtainsteadyimages.Atthesametime,theimagescapturedbythecameraareprocessedonboardinreal-timetoaccomplishagivenhigh-leveltask.Themaincontri-butionofthisworkistodemonstratethroughapracticalimplementationthefeasibilityofa completely autonomous smart drone capable of performing complex missions (in thesample application, targeting people in a crowded environment anddetecting if they arewearing a protectionmask or not) in an energy-effectiveway andwithout the need forconnectingtothegroundduringtheflight.

Therestofthepaperisorganizedasfollows:inSection2,wediscussandcomparetheenergy-efficienttechniquesforUAVs;inSection3,wepresenttheproposedsystem;inSection4,wediscussthesystemvalidationbydescribinganexperimentalsetupandtherelevantsimulationresults;andfinally,inSection5,wedrawtheconclusions.

2. Energy-Efficient Machine Learning Approaches for UAVs Inthissection,webrieflyreviewseveralrecentapplicationsofmachinelearningand

artificial intelligence to UAVs, mainly focusing on energy efficiency. In [12], the authorspresent a long short-termmemory (LSTM) inference algorithm to predict future mobiletrafficusingback-propagation-basedneuralnetwork training. Theyproposed thedivisionof the entire coverage area into clusters using a joint k-means and anEMalgorithmof aGaussianmixturemodel.Astheirapproachrequiredacentralintelligence,theoptimizationwasperformedat the ground station.Q-learning is another techniqueused inmanyUAVapplications to find an optimal flight path and rechargemethod thatmaximizes the totalflightdurationof theUAV.Asanexample, in [13], theUAVreceivedupdated informationateachstepandaUAVbasestationwasused forbatteryrechargingbyawirelesspowertransfer from another flight source. Although the proposed scheme probably requiredonboardcomputation,noimplementationdetailswereprovidedbecausetheauthorsbased

Drones 2021,5,127 3of10

theiranalysisonsimulations.Ameta-gradientinspiredMLmodeloverawirelessnetworkwasproposedin[14],calledhierarchicalnestedpersonalizedfederatedlearning(HN-PFL).Suchatechniqueoptimizesthetrade-offbetweenenergyconsumptionandtheperformanceof a machine learning model by configuring the exchange of data among UAVs.

Clustering is a well-known energy-efficient method in which nodes select a clusterhead. In [15], a deep learningmodel is proposed tomanage cluster-basedUAVnetworksin an energy-effectiveway. Themodel adopts a cluster-level fuzzy logic technique basedonaresidualnetwork.Theycomparetheachievedresultswithothercompetingmethods,demonstratingthattheT2FL-Cguaranteesalowerenergyconsumption;however,animageanalysisisperformedatthebasestationwheretheimagesaretransmitted.Energy-efficient,a fair 3D deployment, and energy replenishment strategies formultiple UAVs are jointlystudiedin[16].Theytakeinspirationfromdeepreinforcementlearning(DRL)todesignaUAV control policybasedon thedeepdeterministic policy gradient (DDPG), a deep actorcriticalalgorithm. Theresultsarepresentedbasedonsimulations.

Tomaximizeenergyefficiencyinterrestrialcommunications,[17]proposesvariouskeyperformanceindicators(KPIs)soughttosatisfytheDRLframeworkanddefineanoptimalenergyefficiencystrategyfromheterogeneousdata.Theyconvertthedeeplearningproblemintodeepqueuelearningtooptimizetheenergyefficiencyforairborneuserswhilstminimizinginterferencewithterrestrialusers.Unfortunately,theirapproachimpliesahighlycomplex,non-convexoptimizationproblemand,duetothehighmobility,theUAVnetworksmustco-existwithterrestrialnetworks.TheproposedapproachiscompletelycentralizedasitrequirestheinterconnectionoftheUAVstoanetworkofinterconnectedbasestations,whicharecoordinatedbyacentralcontrolnode.

Acomparisonofthebattery-basedenergy-efficientmanagementsolutionsispresentedinTable1.

Table 1. Comparisonbetweentheproposedenergy-efficientmethodsusingML,DL,andRLapproaches.

Research Area Objective Outcome Limitations Refs.

MachineLearningFlyingBaseStation

Q-LearningataRechargeHotspot

DistributedMLonUAVNetworksForGeo-Distributed

DeviceClusters

Cluster-BasedUAVNetworksWithDeepLearning(DL)

DeepDeterministicPolicyGradient(UC-DDPG)

InterferenceManagementDeepLearning

Themodelusesback-propagationlearningto

predictfuturetrafficAnoptimizedtrajectoryis

proposedtosolvetheproblem.AMarkovdecisionprocessis

usedforwirelessUAVhotspotservices

MLpersonalizedlocalmodelsHN-PFLareproposedwhereitsplitsthedrone-basedmodeltrainingproblemintoa

network-enabledmacro-trajectoryandlearning

durationdesignsClusteringwitha

parameter-tunedresidualnetwork(C-PTRN)thatworksintwomainphases,clustering

andsceneclassification3DdronebasedontheDDPGalgorithm,whichconsiderstheresidualenergy,mobilitypower,circuitpower,communicationpower,andhoverpowerProposalofvariouskey

performanceindicators(KPIs)toachieveatrade-offbetween

maximizingtheenergyefficiencyand

spectralefficiency

Thesimulationresultshowsthatthepowercanbereduced

by24%Theresultsconfirmthatthetwobenchmarkstrategies,randommotionandstaticlevitation,

outperformSoA

Theirdeepreinforcementlearningapproachshowsa

reductioninthepercentageofenergyconsumptioncompared

withgreedyoffloading

TheresultsshowthattheT2FL-Ctechniquereachesthelowestenergyconsumption

ThesimulationresultsshowthatUC-DDPGinspiredbyreinforcementlearninghasa

goodconvergence

Theapproachmakesthe

advantagesofusingintelligentenergy-efficientsystemsevident

Everybasestation(BS)hasthesamecellulartrafficdistribution

DuetothepowerfulEMwaverequirement,theenergy

transferislimitedbydistance

TheUAVsperformthelocalmodeltrainingand

commonalityamongthedataacrossthedeviceonthe

clustersonly

TheT1FL-Cmodelmethodachievespoorresultswithahighenergyconsumptioncomparedwithrelated

techniques

TheRLmodelisnotsuitableforcomplextasksorworkingin

continuousandhighdimensionalspaces

Ahighlycomplex,non-convexoptimizationproblem.Duetothehighmobility,calculatingthesolutionineverytimeinstantisunrealistic

[13]

[14]

[15]

[16]

[17]

[12]

Drones 2021,5,127 4of10

3. Proposed System In this section, we discuss our proposed system. The objective was to develop an

integratedarchitecture that implementeda fullyautonomoussmartdrone,able toaccom-plishagivenmissiongoal ina completely independentwaywhilstoptimizing thebatteryconsumption to maximize the flight duration. For this purpose, we focused on a mobilearchitecture based on the computation at the edge paradigm, thus seeking an embeddeddevice with acquisition and processing capabilities with appropriate characteristics. Inthis context, the envisaged solution leveragedon the capabilities of TinyML [18] to allowthe MAV to take advantage of the ML without significantly affecting the flight durationor imposing further constraints on the already constrained MAV architecture. The blockdiagramoftheproposedsystemisillustratedinFigure1.

Figure 1. Blockdiagramoftheproposedsystem.

Asabasisforthedevelopmentofourautonomoussystem,wechoseaDJITelloduetoitseasyprogrammabilityandwideavailability.TheTellodrone[19]hasamaximumflighttimeofupto13minutes,aweightofabout80g(withpropellersandbattery),anddimensionsof98mmx92.5mmx41mm.Itmounts3-inchpropellersandhasabuilt-inWIFI802.11n2.4Gmodule.

As for the TinyML platform, we chose an OpenMVmicrocontroller, which actedas a decision unit. TheOpenMV platform [20] is a small, low powermicrocontrollerthatenablestheeasyandintuitiveimplementationofimageprocessingapplications. Itcanbeprogrammedusinghigh-levelPythonscripts(Micro-Python). It isdrivenbyanSTM32H74VIARMCortexM7processorrunningat480MHz,whichissuitableformostmachinevisionapplications.OpenMVwasparticularlysuitableforourproposedapproachduetoitslowpowerconsumptionandweightasOpenMVasapayloadwouldnotpreventthedrone fromtakingoff. Itwasalsoequippedwithahigh-performancecamera thatweusedtocollectdataforthemissionpurposes. Inaddition,OpenMVwasselectedtoensurethatthedronewouldbeabletotakeoffandoperatewhilstcarryingtherelevantpayloaddue to thesharpweight limitationsof theDJITellodrone. Consequently, theenergyconsumptionofthedronewithandwithoutthepayloadwastestedtoverifythattheproposedintegrationwasenergy-effective.

TheOpenMVmicrocontrollerwasconnected to thedroneviaaWiFiUDP link toallowthedatatransfer.Inthisway,theconnectivitywasguaranteedindependentlyoftheavailabilityofaninternetconnection,thusmeetingthegoalofafullyfunctionalsystemregardlessoftheenvironmentorsituation.Communicationwasfundamentaltocontrolthedronethroughthesystemintelligence.

Oneof themost important featuresof thesystemwas theTinyMLmodel,whichwasintegratedaspartofanapplicationtoverify the functionalityof theproposedsystem.Webeganbuildingourmodelwith theTensorFlow(TF) library,whichwas thenconverted tothe lightweight version, TensorFlow Lite (TFL),which is suitable formobile applications.TheTFLmodelwas optimized through a quantization process [21] and finally themodelwasdeployedaspartoftheclassificationapplicationofthemicrocontrollersupportingthe

Drones 2021,5,127 5of10

×

TensorFlowLiteforMicrocontrollers(TFLM)framework[22].TheTFLMcouldinterpretthemodelandperformedtheinferencetaskbasedontheinputdata.

Asforthedronenavigation,thestrategywasdeterminedbythemissionobjectiveandbytheresultsoftheinferenceengine,whichtriggeredacontinuousdetection-controlloop:onceaninferenceresultwasoutput,adecisionwasmadeonthenextdesiredpositioningofthedroneandthisinformationwascommunicatedtotheMAVtoperformthenecessarycontroloperations.Asanexample,adetectiontaskmightrequireabetterframingofthetargetobject;thisinturnimpliesthepositioningofthedroneinamoresuitableposition,whichhastobetranslatedintoasetofcontrolcommands.Afterfurthertargetcapturing,theinferenceenginecanachieveamoreaccurateclassification,whichleadstoafurthertargetaccordingtotheoverallmissiongoal.

Afterperformingafewinitialtrialsofthedronecontrolloop,werealizedthatthenavigationwasnotsatisfactoryduetoinstabilityproblems.Thisinturnledtoproblemsincapturingsufficientlystableimages(e.g.,motionblur,imperfectframing),oftenresultinginthelossofthetarget.Tosolvethisproblem,weintroducedaPIDcontrollerintheloop[23].ThePIDprovidedasimplebuteffectivesolutiontostabilizethedroneaseachvariablecouldbetreatedseparatelywithina limitedrangewheretheMAVactionwasroughlylinear.Foreachcontroliteration,wecalculatedtheerrorbetweentheactualpositionandthetargetposition(whichconsideredthepositionofthetargettobeclassified).Iftheactualvaluewasgreaterthanthetargetvalue,apositiveerrorwassentbacktothesystem.TheerrorwasthennormalizedandsenttothePID,whichcalculatedthenewvaluesforthespeedofthemotors.Fortheproportionalterm,wemultipliedtheerrorbyagainfactor,whichwassetat0.5afterseveralexperimentaltrials.Tocontrolthederivativeterm,thedifferencebetweenthecurrentandthepreviouserrorwascalculatedandagainmultipliedbyagainfactorof0.5.Accordingly,theintegraltermwassetbysummingtheerrorsovervariousstepsandapplyingagainfactortothecumulativeerror.Oncewecalculatedallthecorrectionfactors,wetransmittedthenewspeedvaluestothedrone. Todealwithout-of-rangesituations,wetruncatedthevaluesintherangeof+/-100.

4. Experimental Validation 4.1. Experimental Setup

In this section,we describe the experimental setup implemented to validate theproposedenergy-awaresmartdroneinthecontextofapracticalapplication.Wedefinedamissioninwhichthedronenavigatedoverapopulatedenvironmentandhadtoidentifypeople(facedetectiontask)andclassifythemaswearingaprotectionmaskornot(abinaryclassifier).TheinspirationforsuchanapplicationcamefromthecurrentCOVID-19-relatedrecommendationsasasupportforlocalauthoritiestoenforcerelevantrestrictions[24].Toimplementthisspecificmission,wehadtospecializetheOpenMVmicrocontrollertoperformthefacedetectionandmaskrecognitiontasks.

As for the set of data for training and validation, we adopted the open-source FaceMask Lite [25] dataset, which is based on generated faces for model training withoutprivacy problems. In Figure 2, we represent the dataset. Even after rescaling and colorconversions,imagefeatureshaveahighdimensionalitythatpreventssuitablevisualization.Forthisreason,adimensionalityreductionwasappliedtoreducetheimagetoa3Dspace(the three visualization layers). To train the model, we randomly selected 1000 images,equally divided into images with and without face masks. Each image was resized to96 96pixels. The number of sampleswas chosen to complywith the constraints of theselectedmicrocontrollerandtooptimizethemodelsize.

Drones 2021,5,127 6of10

Figure 2. Datasetvisualizationandseparabilityoftheclasses.

As far as the neural architecture was concerned, in our application we selected a

convolutional neural network (CNN) to train a model compatible with the TinyML de-ployment.ThefacemaskdetectionmodelusesMobileNetV2architecture(seeFigure3),awell-known and proven architecture developed for image classifications [26]. MobileNetV2consistsofastackof16depth-wiseseparableconvolutionallayerswithanaveragepoolfollowed by a fully connected layer and a SoftMax at the end. The width multiplier waschosenat0.35as itwasoptimized forourmicrocontroller in termsofRAMusageandaneffectivecomputationreduction.

Figure 3. TheconvolutionalblockofMobileNetV2.

Drones 2021,5,127 7of10

Oneof thecriticalpoints inbringingadeeparchitectureontoa tinyprocessor is theneed to reduce as much as possible the complexity without significantly affecting theaccuracy.Oneofthecommonpracticesconsistsofreducingtheprecisionrequirementsforboththeweightsandtheactivations[27]. This isanessentialsteptocompressthemodeltomeet the hardware constraints of the OpenMVmicrocontroller. Although quantizationisstillanopenresearchtopic, ithasbecomeastandardcompressionmethodforTinyML-related applications. It allows the consumption of less flash memory and RAM whilstmaintainingalmost the sameaccuracyof theoriginalmodel [28]. Moreover, compressingthenetworkfrom32-bitto8-bitresultsinasignificantlyfasterprocessing,shorterinferencetime,andlowerpowerconsumption.

Inourcase,thenon-optimizedmodelconsumed1.6MBofflashmemory,1.7sofonboardinferencetime,and957KBofRAM.Themodelsizeinthisconditionwasnotcompatiblewithourembeddeddevice.Afterintroducingadynamicquantizationfroma32-bitfloatingpointtoan8-bitinteger,theresultingoptimizedmodelshowedasignificantreductioninsize(585KB);theonboardinferencetimewasreducedto859msecandtheuseofRAMwaslimitedto296KBwithanaccuracyafterthepost-trainingvalidationof97%.

4.2. Experimental Results and Comparisons The main parameters to assess the performance of the system concerned the per-

formance in accomplishing the application task (classification accuracy) and the energyconsumption.As for the firstparameter,we shouldmention that the classificationengineimplemented inbothsystemswas identicalandthetrainingwasperformedexactly in thesameway.We expected, therefore,minimal differencesdue tonumerical precision in therelevantprocessingarchitectures.

To estimate the power consumption of the drone in flight missions, we defined adetailedmodelofthebatteryperformanceofthedrone. Themeasurewasbasedonboth

mathematicalmodels [seealso16]and realmeasures. As for the latter,weexploited thesensorsmounted on the drone used for stabilization purposes and to remotelymonitorthebatterystatusofthedrone(stateofcharge,SoC).Thestabilityofadroneiscontrolled

by three main sensors (i.e., gyroscopes, accelerometers, and barometers); the SoC wasmeasuredbythevoltageandcurrentsensorsviaaTellosoftwaredevelopmentkit(SDK).

Theenergyconsumptionofadronevariesindifferentstatesincludingtakeoff,reach-ingthedesiredaltitude,hovering,andmaneuvering.Therefore,itwasimportanttocreateaconsistenttestscenarioforallthesetups.Forthispurpose,allthetestswereperformedinequalconditionswheretheUAVhoveredatarelativealtitudeof5m. Furthermore,eachscenariowastested10times;thepresentedresultswerecalculatedbyaveragingthemeasuredvalues.

In order to provide a comparative analysis,we selected three cases. The first set ofmeasures referred to the drone without any payload installed. In this case, the missionwassimulatedwitharemotecontroller.Thesecondcasereferredtoanalternative imple-mentation thatwedesignedusing anArduinoNano33BLE and the third referred to theproposedsystembasedontheOpenMVmicrocontroller.

AsfortheArduinoimplementation,itshouldbepointedoutthatthiscouldrunthesameTinyMLmodelforthesameusecaseaftergoingthroughtheoptimizationstepsusingthequantizationmethod.TheArduinoNanowaschosenbecauseofitslightweightandlowpowerconsumption.TheArduinowasequippedwithanOV7675camerathatwasusedtocapturedataastheinputtoourfacemaskdetectionapplication. AlthoughtheArduinowasabletorunthemodel,ittook7secondstoperformtheinferenceduringtheliveclassificationandmostofthetimeitfailedtofullycompletetheclassificationprocessduetothelimitedRAM.Overall, theArduinoNanodidnotworkwhenthemodelwasusedforpracticaltesting.TotesttheOpenMVmicrocontroller,weusedthesamemodeltodeploytheapplicationandruntheinference.Theliveclassificationtimewas859mswiththeRAMpeakstillbelowthemaximumofthedevice.OpenMVoutperformedtheArduinoNanoinallcomparisoncriteria.Nevertheless,thedecisionmatrixfortheliveclassification

Drones 2021,5,127 8of10

ofthemodelshowedthatbothmicrocontrollersachievedthesameresults. Therefore,weconcluded that OpenMVwasmore effective for our proposed system (see Figure 4).

Figure 4. Processingdataandconfusionmatrix for theArduinoNano(above)andOpenMV(below)architectures.

Table2showstheaverageenergyconsumption(kJoule)andflighttime(min:sec)measuredacrossthevarioustestsinthedifferentsetupsofthesystem(nopayload,Arduinopayload,andtheproposedsystembasedontheOpenMVpayload). Theperformancesweremeasuredindifferentstatesandflightoperations.

Table 2. Energyconsumptionandflightdurationcomparison.

Consumption (kJoule) No Payload Arduino OpenMV Idle 60.192 - -

Hovering 77.976 96.307 116.280Maneuvering 89.727 112.449 141.588

Flying time (min:sec) No Payload Arduino OpenMV IdleState 12:00 - -Hovering 09:23 7:50 6:20

Maneuvering 08:05 6:42 5:10

Fortheidlestate,theUAVwaspoweredonwithoutthefourmotorsrotating;only

theinternalprocessingandLEDswereon,thusresultinginaratherstableandlowpowerconsumption.Forthehoveringstate,theUAVwasmadetotakeoffandhoveratacertainaltitudeashoveringbetweenaltitudescausedfluctuationsintheenergyconsumptionduetosuddenconsumptionspikeswhenclimbingtohigheraltitudes.TheresultsshowedthatthehoveringstatemaintainedareasonablystablepowerconsumptionaswhentheUAVhovered,theforcesactingontheUAVwereideallybalanced.Moreover,themaneuveringstatewasthemostenergy-demandingstate.Inhorizontalflightwithalimitedspeed,theenergyconsumptionfluctuatedsomewhatbuttendedtostabilizeovertime.Theenergyconsumptionduringhorizontalflightwasslightlyhigherthantheenergyconsumptionofhoveringatthesamealtitude.Thiswastheresultofthedifferentthrustdevelopedbythedroneinthetwodifferentmodes.

Asexpected,theintegrationofthetwomicrocontrollersincreasedtheenergycon-sumption,whichalsoaffectedtheflightduration.Roughlyspeaking,theproposedsetupreducedtheflyingtimebyaround30%inatypicalmissioninvolvingdifferentnavigationmaneuvers.Nevertheless,itshouldbepointedoutthatsuchalossofperformancewasnotdramaticcomparedwiththeadvantagesintermsofthecapabilitiesofthesmartsystemwithrespecttothecompletelymanualsetup.

Drones 2021,5,127 9of10

Tofurthervalidatetheproposedonboardinferencesystem,wecompareditwithaconventionaldistributedinferenceapproachwhereonlyacameraandacommunicationmodulewereinstalledonthedrone;theprocessingwasperformedataremoteunitthatcommunicatedtheresultsbacktotheUAV.Inthiscase,theenergybudgetintermsofthecommunicationlinkslightlydecreasedtheperformanceofthedistributedsystem. TheresultsintermsoftheenergyconsumptionandflighttimearereportedinTable3andcouldbecomparedwiththeproposedsystem(samedataofTable2,Column3).Wenotedthatthedistributedinferenceapproachwasslightlylessenergy-demandingcomparedwiththecompletelyonboardsystemandclosetotheArduino-equippedsetup.Nevertheless,itshouldbepointedoutthatinthiscasewelostthereal-timecapabilityaseverydecision(detection,classification,navigation)requiredacommunicationloopwithasignificantround-triptime.Furthermore,ifthesystemneededtooperateinanenvironmentwhereitwasnotpossibletoestablishastableconnectionortoplaceagroundstation,thedistributedmodelfellshort.

Table 3. Energyconsumptionandflightdurationcomparisonwithdistributedprocessing.

Consumption (kJoule) Distributed Inference OpenMV Hovering 86.320 116.280

Maneuvering 101.232 141.588

Flying time (min:sec) Distributed Inference OpenMV Hovering 08:36 6:20

Maneuvering 07:13 5:10

5. Conclusions

Inthispaper,wepresentedanovelapproachtoendowdroneswithalargerautonomyandintelligencethankstotheintegrationofajointflightandmissioncontrolembeddedsystem. Thankstotheadoptionofapowerful lightweightprocessingarchitectureandasuitablydesignedML inferenceengine, thesystem implementedanedgecomputingsolutionthatenabledtheachievementofsophisticatedmissiongoalswithoutseverelylimitingtheflightduration.Theenergyconsumptionofthesystemwastestedinvarioussetupsandflyingscenariostodemonstrateitsenergyefficiency. Anexperimentalvali-dationwasperformedonasignificantsampleapplicationandshowedthattheimpactoftheintegrationofthemicrocontrolleronthepayloadwastolerablewithrespecttotheaddedvalue in termsof the intelligenceof thesystem, thusmaking thesystemviableinpracticalcontexts. Futureworkwillincludetheimplementationoffurtherusecasesindifferentapplicationdomainsandtheextensiontootherdronemodelstopromoteabroaderadoptionoftheproposedtechnology.

Author Contributions: Conceptualization,W.R.andA.O.;methodology,W.R.,A.O.andF.D.N.;software,W.R.,A.O.andF.F.;validation,W.R.andA.O.;formalanalysis,W.R.,A.O.andF.D.N.;investigation,W.R.,A.O.andF.D.N.;resources,W.R.,A.O.andF.F.;datacuration,W.R.andA.O.;writing—originaldraftpreparation,W.R.,A.O.andF.D.N.;writing—reviewandediting,W.R.,A.O.andF.D.N.;visualization,W.R.,A.O.andF.F.;supervision,F.D.N.;projectadministration,W.RandF.D.N.Allauthorshavereadandagreedtothepublishedversionofthemanuscript.

Funding: Thisresearchreceivednoexternalfunding.

Institutional Review Board Statement: Notapplicable.

Informed Consent Statement: Notapplicable.

Data Availability Statement: Some or all data, models, or code that support the findings of thisstudyareavailablefromthecorrespondingauthoruponreasonablerequest.

Conflicts of Interest: Theauthorsdeclarenoconflictofinterest.

Drones 2021,5,127 10of10

References 1. Atzori,L.;Iera,A.;Morabito,G.Theinternetofthings:Asurvey.Comput. Netw. 2010,54,2787–2805.[CrossRef]2. Maddikunta,P.K.R.;Hakak,S.;Alazab,M.;Bhattacharya,S.;Gadekallu,T.R.;Khan,W.Z.;Pham,Q.V.UnmannedAerialVehicles

inSmartAgriculture:Applications,Requirements,andChallenges.IEEE Sens. J. 2021,21,17608–17619.[CrossRef]3. Roberge,V.;Tarbouchi,M.;Labonté,G.FastGeneticAlgorithmPathPlannerforFixed-WingMilitaryUAVUsingGPU.IEEE

Trans. Aerosp. Electron. Syst. 2018,54,2105–2117.[CrossRef]4. Qu,T.;Zang,W.;Peng,Z.;Liu, J.;Li,W.;Zhu,Y.;Zhang,B.;Wang,Y.ConstructionSiteMonitoringusingUAVOblique

PhtogrammetryandBIMTechnologies.InProceedingsofthe22ndCAADRIAConference,Suzhou,China,5–8April2017.5. Manfreda,S.;McCabe,M.F.;Miller,P.E.;Lucas,R.;PajueloMadrigal,V.;Mallinis,G.;BenDor,E.;Helman,D.;Estes,L.;Ciraolo,

G.;etal.OntheUseofUnmannedAerialSystemsforEnvironmentalMonitoring.Remote Sens. 2018,10,641.[CrossRef]6. Boukoberine,M.N.;Zhou,Z.;Benbouzid,M.Acriticalreviewonunmannedaerialvehiclespowersupplyandenergymanage-

ment:Solutions,strategies,andprospects.Appl. Energy 2019,255,113823.[CrossRef]7. Voghoei,S.;Tonekaboni,N.H.;Wallace,J.G.;Arabnia,H.R.Deeplearningattheedge.InProceedingsofthe2018International

ConferenceonComputationalScienceandComputationalIntelligence(CSCI),LasVegas,NV,USA,12–14December2018.8. Banbury, C.R.; Reddi, V.J.; Lam,M.; Fu,W.; Fazel, A.;Holleman, J.;Huang, X.;Hurtado,R.; Kanter,D.; Lokhmotov,A.; et al.

BenchmarkingTinyMLsystems:Challengesanddirection.arXiv 2020,arXiv:2003.04821.9. Abadi,M.;Barham,P.;Chen,J.;Chen,Z.;Davis,A.;Dean,J.;Devin,M.;Ghemawat,S.;Irving,G.;Isard,M.;etal.Tensorflow:A

systemforlarge-scalemachinelearning.InProceedingsofthe12th{USENIX}SymposiumonOperatingSystemsDesignandImplementation({OSDI}16),Savannah,GA,USA,2–4November2016.

10. Ketkar,N.Introductiontokeras.InDeep Learning with Python;Apress:Berkeley,CA,USA,2017;pp.97–111.11. Marchisio,A.;Hanif,M.A.;Khalid,F.;Plastiras,G.;Kyrkou,C.;Theocharides,T.;Shafique,M.DeepLearningforEdgeComputing:

CurrentTrends,Cross-LayerOptimizations,andOpenResearchChallenges.InProceedingsofthe2019IEEEComputerSocietyAnnualSymposiumonVLSI(ISVLSI),Miami,FL,USA,15–17July2019;pp.553–559.[CrossRef]

12. Lu, L.; Hu, Y.; Zhang, Y.; Jia, G.; Nie, J.; Shikh-Bahaei,M.Machine Learning for Predictive Deployment of UAVswith RateSplittingMultipleAccess.InProceedingsofthe2020IEEEGlobecomWorkshops,Taipei,Taiwan,7–11December2020.

13. Hoseini,S.A.;Hassan,J.;Bokani,A.;Kanhere,S.S.Trajectoryoptimizationofflyingenergysourcesusingq-learningtorechargehotspotuavs.InProceedingsoftheIEEEINFOCOM2020-IEEEConferenceonComputerCommunicationsWorkshops,Toronto,ON,Canada,6–9July2020.

14. Wang,S.;Hosseinalipour,S.;Gorlatova,M.;Brinton,C.G.;Chiang,M.UAV-assistedOnlineMachineLearningoverMulti-TieredNetworks:AHierarchicalNestedPersonalizedFederatedLearningApproach.arXiv 2021,arXiv:2106.15734.

15. Pustokhina, I.V.;Pustokhin,D.A.;KumarPareek,P.;Gupta,D.;Khanna,A.;Shankar,K.Energy-efficientcluster-basedunmannedaerialvehiclenetworkswithdeep learning-basedsceneclassificationmodel. Int. J. Commun. Syst. 2021,34, e4786. [CrossRef]

16. Qi,H.;Hu,Z.;Huang,H.;Wen,X.;Lu,Z.Energyefficient3-DUAVcontrolforpersistentcommunicationserviceandfairness:Adeepreinforcementlearningapproach.IEEE Access 2020,8,53172–53184.[CrossRef]

17. Ghavimi, F.; Riku, J. Energy-efficient uav communications with interference management: Deep learning framework. InProceedingsofthe2020IEEEWirelessCommunicationsandNetworkingConferenceWorkshops(WCNCW),Seoul,Korea,6–9April2020.

18. Sanchez-Iborra,R.;Skarmeta,A.F.Tinyml-enabledfrugalsmartobjects:Challengesandopportunities.IEEE Circuits Syst. Mag. 2020,20,4–18.[CrossRef]

19. RAZE.Tello.2019.Availableonline:https://www.ryzerobotics.com/kr/tello(accessedon13March2021).20. Abdelkader, I.;El-Sonbaty,Y.;El-Habrouk,M.Openmv: APythonpowered,extensiblemachinevisioncamera. arXiv 2017,

arXiv:1711.10464.21. Krishnamoorthi,R.Quantizingdeepconvolutionalnetworksforefficient inference: Awhitepaper. arXiv 2018,arXiv:1806.08342.22. David,R.;Duke,J.;Jain,A.;Reddi,V.J.;Jeffries,N.;Li,J.;Kreeger,N.;Nappier,I.;Natraj,M.;Regev,S.;etal.Tensorflowlitemicro:

Embeddedmachinelearningontinymlsystems.arXiv 2020,arXiv:2010.08678.23. Joyo,M.K.;Ahmed,S.F.;Bakar,M.I.A.;Ali,A.HorizontalMotionControlofUnderactuatedQuadrotorUnderDisturbedand

NoisyCircumstances.InInformation and Communication Technology;Springer:Singapore,2018;pp.63–79.24. Mohan,P.;Aditya,J.P.;Abhay,C.Aditya,J.P.;Abhay,C.AtinyCNNarchitectureformedicalfacemaskdetectionforresource-

constrainedendpoints.InInnovations in Electrical and Electronic Engineering;Springer:Singapore,2021;pp.657–670.25. Available online: https://www.kaggle.com/prasoonkottarathil/face-mask-lite-dataset (accessed on 8 July 2021).26. Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. Mobilenetv2: Inverted residuals and linear bottlenecks. In

Proceedingsof the IEEEConferenceonComputerVisionandPatternRecognition (CVPR), Salt LakeCity,UT,USA,18–23June2018.

27. Warden,P.;Situnayake,D.Tinyml: Machine Learning with Tensorflow Lite on Arduino and Ultra-Low-Power Microcontrollers;O’ReillyMedia:Newton,MA,USA,2019.

28. Heim,L.;Biri,A.;Qu,Z.;Thiele,L.MeasuringwhatReallyMatters:OptimizingNeuralNetworksforTinyML.arXiv 2021,arXiv:2104.10645.