SHM WITH AUGMENTED REALITY FOR AIRCRAFTMAINTENANCE

G.O.C. Prado∗, P.A. Silva∗, F.M. Simomura∗, K.L.O. Pereira∗∗Embraer S.A., Brazil

Keywords: Structural Health Monitoring, Augmented Reality, Aircraft Maintenance, AircraftInspection, Nondestructive Testing

Abstract

Structural Health Monitoring (SHM) has a greatpotential to reduce the costs of the current aero-nautical maintenance procedures that are donethrough Non Destructive Test (NDT). Thesetasks use to be complex, as they are normallydone in areas with restrict access that demandthe disassembly of several parts of the aircraft.

With the SHM trustworthy systems, it ispossible to evaluate the current conditions of thestructure without the need of disassembly andthus inform the maintenance company about thepresence and location of a structural failure.

Another technology that has shown greatpotential of use in the aeronautical industry isthe Augmented Reality (AR). The integrationof the real world with the virtual world allowsthe time reduction in tasks of manufacturing,maintenance, and training.

This paper shows that, when SHM and ARare combined, the data about the structure -obtained by the SHM technology - can be shownin real time without the need of disassembly, thusreducing the maintenance costs and the “humanfactors” risks during an inspection.

1 Introduction

This section presents the concept of aero-nautical maintenance, followed by details ofNon-Destructive Testing and Structural HealthMonitoring. The section finishes with a briefintroduction on Augmented Reality.

1.1 Maintenance

According to Blanchard [1], “maintenance in-cludes all actions necessary for retaining a sys-tem or product in, or restoring it to, a desired op-erational state”. The basic maintenance types areshown below:

• Corrective maintenance - This kind ofmaintenance is used after the item breaksdown or presents malfunction. It includesall unscheduled maintenance actions. Thecorrective maintenance cycles includefailure identification, verification, localiza-tion, fault isolation, disassembly to gainaccess to the faulty item, item removal,item replacement with a spare or repair inplace, reassembly, checkout, and conditionverification.

• Preventive maintenance - This kind ofmaintenance is used to prevent failures,safety violations, malfunction, or unnec-essary production costs and losses of theitem. It includes all scheduled mainte-nance actions performed to retain a system

1

G.O.C. PRADO , ET AL.

or product in a specified operationalcondition. Scheduled maintenance coversperiodic inspections, condition monitor-ing, critical-item replacements (prior tofailure), periodic calibration, and the like.In addition, servicing requirements (e.g.fueling & and lubrication) may be includedunder scheduled maintenance.

• Predictive maintenance - This kind ofmaintenance constantly monitors theitem to determine the exact status of theitem, predicting possible degradation. Toestablish requirements in this area, it isnecessary to know how various systemcomponents fail (i.e. physics of failure)and to have available the use of such testmethods as vibration signature analysis,thermography, and tribology. The ob-jective is to predict when failures willoccur and to take preventive measuresaccordingly.

Maintenance levels pertain to the division offunctions and tasks for each area where mainte-nance is performed. There are 3 maintenance lev-els, as follows [2]:

• Organizational level maintenance -generally performed by line mainte-nance personnel and normally used onscheduled inspections/servicing andquick-maintenance actions on the aircraftdiscrepancies identified during or betweenaircraft flights (i.e. unscheduled mainte-nance).

• Intermediate level maintenance - involvesmore extensive maintenance work such asmajor inspections and component repairand overhaul (on-wing maintenance).

• Depot or factory level - involves extensivemodification or overhaul to major aircraft

components. The component is removedfrom the aircraft and the maintenanceis performed in a workshop (off-wingmaintenance).

1.1.1 Non-Destructive Testing

The American Society for Nondestructive Test-ing (ASTM) [3] definition for NDT is “theprocess of inspecting, testing, or evaluatingmaterials, components, or assemblies for dis-continuities, or differences in characteristics,without destroying the serviceability of the partor system. In other words, when the inspectionor test is completed, the part can still be used”.

According to ATA MSG-3 [4], inspection isdivided in:

• General Visual Inspection (GVI) - “a vi-sual examination of an interior or exteriorarea, installation, or assembly to detectobvious damage, failure, or irregularity.This level of inspection is made fromwithin touching distance, unless otherwisespecified. A mirror may be necessaryto enhance visual access to all exposedsurfaces in the inspection area. This levelof inspection is made under normallyavailable lighting conditions such asdaylight, hangar lighting, flashlight, ordrop-light, and may require removal oropening of access panels or doors. Stands,ladders, or platforms may be required togain proximity to the area being checked.Basic cleaning may be required to ensureappropriate visibility”.

• Detailed Visual Inspection (DET) - “anintensive examination of a specific item,installation, or assembly to detect dam-age, failure, or irregularity. This couldinclude tactile assessment in which acomponent or assembly can be checkedfor tightness/security. Available lightingis normally supplemented with a direct

2

SHM WITH AUGMENTED REALITY FOR AIRCRAFT MAINTENANCE

source of good lighting at an intensitydeemed appropriate. Inspection aids suchas mirrors and magnifying lenses may benecessary. Surface cleaning and elaborateaccess procedures may be required”.

• Special Detailed Inspection (SDI) -“an examination of a specific item, in-stallation, or assembly, making use ofspecialized inspection techniques such asNon-Destructive Testing and/or equipment(e.g. boroscope, videoscope, tap test) todetect damage, failure, or irregularity.Intricate cleaning and substantial access ordisassembly procedure may be required”.

GVI and DET are classified as visual inspec-tion. Approximately 80% of all NDT proceduresare accomplished by using visual inspectionmethods [5].

1.1.2 Structural Health Monitoring

For Embraer, the SHM means a monitoring andmanagement technique that uses calculationmethods and on-board sensors to assure the air-craft continued airworthiness from the structuralviewpoint, aiming at the improvement of design,operation, and maintenance [6].

For years, Embraer has been studying differ-ent SHM technologies with possible applicationscenarios for structural damage detection.

There are some SHM damage monitoringtechnologies under evaluation by Embraer’sR&D department, such as Electro-MechanicalImpedance, Acoustic Emission, Lamb Waves(LW), Fiber Bragg Gratings, Comparative Vac-uum Monitoring (CVM), etc.

The SHM application scenarios investigatedby Embraer are related to the type of structuremonitored.

Metallic structures can be monitored by SHMtechnologies in order to enable the detectionof damage such as cracks (including size andlocation), crack growth, corrosion (includinglocation and severity), and accidental damage(including location and intensity).

Composite structures can also be moni-tored by SHM technologies in order to enabledetection of delamination (including size and lo-cation), debonding (including size and location),and accidental damage (including location andintensity).

The ATA MSG-3 [4] document has alreadybeen updated in 2009 to allow initial use ofScheduled SHM (S-SHM). This inclusiondemonstrates the maturation of SHM tech-nologies as promising ones to improve aircraftstructural maintenance [7].

It was recognized that SHM can be imple-mented in two operation modes:

• Scheduled SHM (S-SHM): The act touse/run/read out an SHM device at an in-terval set at a fixed schedule [4].

• Automated SHM (A-SHM): that relies onthe SHM system to inform maintenancepersonnel that action must take place [8].

In terms of airplane design, SHM offerspotential benefits such as structural efficiencyimprovements and weight savings.

In order to incorporate SHM technologiesinto aeronautical structures, some SHM issuesstill must be considered for system reliability(for the entire life of the aircraft), durability, andqualification, in accordance with the applicableaviation regulations.

After demonstrating strong results on groundtests and in the flight test aircraft, Embraer de-cided to develop a project using the S-SHM ap-proach for the qualification of CVM and LW

3

G.O.C. PRADO , ET AL.

technologies and to validate the performance ofsuch systems in real-life operational environ-ment. This project included laboratory tests forthe assessment of detection capabilities and testswith systems installed on a number of operator’saircraft to check operational behavior [9].

1.2 Augmented Reality

AR is the combination of the physical real worldand the virtual world in real time with the userinteraction. The virtual information is spatiallyrendered in the physical world, usually using atablet, smartphone, or augmented-reality glasses.

The AR-process basic flow starts in the videocapture by a camera. This video is split in framesin which each frame/image is processed so that aposition is identified, normally through a marker.Once this marker is detected, the camera positionis calculated considering its inherent parameters.After this position is found, the virtual objectis then rendered in the same frame/image byapplying perspective angles, translations, androtations. As shown in the Fig. 1.

The fiducial marker is one of the most-usedtechniques to position a virtual object in thephysical world. It is a bidimensional image thatis added to the scene so that it can be capturedby the camera and then, when it is processed, itcan identify its position.

There are also initiatives for the use of ARwithout markers. In this case, the application iscapable of getting the position of the physicalreal world. One example of this technique isthe recognition of the edges of an object basedon its previously-known geometry. The matchof these edges makes the object itself be a marker.

Many advances in the technology havemade feasible the use of AR in the aeronauticalindustry. The 3D models created in the aircraftdesign can be reused in this context, allowingnew possibilities of use in aircraft manufacture,training, and maintenance. Combined with the

use of mobile devices, it is possible to expand theaccessibility and make it easier the presentationof virtual information in the physical world.

Fig. 1 Augmented-Reality-process basic flowwith fiducial marker

2 Structural Health Monitoring With Aug-mented Reality For Aircraft Maintenance

Scheduled maintenance and inspection activitiescan take advantage of the SHM technologies byevaluating the structural integrity of an aircraftand performing less time consuming procedures,compared to the current NDT technologies.

The SHM technologies cannot only reducethe amount of time and burden of those activities,but can also early detect, without great effort,the possibility of finding structural damage in re-stricted access areas, reducing maintenance costsdue to less time-consuming and less complexmaintenance procedures.

Another important advantage is the possibil-ity of minimizing the effects of “human factors”during an inspection, because it is an automateddamage detection application and the automated

4

SHM WITH AUGMENTED REALITY FOR AIRCRAFT MAINTENANCE

data analysis has the potential of reducing humanerrors induced by fatigue and repetitive tasks.[10].

Fig. 2 shows an example of a maintenanceinspection task and it is possible to understandthe complexity and time required to perform theactivity.

Fig. 2 Example of traditional disassembly to per-form a maintenance procedure with NDT

Fig. 3 shows an example of an inspectiontask with SHM technology. During the in-stallation of the SHM on the aircraft, the dataacquisition points are left in an easily-accessiblelocation for the inspector. In this example, itis necessary to remove only a ceiling panel toperform the structural inspection with SHM.

It is important to note that SHM can preventdisassembly processes, which may eventuallycause undue damage to the structure.

Fig. 3 Example of an easy disassembly to per-form a maintenance procedure with SHM

In the future, SHM will be capable of pro-viding means for the replacement of the currenttime-based maintenance practices by ConditionBased Maintenance (CBM) [11].

2.1 Structural Health Monitoring enhancedby Augmented Reality

Embraer is always focused on exploring ways toreduce maintenance costs and increase efficiencyof its services. The use of AR technology is di-rectly connected with this Embraer’s philosophy.

According to this view, Embraer has agranted patent on the use of Structural HealthMonitoring system with the identification of thedamage through a device based on AugmentedReality technology (US20170322119 A1)[12].

The augmented reality combined withStructural Health Monitoring sensors allows themaintenance technician to perform maintenancetasks much quicker and in a more reliable way.

This technology enables the inspector toknow your position in relation to the structurethat needs to be checked and, with this ref-erence, the SHM data is shared with the ARdevice. Then, the technician has the real-timeinformation about the structure without anynecessary components disassembly, allowingmore maintenance tasks to be performed in thattime period.

5

G.O.C. PRADO , ET AL.

Fig. 4 Example of a non-need disassemblyto perform a maintenance procedure with SHMwith AR

Fig. 4 shows an example of how the in-formation can be quickly reached through areal-time connection between the data acquiredby SHM technology installed in the aircraft andthe augmented reality device, without the needof disassembling the aircraft.

Fig. 5 shows a simplified proposal of asystem using SHM enhanced by AR that can alsogenerate a maintenance report with the aircraftstructural condition, to facilitate the maintenancetechnician work in registering the inspectiondata after the end of the inspection activity. Thisreport can also support the engineering group togive a fast resolution of the detected damage andreturn the aircraft to an airworthiness condition.

3 Conclusion

The combination of the Structural HealthMonitoring with Augmented Reality tech-nologies gives the opportunity to decrease themaintenance-activities time in opening andclosure of accesses, disassembly and assemblyof components, inspection activities, elaborationof reports, register of maintenance activities, etc.,and the possibility to decrease the introductionof the damage or wear in the components, due tothese activities.

Fig. 5 Simplified proposal of a system usingSHM enhanced by AR

Therefore, the combination of these tech-nologies can increase the aircraft availability toEmbraer’s Customers in the future.

Embraer’s team has developed initial studiesto present SHM data results with an AR deviceto the maintenance technicians. In addition toassisting in the accomplishment of maintenanceactivities at a reduced cost, the main idea of thesestudies is to share the knowledge of the technol-ogy for use in several areas of the company.

4 Contact Author

For further information please contactgabriel.cruz@embraer.com.br.

References

[1] Blanchard, B., D. Verma, and E. Peterson.Maintainability: A Key to Effective Serviceabil-ity and Maintenance Management, 1995.

[2] National Aeronautics and Space Admin-

6

SHM WITH AUGMENTED REALITY FOR AIRCRAFT MAINTENANCE

istration. Aircraft Maintenance. Available at:https://www.nasa.gov/../aircraft_maintenance.html[Accessed on Jun, 15, 2018].

[3] The American Society of Non-destructive Testing. Available at:https://www.asnt.org/../AboutASNT/Intro-to-NDT[Accessed on Jun, 15, 2018].

[4] Airlines for America. ATA MSG-3 Revision2015.1: Airline and Operator Maintenance Pro-gram, Washington, USA, 2015.

[5] Federal Aviation Administration. Acceptablemethods, techniques, and practices âAS aircraftinspection and repair. Oklahoma, USA, 1998.

[6] Rulli, R. P., and Silva, P. A. Embraer Perspec-tive for Maintenance Plan Improvements by Us-ing SHM. 3rd Asia-Pacific Workshop on SHM,2010.

[7] L.G. dos Santos, Embraer perspective on the in-troduction of SHM into current and future com-mercial aviation programs, in 8th InternationalWorkshop on SHM, 2011.

[8] SAE International. ARP-6461 - Guidelines forImplementation of Structural Health Monitor-ing on Fixed Wing Aircraft. 2013.

[9] R. P. Rulli, F. Dotta, P. A. da Silva, L. C. Vieira,A. K. F. Tamba, G. O. C. Prado, D. Roach, T.Rice. Structural Health Monitoring Qualifica-tion Process and the Future of Aircraft Main-tenance. Brazil, 31st Congress of ICAS 2018.

[10] Schmidt, H.-J., et al., 2004. Application ofStructural Health Monitoring to Improve Ef-ficiency of Aircraft Structure. 2nd EuropeanWorkshop on SHM, July 2004, Munich, Ger-many.

[11] Roach, D., 2007. Smart Aircraft Structures: AFuture Necessity. High Performance Compos-ites.

[12] Silva, P. A. et al. Structural health monitor-ing system with the identification of the dam-age through a device based in augmented realitytechnology. U.S. Patent US20170322119 A1, 9nov. 2017.

Copyright Statement

The authors confirm that they, and/or their companyor organization, hold copyright on all of the origi-nal material included in this paper. The authors alsoconfirm that they have obtained permission, from thecopyright holder of any third party material includedin this paper, to publish it as part of their paper. Theauthors confirm that they give permission, or have ob-tained permission from the copyright holder of thispaper, for the publication and distribution of this pa-per as part of the ICAS proceedings or as individualoff-prints from the proceedings.

7