reverse engineering - microway systems · everything is reverse engineered from something similar...

Copyright © 2003 by Sunrise Certification & Consulting, Inc. All Rights ReservedCopyright © 2008 by Sunrise Certification & Consulting, Inc. All Rights Reserved

Reverse Engineering

How does it fit with RTCA/DO-178B?(with a bit of RTCA/DO-254)

2Copyright © 2008 by Sunrise Certification & Consulting, Inc. All Rights Reserved

What is “Reverse Engineering”?

“Reverse Engineering is the process of taking a finished product and reconstructing design data in a format from which new parts or molds can be produced.”

- The Society of Manufacturing Engineers (SME)

“… the process of duplicating an item functionally and dimensionally by physically examining and measuring existing parts to develop the technical data (physical and material characteristics) for competitive procurement.”

– Military Handbook MIL-HDBK-115

“Reverse engineering - The method of extracting software design information from the source code.”

- RTCA/DO-178B Glossary

“Reverse engineering is the process of design recovery.”- Roger Pressman, Software Engineering: A Practitioner’s Approach

“Reverse engineering - Re-implementation of a hardware item by study of its construction, function and performance within a particular environment.”

- RTCA/DO-254 Glossary


Reverse Engineering – RTCA/DO-178B

• Bottom line -- we have a model of some kind that works but we have insufficient data available to reproduce that model

• Apparently reverse engineering can provide a structured means of creating the necessary data to allow creation of production quality devices

• Basic guidance is provided in section 12.1.4(d) of RTCA/DO-178B…

Reverse engineering may be used to regenerate software life cycle data that is inadequate or missing in satisfying the objectives of this document. In addition to producing the software product, additional activities may need to be performed tosatisfy the software verification process objectives.

Reverse engineering may be used to regenerate hardware life cycle data that isdeficient or missing to satisfy the design assurance objectives of this document (DO-254)


Reverse Engineering – RTCA/DO-248B and CAST

• Additional information is provided in section 4.5.4.3 RTCA/DO-248B but it is of dubious value (presenter’s opinion)…

• In addition to the RTCA documents there is CAST paper, CAST 18, entitled “Reverse Engineering in Certification Projects”

From the presenter’s perspective, the CAST paper is much closer toaddressing the Reverse Engineering issues for demonstrating RTCA/DO-178B compliance than RTCA/DO-248B (and more useful)


Why “Reverse Engineer”?

• Generically if we want to reverse engineer we must have a previously developed* something…

• A typical perspective is that if we have something that is previously developed, reuse of that something must be a bargain compared to new development

• Follow the money…

• And there is some logic to the argument - at some point nearly everything is reverse engineered from something similar or a prototype model

• If there was not a perception more money could be made through reverse engineering as opposed to forward path development, we would not try to employ reverse engineering

* Reverse engineering falls under the global umbrella of PDS in the current regulatory lexicon


Likely RTCA/DO-178B Reverse Engineering Suspects

• Commercial off-the-shelf packages– Operating systems– Libraries– Frameworks

• Military to civil transition packages– Anything and everything that has a civil counterpart…

• Fuel gauging equipment• Inertial reference systems• Engine controllers• Flight control components• Etc, etc, etc

• Automotive or marine transition packages– CAN bus drivers– Reciprocal engine FADECs– And so on

• “Research projects” (even some initially commissioned/sanctioned by the FAA)

• Incomplete or unfinished projects which started as “forward path” developments

• Updates to obsolete product lines (really covered elsewhere in RTCA/DO-178B)

It is likely any function one canthink of which is applicable to aircraft has attempted to gain regulatory approval via reverse engineering!

All share the common thread of “saving time” and/or “saving money”.


Reverse Engineering – Some typical issues

• Tight focus on the production life cycle data items vs. systematic integral activities (CM, quality assurance, certification liaisons, verification) – After all, the existing product “works” just fine…

• Lack of domain expertise – We acquired the technology but not the intellectual capital…

• A variation of the second bullet: We cannot afford to do the reverse engineering in-house so we will outsource it – After all it is just generation of missing life cycle data

• Inadequate planning with regard to an appropriate life cycle or even the effort that will be required – I don’t need no stinkin’ plans to do reverse engineering!

• The customer liked the prototype therefore the design (read source code) is frozen and any deficiencies can be addressed through a Problem Report (PR) – the code must not change!

• Safety issues or hazards are always identified in the SSA for a specific airframe but those safety issues may not have been considered as part of the reverse engineering or adaptation process – The customer liked the prototype, don’t change the code!

• Linkages to aircraft level requirements are not considered – We know best!


Reverse Engineering – Some possible opportunities

• Long and successful service history of baseline product

• Significant aircraft level experience or solid partnerships with aircraft OEMs

• Significant domain expertise (e.g., military to civil applications)

• Conscientious commitment to satisfying the intent of the regulatory guidance (more than just RTCA/DO-178B and life cycle data)

• Sincere effort to design and document appropriate life cycle processes to satisfy the objectives of RTCA/DO-178B

• Conscious intent to “idiot and/or exception proof” your design even if it means code changes – this is particularly germane for COTS operating systems, libraries, frameworks, and so on

– Well-defined APIs and data dictionaries– Documented pre- and post- conditions as well as invariants– Runtime validation of interfaces where possible, and so on

• Willingness to make the design (and implementation) more robust as verification activities identify holes, chinks and deficiencies


Successful Reverse Engineering

Do you want to develop a (sub)system that is demonstrably compliant to regulatory objectives and customer needs or do you want to fill “ticky-boxes” as inexpensively as possible?


Reverse Engineering – Where do we start?

• Place the existing data, no matter its state, under firm change control processes – this will be the initial baseline for all further activities

• Perform an honest gap analysis of the existing design using a small team of very experienced DO-178B team members with domain expertise

– Consider the design assurance level that will be required for the end product

– Evaluate processes used to develop the existing design if it is available (that process definition may or may not be)

– Evaluate the existing life cycle data and potential or actual driving requirements with particular attention paid to safety specific issues


Where do we start continued…

• Performing the honest gap analysis…– Utilize the software job aid or similar mnemonic to ensure all

appropriate objectives have been evaluated – identify both compliant areas and especially non-compliant areas

– Consult the appropriate PSSA, SSA, or worst case, FHA to determine hazards which may need to be addressed or mitigated

– If there is no safety information available, develop a product level PSSA to capture potential safety issues or hazards the design might need to address

– If available examine aircraft or higher level system requirements to determine where gaps might exist – concentrate on robustness and safety critical issues

– If aircraft or higher level system requirements data is not available, consider TSO data, RTCA MOPS data or other industry standards –worst case consider the effort of developing this level of requirements


Reverse Engineering – After the gap analysis

• Document the results of the gap analysis

• From both a business perspective and compliance perspective, develop a preliminary task breakdown

– The task breakdown will serve as the basis for multiple activities• From a business perspective preliminary cost estimates can be developed – they will

not be final but they should provide a read on whether the approach is financially feasible (follow the money)

• A compliance perspective can be developed that will serve as the basis of the PSAC – the perspective should provide a good snapshot in terms of the life cycle activities that will be required to satisfy the appropriate RTCA/DO-178B objectives

• Develop a white paper or set of slides that can be used to discuss the previous points with both the management team and the certification authorities

• Establish a meeting with the appropriate certification authorities and solicit their feedback – after their initial shock they will likely have some legitimate feedback!


Reverse Engineering – Go forward plan

Consider both management feedback and certification authority feedback – assuming there is still a desire to continue…


Reverse Engineering – Planning Documents

• Develop an outline for the PSAC, Development, and Verification plans

1. Whiteboard a life cycle process diagram before writing a single sentence in any planning document – do not skip this step

2. If you did not whiteboard a life cycle process diagram, see step 1

3. Make sure the life cycle process(es) address all of the gaps you have identified in the gap analyses, issues the management team may have noted, and issues the certification authorities may have noted in your earlier meetings

4. For each process (block) in the life cycle identify the following:a) Inputs (driving requirements)b) Processing (work to be performed)c) Outputs (data and evidence to be produced)d) Entry criteria (state of driving requirements)e) Exit criteria (state of this process)f) Associated verification activities (reviews, tests, analyses)g) Associated CM activities (baselining and change control)

This is a good approach even if you are not reverse engineering but sometimes it is very hard to convince anorganization to do this…


Reverse Engineering – Planning Documents Continued

5. Iterate the whiteboard exercise until it appears all gaps and objectives of Table A-1 of RTCA/DO-178B can satisfied

6. Allocate the life cycle processes to the appropriate document(s) (the PSAC will contain a summary of the full process) – some organizations have found it is very practical and expedient to combine the development and verification plans…

7. Evaluate the PSAC, Development, and Verification plans and ensure they are coherent, consistent, and will satisfy both the objectives of RTCA/DO-178B and the gaps identified earlier

• Create a CM plan that allows the defined processes to be controlled (most of this should already be in place piece-wise through earlier work)

• Develop an SQA Plan with sufficient rigor and teeth to ensure specifics of the Development, Verification, and CM plans will be adhered to

• Submit the PSAC and gain certification authority concurrence


Reverse Engineering – Planning Document Hints

• As you do the whiteboard exercise make sure you account for potential pitfalls…

– Missing safety constraints (FHA, PSSA, SSA) besides the potential for a missing hazards that could drive a criticality level change

– Missing aircraft or higher level system requirements– Missing implementation robustness in terms of error tolerance and recovery

(thereby reverse engineering less than robust requirements because robustness was not in the code to start with

– Availability of requirements that specify what (intent) versus how (implementation) – this is a bigger and bigger deal the higher (more abstract) you get in the requirements hierarchy

– Incomplete or missing traceability relationships (Do various requirement levels align?)

– Incomplete or non-existent verification activities– Insufficient domain expertise– Weak CM controls and processes on the starting baseline (whatever it is)– Outsourcing of labor for the reverse engineering effort (can be a really big

deal!)

• There are no “one size fits all” answers for these issues but they are important in terms of creating a system that operates nominally as opposed to a system that is safe (assuming a safe system is something other than a system that does not run…)


Reverse Engineering – We have plans!

• Assuming there is certification authority buy-in (remember the PSAC you submitted which was complete and addressed the objectives at a high level?)…

• It should be a simple process of turning the crank... Yeah sure, isn’t it always????

• Reverse engineering done well is every bit as tough, and perhaps harder than forward path engineering

• Plans must be adhered to, integral processes must be followed - that is simply a lot of work

• The best laid plans can have problems… Communicate with your certification authorities, communicate with your own team(s) and ensure rigorous process adherence to help ensure success

• At the end of the day there will still be one last path, top-down, which must be taken to ensure everything is in synch and fully verified from system level requirements to structural coverage analysis metrics and traceability

• End-to-end reverse engineering is not easy and cost savings may be debatable


Reverse Engineering – Wrapping up

• Assuming everything went to plan (and it generally doesn’t)…

• We need to finalize a SAS and SCI

• The SAS and SCI documents should not really be any different than what a traditional approach would yield– Were the plans followed and adhered to, and if not what

happened and how was the non-compliance mitigated?– What are the outstanding problem reports and what is their

impact?– What is the configuration of the life cycle data and tools

comprising the system?


Reverse Engineering - Notes

• Outsourcing issues complicate matters – the further outsourced (or multiple layers) the more difficult

• Good plans (and adherence to those plans) are critical to success

• Tools can help identify structure and relationships of elements but generally are useless for determining intent

• A data dictionary seems to be an uncommon document in DO-178B land and should not be – it can save a lot of work in both forward and reverse engineering approaches

• System safety issues (PSSA/SSA/FHA, safety cases) are critical but often not readily available in reverse engineering efforts – figure out up front how you will account for this

• Be prepared for code changes – your reverse engineering process may have to have a forward path defined to accommodate post-baseline and post-approval changes

• TSOs, military standards, industry standard (e.g., RTCA MOPS and SAE documents) can help in the creation of solid requirements (but be prepared for code changes)

• Higher design assurance levels are typically more difficult and more expensive (that should be a no-brainer)

• Relative size and complexity drive reverse engineering effort (another no-brainer)

• Organizational ability and discipline are key to being successful

• Planning processes and integral processes still required (see bullet 2)

• Top-down activities will eventually be required – there will be at least on top-down verification path

• Read the CAST paper

• Keep the certification authorities in the loop


Reverse Engineering – Discussion

This presentation provided a method that might be employed but short on specifics because every situation is different…

It is your time to question, pushback or otherwise enlighten the world

Copyright © 2003 by Sunrise Certification & Consulting, Inc. All Rights ReservedCopyright © 2008 by Sunrise Certification & Consulting, Inc. All Rights Reserved

Contact Info

Jeff Knickerbocker

[email protected]

Office – (620)229-8684Cell – (651)470-9855