injecting faults for error evaluation

Injecting Faults for Error Evaluation

NASA Glenn Research CenterKalynnda Berens, SAICRichard Plastow, SAIC

SAS 2004 - Fault Injection 2

Mission Success Starts With Safety

Introduction Applications often consists of software components

plus custom development, merged into a coherent package. COTS, GOTS, open source, etc.

Source code is usually not available for review of quality and reliability. Visibility into the component is only what’s available via a

public interface What is the quality of that component? What faults lay inside the component?

Applications interface with hardware and other software and can be influenced by failures in those systems.



Fault Injection on Interfaces Interfaces (hardware, software, human) are a

major source of errors and induced faults Software and system testing looks at anticipated

off-nominal situations, but often misses unusual situations or combinations of faults

Mishap investigation has shown that multiple faults or unexpected anomalies are key players in accidents and mission failures



Example System

ApplicationCOTSLibrary

COTS Operating System

Other Applications

on same system

External Systems

System Hardware

Input Sensors

Control Outputs



Fault Injection Flow Diagram

No

Identify Interfaces and Critical Sections

Error/Fault Research

Estimate Effort Required

Obtain Source Code and Documentation

Start

Sufficient time and funds?

Importance Analysis

Select Subset

Test Case Generation

Fault Injection Testing

Document Results, Metrics, Lessons Learned

Feedback to FCF Project

End

Yes

Results



Interface Identification Artifacts and Documentation

Software and System Requirements and Design specifications

Interface SpecificationsUser and Training ManualsHardware DocumentationOther project documentation

For FCF, “Signals List” Source code



Error Research Sources of Error/Fault Information

Vendor documentationPublic bug list Internet SourcesSoftware logsError databasesProject ExperiencePrevious Test ResultsPersonnel Experience



Estimation of Effort Determine level of effort, funding, time

constraints If complete effort not possible

Perform importance analysis of interfaces, software units

Safety Complexity Use by other system elements Expected number or types of faults

Prioritize and select by importance



Testing Test case generation based on identified errors plus

permutations on possible input values Consider multiple faults Consider faults while system is off-nominal from a

previous fault Consider effects of system load/stress Consider state-specific effects Instrument software to observe effects of injected

faults External or observable effects State changes (or lack of) Effects on safety-critical functions



Results: First Project: Tempest

Written in Java 1.1 Configurable Cross platform operability Implements HTTP GET and HEAD Request and

Server Side Includes Has some Basic Security Features Debug Mode monitoring Commercially available



Tempest Critical Errors Inappropriate system operation with modified

configuration file Non-compliance with HTTP standard System crash with invalid port numbers

Port 49151.45 -> opened port 80 File access in server machine outside of

authorized directories System did not operate as per user

documentation



Results: Fluids and Combustion Facility Permanent, multi-user facility for ISS microgravity

experiments Fluids Integrated Rack (FIR) and Combustion Integrated

Rack (CIR) Operates for 10 years, so robustness important CANbus processors selected for fault injection

Health and Status Monitoring Cannot be upgraded in flight Mature requirements, design, and interface definition Source code available



CANbus Processors Air Thermal Control Unit (ATCU) Color Camera Package (CCP) FOMA Control Unit (FCU) FSAP Diagnostic Board Input/Output Processor (IOP) IPSU Diagnostic Board* Mass Data Storage Unit (MDSU)* Nd:YAG Laser Package* Water Thermal Control System (WTCS) White Light Package

* Not yet tested



FIR System Diagram

IOP Main Processor IOP HRDL

Processor

IOP Video Switch

Processor

IOP CAN Node

Processor

Input-Output Processor (IOP)

FSAP Main Processor

FSAP CAN Node

Processor

FSAP

IPSU Main Processor

ISPU CAN Node

Processor

Common IPSULaser Diode

CAN Processor

White Light CAN

Processor

DCMCAN

Processor

Nd:Yag CAN Processor

PI Package

ECS CANbus

ATCU CAN Processor

WTCS CAN Processor

Optics Bench CANbus

Ethernet

MDSU Main Processor

MDSU CAN Node

Processor

MDSU



CANbus Processor State Diagram

Off-Nominal (O-N)

Power Down (P)

Power On

Initialization

Power Off

Operational (OP)

Power Down Cmd

Error

Success

Operational Cmd

Error

Error

Operational Cmd

Power Down Cmd



Testing Software



Test Setup



FCF Fault Injection Process Interface Identification and prioritization Obtain hardware, source code for testing

environment Error/Fault search on selected interfaces and

components Static analysis using Understand™ tool Analysis of previous testing, defects Test case generation, source code

instrumentation, and test execution



Types of faults injected Out-of-range Unexpected input Multiple errors Timing Flood the input with values Remove Input/Output Interrupt Input/Output



FCF Results Software previously qualified 35 errors, 3 critical

Loss of the output connection caused a continuous reboot

Changing the processor address caused a hang condition

Going to the input limits caused invalid telemetry to be sent.

Project corrected 20 errors 4 errors still in process

Testing still in progress



Final Steps In-depth case study (ISS flight payload)

Update Fault Injection Methodology documentRecord all the details – problems as well as

successesCompare results to other defect detection

mechanismsWritten for those who want to try the technique

Release FI Methodology and Case StudyDecember, 2004



Passing the torch Potential applications

Any software project using COTS software or with hardware interfaces

Data and Case StudiesFault Injection Methodology (draft)

Available through SARPCase Study (FCU main processor)

Available December, 2004

injecting faults for error evaluation

Documents