dependable software verification
TRANSCRIPT
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Software Verification
22nd May 2012
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Agenda
When Things Go Wrong...
Certifying Software
Safety Critical Systems
ISVV Demonstration
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
When Things Go Wrong...
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
4
Therac 25
Radiation treatment machine for Cancer
Was a new unit with greater dependence on software
Six massive overdoses between 1985 and 1987
Three deaths directly due to radiation poisoning
Root Causes
Code not independently reviewed
Failure modes not considered
Failure codes not explained
No system integration testing
Complaints not heeded
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Ariane 5 Flight 501
Software code for the Ariane 4 rocket is reused
in the Ariane 5
Ariane 5's faster engines trigger a bug in an
arithmetic routine inside the rocket's flight
computer. The error is in the code that converts
a 64-bit floating-point number to a 16-bit signed
integer.
The faster engines cause the 64-bit numbers to
be larger in the Ariane 5 than in the Ariane 4,
triggering an overflow condition that results in
the flight computer crashing.
Result?
5
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Ariane 5 Flight 501
Fireworks!
6
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Lauda Air Flight 004
Flight crashed into the jungles of Thailand after the
No. 1 thrust reverser inadvertently deployed while the
aircraft was at 31,000 feet.
All 223 people aboard were killed.
7
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Blue Screen of Death
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Blue Screen of Death
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Certifying Software
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
11
Why certify software?
Today’s software:
Key component in complex systems
Increasing role in reliable systems
Impossible to fully test
Therefore:
We need to have confidence in the software
Define and analyse all requirements
Develop in a controlled and robust manner
Comprehensively test against the requirements
The whole process needs to be evidenced
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
12
Why is this important?
A steady drive to:
Higher levels of automation
Ever complex systems
Traditional mechanical systems being replaced by software
controlled systems
Provide confidence to deliver expected functionality
safely
Meet stakeholder demands (public, environment, business,
economic)
Long product lifetimes
Very expensive if it goes wrong
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
The future
Further integration of complex systems
Higher levels of automation
Higher level of functionality
Higher feature content
13
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
14
Key Safety Standards
EN ISO 61508
Safety Integrity Level (SIL)
SIL 1 to SIL4
Derivatives (Domain Specific)
EN ISO 50128 - Railway
EN ISO 26262 – Automotive
OLF 070 – Oil & Gas
DO-178B/C & DO-254
Level A to Level E
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Key Standards:
DO-178B / DO-254
Software Considerations in Airborne Systems and Equipment
Certification / Design Assurance Guidance For Airborne
Electronic Hardware
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
DO-178B and DO-254
DO-178B
Developed 1980 – 1992
Many compromises to satisfy goals
Able to accommodate different development approaches
Objective based
Widely adopted
DO-178C released January 2012
DO-254
Developed 1996 – 2000
Hardware focussed but very similar to DO-178B for software
Covers all electronic hardware 16
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Why DO-178B & DO-254?
Mature and robust
Not avionics specific and so easily transferred for use
in other industries
Proven track record
Not a single airline accident caused by software since
standards were introduced
Used as a benchmark in other industries (e.g. Space)
17
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
DO-178B/DO-254 in Context
18
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
DO-178B Key Features
Detailed Planning
Five Criticality Levels ( A to E )
Consistency and Determinism
Traceability
Independence
Proven Tools (“Qualification”)
66 Objectives
Up to 20 Artefacts
19
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Criticality Levels
A. Resulting in Catastrophic Failure
B. Resulting in Hazardous/Severe-Major Failure
C. Resulting in Major Failure
D. Resulting in Minor Failure
E. No Effect
20
Level A
≤1 X 10-9
Level B
≤1 X 10-7
Level C
≤1 X 10-5
Level D
>1 X 10-5
Level E
NA
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
DO-178B
Objective based standard
21
DO-178B Assurance level
Number of Objectives
Independence Cost +
Level A 66 25 40%
Level B 65 14 35%
Level C 57 2 25%
Level D 28 2 10%
Level E 0 0 0%
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Five Key Plans
PSAC: Plan for Software Aspects of Certification
SQAP: Software Quality Assurance Plan
SCMP: Software Configuration Management Plan
SDP: Software Development Plan
SVP: Software Verification Plan
22
PSAC SQAP SCMP SDP SVP
3. Correctness Process
1. Planning
Process
2. Development
Process
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Additional Documents
Software Requirements Standard
Software Design Standard
Software Coding Standard
Software Configuration Index (SCI)
Software Accomplishment Summary (SAS)
Software Traceability Matrix (STM)
Requirements, Design, Code and Test Results
Tool Qualification Plan
CM Records and PR
QA & Design Authority audit records
Checklists
23
3. Correctness Process
1. Planning
Process
2. Development
Process
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
V&V Methods
Code Inspection
Unit Testing
Integration Testing
Final Integration Testing
24
3. Correctness Process
1. Planning
Process
2. Development
Process
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Safety Critical Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Get it Right!
26
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
V Model
27
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
Verification and Validation
28
Document
Plan for each defined life-cycle phase
Methods for doing and recording
Acceptance criteria
Tests
For functionality and performance
To challenge (stress) the system
Validate system to
Requirements specifications
Intended uses
Customer needs
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
29
Reviews
Design reviews
Walkthroughs
Prototyping
Traceability matrices
Requirements specifications
Mitigation of identified hazards
•Testing
• Code Scrutiny
• Module Test
• Integration Test
Verification Activities
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
30
Testing
Boundary Testing
Robustness Testing
Stress Testing
Discontinuity Testing
Statement Coverage
Branch Coverage
MC/DC Coverage
Unit Testing
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
31
White Box
Access to all variables including local variables
Grey Box
Access to global variables, input/output
parameters and functions called.
Black Box
Only access to global variables and input/output
parameters.
Type of Testing
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
32
If( A & B )
{
Do this....
}
Else
{
Do that....
}
Coverage
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
33
If( A & B )
{
Do this....
}
Else
{
Do that....
}
Coverage - Statement
Test only needs to
cover this branch or
the next branch to
achieve 100%
statement coverage.
1 test case
A & B = TRUE
OR
A & B = FALSE
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
34
If( A & B )
{
Do this....
}
Else
{
Do that....
}
Coverage - Branch
Test needs to cover
this branch and the
next branch to
achieve 100% branch
coverage. Therefore
conditions must be
TRUE and FALSE.
2 test cases
A & B = TRUE
A & B = FALSE
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
35
If( A & B )
{
Do this....
}
Else
{
Do that....
}
Coverage – MC/DC
Test needs to verify that
both A and B affect the
condition.
At least 3 test cases
A = TRUE, B = TRUE
A = FALSE, B = TRUE
A = TRUE, B = FALSE
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
36
Examples of Validation Functional and safety tests
User acceptance tests
Installation and checkout tests
IMPORTANT Validation is performed before delivery to the customer
Validation is performed by personnel that are
independent from the design team
Validation
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
37
Sources of Faults
Requirements
Errors in Conversion
Design
Incorrect algorithms and interfaces
Coding
Syntax errors, incorrect signs, endless loops
Timing
Missed deadlines
Dependable
Technologies
For Critical
Systems
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
ISVV Demonstration
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
ISVV
39
Conclusion
Technical Specification Verification
Software Design Verification
Source Code Verification
Validation Tests Specification
Validation Procedures
Implementation
Test Execution and Results
Analysis
System Concept System in Operation
ISVV introduces
a small added cost...
...and brings a
high added value
@ 2
011 C
ritical
Soft
ware
Technolo
gie
s L
td S
outh
am
pto
n
40
Contacts
Russell Jugg
Ricardo Silva
www.critical-software.co.uk