isvv tutorial - web.fe.up.pt · isvv tutorial nuno silva, engineering manager, dependability &...

Dependable Technologies for Critical SystemsCopyright Critical Software S.A. 1998-2006 All Rights Reserved.

ISVV Tutorial

Nuno Silva,

Engineering Manager,

Dependability & Embedded Systems

Document Reference: CSW-2006-PRS-2495

© Copyright Critical Software S.A. 1998-2006 All Rights Reserved.

Tutorial outline

� Part I – Basic concepts

� Part II – The ISVV process

� Part III – Verification methods & tools

� Part IV – Validation methods & tools

� Part V – ISVV@CSW

� Part VI – Conclusion

� Part VII – Bibliography


Part IBasic concepts


Part I • Outline

� ISVV

� Independence

� Verification

� Validation

� ISVV motivation

� Brief history of ISVV

� Q&A


Part I • ISVV

Independent

Performed by an independent entity

Software

Applicable to the software

Verification &

Analysis of requirements, design, code, etc.

Validation

Definition and execution of validation tests


Part I • Independence

� Technical independence� ISVV personnel use their expertise to assess development processes and products independently of the developer.

� Managerial independence� ISVV effort shall be vested in a organisation separated from the organisation responsible for the system development.

� Independent selection of SW components to analyse.

� Independent selection of the IV&V techniques to use.

� Independent definition the schedule of IV&V activities.

� Financial independence� All work including quality assurance is directly founded by the project.


Part I • Verification

Are we building the product right?

� Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled. [ISO 8402]

� Requirements Verification

� Design Verification

� Code Verification

[ISO 8402:1994, IEEE 1012:1998, ECSS-P-001A:1997]


Part I • Verification (cont.)

Verification

Requirements

Design

Code

Test Specifications

Standards and guidelines

SW Verification Plan

Traceability Matrices

Safety & Dependability

Verification Reports


Part I • Validation

Are we building the right product?

� Confirmation by examination and provisions of objective evidence that the particular requirements for a specific intended use are fulfilled. [ISO 8402]

� Definition of validation test cases

� Implementation of validation test procedures

� Tests execution and results analysis

[ISO 8402:1994, IEEE 1012:1998, ECSS-P-001A:1997]


Part I • Validation (cont.)

Validation

Requirements

Design

Code

SW ValidationPlan

Test Results Reports

TestSpecifications

SVF Documentation


Part I • ISVV motivation

� Complexity and size of SW is growing

� SW role in critical systems is increasing

� Cost of failure is high for some systems

� Developer’s viewpoint is partial

� Dependability & safety analysis are not complete


Part I • Brief history of ISVV

ISVV derives from the application of IV&V to the software. IV&V was we know it today dates back to the early 1970s when the U.S. Army sponsored the first significant such IV&V program for the Safeguard Anti-Ballistic Missile System.

By the end of the 1970s IV&V was rapidly becoming popular. The increasing complexity, size and importance of the software lead to an increasing demand on IV&V.

Meanwhile IV&V gets consolidated and is now widely used by organisations such as the DoD, FAA, NASA and ESA. IV&V is mentioned in [ISO/IEC 12207] and formalised in [IEEE 1012].


Part I • Q&A

??


Part IIThe ISVV process


Part II • Outline

� ISVV process layout

� Independent Verification process� Technical Specification Analysis

� Design Analysis

� Code Analysis

� Independent Validation process� Test cases specification

� Test procedures implementation

� Test execution and results analysis


Part II • ISVV process layout (CSW SDP)

OPERATIONS &MAINTENANCE

PHASE

ACCEPTANCEPHASE

VALIDATIONPHASE

DESIGNENGINEERING

PHASESOFTWARE

REQUIREMENTSENGINEERING

PHASE

SYSTEMENGINEERING

PHASE

SRR PDR CDR QR AR

Independent Verification

Independent Validation

Starts at least when a work product is available

for verification

Tests Specification may be performed during requirements

phase

[CSW-SDP:2005]


Part II • ISVV process layout (schedule)

SystemEngineering Requirements

EngineeringDesign

EngineeringValidation Acceptance

Operations &Maintenance

IVE. Independent Verification

SYSTEM LIFE CYCLE

DEVELOPMENT LIFE CYCLE

SRR CDRPDR QR ARDDR

Tech. Spec. Analysis

Design Analysis Code Analysis

IVA. Independent Validation

Test Cases Definition

Test Cases Execution

[ECSS-E-40B:2003]

Test Procedures


Part II • Independent Verification process

Are we building the product right?

� Technical Specification Analysis

� Design Analysis

� Architectural Design Analysis

� Detailed Design Analysis

� Code Analysis


Part II • Independent Verification process

MajorSeverity

ConsistencyProblem type

Description: The DDD says that PROM checksum only refers to the used memory size and not to the full memory (as currently implemented).

Recommendation: Compute PROM checksum as defined in the DDD.

Problem description

ot_mem_s.c:201-250Problem location

ot_mem_s.cSource code references

PKG_INTERFACE::PKG_MEMORYUnit ID

[RD.09]Document Reference

Incorrect computation of the PROM checksumTitle

João EstevesAuthor

30.09.2004Date

14Item NºReviewItemDiscrepancy

RID.14 Incorrect computation of the PROM checksum

Software Source Code

Detailed Design Document

ISVV


Part II • Technical Specification Analysis

Technical Specification Analysis

System Requirements allocated to SW [RB;SRR]

Technical Requirements Specification [TS;PDR]

• External Consistency with HW/SW interfaces

• Internal Consistency• Correctness• Completeness• Testability• Feasibility• Readability

• External Consistency with System Requirements

Interface Control Documents [TS;PDR]

HW/SW Interface Requirements [RB;SRR]


Part II • Technical Specification Analysis

� Traceability Verification� With System Requirements allocated to SW

� With HW/SW Interface Requirements

� Technical Specification Verification� Internal consistency

� Correctness with respect to system requirements

� Completeness of the technical specification

� Readability of the technical specification document

� Feasibility of testing each requirement

� Feasibility of implementing each requirement

� Conformance with applicable standards and conventions


Part II • Architectural Design Analysis

SW Architectural Design Analysis

Technical Specification (PDR)

SW Architectural Design (PDR)

Interfaces Control Doc (PDR)

• External Consistency with interfaces

• Internal Consistency• Correctness• Completeness• Testability• Feasibility• Readability

• External Consistency with Technical Specification


Part II • Architectural Design Analysis

� Traceability Verification� With the Technical Specification� With the Interface Control Documents

� Design verification� Interfaces consistency between different SW components� Architectural design correctness with respect to Technical Spec.� Architectural design completeness� Dependability & safety of the design� Readability of the design document� Feasibility of testing� Feasibility of producing a Detailed Design� Architectural design conformance with applicable standards� Independently perform timing and sizing budgets analysis


Part II • Detailed Design Analysis

Detailed Design Analysis

Technical Specification (DDR)

Interfaces Control Doc (PDR)

Detailed Design (DDR)

Interfaces Control Doc (DDR)


SW Architectural Design (DDR)

• External Consistency with SW Architectural Design

• External Consistency with Technical Specification

• Internal Consistency• Correctness• Completeness • Accuracy• Testability• Feasibility• Readability


Part II • Detailed Design Analysis

� Traceability Verification� With the Technical Specification & ICDs� With the Architectural Design

� Design Verification� Interfaces consistency between different SW components� Detailed design correctness with respect to Technical Spec.� Detailed design completeness� Dependability & safety of the design� Readability of the design document� Feasibility of testing� Feasibility of coding� Detailed design conformance with applicable standards� Independently perform timing and sizing budgets analysis


Part II • Code Analysis

Code Analysis

Detailed Design Document (CDR)

Interfaces Control Doc (CDR)

SW Source Code (CDR)

• Internal Consistency• Correctness• Completeness• Accuracy• Testability• Feasibility• Readability

• External Consistency with detailed design


Technical Specification (DDR)

SW Architectural Design (CDR)

• External Consistency with SW Architectural Design

• External Consistency withTechnical Specification


Part II • Code Analysis

� Traceability Verification� With the Technical Specification & ICDs

� With the Architectural and Detailed Design

� Code Verification� Interfaces consistency between different SW units

� Source code correctness with respect to TS, AD and DD

� Source code readability, maintainability and conformance with the applicable standards

� Dependability & safety of the source code

� Accuracy of the source code

� Feasibility of testing

� Timing and sizing budgets of the software


Part II • Independent Validation process

Are we building the right product?

� Test cases specification

� Test procedures implementation

� Test execution and results analysis


Part II • Test cases specification

None.

Inter-case dependency

None.

Special procedural requirements

Other:None.

Software:None

Hardware:None.

Environment needs

a. Single slot overrun detectedSingle overrun error should be logged, a TM generated. The recovery mechanism is activated. The system is able to continue.

Outputs

a. TS interrupt with higher frequency:A single overrun can be forced by increasing the frequency of the TS interrupt (e.g., delivering two TS interrupts in less than 100 ms, say 25 ms).

Inputs

[16] Section 3.3.3 – Dispatcher

Test items

Test whether the CDMU dispatcher correctly detects single slot overrun.Purpose:

Critical Software S.A.Responsibility:

CS-TCS-CSW-DH-01-01aTest case ID:

TEST CASE SPECIFICATION

ISVV

Software Source Code

Architectural / Detailed Design

Software requirements

. . .


Part II • Test procedures implementation

SVF User Manuals and/or SVF Tools

Validation Test Cases

ISVV

Name ISVV_DH_01_01ATypeLib_NameAuthor VelkovSubsystem CDMUDescription Force a single slot overrun by generating a TS interruptPreconditionPostconditionCalling_ArgChange Log 02-03-2004,Velkov,Draft,Initial Version;

13-12-2004,Nuno Silva,1.1,Corrected compilation errors;

STEP_HEADER ("CS-TCS-CSW-DH-01-01a","1") Force a single slot overrun.COMMENT "Initiate service five, sub-service three TM wildcard."INIT_PACKET CS2_EVT20804_ACOMMENT "Inject a TS interrupt."SCOE_CMD CDMUS_SAS,"inject CPU forceirq 3;"COMMENT "Wait ten seconds for a service five, sub-service three TM.

(TM indicating SGM corruption not implemented.)"WAIT_FOR_PACKET CS2_EVT20804_A,10.0 [s],"print","",-1STEP_FOOTER


Part II • Test execution and results analysis

SVF Tools

Validation Test Procedures

ISVV

|vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv|**** Test campaign for Dispatcher, Slot Timer, and ****|**** System Start Up ****|vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv|>:- COMMENT:| >:- CVS information:|>:- COMMENT:| >:- $Id$|>:- COMMENT:| >:- 02 Mar 2004 00:00:00 Velkov Draft Initial Version|>:- COMMENT:| >:- 20 Mar 2004 11:10:22 ==> Proc_Lite to txt|----------------------------------------------------------------------------------------------------|---- STEP 01a ----|---- CS-TCS-CSW-DH-01-01a ----|----------------------------------------------------------------------------------------------------|>:- COMMENT:| >:- Initiate service five, subservice three TM wildcard.|INIT_PACKET: CS2_EVT20804_A|>:- COMMENT:| >:- Inject a TS interrupt.|SCOE_CMD:|WAIT_FOR_PACKET: CS2_EVT20804_A| Execution completion in max 10.00 secCST50012: 611(EVENT_ID) | CST50014: 11(EXPECTEDSLOTNUM) | CST50003: 12(ACTUAL_SLOT_NUM)DST50012: 620(EVENT_ID) | CST52002: 1(SGM CHECK RESULT)|****************************************************************************************************|***** Error summary of procedure : >>> SWC_DH_01_01A <<< *****|***** STEP 01a 1 errors *****|***** _______________ *****|***** ALL 1 errors *****|****************************************************************************************************

38%

25%7%

8%

5%

17% SUCCESS

DEBUG

FAILURE

PENDING

REDUNDANT

DISCARDED

None.SUCCESSCS-TCS-CSW-DH-01-03a

The expected Slot_Overrun_Detected haven't arrived. As in the previous test case, depending upon the time at which the extra interrupts are injected, different behaviours may occur. In fact four executions of this test case have produced three different results.

FAILURECS-TCS-CSW-DH-01-01b

Instead of a Recoverable_Slot_Overrun event, a Dispatcher_Sync_Fault was received. If the current slot number is different than the expected slot number, the current execution of the Handle_Slotprocedure shall be aborted (the procedure shall return). Anyway, this test shall be executed further times. Depending upon the time the extra interrupt is injected, different behaviours may occur.

FAILURECS-TCS-CSW-DH-01-01a

CommentsVerdictTest Case ID


Part II • Q&A

??


Part IIIVerification methods & tools


Part III • Outline

� Inspection

� Modelling

� Traceability Analysis

� Schedulability Analysis

� Static Code Analysis

� Fault/Failure Analysis (SFTA, FMECA, HSIA)

� Other methods

� Q&A


Part III • Inspection

� Purpose

� To assess one or more characteristics of an entity comparing the results with the requirements in order to check their conformity.

� How to

� By conducting a formal review meeting.

� Individual reviewers are assigned specific responsibilities.


Part III • Inspection

� Supporting tools

� The method is manual

� No specific SW tools exist for supporting it

� However, checklists can be used

� Applicability

� Requirements verification

� Design verification

� Code inspection


Part III • Modelling

� Purpose

� To facilitate the understanding of the system, to analyse interfaces consistency, to analyse data and control flows within the system.

� How to

� Building up a model of the system using a modelling tool or language.

� UML is a de facto standard but other languages may be used.


Part III • Modelling

� Supporting tools� Enterprise Architect

� Rational Rose

� STOOD / CP HOOD

� Telelogic TAU Architect

� Applicability� Interfaces analysis

� Design correctness evaluation

� Data and control flow analysis


Part III • Traceability Analysis

� Purpose

� To check external consistency between work products throughout the development life-cycle.

� How to

� Mapping the outputs of a phase to the inputs of that phase (backward traceability).

� Mapping the inputs of a phase to the outputs of that phase (forward traceability).


Part III • Traceability Analysis


� It depends on the work documents format

� A spreadsheet is always useful

� Applicability

� External consistency evaluation


Part III • Schedulability Analysis

� Purpose� To check whether a real-time system is schedulable or not, i.e. to check if all tasks can meet their deadlines.

� How to� Computing the worst case execution time (WCET) of each task (or use provided values).

� Exercising the system scheduling algorithm (e.g. RMA) using the computed WCETs.


Part III • Schedulability Analysis


� Absint AIT, Bound-T

� SCAN, Rapid RMA

� Applicability

� Schedulability verification


Part III • Static Code Analysis

� Purpose

� To evaluate the dependability and safety of the source code without executing it.

� To check whether the source code conforms with the applicable standards (e.g. MISRA C)

� How to

� Conducting inspections of the source code.

� Using static analysis tools


Part III • Static Code Analysis


� PolySpace (C, C++, Ada)

� Understand (C, C++, Ada)

� LDRA, PC-Lint, Splint, etc.

� Applicability

� Dependability and safety analysis

� Control and data flow analysis

� Coding standard conformance


Part III • Fault/Failure Analysis (SFTA)

� Purpose� To identify the probability of a top event to occur, its causes and the logical combination of these.

� How to� Start by identifying top level events and then iteratively identify the potential causes and they logical organization. The iterative process ends with the identification of the basic events.



Gate5

Door Close Failure

Q:0,00989076

Door Open

Door Open Error

Q:0,00121007

Gate6

Box Not at Level

Q:0,122344

Event10

Latch Failure

Q:0,00249688

Event11

Controller Failure

Q:0,00741239

Event13

Cable Slips

Q:0,0054849

Event12

Box Early or Late Stop

Q:0,117503

Gate5

Door Close Failure

Q:0,00989076

Door Open

Door Open Error

Q:0,00121007

Gate6

Box Not at Level

Q:0,122344

Event10

Latch Failure

Q:0,00249688

Event11

Controller Failure

Q:0,00741239

Event13

Cable Slips

Q:0,0054849

Event12

Box Early or Late Stop

Q:0,117503




� BlockSim FTI

� RAM-Tools FTA

� Relex Fault Tree/Event Tree

� item Fault Tree Module

� Applicability

� Dependability & Safety analysis

� Criticality analysis


Part III • Fault/Failure Analysis (SFMECA)

� Purpose� To identify failure modes effects propagation within a system and to assign a criticality level upon the end effects on the system.

� How to� Start by identifying high level failure modesfor each SW function and their potentialcauses.

� Identify the propagation of the failure effectsand assign a criticality level based on that.



# Component ID Function Failure Mode Failure Cause Local Effects End Effects Severity Classification

ST.0209 GCS.ST.TT Support the definition of training plans and the production of related training courses and documentation

Training plans support function is not performed or not performed on-time

Training plans support function is not executed,or is incorrectly setup,or the environment where it runs is incorrectly setup,or outputs are not produced,or outputs are not made available on time

A training plan cannot be executed

Trainning session cannot be executed.Staff cannot get initial training, refresher training or upgrade training.There are no operational on-line staff certified to operate GCS. Loss of operators for the GCS.

Major

ST.0210 GCS.ST.TT Support the definition of training plans and the production of related training courses and documentation

Training plans support function is wrongly performed

The Training plans support function generates the wrong outputs,or the inputs are not available,or the Training plans support function is incorrectly implemented

Execution of training plan with errors or incorrect training plan.

Trainees get incorrect or wrong training, which can lead to an incorrect TC or flight control procedure being executed. Wrong TCs and procedures are not validated by the CSIM chain. In a worst case nothing is validated and no control can be exercised over

Minor




� Relex FMEA/FMECA

� RAM-Tools FMEA/FMECA

� ReliaSoft Xfmea

� item FMECA

� Applicability


� Criticality analysis


Part III • Fault/Failure Analysis (HSIA)

� Purpose

� To analyse the reliability of the interfaces between hardware and software. To verify whether the SW correctly handles HW failures and that the SW is not harming the HW.

� How to

� Inspect each HW/SW interface with the guidance of a checklist of questions.



10. Does the HW/SW system remove any possibility of Single Point of Failure?

9. Is the failure detection and reaction executed within appropriate time limits?

8. Is it verified that software does not stress the hardware?

7. Is the recovery completely performed within the system?

6. Is the immediate corrective action not built on top of the function that failed?

5. Is there possibility of external intervention (especially when recovery/compensation fails)?

4. Are there fault tolerance characteristics in order to compensate the failure mode?

3. Does the software initiate a corrective action that safe-keeps the system and negate the effects of the failure?

2. Is failure mode information reported beyond the system?

1. Is sufficient and reliable information about the failure available?

HSIA Checklist




� No specific tools exist

� A spreadsheet is very helpful

� Applicability


� HSIA can use and complement FMECA SFMECA analysis.


Part III • Other methods

� Simulation

� Design execution

� Numeric analysis

� Error propagation in numerical computation

� Formal Methods

� Analytical methods for building SW systems

� Examples: B, CCS, Z, SDL, Object-Z

� See: http://vl.fmnet.info/


Part III • Q&A

??


Part IVValidation methods & tools


Part IV • Outline

� Testing methods

� Boundary testing

� Robustness testing

� Stress testing

� Testing tools

� Q&A


Part IV • Testing methods

� Boundary testing

� Testing strategy based on testing a function at its boundaries, i.e. below the limit, on the limit above the limit.

� This method aims at validating the accuracy of the inputs validation as well as the feasibility of the specified limits.

� To test whether a system or function copes with the required operational limits.



� Robustness testing

� To test the reliability and availability of a SW product when exposed to abnormal conditions (e.g. unexpected inputs).

� This method aims at exercising and validating the effectiveness of the error handling mechanisms.

� Fault Injection techniques and tools can be used to apply robustness testing.



� Stress testing

� A testing strategy that aims at testing the system behaviour when submitted to heavy load conditions.

� This method can be used to quantify the availability of a system.

� The method can also be used to obtain performance values.


Part IV • Testing tools

� XceptionTM

� Provides an automated fault injection / robustness testing environment.

� Generates sets of test cases (faults) according to user-defined criteria

� Executes the tests them in a automatic way

� Collects the tests results

� Provides support for results analysis

http://www.xception.org



� LDRA Testbed� Provides system and integration test functionalities

� Extracts test coverage metrics� Statement

� Branch Condition

� Dynamic flow (which variables are used in run-time)

� Also provides static analysis features

� Supports DO-178B

http://www.ldra.co.uk



� Cantata++/AdaTEST 95� Provides integration and unit-test functionalities for C/C++ and Ada

� Extracts code coverage metrics� Entry points and Call Returns

� Decisions (branches)

� MC/DC (DO-178B)

� Provides static analysis features as well

� Supports stubbing to simulate externalinterfaces

http://www.ipl.com


Part IV • Q&A

??


Part VISVV@CSW


Part V • ISVV@CSW

� CryoSat ISVV

� ISVV of the ESA CryoSat Satellite

� SGMP ISVV

� ISVV of a banking application for CGD

� ISVVP&F

� Contribution for the definition of the ISVV Process & Facility to be used by ESA


Part V • Other ISVV related projects

� STADY� Static and Dynamic Analysis of Critical Software (ORK/OBOSS)

� RAMS01� SFMECA evaluation (SCOS 2000)

� RAMS02� Robustness and stress testing methodology (RTEMS)

� RAMS03� HSIA evaluation and improvement (Herschel PACS)

� RAMS04� HA and HAZOP evaluation (ATV)


Part V • Q&A

??


Part VIConclusion


Part VI • Conclusion

� ISVV is carried out by an independent entity:

� Technical, managerial and financial

� Independent verification encompasses:

� Requirements, design and code

� Independent validation encompasses:

� Tests definition, implementation, execution and results analysis



� ISVV plugs into the SW development life-cycle� Complementing developper’s RAMS activities

� Contrubutting for the delivery of a betterproduct

� There are several methods and tools for supporting ISVV� Tailoring shall be performed according to project and customer needs


ISVV


Technical Specification Verification

Software Design Verification

Source Code Verification

Validation Tests Specification

Validation Procedures

Implementation

Test Execution and Results Analysis

System ConceptSystem in Operation

ISVV introduces

a smalladdedcost...

ISVV introduces

a smalladdedcost...

...andbrings a

highaddedvalue

...andbrings a

highaddedvalue


Part VIIBibliography


Part VII • Bibliography

The World Wide Web Virtual Library – Formal Methods, http://vl.fmnet.info/

[F-METHODS:2005]

MISRA Report 6, Verification and Validation[MISRA-RPT6:1995]

IEEE Standard for Software Verification and Validation

[IEEE-1012:1998]

Space product assurance, Software product assurance

[ECSS-Q-80B:2003]

Space engineering, Software – Part 1: Principles and requirements

[ECSS-E-40B:2003]

QMS, Software Development Process,CSW-2002-SDP-0909

[CSW-SDP:2005]


Part VII • Bibliography

IEEE Guide for Software Verification andValidation Plans

[IEEE-1059:1993]

Independent Verification and Validation, A Life Cycle Engineering for Quality Software

[LEWIS:1992]

ECSS Glossary of Terms[ECSS-P-001A:1997]

CSW ISVV web site, http://www.isvv.org[CSW-ISVV:2005]

Information technology – Software life-cycle process

[ISO/IEC 12207:1995]

NASA Independent Verification and Validation Facility, http://www.ivv.nasa.gov/index.php

[NASA-IVV:2005]


The

End

The

End


Revision history [CSW-2006-PRS-2495]

João EstevesHarmonised with the Ada and RAMS layout.26.09.20051.2-

João EstevesFirst presentation of the tutorial.13.06.20061.11

AuthorDescriptionDateRevisionIssue