6. verify and sell-off 1 agenda for verify activity r1. objectives r2. verification vs validation...

6. Verify and sell-off 1

Agenda for Verify Activity

1. Objectives 2. Verification Vs validation 3. Verification methods 4. Documentation of tests 5. Test execution 6. Test results 7. Reviews 8. Special tests


1. Objectives

Verify activity Product-based activities Products used to manage Completion criteria

1. Objectives


Verify Activity

Verifies that the product complies with all its requirements, and documents these test results

1. Objectives


Verify Tasks

Write test plan

Write test spec

Write test procedure

product spec

Review test

readiness

Verifyproduct

by analysis

Review verification

results

Verifyproduct by ins, demo,

test, sta

test results

test plan

product

test spec

test proc

insdemoteststa

ana

product design

1. Objectives


Completion Criteria

Customer agrees testing is complete

1. Objectives


Pseudo- Completion Criteria

Testing complete Product verification review complete

1. Objectives


2. Verification Vs Validation

Verification Validation Heuristic Government Vs civilian validation

2. Verification vs validation


Verification

Formal collection of test results that show compliance to requirements

Can we prove that the product meets the requirements of the specification



Validation

Assessment of the correctness of a product for its end-use.

Can we show that the requirements reflect the needs of the end user



Heuristic

Verification -- Have we built the system right Validation -- Have we built the right system



Government Vs Civilian Validation

Government -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL)

Civilian -- Contractor often validates needs



3. Verification Methods

Inspection Analysis Demonstration Test

3. Verification methods


Inspection (INS)

Verification without the use of special laboratory appliances or procedures.

Generally non-destructive. Includes sight, hearing, smell, touch, and taste May include simple physical manipulation May include mechanical & electrical measurement May include other forms of investigation

Done when convenient, normally when no further changes to the configuration are planned



Analysis (ANA)

Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals

Verification through the technical evaluation of data

Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation

MTBF and MTBCF analysis may include failures from testing



Analysis Categories

Math analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements

Design analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements

Extrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements

Analysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests.



Demonstration (DEM)

Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.



Test (TST)

Verification by technical means, including operation, and involves application of scientific principles and procedures.



Selected Test and Analysis (STA)

Verifies a quantitative requirement by selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program

Conducted under controlled conditions Analysis needed to interpret test results and to

extend test results to points not tested Models need to be validated



Distinctions of Methods

Subjective Agreement between customer &contractor



Crude Distinction

Analysis -- Verification that can occur without the product being present

Inspection -- Simple verification without exercising the product

Demonstration -- Simple verification while exercising the product

Test -- Everything else



Test

Many opinions say that test involves exercising the product and collecting data that needs to be reduced



Importance of Distinguishing among Methods

Distinction is not terribly important Analysis represents less risk Inspection and demonstration are easier than test Many methods change when test spec written



Analysis Examples

Test coverage is greater than 95 percent Design conforms to best commercial practices



Inspection Examples

Weighs less than 10 pounds Surface free of visible defects



Demonstration Examples

Closed caption text appears when commanded Light goes off when safety switch activated



Test Examples

Signal to noise ratio > 2 Vibration power spectral density <0.2 g2/Hz at 20 Hz



4. Documentation of Tests

Test plan Test specification Test procedure

4. Documentation of tests


Test Plan (1 of 2)

Generated by the contractor Defines approach to testing Establishes approach to test methods

Demonstration, test, & STA -- defined in test spec Analysis -- defined in verification by analysis plan Inspection -- defined as inspection directives



Test Plan (2 of 2)

Defines roles of test specs and test procedures Defines how to capture test results Defines reviews



Test and Evaluation Master Plan

Generated by customer Defines customer plan for evaluating product Applies to the highest level government product



Test Specification

Generated by contractor Defines agreement between customer and

contractor about what each demonstration, test, and STA must do

Includes the pass/fail criteria Is not a substitute for the product specification



Test Specification Outline

1.0 Purpose -- gives high level purpose 2.0 Scope -- covers DEM, TST, STA test methods 3.0 Objectives -- states to verify requirements 4.0 Document organization -- describes each section 5.0 Overview -- describes test approach and strategy 6.0 Detailed tests -- defines tests - 7.0 Verification map -- assigns requirements to tests 8.0 Applicable documents -- lists cited documents 9.0 Acronyms -- lists acronyms used



Detailed Test Group Outline (1 of 2)

1. Test objectives -- requirements verified by test group 2. Test prerequisites -- tests or activities that must be

completed prior to this test 3. Test requirements -- what is required to execute the

test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions



Detailed Test Outline (2 of 2)

4. Test conditions -- test conditions and ordering 5. Analysis requirements -- analysis techniques

and tools needed to determine pass/fail 6. Pass/fail criteria -- exit criteria for test

DEM -- what should be observed TST, STA -- what should be measured ALL -- margin of error, tolerance



Analysis Plan


contractor about what each analysis must do Includes the pass/fail criteria Is not a substitute for the product specification



Inspection Directive


contractor about what each inspection must do Includes the pass/fail criteria Usually a single sheet that also documents

results of test Is not a substitute for the product specification



Test Description

Requirements Design Build

Test spec Test description Test procedure

Expands detail between the test spec & procedures Analogous to design Allows people to agree on details of test approach

before starting to write test procedure

is similar to



Test Procedure

Generated by the contractor Defines specific test set up Defines specific test process for running test Applies to demonstration, test, and selected test

and analysis



5. Test Execution

Simulation Configuration Tests Regression tests

5. Test execution


Simulation

A test tool that reduces cost Verifies extremes that would be hard to produce

in actual tests Support s performance and environmental testing

5. Test execution


Simulation Verification

Simulations need to be validated for accuracy Customer may not have confidence in simulation

if the phenomena being simulated is complicated; e.g. software throughput

5. Test execution


Configuration

Tests should be done on configured hardware and software to ensure repeatability

5. Test execution


Tests

May be run in a lab, in the manufacturing environment, when installed, or when used in the field

Important tests may warrant a quick-look report giving test results within 24 hours of the test

5. Test execution


Regression Tests

Tests that are run to confirm that a product performance is OK after a change to the product

Run after originally scheduled verification test For example, if a new device driver is added to an

already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver

5. Test execution


6. Test Results

Test method

Analysis (ANA)

Inspection (INS)

Demonstration (DEMO)

Test (TEST)

STA

Pass/fail

VDB

VDB

VDB

VDB

VDB

Analysis

Study

n/a

n/a

Study

Study

Test

Sheet

Sheet

Sheet

Report

Report

Pass/fail status for each type of test is recorded in a single place such as a verification data base (VDB), Analysis is documented in a study. Tests having methods ANA, INS, or DEMO are documented in a single sheet report. Test and STA are documented in a separate report

6. Test results


7. Reviews

Readiness review Product verification review Failure review board

7. Reviews


Readiness Review

Obtains customer concurrence that contractor is ready to perform test

Confirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test

Presents schedule of testing, analysis, and review

7. Reviews


Product Verification Review (1 of 2)

Obtains customer concurrence that testing is satisfactory

Held after test Examines results of tests, including analysis

7. Reviews


Product Verification Review (2 of 2)

List tests that pass Explains exceptions, problems, and failures;

addresses deviations and waivers May be executed as a series of reviews with each

review associated with one or more tests

7. Reviews


Failure Review Board

Examines each hardware failure Defines failure mechanism such as electrostatic

discharge, manufacturing defect, or manufacturing Defines impact of failure Defines options for preventing future failures

7. Reviews


8. Special Tests

Reliability testing Design of experiments Accelerated life testing Maintainability and testability Environmental stress screening (ESS) Electromagnetic testing Failure report and corrective action system

8. Special tests


Agenda for Sell Off Activity

1. Objectives 2. Sell off Vs close out 3. Audits 4. Importance of sell off 5. Homework


1. Objectives

Sell-off activity Product-based activities Products used to manage Completion criteria

1. Objectives


Sell-Off Activity

Confirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product

1. Objectives


Sell-Off Tasks

ConductPCA

ConductFCA

Collect SOW deliverables and

test results

Agree

product

FCA audit results

SOW deliverables and test results

SOW deliverables and test results

agreement

PCA audit results

1. Objectives


Completion Criteria

Sell off agreement complete

1. Objectives


Pseudo-Completion Criteria

Product delivered FCA and PCA complete

1. Objectives


2. Sell Off Vs Close Out

Sell off occurs when the product has completed verification

Contract close out comes later when all contractual issues are closed

2. Sell off vs close out


3. Audits

Surveys to confirm delivered product is as it should be Major government audits

Functional configuration audit (FCA) Physical configuration audit (PCA)

May alternately be considered part of verification

3. Audits


FCA

Verifies that the each hardware and software configuration item complies with its requirements

Confirms that product passed its tests Example items in audit

Test procedures agree with spec & test spec Test results Configuration control of testing Action items

3. Audits


PCA

Establishes the product baseline, ensures as-built product matches documentation, verify that acceptance testing is adequate

Confirms we built what we designed Example items in audit

CI design documentation History of changes made and pending Operating and support manuals Parts selection list CI as-built specifications and interfaces Manufacturing and assembly instructions Acceptance test procedures and data

3. Audits


Types of PCAs

Build up -- Perform PCA as product built up Tear down - Perform PCA after product built

3. Audits


4. Importance of Sell Off

Important for completing contract Not a difficult concept Requires planning to assemble documentation

Suggestions Make the customer a team member in sell off

so that the customer will know what to expect at the sell off

Begin at CDR to monitor the percentage of all sell-off items that are complete

4. Importance of sell off


Example 1 -- Government Product

Contract items Product Test results Warranty Spare parts Support agreement Government furnished equipment Facilities Contract data

Sell-off agreement DD 250 for each contract line item



Example 2 -- House

Sell-off items House Title policy Warranty deed

Sell-off agreement Bill of sale



5. Homework

1. For buying a car for personal use, list three types of tests that might apply for each of the four test methods.

2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use?

3. In building a house, what is an example of a verification test that we might require before the house is complete?

5. Homework

6. verify and sell-off 1 agenda for verify activity r1. objectives r2. verification vs validation...

Documents