9. verify and sell-off1 agenda for verify activity r1. objective r2. verification vs validation r3....
TRANSCRIPT
9. Verify and sell-off 1
Agenda for verify activity1. Objective2. Verification Vs validation3. Verification methods4. Test documentation5. Test execution6. Test results7. Reviews8. Special tests
9. Verify and sell-off 2
1. ObjectiveVerify activityVerify tasksCompletion criteriaPseudo-completion criteria
1. Objective
9. Verify and sell-off 3
Verify activityVerifies that the product complies with all its
requirements, and documents these test results
1. Objective
9. Verify and sell-off 4
Verify tasks
Write test plan
Write test spec
Write test procedure
product spec
Review test
readiness
Verifyproduct
by analysis
Review verification
results
Verifyproduct by ins, demo,
test, sta
test results
test plan
product
test spec
test proc
insdemoteststa
ana
product design
1. Objective
9. Verify and sell-off 5
Completion criteriaCustomer agrees testing is complete
1. Objective
9. Verify and sell-off 6
Pseudo-completion criteriaTesting completeProduct verification review complete
1. Objective
9. Verify and sell-off 7
2. Verification vs validationVerificationValidationHeuristicGovernment vs civilian validation
2. Verification vs validation
9. Verify and sell-off 8
VerificationFormal collection of test results that show
compliance to requirementsCan we prove that the product meets the
requirements of the specification
2. Verification vs validation
9. Verify and sell-off 9
ValidationAssessment of the correctness of a product
for its end-use.Can we show that the requirements reflect
the needs of the end user
2. Verification vs validation
9. Verify and sell-off 10
HeuristicVerification -- Have we built the system rightValidation -- Have we built the right system
2. Verification vs validation
9. Verify and sell-off 11
Government vs civilian validation
Government -- Contractor might not participate directly in formal validation or operational evaluation (OPEVAL)
Civilian -- Contractor often validates needs
2. Verification vs validation
9. Verify and sell-off 12
3. Verification methodsInspectionAnalysisDemonstrationTestSelected test and analysisDistinction among methods
3. Verification methods
9. Verify and sell-off 13
Inspection (INS) (1 of 2)
Definition: Verification without the use of special laboratory appliances or procedures. • Generally non-destructive. • Includes sight, hearing, smell, touch, and taste • May include simple physical manipulation• May include mechanical & electrical
measurement• May include other forms of investigation
3. Verification methods
9. Verify and sell-off 14
Inspection (INS) (2 of 2)Done when convenient, normally when no
further changes to the configuration are planned
Examples• Weighs less than 10 pounds• Surface free of visible defects
3. Verification methods
9. Verify and sell-off 15
Analysis (ANA) (1 of 3)Definition: Verification by technical or
mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals • Verification through the technical
evaluation of data• Data may may include lower-level product
test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation
• MTBF and MTBCF analysis may include failures from testing
3. Verification methods
9. Verify and sell-off 16
Analysis (ANA) (2 of 3)Subtypes
• Math analysis (ANA/M) -- Uses a model of formula to verify a design meets requirements
• Design analysis (ANA/D) -- Analysis of the design of a component to verify a design meets requirements
• Extrapolation analysis (ANA/T) -- Extrapolates the results of lower level tests to verify a design meets requirements
3. Verification methods
9. Verify and sell-off 17
Analysis (ANA) (3 of 3)
• Analysis based on test data (ANA_T) -- Verifies a design based on testing of lower products. This approach is often used with environmental tests.
Examples• Test coverage is greater than 95 percent• Design conforms to best commercial
practices
3. Verification methods
9. Verify and sell-off 18
Demonstration (DEM)Definition: Verification using actual operation,
adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.
Examples• Closed caption text appears when
commanded• Light goes off when safety switch activated
3. Verification methods
9. Verify and sell-off 19
Test (TST)Definition: Verification by technical means,
including operation, and involves application of scientific principles and procedures.
Many opinions say that test involves exercising the product and collecting data that needs to be reduced
Test examples• Signal to noise ratio > 2• Vibration power spectral density <0.2 g2/Hz at 20
Hz
3. Verification methods
9. Verify and sell-off 20
Selected test and analysis (STA)Definition: Verifies a quantitative requirement by
selected or sample tests when it is not feasible to perform comprehensive testing or when statistical significance cannot be obtained through a reasonable test program
Conducted under controlled conditionsAnalysis needed to interpret test results and to
extend test results to points not testedModels need to be validatedSTA example
• Performance shall be met at all altitudes from 1000 meters to 5000 meters
3. Verification methods
9. Verify and sell-off 21
Distinctions among methods (1 of 2)SubjectiveAgreement between customer &contractorCrude distinction
• Analysis -- Verification that can occur without the product being present
• Inspection -- Simple verification without exercising the product
• Demonstration -- Simple verification while exercising the product
• Test -- Everything else
3. Verification methods
9. Verify and sell-off 22
Distinctions among methods (2 of 2) Importance of distinguishing among methods
• Distinction is not terribly important• Analysis represents less risk • Inspection and demonstration are easier
than test• Many methods change when test spec
written
3. Verification methods
9. Verify and sell-off 23
4. Test documentationTest planTest specificationTest procedure
4. Test documentation
9. Verify and sell-off 24
Test Plan (1 of 3)Generated by the contractorDefines approach to testingEstablishes approach to test methods
• Demonstration, test, & STA -- defined in test spec
• Analysis -- defined in verification by analysis plan
• Inspection -- defined as inspection directives
4. Test documentation
9. Verify and sell-off 25
Test Plan (2 of 3)Defines roles of test specs and test
proceduresDefines how to capture test resultsDefines reviews
4. Test documentation
9. Verify and sell-off 26
Test plan (3 of 3)Test and evaluation master plan (TEMP
• Special case of test plan• Generated by customer• Defines customer plan for evaluating product• Applies to the highest level government
product
4. Test documentation
9. Verify and sell-off 27
Test specification (1 of 6)Generated by contractorDefines agreement between customer and
contractor about what each demonstration, test, and STA must do
Includes the pass/fail criteriaIs not a substitute for the product specification
4. Test documentation
9. Verify and sell-off 28
Test specification (2 of 6)Outline
• 1.0 Purpose -- gives high level purpose• 2.0 Scope -- covers DEM, TST, STA test
methods• 3.0 Objectives -- states to verify requirements• 4.0 Document organization -- describes each
section• 5.0 Overview -- describes test approach and
strategy• 6.0 Detailed tests -- defines tests -
4. Test documentation
9. Verify and sell-off 29
Test specification (3 of 6)
• 7.0 Verification map -- assigns requirements to tests
• 8.0 Applicable documents -- lists cited documents
• 9.0 Acronyms -- lists acronyms used
4. Test documentation
9. Verify and sell-off 30
Test specification (4 of 6)Detailed tests outline
• 1. Test objectives -- requirements verified by test group
• 2. Test prerequisites -- tests or activities that must be completed prior to this test
• 3. Test requirements -- what is required to execute the test group; e.g. facilities, environment, test equipment including accuracy, configuration, repetitions
4. Test documentation
9. Verify and sell-off 31
Test specification (5 of 6)
• 4. Test conditions -- test conditions and ordering
• 5. Analysis requirements -- analysis techniques and tools needed to determine pass/fail
• 6. Pass/fail criteria -- exit criteria for test• DEM -- what should be observed• TST, STA -- what should be measured• ALL -- margin of error, tolerance
4. Test documentation
9. Verify and sell-off 32
Test specification (6 of 6)Analysis plan
• Generated by contractor• Defines agreement between customer and
contractor about what each analysis must do
• Includes the pass/fail criteria• Is not a substitute for the product
specification
4. Test documentation
9. Verify and sell-off 33
Test procedure (1 of 3)Generated by the contractorDefines specific test set upDefines specific test process for running testApplies to demonstration, test, and selected
test and analysis
4. Test documentation
9. Verify and sell-off 34
Test procedure (2 of 3)Inspection directive
• Generated by contractor• Defines agreement between customer and
contractor about what each inspection must do
• Includes the pass/fail criteria• Usually a single sheet that also documents
results of test• Is not a substitute for the product
specification
4. Test documentation
9. Verify and sell-off 35
Test procedure (3 of 3)
Requirements Design Build
Test spec Test description Test procedure
Test description• Expands detail between the test spec & procedures• Analogous to design• Allows people to agree on details of test approach before starting
to write test procedure
is similar to
4. Test documentation
9. Verify and sell-off 36
5. Test ExecutionSimulationConfigurationTestsRegression tests
5. Test execution
9. Verify and sell-off 37
Simulation (1 of 2)A test tool that reduces costVerifies extremes that would be hard to
produce in actual testsSupport s performance and environmental
testing
5. Test execution
9. Verify and sell-off 38
Simulation (2 of 2)Simulation verification
• Simulations need to be validated for accuracy
• Customer may not have confidence in simulation if the phenomena being simulated is complicated; e.g. software throughput
5. Test execution
9. Verify and sell-off 39
ConfigurationTests should be done on configured hardware
and software to ensure repeatability
5. Test execution
9. Verify and sell-off 40
TestsMay be run in a lab, in the manufacturing
environment, when installed, or when used in the field
Important tests may warrant a quick-look report giving test results within 24 hours of the test
5. Test execution
9. Verify and sell-off 41
Regression testsDefinition: Tests that are run to confirm that a
product performance is OK after a change to the product
Run after originally scheduled verification testFor example, if a new device driver is added to
an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver
5. Test execution
9. Verify and sell-off 42
6. Test Results
Test method
Analysis (ANA)
Inspection (INS)
Demonstration (DEMO)
Test (TEST)
STA
Pass/fail
VDB
VDB
VDB
VDB
VDB
Analysis
Study
n/a
n/a
Study
Study
Test
Sheet
Sheet
Sheet
Report
Report
Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports
provide supporting data
Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports
provide supporting data
6. Test results
9. Verify and sell-off 43
7. ReviewsReadiness reviewProduct verification reviewFailure review board
7. Reviews
9. Verify and sell-off 44
Readiness reviewObtains customer concurrence that contractor
is ready to perform testConfirms that test procedures are satisfactory,
that product to be tested will be configured, and that correct people will witness the test
Presents schedule of testing, analysis, and review
7. Reviews
9. Verify and sell-off 45
Product verification review (1 of 2)Obtains customer concurrence that testing is
satisfactoryHeld after testExamines results of tests, including analysis
7. Reviews
9. Verify and sell-off 46
Product verification review (2 of 2)List tests that passExplains exceptions, problems, and failures;
addresses deviations and waiversMay be executed as a series of reviews with
each review associated with one or more tests
7. Reviews
9. Verify and sell-off 47
Failure review board
Examines each hardware failureDefines failure mechanism such as
electrostatic discharge, manufacturing defect, or manufacturing
Defines impact of failureDefines options for preventing future failures
7. Reviews
9. Verify and sell-off 48
8. Special testsReliability testingDesign of experimentsAccelerated life testingMaintainability and testabilityEnvironmental stress screening (ESS)Electromagnetic testingFailure report and corrective action system
8. Special tests
9. Verify and sell-off 49
Agenda for sell-off activity1. Objective2. Sell off vs close-out3. Audits4. Importance of sell-off5. Examples5. Homework
9. Verify and sell-off 50
1. ObjectiveSell-off activitySell-off tasksCompletion criteriaPseudo-completion criteria
1. Objective
9. Verify and sell-off 51
Sell-off activity
Confirms that all elements of contract have been met and obtains written agreement from the customer that the customer accepts the product
1. Objective
9. Verify and sell-off 52
Sell-off tasks
ConductPCA
ConductFCA
Collect contract deliverables and
test results
Agree
product
FCA audit resultscontract
deliverables and test results
contract deliverables and
test results
agreement
PCA audit results
1. Objective
9. Verify and sell-off 53
Completion criteriaSell off agreement complete
1. Objective
9. Verify and sell-off 54
Pseudo-completion criteria
Product deliveredFCA and PCA complete
1. Objective
9. Verify and sell-off 55
2. Sell-off vs close-out Sell off occurs when the product has completed
verificationContract close out comes later when all
contractual issues are closed
2. Sell-off vs close-out
9. Verify and sell-off 56
3. Audits DefinitionFCAPCA
3. Audits
9. Verify and sell-off 57
DefinitionSurveys to confirm delivered product is as it
should beMajor government audits
• Functional configuration audit (FCA)• Physical configuration audit (PCA)
May alternately be considered part of verification
3. Audits
9. Verify and sell-off 58
FCAVerifies that the each hardware and software
configuration item complies with its requirements
Confirms that product passed its testsExample items in audit
• Test procedures agree with spec & test spec
• Test results• Configuration control of testing• Action items
3. Audits
9. Verify and sell-off 59
PCA (1 of 2)
Establishes the product baseline, ensures as-built product matches documentation, verify that acceptance testing is adequate
Confirms we built what we designed Example items in audit
• CI design documentation• History of changes made and pending• Operating and support manuals• Parts selection list
3. Audits
9. Verify and sell-off 60
PCA (2 of 2)
• CI as-built specifications and interfaces• Manufacturing and assembly instructions• Acceptance test procedures and data
Types of PCAs• Build up -- Perform PCA as product built up • Tear down - Perform PCA after product
built
3. Audits
9. Verify and sell-off 61
4. Importance of sell-offImportant for completing contractNot a difficult conceptRequires planning to assemble
documentation Suggestions• Make the customer a team member in sell
off so that the customer will know what to expect at the sell off
• Begin at CDR to monitor the percentage of all sell-off items that are complete
4. Importance of sell off
9. Verify and sell-off 62
5. ExamplesExample 1Example 2
5. Examples
9. Verify and sell-off 63
Example 1Government product
• Contract items• Product
• Test results
• Warranty
• Spare parts
• Support agreement
• Government furnished equipment
• Facilities
• Contract data
• Sell-off agreement• DD 250 for each contract line item
5. Examples
9. Verify and sell-off 64
Example 2House
• Sell-off items• House• Title policy• Warranty
deed
• Sell-off agreement
• Bill of sale
5. Examples
9. Verify and sell-off 65
6. Homework1. For buying a car for personal use, list three
types of tests that might apply for each of the four test methods.
2. If we said that we were going to verify the weight of a car by adding up the weights from the spec sheets for each component of the car, what test method would we use?
3. In building a house, what is an example of a verification test that we might require before the house is complete?
6. Homework