1 agenda for verify activity r1. objective r2. verification methods r3. test documentation r4. test...
TRANSCRIPT
1
Agenda for verify activity
1. Objective2. Verification methods3. Test documentation4. Test execution5. Test results6. Reviews7. Special tests
2
1. Objective
Verify activityVerify tasksCompletion criteriaPseudo-completion criteria
1. Objective
3
Verify activity
Verifies that the product complies with all its requirements, and documents these test results
1. Objective
4
Verify tasks
verifyproduct by ins, demo,
test
write test plan
write test spec
verifyproduct
by analysis
writetest
procedure
product spec
test readinessreview (TRR)
verification review (VR)
test results
test plan
product
test spec
test proc
insdemoteststa
ana
product design
1. Objective
5
Completion criteria
Customer agrees testing is complete
1. Objective
6
Pseudo-completion criteria
Testing completeProduct verification review complete
1. Objective
7
2. Verification methods
InspectionAnalysisDemonstrationTestDistinction among methods
2. Verification methods
8
Inspection (INS) (1 of 2)
Definition: Verification without the use of special laboratory appliances or procedures. • Generally non-destructive. • Includes sight, hearing, smell, touch, and taste • May include simple physical manipulation• May include mechanical & electrical
measurement• May include other forms of investigation
2. Verification methods
9
Inspection (INS) (2 of 2)
Done when convenient, normally when no further changes to the configuration are planned
Examples• Weighs less than 10 pounds• Surface free of visible defects
2. Verification methods
10
Analysis (ANA) (1 of 2)
Definition: Verification by technical or mathematical models, simulations, algorithms, charts, circuit diagrams, or other scientific principals
2. Verification methods
11
Analysis (ANA) (2 of 2)
• Verification through the technical evaluation of data
• Data may may include lower-level product test results, historical data, simulation data, engineering drawings, flow-down studies, or other documentation
• MTBF and MTBCF analysis may include failures from testing
12
Demonstration (DEM)
Definition: Verification using actual operation, adjustment, or reconfiguration. Products may be instrumented and quantitative limits of performance monitored.
Examples• Closed caption text appears when
commanded• Light goes off when safety switch activated
2. Verification methods
13
Test (TST)
Definition: Verification by technical means, including operation, and involves application of scientific principles and procedures.
Many opinions say that test involves exercising the product and collecting data that needs to be reduced
Test examples• Signal to noise ratio > 2• Vibration power spectral density <0.2 g2/Hz at 20
Hz
2. Verification methods
14
Distinctions among methods (1 of 2)
SubjectiveAgreement between customer &contractorCrude distinction
• Analysis -- Verification that can occur without the product being present
• Inspection -- Simple verification without exercising the product
• Demonstration -- Simple verification while exercising the product
• Test -- Everything else
2. Verification methods
15
Distinctions among methods (2 of 2)
Importance of distinguishing among methods• Distinction is not terribly important• Inspection represents less risk • Inspection and demonstration are easier
than test• Many methods change when test spec
written
2. Verification methods
16
3. Test documentation
Test planTest specificationTest procedure
4. Test documentation
17
Test plan (1 of 3)
Generated by the contractorDefines approach to testingEstablishes approach to test methods
• Demonstration and test -- defined in test spec• Analysis -- defined in verification by analysis
plan• Inspection -- defined as inspection directives
4. Test documentation
18
Test plan (2 of 3)
Defines roles of test specs and test procedures
Defines how to capture test resultsDefines reviews
4. Test documentation
19
Test plan (3 of 3)
Test and evaluation master plan (TEMP)• Special case of test plan• Generated by customer• Defines customer plan for evaluating product• Applies to the highest level government
product
3. Test documentation
20
Test specification (1 of 2)
Generated by contractorDefines agreement between customer and
contractor about what each demonstration and test must do
Includes the pass/fail criteriaIs not a substitute for the product specification
3. Test documentation
21
Test specification (2 of 2)
Analysis plan• Generated by contractor• Defines agreement between customer and
contractor about what each analysis must do
• Includes the pass/fail criteria• Is not a substitute for the product
specification
3. Test documentation
22
Test procedure (1 of 3)
Generated by the contractorDefines specific test set upDefines specific test process for running testApplies to demonstration, test, and selected
test and analysis
3. Test documentation
23
Test procedure (2 of 3)
Inspection directive• Generated by contractor• Defines agreement between customer and
contractor about what each inspection must do
• Includes the pass/fail criteria• Usually a single sheet that also documents
results of test• Is not a substitute for the product
specification
3. Test documentation
24
Test procedure (3 of 3)
Requirements Design Build
Test spec Test description Test procedure
Test description• Expands detail between the test spec &
procedures• Analogous to design• Allows people to agree on details of test before
starting to write test procedure
is similar to
3. Test documentation
25
4. Test execution
SimulationConfigurationTestsRegression tests
4. Test execution
26
Simulation (1 of 2)
A test tool that reduces costVerifies extremes that would be hard to
produce in actual testsSupports performance and environmental
testing
4. Test execution
27
Simulation (2 of 2)
Simulation verification• Simulations need to be validated for
accuracy• Customer may not have confidence in
simulation if the phenomena being simulated is complicated; e.g. software throughput
4. Test execution
28
Configuration
Tests should be done on configured hardware and software to ensure repeatability
4. Test execution
29
Tests
May be run in a lab, in the manufacturing environment, when installed, or when used in the field
Important tests may warrant a quick-look report giving test results within 24 hours of the test
4. Test execution
30
Regression tests
Definition: Tests that are run to confirm that a product performance is OK after a change to the product
Run after originally scheduled verification testFor example, if a new device driver is added to
an already tested operating system, tests would be run to confirm that the operating system still opiates with the added device driver
4. Test execution
31
5. Test results
Test method
Analysis (ANA)
Inspection (INS)
Demonstration (DEMO)
Test (TEST)
Pass/fail
VDB
VDB
VDB
VDB
Analysis
Study
n/a
n/a
Study
Test
Sheet
Sheet
Sheet
Report
Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports
provide supporting data
Pass/fail is recorded in single place like a verification data base (VDB). Studies and single-sheet reports
provide supporting data5. Test results
32
6. Reviews
Readiness reviewProduct verification reviewFailure review board
6. Reviews
33
Readiness review
Obtains customer concurrence that contractor is ready to perform test
Confirms that test procedures are satisfactory, that product to be tested will be configured, and that correct people will witness the test
Presents schedule of testing, analysis, and review
6. Reviews
34
Product verification review (1 of 2)
Obtains customer concurrence that testing is satisfactory
Held after testExamines results of tests, including analysis
6. Reviews
35
Product verification review (2 of 2)
List tests that passExplains exceptions, problems, and failures;
addresses deviations and waiversMay be executed as a series of reviews with
each review associated with one or more tests
6. Reviews
36
Failure review board
Examines each hardware failureDefines failure mechanism such as
electrostatic discharge, manufacturing defect, or manufacturing
Defines impact of failureDefines options for preventing future failures
6. Reviews
37
7. Special tests
Reliability testingDesign of experimentsAccelerated life testingMaintainability and testabilityEnvironmental stress screening (ESS)Electromagnetic testingFailure report and corrective action system
7. Special tests