System test, Evaluation, and

ValidationBy:

Ocampo, Abbygail

Zabat, Jenilyn

System test, Evaluation and Validation

System test, evaluation, and validation activities should be established during the conceptual design phase of the life cycle, concurrently with the definition of the overall system design requirements. From that point on, The test and evaluation continues by testing of individual components, the testing of various system elements and major subsystems and then by the testing of overall system as an integrated entity.

The objective is to adopt a “progressive” approach that will lend it self to continuous implementation and improvement as the system design and development process evolves.

Test and evaluation activities in this chapter can be aligned initially with the design activities describe in chapter3-5 and then extended through production/construction and system utilization.

The purpose of this chapter is to:

• to present an integrated approach for system test and evaluation

• to facilitate the necessary validation of the proposed system configuration and

• to provide assurance that will indeed meet the customer requirements

System Requirements and Evaluation Relationship

Stages of System Evaluation During Life Cycle

Categories of System test and Evaluation

• 6.2.1 Analytical and Simulation Evaluation

The utilization of analytical models, CAD methods, and various combinations of these can be effective in the verification of certain design relationships before system is finalized.

for example, the development of system through computer graphics can illustrate the relationships among various system elements and accesses for maintenance and support. From this, potential interferences, space utilization conflicts can be detected.

Categories of System test and Evaluation

• Type 1 Testing

During the early phase of detail design, breadboards, bench-test models, engineering models, engineering software, and service test models are build with the intent of verifying certain performance and physical design characteristics. These models, representing either an entire system or a designated system component, usually operate functionally (electrically and mechanically) but do not by any means represent production equipment. In the development of software, the application of rapid prototyping is sometimes used to verify design adequacy.

Categories of System test and Evaluation

• Type 2 Testing

Formal tests and demonstrations are accomplished during the latter part of detail design when preproduction prototype equipment, software, and similar to production equipment (that which will be deployed for operational use), but it is not necessarily fully qualified at this point in time. A test program may constitute a series of individual tests tailored to the need. Such a program might include the following:

Type 2 test..• 1. Performance tests are accomplished to verify individual

system performance characteristics. For instance, tests are designed to determine whether the electric motor will provide the necessary output, whether the airplane will perform its intended mission successfully, and so on. Also, it is necessary to verify form, fit, interchangeability, product safety, and other comparable features.

• 2. Environmental qualification – temperature cycling, shock and vibration, humidity, wind, salt, spray, dust and sand, fungus, acoustic noise, pollution emission, explosion proofing, and electromagnetic interference tests are conducted. These factors are oriented to what the various system elements will be subjected to during operation, maintenance, and during transportation and handling function.

Type 2 test..

3. Structural tests are conducted to determine material characteristics relative to stress, strain, fatigue, bending, torsion and general decomposition.

4. Reliability qualification are accomplished on one or more system elements to determine MTBF. Also, special tests are often designed to measure component life, to evaluate degradation, and to determine models of failure.

5. Maintainability demonstration – tests are conducted on one or more system elements to assess the values for mean active maintenance time (M), mean corrective maintenance (Mct), mean preventive maintenance time (Mpt), maintenance labor-hours per operating hour (MLH/OH), and so on

Type 2 test..

6. Support equipment compatibility tests are often accomplished to verify compatibility among the prime equipment, test and support equipment, and transportation and handling equipment

7. Personnel test and evaluation are often accomplished to verify the relationships between people and equipment, the personnel quantities and skill levels required and training needs, both operator and maintenance tasks are evaluated.

8. Technical data verification – the verification of operational and maintenance procedures is accomplished.

9. Software verification – the verification of operational and maintenance software is accomplished.

Categories of System test and Evaluation

• Type 3 Testing

Formal tests and demonstrations, started after initial system qualification and prior to the completion of production, are accomplished at a designated field test site by user personnel, operation.

Categories of System test and Evaluation

• Type 4 Testing

During the operational-use phase, formal tests are sometimes conducted to gain further insight in a specific area. It may be desirable to vary the mission profile or the system utilization rate to determine the impact on total system effectiveness, or it might be feasible to evaluate several alterative support policies to see whether system operational availability can be improved

Planning for Testing and Evaluation

1. The definition and schedule of all tests requirements, including anticipated test output for each individual test and integrated where possible.

2. The definition of organization, administration and control responsibilities.

3. The definition of test conditions and maintenance and logistic resource requirements.

Planning for Testing and Evaluation

4. A description of the test preparation phase for each type of testing method, training of testing.

5. A description of the formal test phase.

6. A description of condition and provisions for a retest phase.

7. The identification of test documentation

Evaluation of Test Requirements

Preparation for system test and evaluation

• The selection of the items to be tested,

• Establishment of test procedures,

• Test site selection,

• Selection and training of test personnel,

• Preparation of test facilities and resources,

• Acquisition of support equipment,

• Test supply support.

Test performance and reporting

• This requires operating and supporting the system in a prescribed manner as defined in the system test and evaluation plan.

Test performance and reporting

• The following questions will be answered:

1. what is the true performance and effectiveness of the system?

2. what is the true effectiveness of the maintenance and logistic support capability?

3. are the initial specified requirements being met?

Test data requirements

1. It provides ongoing data that are analyzed to evaluate and assess the performance, effectiveness, operation, maintenance, logistic support capability, and so on.

2. It provides historical data that are applicable in the design and development of new systems having a similar function and nature.

Development of a data subsystem

1. Success data constitute information covering system operation and use on a day-to-day basis.

2. Maintenance data cover each event involving scheduled and unscheduled maintenance.

Test reporting

1. The final effort in the evaluation process,

2. It should reference the initial system evaluation planning document,

3. It should describe all test conditions, incorporated system modifications during the test, test data, and the results of data analysis.

System Evaluation and Corrective Action Loop

Summary