an annotated selective bibliography on human … · from this bibliography. where possible, we used...

Technical Report 435

AN ANNOTATED SELECTIVE BIBLIOGRAPHY

ON HUMAN PERFORMANCE IN FAULT

DIAGNOSIS TASKS

William B. Johnson, Sandra H. Rouse,and William B. Rouse

0 Coordinated Science Laboratory, University of Illinois

MANPOWER & EDUCATIONAL SYSTEMS TECHNICAL AREA

)o,, '

U.S. Army

Research Institute for the Behavioral and Social Sciences

January 1980

Approved for public release; distribution unlimited.

80 7 10 012

L IIA'C

U. S. ARMY RESEARCH INSTITUTEFOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of theDeputy Chief of Staff for Personnel

FRANKLIN A. HARTJOSEPH ZEIDNER Colonel, US ArmyTechnical Director Commander

NOTICES

DISTR IOU TI ON Prime(ry distribution of th~is report has been Made by AMt. Please address correspondene

concerning distribution of reports to. U. S. Army Research Institute for the Blehavioral and Social Sciences,0 ATTN: PERI-TP, 5001 Eisenhower Avenue, Alexandria. Virginia 22333.

FINAL DISPOSITION This report may be destroyed whon it is no longer needed. Please do not return it to

the U. S. Army Research Insitute f or the Behavioral and Social Sciences.

I N=L The findings in this report are not to be construed as an official Department ml rhe Army Position.unless so designated by other euthorized documents.

LL LIN"

UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE (When Data BamK )___________________

READ INSTRUCTIONSREPORT DOCUMENTATION PAGE BEFORE COMPLETING FORK

REPORTNUM2. GOVT ACCESSION NO. 3. RECIPI"T-S CATALOG NUMBER

Technical Report 435 &

TST ITYPf.

_%/ AN,*M$OTATED -jELECTIVE JIBLIOGRAPHY ON HUMAN'PERFORMANCE IN FAULT DIAGNOSIS TASKS -N Final a# 8Jul#79

__ S. PERFORMING OR G.REPT UMR&

William B./Johnson Sandra H. /Rouse' and -- -/DAHCl9-78-G1 Z'-'

3PERFORMING ORGANIZATION NAME AND ADDRESS ) 10. PROGRAM ELEMENT. PROJECT, TASK

Coordinated Science Laboratory, College ofA&WOKUINMBREngineering . -. I j AR

University of Illinois7l Urbana, IL 61801

ii. CONTROLLING OFFICE NAME AND ADDRESS62R

US Army Research Institute for the Behavioral andli-jnNj~

Social Sciences (PERI-OK) 1.NM

5001 Eisenhower Avenue.,fAlexand _ VA 223'13 4914 ONIWING AGENCY NAME & ADDESI diffn CroCntrolling Office) 1S. SECURITY CLASS. (of Ibis report)

~UnclassifiedT~a DCASFCATION/DOWNGRADINGSCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abstract entered I Block It, iffferent from Rep'o")

I*. SUPPLEMENTARY NOTES

Monitored technically by Leon H. Nawrocki, Richard M. Johnson, and Bruce W.Knerr of the Manpower & Educational Systems Technical Area, 'RI

I9. KEY WORDS (Continuiie an revers, aide It nec.esar mnd Identity by block number)

Maintenance TrainingDiagnostic TrainingTroubleshooting TrainingFault Diagnosis

2j- 4kSTUACr Cmietwe eareerse ag N noseaey md identify by block numnber)

-2f/This is an annotated bibliography of 61 published works about humanperformance in fault diagnosis. The sources are predominatly from thedisciplines of engineering, psychology and education. Computer searches wereconducted of the past 10 years of Psychological Abstracts and ERIC Documents.Manual searches using references from reports, University of Illinois librarycard catalog,0 and solicitation of references from experts in the fieldextended' the search to include references dating from the early fifties .

DOR W~ 43 RIINTIOM OFI NOV %IS OBSOLETE Ucasfe

i SECURITY CLASSIFICATION OF THIS PAGE (Whent 0aq0 Eat~r

Technical Report 435

AN ANNOTATED SELECTIVE BIBLIOGRAPHY

ON HUMAN PERFORMANCE IN FAULT

DIAGNOSIS TASKS

William B. Johnson, Sandra H. Rouse,and William B. Rouse

Coordinated Science Laboratory, University of Illinois

Submitted by:James D. Baker, Chief

MANPOWER & EDUCATIONAL SYSTEMS TECHNICAL AREA

Approved by:Robert M. Sasmor, DirectorBASIC RESEARCH

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for PersonnelDepartment of the Army

January 1980

Army Project Number Basic Research in Decision-Making20161102B74F

AoprowAid for publmicle: dlnnbuton unlmid., iii

ARI Research Reports and Technical Reports are intended for sponsors ofR&D tasks and for other research and military agencies. Any findings readyfor implementation at the time of publication are presented in the last partof the Brief. Upon completion of a major phase of the task, formal recom-mendations for official action normally are conveyed to appropriate militaryagencies by briefing or Disposition Form.

iv

A1

: ... : ... : ...• _ .,- .... ., .,.,' . t ,. - .:... ,, - : -

FOREWORD

The Manpower & Educational Systems Technical Area of the ArmyResearch Institute for the Behavioral and Social Sciences (ARI) performs

research and development in areas that include educational technologyand training simulation with applicability to military training. Ofspecial interest is research in the area of computer-based systems formaintenance training. The development and implementation of suchsystems is seen as a means of reducing training time and costs byproviding more highly individualized training than would be otherwisepossible, while at the same time reducing the need for operationalequipment for training.

The report provides an annotated bibliography of research directlyrelated to human problem-solving behavior in fault diagnosis (trouble-shooting) tasks. It provides an information source to assist plannersof future research efforts in this area.

This research is responsive to the requirements of RDT&E Project2QI61102B74F, "Basic Research in the Behavioral and Social Sciences."

JO INERchnical Director

tI I'S

,A&,J i r.0 :C,r.

XA~ 1

V ~ 7-x

* .r." -- * . .. - - - -~ - - -- -

* . **L

AN ANNOTATED SELECTIVE BIBLIOGRAPHY ON HUMAN PERFORMANCE IN FAULTDIAGNOSIS TASKS

BRIEF

Requirement:

In conjunction with an ongoing program of research on the use of

computer simulations for fault diagnosis training, a thorough review ofthe existing literature was necessary to provide historical input and toidentify current trends in fault diagnosis work. The results of thissearch are presented here as a selective annotated bibliography.

Procedure:

The search procedure consisted of on-line computer search of thepast 10 years of Psychological Abstracts and ERIC Documents throughmid-1978. Manual searches used references from reports as well as from

the card catalog of the University of Illinois library. In addition,experts in the area of fault diagnosis were sent the references resultingfrom the computer and manual searches and asked to suggest any additionalreferences that should be included.

Findings:

The references revealed by the search procedure were predominatelyfrom the areas of engineering, psychology, and education. Three majorcategories of materials are presented. These consist of (1) researchreports, (2) manuals for fault diagnosis, and (3) material indirectlyrelated to fault diagnosis, such as medical diagnosis and mathematicalproblem solving, or items directly related to fault diagnosis that are

not research reports.

Utilization of Findings:

This annotated bibliography will serve as an information source

for planners of future R&D efforts in this area.

vii

INTRODUCTION

The authors and several colleagues at the University of

Illinois are investigating the use of computer simulations for

fault diagnosis training. This work runs the gamut from

context-free simulations and context-specific simulations into

the live environment of real equipment troubleshooting. This

research, supported by the U.S. Army Research Institute for the

Behavioral and Social Sciences, includes the development of

models of human fault diagnosis performance and ultimately an

intelligent interactive training simulator. To this end, we have

already run six experiments involving almost 160 subjects who

solved approximately 13,000 fault diagnosis problems.

As part of our work, we have prepared this annotated

selected bibliography on human performance in fault diagnosis

tasks. The sources in this bibliography are predominantly from

the disciplines of engineering, psychology, and education.

Although the primary emphasis is on the research literature, we

have included a representative number of actual troubleshooting

manuals from the automotive and aviation fields.

The literature search included several phases, involving

both on-line and manual searches. Computer searches of the past

ten years of Psychological Abstracts and ERIC documents through

mid-1978 were performed. Manual searches were conducted using

bibliographies from various reports. Using the card catalog at

the University of Illinois library, we obtained materials dating

from the early fifties.

PI

The third phase of this literature search involved the

solicitation of additional references from noted fault diagnosis

and/or computer simulation experts. Over twenty such individuals

were sent the list of references resulting from the first two

phases of our literature search. 'They were asked to skim the

list of references and suggest any additional references that

they felt to be essential for a selected bibliography on human

performance in fault diagnosis tasks. Approximately one-third of

those queried responded. We greatly appreciate their

contributions.

The materials in this bibliography are divided into the

following three major areas:

1. Research Reports

A. Primary Research

B. Secondary Research

2. Manuals for Fault Diagnosis

3. Materials Indirectly Related to Fault Diagnosis

Primary research reports include empirical and/or theoretical

studies aimed directly at human problem solving behavior in fault

diagnosis tasks. Secondary research reports refer to literature

reviews. As noted earlier, a representative set of training and

field manuals is included in the "manuals" category. Indirectly

related materials refer to studies of human behavior in related

domains (e.g. medical diagnosis or problem solving in

mathematics) -, reports that directly relate to fault diagnosis

A L "4 . "

but do not provide research results of a type necessary for the

report to be categorized as a "research report." Since the number

of items that could be placed in the "indirectly related"

category was potentially enormous, we limited consideration tP

items that related to human behavior and particularly matched our

interests. Thus, for example, the large body of literature on

maintainability and automated fault diagnosis has been omitted

from this bibliography.

Where possible, we used the abstract that accompanied an

item. However, we usually had to supplement the abstract

considerably in order to make it self-explanatory. The indexing

of each item was based on a consensus of the authors.

We hope to periodically update this bibliography and

re-issue it. To that end, we would appreciate suggestions and

contributions. If possible, when submitting a suggestion for

inclusion of a new item, please include a copy of the item.

. gghh.. A4

,

PRIMARY RESEARCH

Bond, N.A., Jr.; Rigney, J.W. (1966) "Bayesian Aspects ofTroubleshooting Behavior." Human Factors. 8:377-383.

Thirty-nine Navy technician trainees filled out asymptom-malfunction matrix on a blocking oscillatorcircuit. The particular sequence of checks used by eachman on each problem was combined with hissymptom-malfunction matrix, via a Bayesian algorithm, toyield computer estimates of failure likelihoods for eachcomponent. The computer program predicted actualparts-replacement behaviors in about half of the cases.Those technicians who start out with validsymptom-malfunction matrices are more likely to resemblethe Bayesian processor. Subject's troubleshootingbehavior that was characterized as Bayesian correlatedwith higher class standing and longer time between tests.It is suggested that symptom-.malfunction matrix accuracycould be a training criterion. Students could be traineduntil they could produce a symptom-malfunction matrix toa high standard of physical validity.

7 pages, 3 references

Brooke, J.B.; Duncan, K.D.; Marshall, E.C. (1978) "InteractiveInstruction in Solving Fault Finding Problems." InternationalJournal Man-Machine Studies. 10:603-611.

Thirty subjects were divided into two groups which solved60 trials of fault diagnosis problems. One group wasgiven process information provided by the interactivesoftware system when an incorrect diagnosis was made. Atthat point during the interaction, feedback regarding thepossible failures was given to the subjects. Also, ifsubjects continued to ask for additional information whenthe solution was obvious, the system informed subjects ofthe fact that the failure could be identified. Thecontrol group had no feedback and were not informed aboutthe correctness of their diagnosis. Analysis of varianceshowed a significant effect of training on learning.


Brown, J.S.; Burton, R.R.; Bell, A.G. (1975) "SOPHIE: A StepToward Creating a Reactive Learning Environment." InternationalJournal of Man-Machine Studies. 7:675-696.

This paper describes a fully operational AI-CAI systemwhich incorporates artificial- intelligence techniques toperform question answering, hypothesis verification andtheory formation activities in the domain of electronictroubleshooting. Much of its logical or inferencingcapabilities are derived from uses of simulation modelsin conjunction with numerous procedural specialists. Thesystem also includes a highly tuned structural parser forallowing the student to communicate in natural language.Students can test components in an electronic circuit,develop hypotheses regarding the fault, and obtainfeedback concerning why a hypothesis is incorrect.Replacement of parts is allowed only when the student cananswer correctly queries about why he thinks the partsshould be replaced. Although the system is extremelylarge it is sufficiently fast to be thoroughly exercisedin a training or classroom environment.


1 -. ,- ' "',

Brown, J.S.; Rubinstein, R.; Burton, R. (1976) "ReactiveLearning Environment for Computer Assisted ElectronicsInstruction." Lowry Air Force Base, CO: AFHRL-TR-76-68.

The development of several new computer based strategiesfor teaching .troubleshooting principles to electronicstechnicians is described. The report documents anexperiment in which those materials were presented tostudent technicians to determine their attitudes towardthe techniques, and to determine whether the resultingtraining resulted in improvement of their technicalskills. Results in both cases were positive withstudents responding very favorably to the materials andwith their performance improving quantitatively andqualitatively after the instruction.

Seventeen college level students in an electronicscurriculum were subjects in an experiment for evaluatingalternative training and teaching methodologies ontroubleshooting behavior. Subjects were divided intofour groups, each group receiving a different sequence ofteaching and training activities. Most of the evaluationis qualitative and based on extensive interviews andquestionnaires. However, the fourth group wasadministered a pre-test for measuring basic understandingof power supply troubleshooting. This pre-test alsoincluded two faults to troubleshoot before the laboratorygames began. A post-test was administered to measure theeffect of computer-based activities on learning.Students' scores reflected their ability to make correctfault hypotheses given a set of symptoms. Improvement inall subjects' scores was found to be significant at the.05 level. Also, subjects experienced an averagedecrease in task time by 41%, significant at the .05level.

140 pages

!.' A I

Crawford, A.M.; Crawford, K.S. (1978) "Simulation ofOperational Equipment with a Computer-Based System: A Low CostTraining Technology." Human Factors. 20(2):215-224.

The feasibility and effectiveness of teaching performanceskills using a computer-based training (CBT) methodologywere investigated. Graphic simulations of the appearanceand functions of a system in an anti-submarine aircraftwere presented to students within an instructionalframework. The objective was to determine whether CBTcould be used for low cost, part-task training. Thesubjects were new co-pilot trainees for the S-3Aanti-submarine aircraft. The performance of the 22 CBTstudents, as measured on a high fidelity simulator, wascompared to that of 22 students who had gone throughconventional training consisting of workbook study andhands-on practice in the simulator. Results showed thatCBT students performed the necessary skills as wellbefore practice in the simulator as conventionallytrained students could after this practice. Performancemeasures were the number of tasks completed and the timeto completion. The authors felt that CBT was aneffective and low-cost means of part-task training ofcertain performance skills.


Crooks, W.H.; Kuppin, M.A.; Freedy, A. (1977) "Application ofAdaptive Decision Aiding Systems to Computer-AssistedInstruction: Adaptive Computerized Training System (ACTS)."Arlington, VA: U.S. Army Research Institute for the Behavioraland Social Sciences. PATR-1028-77-1.

This report describes the Adaptive Computerized TrainingSystem (ACTS) which combines the techniques of circuitsimulation, artificial intelligence, decision modeling,and adaptive computer-assisted instruction to providetraining in decision making. The ACTS incorporates anadaptive computer program which learns a student'sdiagnostic value structure and uses th'is structure totrain the student in practical decision making. Thisreport describes the development and operation of theACTS as it is applied to training electronicstroubleshooting.

Six college students with electronics backgrounds servedas subjects in solving 45 fault diagnosis problems.Subjects were divided into three groups receiving thefollowing initial aiding: 1) the expert's three bestalternatives were displayed to the subject and after thesubject's choice was made, additional feedback wasprovided by identifying the expert's best alternative;2) subjects first indicated alternatives before expert'salternatives were displayed; and 3) no feedback orindication of expert's choice. Results indicate thatthose who received aiding or feedback solved problems atlower costs than subjects who received neither aiding norfeedback. Practice was shown to have a positive effectfor all groups in terms of reducing costs.

54 pages

Elliott, T.K. (1966) "A Comparison of Three Methods forPresenting Procedural Troubleshooting Information." AerospaceMedical Research Laboratories, Wright-Patterson Air Force Base,Ohio. Report No. AMRL-TR-66-191.

The effects of subject aptitude and performance aide modeof presentation on the performance of procedural,between-stage troubleshooting tasks on a real piece ofelectronic equipment were studied. The study usednondecison aids presented in three modes, namely, anautomatic retrieval of visual information, an automaticretrieval of audio information, and a manual retrieval ofvisual information. Two aptitude groups (Air Forceelectronic index 40-60 and 75-90) with no previouselectronic training or experience were given from 3 to 5hours of task training before the experiment. Eachsubject solved 26 actual, and 11 synthetic problems. Thestudy indicated no difference in effectiveness of aidsusing visual representations. Both were superior toaudio presentations. The two aptitude groups wereequally effective in performing troubleshooting tasksusing nondecision aids. In comparison with a previousstudy using decision aids, the study showed nondecisionaids produced superior performance on the samebetween-stage troubleshooting problems using similarsubjects.


rii:

I

Elliott, T.K.; Joyce, R.P. (1971) "An Experimental Evaluationof a Method for Simplifying Electronic Maintenance." HumanFactors. 13(3):217-227.

Two groups of subjects solved the same set of 13troubleshooting and repair problems in sevensolid-state-circuit modules which contained as many asfive stages each. Both groups used the same hand toolsand test equipment. One group was composed of 41conventionally trained Air Force 5- and 7-leveltechnicians who normally maintain such equipment as partof their jobs. The technicians used the sametroubleshooting techniques they ordinarily used on theirjobs, and they were provided with a performance aidresembling an Air Force technical order. The other groupwas composed of 20 high-school students with no priortraining or experience in electronics. Their trainingfor this study consisted of a 12-hour course in the useof hand tools and test equipment and in the use of theproceduralized troubleshooting aid evaluated in thisstudy. The aid indicated which check to make based uponthe outcome of previous checks. Using the proceduralizedtroubleshooting aid, the high-school students tooksignificantly less troubleshooting time than did theexperienced technicians using normal techniques;however, the technicians requirdd significantly lessrepair time and made significantly fewer errors than didthe students. These differences may be accounted for inpart by differences in the tasks and scoring methods for

the two groups, and there were no differences between thetwo groups on a number of other measures. However, thedifference in training time and, therefore, cost oftraining between the two groups was so great as to

suggest the possibility that job-relevant training andproceduralization of the task can introduce substantialsavings, even after the cost of developing the specialperformance aids required by proceduralizedtroubleshooting is subtracted.


I

r ., ., -,-

Ely, J.H.; Hall, N.B.; Van Albert, C.E. (1960) "CodingElectronic Equipment to Facilitate Maintenance." IRE Transactionson Human Factors in Electronics. HFE-1(2):66-69.

The aim of this study was to improve maintenance ofelectronic equipment by detcrmining what information toplace on the equipment and developing techniques for itsdisplay. Trips to various communications installationswere made in order to examine equipment, observemaintenance activities, examine mAnuals, and interviewpersonnel. Detailed data were collected showing themarked variation between observed test point readings andthose called for in maintenance manuals. Recommendationswere developed concerning information to be displayed onprime electronic equipment, covering 1) designation offunctional groupings, 2) identification of signal paths,3) identification and sequence for using test points, and4) presentation of historical information for each testpoint. These recommendations were evaluated byincorporating them on an oscilloscope and evaluating theperformance of 18 technicians. Comparing this "coded"scope with an identical uncoded one showed thattroubleshooting time was reduced on the average toapproximately one-half that for the uncoded equipment.


Finch, C.R. (1971) "Troubleshooting Instruction inVocational-Technical Education via Dynamic Simulation."Pennsylvania Department of Education, Bureau of Vocational,Technical and Continuing Education. ERIC Document ReproductionService ED 055 204.

This study used a commercial real equipment simulator totrain automechanics for fault diagnosis. The researchalso explored the attitude of the students and theinstructors towards the simulator. The experimentalgroup received simulator practice, while the control

4 group received no treatment. Those students whopracticed on the simulator had better troubleshootingperformance on four of the six criteria measures. Theinstructors were satisfied with the simulator, and thestudents believed that it was at least as good astraditional classroom or laboratory instruction.


I

-. - .-. - -- .

Finch, C.R. (1972) "The Effectiveness of SelectedSelf-Instructional Approaches in Teaching Diagnostic ProblemSolving." The Journal of Educational Research. 65:219-223.

This study examined the effects of threeself-instructional approaches of teaching diagnosticproblem-solving (troubleshooting). Two of the approachesattempted to simulate student-equipment interaction viaprogrammed and text instruction while a third approachutilized actual equipment. After forty-five communitycollege students received an orientation to equipmentoperation, fifteen were randomly assigned to each of thethree treatments. Criterion variables includedtroubleshooting knowledge, troubleshooting performance,and attitude toward instruction received. Analysis ofcovariance indicated that the equipment approach wassuperior to non-equipment simulation approaches in termsof troubleshooting performance scores. Correlationalanalyses of criterion variables provided some evidencethat relationships between attitude and behavior weredependent upon the instructional approach used.


Freedy, A.; Crooks, W. H. (1975) Use of an Adaptive DecisioQModel in Computer-Assisted Tr-aining on ElectronicsTroulesh53ting. Woodland Hills, CA: Perceptio--ns Inc., Apri1975.

The report discusses a system for Computerized DecisionTraining (CDT). This system has the capability tocompare subjects' troubleshooting decisions with theoptimal as defined by "experts." The goal was to trainfor quality decision making rather than cookbooktroubleshooting. The simulations are computer generatedproblems applicable to electronic circuits. The problemsare presented on a CRT and permit the subject to: checksymptoms, measure a component value, replace a module, ordeclare circuit to be fault free. The adaptive decisionmodel was found to converge on the decision behavior ofthe students who quickly settle into consistent decisionstrategies. The authors' conclusion notes that there isa need for considerably more research in this area.


!I

* K~ ..

Glaser, R.; Damrin, D.E.; Gardner, F.M. (1954) "The Tab Item:A Technique for the Measurement of Proficiency in DiagnosticProblem Solving Tasks." Educational and PsychologicalMeasurement. 14:283-293.

This work describes a method of fault diagnosis trainingrelying on the "tab item." The tab item is a foldoutcardboard flow chart which allows the troubleshooter togather information by pulling tabs. This system providesthe following: A symptom, check procedures, and a listof possible failures. This system therefore attmepts toreproduce the troubleshooting situation on paper.Measures for scoring troubleshooting performance whichare discussed are: number of tests performed, number ofredundant tests, test relevancy, and test adequacy.Although this article offers no empirical evidence of theinstructional value of the tab item, it does speculate asto its potential value across a broad spectrum oftechnical training. Addit-ional uses of the Tab Test canbe found in Standlee (1956) and Fattu (1956).

10 pages

* "~

Glass, A.A. (1967) "Problem-Solving Techniques andTroubleshooting Simulators in Training Electronic Repairmen."Doctoral Thesis. Columbia University.

The purpose of this project was to study the practicaland operational application of problem-solving conceptsby introducing into the regular training of militaryelectronic repairmen two experimental methods of teachingtroubleshooting and to evaluate their effectiveness inpromoting success. These two experimental methods were:1. a series of recorded lectures dealing with problemsolving methods, and 2. a series of paper and penciltroubleshooting simulators to provide the opportunity toapply problem solving techniques. There were twosimulators, one commercial and the other custom-built.The main criteria for evaluation of the experimentalvariables were performance tests on actual electronicequipment (N:240). The method for administering andscoring the performance tests was the result of extensivepilot studies. The essence of the scoring method was toscore for sequence of logic in troubleshooting, time, andsolution success. The conclusions were: 1. Subjectsreceiving problem-solving lectures were superior onperformance test to subjects not receiving them. 2.Subjects using simulators were superior on performancecriteria tests to those not using the devices, but onlywhen the custom-built device was used. 3. Subjectsreceiving a combination of lectures and devices weresuperior on performance criteria tests to subjects notreceiving the combination. 4. The use of thecustom-built simulator produced superior performancecriteria test scores as compared to subjects using thecommercial version, only when the lectures were used aswell.


* ~*.*; I

.74

Goldbeck, R.A.; Bernstein, B.B.; Hillix, W.A.; Marx, M.A.(1957) "Application of the Half-Split Technique toProblem-Solving Tasks." Journal of Experimental Psychology.53(5):330-338.

An experiment was designed (1) to test the hypothesisthat the half-split instruction increased efficiency forsimple problems and (2) to determine half-splitefficiency for more complex problems. A secondexperiment was designed to determine whether additionalpractice, different methods of instruction, and morecomplex training materials would produce an improvementin the subject's ability to arrive at a set ofalternatives, and, if so, whether the half-split methodwould then be more efficient.

In experiment I, 90 male university students participatedas subjects and were instructed to replace a faulty unitin a troubleshooting task. Thirty were assigned problemsto each of three complexity levels. Based on scoresobtained from a problem solving test, subjects wererandomly assigned to a control group and a group whichreceived explicit instructions regarding the applicationof the half-split technique. Solution time for thehalf-split group was significantly longer than thecontrol group. Increasing complexity resulted in anincrease in the number of tests only for those subjectswho scored low on the problem solving test. Resultssuggest that the half-split technique is superior undercertain conditions, such as with diagrams of lowcomplexity where subjects apparently had littledifficulty in finding the trouble-possible units, andwhere subjects avoided incorrect replacements.

Experiment II was an attempt to improve the effectivenessof the half-split method by improving instruction andgiving extended practice. Only problems of the highestlevel of complexity were used. Forty male-universitystudents participated and were randomly assigned to fourtraining groups. The deductive half-split group was likethe previous half-split group but given more practice.The instructions and practice given the deductivehalf-split group were more effective in reducing thenumber of tests than were the less extensive instructions

* in Experiment I. Results of both experiments indicatethat the empirical superiority of the half-splittechnique is a function of the nature of the task, andthe degree and type of training given the troubleshooter.From these experiments, it is now clear that futureresearch on training methods should emphasize the morebasic deductive phase rather than the relativelymechanical half-splitting aspect of the process.


Landa, S. (1972) CATTS: Computer-Aided Training inTroubleshooting. (RO18-PR) Santa Monica, CA: Rand Corporation.ERIC Document Reproduction Service No. ED 066 888.

The Rand Corporation's Programmer-Oriented GraphicsOperation (POGO) was used in the design, implementationand testing of a computer-assisted instruction course totrain airmen in malfunction diagnosis - CATTS (ComputerAided Training in Troubleshooting). The design of thecourse attempted to reduce the problems of computergraphics for both instructor and student. Theobservations of Air Force instructors using the systemsuggest that the system is workable. No empiricalevidence was gathered. The report emphasizes the POGOsoftware more than troubleshooting.


Mallory, W.J.; Elliott, T.K. (December 1978) "MeasuringTroubleshooting Skills Using Hardware-Free Simulation." Air ForceHuman Resources Laboratory, AFHRL-TR-78-47.

The hypothesis formulated was that troubleshootingperformance on simulation would be predictive of the sametasks on real equipment. Sixteen instructors and fifteenbeginning students in a precision measurement equipmentcourse were given the same problems with two treatmentlevels. Problems were solved for criterion of job sampletest (JST) and simulation or symbolic performance test(SPT). The task in each case was to locate the fault atthe stage level rather than component level. Performancemeasures included mean percent correct solution, time tocomplete problems, and number of steps to completion.Statistically significant correlations indicate a highdegree of similarity between the job sample test andsymbolic performance test. An algorithm is presentedwhich is used to evaluate problem difficulty.Correlation of the hypothesis problem difficulty withempirical problem difficulty (obtained by using testresults of subjects) did not produce significantagreement. It was noted that the problem difficultyalgorithm was insensitive to feedback loops whichoccurred in some of the problems and would be modified inthe future.


Mills, R.G. (1971) "Probability Processing and DiagnosticSearch: 20 Alternatives, 500 Trials." Psychonomic Science.24(6):289-292.

Ss' capabilities for processing probability informationin performing a diagnostic search task similar totroubleshooting is explored. Six Ss searched (queried)among 20 components in a series circuit to isolate asingle component failure per trial. Failures weregenerated according to 10 alternativeprobability-of-component-failure distributions (D), eachN = 500. Prior to performing their search on a trial, Sspredicted the failure for that trial. Of primaryinterest was the influence of varying D on predictionsand search performance. Results indicate that: (1)distributions of predictions tend-to match actual D inshape and location but exhibit certain characteristics;(2) probability of correct prediction is low and isinfluenced by D; (3) search strategies are suboptimal(The optimal solution required an average of 14% fewertests.) but "logical" and to some extent predictable;and (4) in some instances, search strategy appears to berelated to predictions.


Osborn, W.C. (1970) "An Approach to the Development of SyntheticPerformance Tests For Use in Training Evaluation." presented atthe 12th Annual Military Testing Association Conference, FrenchLick, Indiana.

This paper describes an approach to the selection ofinexpensive alternatives to fully job-relevantperformance tests. Based on the concept of contentvalidity, the approach is to utilize the skill andknowledge required of the trainee as a criterion forassessing the cost and job-relevance of fabricatedalternatives to performance tests. Two job tasks areanalyzed to illustrate the concept.

6 pages

4.

AUX

IL. I

Rasmussen, J.; Jensen, A. (1974) "Mental Procedures inReal-LIfe Tasks: A Case Study of Electronic Troubleshooting."Ergonomics, 17(3):293-307.

An experiment was conducted at a nuclear researchinstitute to investigate troubleshooting approaches usedby skilled electronics repairmen during on-the-job tasks.A verbal record of the repairmen's strategies wascollected for data analysis. Codes were applied todescribe the heuristics employed: 1) topographic search- use the topographic location of measuring points; 2)functional search - use the normal functional relationbetween a feature in the system's response and a specificpart in the system; and 3) search by evaluation of afault. This last heuristic implies the human is using amental model of the system relating changes to internalsignals, parts or components to the changes observed inthe system response. Results suggest that proceduresconsidered unsystematic in a laboratory study may turnout to be rational and systematic when one listens to theindividual's rationalization. Also, it was found thatverbal records appear to be useful for recording rapidsequences of simple decisions based upon informationallyredundant observations.


Rasmussen, J. (1978) Notes on Diagnostic Strategies in ProcessPlant Environment. Roskilde, Denmark: Riso National Laboratory,R iso-M-T83.

In the report are discussed some aspects of stateidentification and diagnosis in process plant controlwhich must be considered in connection with automaticdisturbance analysis and man-machine interface systems.

The content of the diagnostic process depends upon theoverall goal of the diagnostician - whether it is toprotect the plant, to maintain operation or to repair,and it may not necessarily include a determination of thecause itself. Important aspects of diagnosis includecritical variables and causal flow paths which areintimately related to the paths along which events andchanges propagate through the system, i.e. to the flowof energy, matter and information which together formcomplex, interacting flow structures.

In the process plant environment, a diagnostic taskimplies a search to identify a change from a normal orplanned plant state. Several elementary strategies canbe identified. In the report a distinction is drawnbetween two main groups - topographic search strategies,performed as search through the system with reference toa model of normal plant state; and symptomatic searchstrategies, performed as a search through a library ofabnormal state models with reference to the actual plantstate.

Typical properties of the different strategies arediscussed such as processing capacity requirements anddependence upon a priori analysis.

The role of the elementary strategies in the overalldiagnostic task is not discussed since this is consideredpossible only with reference to selected scenariosdescribing realistic and comnplex real life situationsarising from disturbed plant operations. The aim of thereport is to identify and formulate some of the issueswhich should be taken into account when creating suchscenarios based on careful field studies and analysis ofincident reports. Based on these scenarios, differentoverall diagnostic strategies can then be formulated andtested by simulator experiments.


* I

Rigney, J.W. (1973) "A Discussion of Behavioral TechnologyLaboratories CAI Projects in Relation to a CAI Test-Bed Concept."Behavioral Technology Laboratories, Report No. 71, University ofSouthern California, Los Angeles, CA.

The elements of an instructional system are discussed and.some literature bearing on the functions of theseelements is reviewed, with the objectives of stimulatingthought about an instructional system as a CAI test-bedand of pointing out some results of laboratory research,particularly in cognitive psychology, that are believedto be worthy of the instructional technologist'sattention.

In the second part of the report, selected parts of CAIprojects underway at the Behavioral TechnologyLaboratories are described in relation to instructionalsystem elements, to give an overview of this research.The projects include: work on computer grahics to (1)simulate front-panel topography and functionalorganization of circu-its (2) facilitate interpretationof relative motion, and (3) create visual analogies forabstract concepts and processes to serve as the basis forstudents to develop their own internal representations;work on new methods for recording cortical evokedpotentials and correlating them with learning and memoryprocesses; work on a dynamic programming model foradaptive control of problem-solving types of CAI; andwork on three-dimensional mock-ups of electronicequipment to be placed on-line with a CAI system.


r .

Rigney, J.W.; Towne, D.M. (1969) "Computer Techniques forAnalyzing the Microstructure of Serial-Action Work in Industry."Human Factors. 11(2):113-122.

Three computer-based techniques for analyzing andsimulating serial-action tasks are described. The first,called BETS (a Bayesian model), measured the efficiency,in terms of -expected information, of tests made bytechnicians who -were troubleshooting malfunctioningelectronic circuits. It computed efficiency ratios for atechnician's test sequences in comparison to optimalsequences. The second method, ARMAN, generated detailedtime-and-motion analyses from gross descriptions ofserial-action tasks and man-machine interfaces andcomputed the time costs of these tasks.

The third technique incorporates a general model of theaction-goal structure of serial-action work. It operateson this structure with special functions composed in asymbol manipulation language. This program, calledTASKSYM, can generate all alternative correct ways toaccomplish serial-action work and can track a subjectthrough the performance of this work. The model includesan anti-goal structure which identifies action sequencesleading to catastrophic error, e.g., electrical damage tothe equipment or shock hazard to the performer. TASKSYMis extended with LISP to develop TASKTEACH which providesfeedback on tests and errors, and offers various levelsof help.


L++%• ),

Rigney, J.W.; Towne, D.M. (1970) "TASKTEACH: A Method forComputer-Assisted Performance Training." Human Factors.12 (3 ): 285-296.

A method for using a computer time-sharing system toassist the learning of serial tasks, from operatingequipment through troubleshooting, is described. Themethod is based on mediational theory. The currentresurgence of interest in mediational theory inpsychological research is noted, and parallels betweenexperimental strategies and instructional strategies forevoking and manipulating mediating processes are pointedout. Categories of processes which mediate performanceof serial tasks are described. Procedures forfacilitating the learning of these processes areimplemented by a computer program, called TASKTEACH. Theprogram sustains the student's performance of complexserial tasks by giving him variable amounts of supportwhile helping him learn and organize the processes whichmediate his performance of these tasks. The programgenerates output to the student during the learningsession by processing short lists and the prior responsesof the student. The lists, which are input to theprogram replace the conventional frame-by-framedescription of an instructional sequence written in a"CAI language."


* 1'. i

Rigney, J.W.; Towne, D.M.; King, C.A.; Moran, P.J. (1978)"Field Evaluation of the Generalized MaintenanceTrainer-Simulator: I. Fleet Communications System." BehavioralTechnology Laboratories, Report No. 89, University of SouthernCalifornia, Los Angeles, CA.

The Generalized Maintenance Trainer-Simulator (GMTS) is aconcept for giving students in Class C schools intensivepractice in troubleshooting equipment and systems taughtin those schools. It can be used for any device in whichsignal paths and their relationships to controls,indicators, and test points can be defined. The GMTSuses generative CAI. That is, it generates theinteraction with the student by referring to his lastinputs and to its stored history of interactions with himup to that point. To the extent that these individualstudent histories constitute models of individualstudents, GMTS constructs a model of each student tointeract with that student. In addition to thesefeatures, GMTS is uniquely suitable for use in Class Cschool training. The computer program that implementsthe instructional system is indifferent to the specificequipment being taught. What specific equipment that issimulated by the GMTS is determined by loading two databases for that equipment: one describing essentialinternal features of the equipment and the otherdescribing the external features.

This is a report of a field evaluation of the GMTSapplied to systems level troubleshooting in the UHFcommunications side of the Fleet Communications System.Twenty Class A school students waiting to enter C schoolspracticed solving thirty-five troubleshooting 'problems.Results were generally positive. With respect topractice, the students became uniformly more fluent attroubleshooting; mean times to solve a problem weredecreased by a factor of two,.and standard deviations ofthese time were decreased by a factor of five. Similarreductions existed, for average number of extrareplacements per problem. Also, students' attitudestoward the trainer were generally favorable. Anadditional field trial is underway, using the AN/SPA-66radar repeater as the subject matter.


Robinson, J.E.,Jr.; Deutsch, W.E.; Rogers, J.G. (1970) "TheField Maintenance Interface Between Human Engineering andMaintainability Engineering.,' Human Factors. 12(3):253-259.

Field maintenance actions reported in a deployed airdefense system were reviewed to determine and quantify(1) the nature of operator and maintainer influences onsystem malfunctions, and (2) the nature of man-machineproblems being encountered by system personnel in actualdiagnosis and repair activities. Maintenance eventsdescribed in 213 problem reports from the field werecategorized according to three probable sources ofmalfunction: primarily human, combination human andequipment, and primarily equipment. Human involvement insystem malfunctions was clearly established in 25% of theevents reported. Additional information contained in 21documents analyzing system maintenance was examined. Thenature of human influence on system malfunction isdescribed by examples of "unfortunate" field maintenance,"ordinary" field maintenance, and troubleshootingdeficiencies. Troubleshooting deficiencies were cited asa frequent problem observed by equipment contractors whohad more knowledge and understanding than the technicalpersonnel. The nature of reported man-machine problemshighlighted the fact that human engineering participationin establishing maintainability design requirements canserve to make equipment that will be suitable to thepresence and actions of human users.


Rouse, W.B., (1978) "Human Problem Solving Performance in a FaultDiagnosis Task." IEEE Transactions on Systems, Man, andCybernetics. SMC-8(4T -58-27-.

It is proposed that humans in automated systems will beasked to assume the role of troubleshooter or problemsolver and that problems which they will be asked tosolve in such systems will not be amenable to rotesolution. The design of visual displays for problemsolving in such situations is considered, and the resultsof two experimental investigations (N=20) of humanproblem solving performance in the diagnosis of faults ingraphically displayed network problems are discussed.The effects of problem size (9,25,and 49 components),forced-pacing, computer aiding, and training areconsidered. Results indicate that human performancedeviates from optimality as problem size increases.Forced-pacing appears to cause the human to adopt fairlybrute force strategies, as compared to those adopted inself-paced situations. Computer aiding substantiallylessens the number of mistaken diagnoses by performingthe bookkeeping portions of the task. Further, forself-paced situations, skills developed withcomputer-aided training are transferred to unaidedsituations.


Rouse, W.B., (1978) "A Model of Human Decisionmaking in a FaultDiagnosis Task." IEEE Transactions on Systems, Man, andCybernetics. SMC-8(5T-77-3b1.

Utilizing elementary concepts from the theory of fuzzysets as well as several nonfuzzy heuristics, a model ispresented of human decisionmaking in the task oftroubleshooting graphically displayed networks. Theperformance of the model is compared to the results oftwo previously reported experimental studies. Theability of the model to represent human decision makingas a function of network size, forced-pacing, andcomputer aiding is considered. The model is used tosupport the hypothesis that humans tend to discount thevalue of information about what has not failed insearching for the sources of malfunctions.


i "

- j

Rouse, W.B. (1979) "A Model of Human Decision Making in FaultDiagnosis Tasks that Include Feedback and Redundancy." IEEETransactions on Systems, Man, and Cybernetics. SMC-9(4):237-21.

A previously reported model of human decision making infault diagnosis tasks is extended to include situationswhere feedback and component redundancy are importantconsiderations. The model is based on concepts from thetheory of fuzzy sets. The results of an experiment withhuman subjects are reported and used to estimate theparameters of the model. The model is used to illustratethe notion that subjects who discount the importance offeedback loops tend to have considerable difficulty whenthey encounter problems having ajlarge number of loops.


Rouse, W.B. (1979) "Problem Solving Performance of MaintenanceTrainees in a Fault Diagnosis Task." Human Factors.21(2):195-203.

Forty trainees in an FAA certificate program participatedin an experimental study of troubleshooting ofgraphically displayed networks. The effects of networksize, computer aiding, and training were considered. Itwas found that performance degraded as network sizeincreased, improved with the use of computer aiding, andthat skills developed with computer aiding weretransferred to the unaided situation.


Rouse, W.B. (1979) "Problem Solving Performance of FirstSemester Maintenance Trainees in Two Fault Diagnosis Tasks."Human Factors. 21(5).

Forty-eight first semester trainees in an FAA certificateprogram participated in an experimental study oftroubleshooting of two different types of graphicallydisplayed networks. The effects of network size, redundancy,feedback, computer aiding, and training were considered. Itwas found that performance degraded as network sizeincreased, degraded as the level of feedback was reduced,improved with the use of computer aiding, and that skillsdeveloped with computer aiding in one task were transferredto the other task.

4 references

LIS W~k

Rouse, W.B.; Rouse, S.H. (1979) "Measures of Complexity ofFault Diagnosi3 Tasks." to appear in IEEE Transactions on System,Man, and Cybernetics, SMC-9(11).

The literature of complexity is reviewed and thedistinction between perceptual complexity and problemsolving complexity is discussed. Within the context oftwo particular fault diagnosis tasks, four measures ofcomplexity are considered. These measures are evaluatedusing data from two previously reported experiments whichemployed eighty-eight subjects. It is shown that twoparticular measures of complexity, one based oninformation theory and the other based on the number ofrelevant relationships within the problem, are reasonablygood predictors of human performance in fault diagnosistasks. The success of these measures is explained by thefact that they incorporate the human's understanding ofthe problem and specific solution strategy as well asproperties of the problem itself.

54 references

Saltz, E.; Moore, J.V. (1953) A Preliminary Investigation ofTroubleshooting. Technical Report 53-2. Lackland Air Forceease, Texas: Human Resources Research Center.

Troubleshooting was investigated using radar,reciprocating engines, and remote control turrets as thehardwa-e for malfunction analysis' observation. Thediscrete types of equipment were chosen in an effort topromote generalizability. The following hypotheses wereformulated and tested:

1. Good troubleshooters know more about thefunctioning of the equipment upon which theywork than do Ooor troubleshooters.

2. Good and poor troubleshooters differ inprevious experience.

3. Good and poor troubleshooters differ inintelligence.


LW fU _ Ail ' i

Saupe, J.L. (1954) Troubleshooting Electronic Equipment: AnEmpirical Approach to the Identification of Certain Requirementsof a Maintenance O-cu--ation. Urbana, IL: University OfTtliiiois.

The study attempts to identify the criteria forproficiency in a troubleshooting situation. The studyprovides "leads for the construction cf proficiency testswhich are important aspects of the efficient selection,training, and distribution of maintenance personnel." Thegoals of the research were:

1. to develop a psychological model by whichto view troubleshooting behavior.

2. to test certain hypotheses taken from this modelrelative to aspects of the troubleshootingprocess

3. to illustrate an empirical approach to theidentification of important objectivesof training programs

Subject performance was evaluated on a simulated radioreceiver and on a paper and pencil test. Empiricalresults were presented.

Standlee, L.S.; Bilinski, C. (1966) "Note on Simulated Vs.Actual Electronic Troubleshooting Performance." Perceptual andMotor Skills. 23:532.

This is a brief note on electronic troubleshootingperformance. It discusses the two basic approaches tomeasuring this performance (1) measurement of performanceon simulated and (2) actual electronic equipment. 114subjects (Naval electronics technicians) were dividedinto two groups, one group troubleshooting with thesimulator, and the other with actual equipment. Resultsindicated poor troublshooting performance with thesimulator and proficient performance with the actualequipment.

1 page

..... ' ' - * . , -.. . ,. :". .

Steinemann, J.H., (1966) "Comparison of Performance on AnalogousSimulated and Actual Troubleshooting Tasks." U.S. NavalPersonnel Research Activity, Memorandum SRM 67-1, July 1966.

This investigation compared the performance of a group ofsubjects assessed on a simulated troubleshooting task andon the identical actual troubleshooting task using realequipment. Subjects were 14 students in the experimentaltraining program for Electronics Technicians, conductedby the Navy Training Research Laboratory, San Diego.

Analysis of results revealed that the simulatedperformance measure did not provide a valid estimate ofperformance proficiency on the actual task. Obtainednegative inter-test correlations indicate that simulatedtest results would actually be misleading in terms ofestimating actual performance scores. In addition toperformance score discrepancies, there were observabledifferences in specific performance procedures andoverall troubleshooting strategy attributable to thedifferences in test mode, Although the research designwas weak, the evidence strongly -suggests caution inassuming that a simulated performance measure, even withconsiderable face validity, will provide a valid estimateof actual performance on a common task.

12 pages

C -

Stolurow, L.M.; Bergum, B.; Hodgson, T.; Silva, J. (1955)"The Efficient Course of Actioh in 'Troubleshooting' as a JointFunction of Probability and Cost." Educational and PsychologicalMeasurement, 15(4):462-477.

This research focuses on the development of aprobabilistic model of troubleshooting behavior whichprescribes an efficient course of action in locatingdefects based on probabilities of failures and work timegiven particular symptoms. Data were obtained frommaintenance records for 46 four-engine aircraft to obtainthe probability of each possible cause of a particularset of diagnostic signs. Mean repair times were alsocompleted for each type of repair. A complete method anda simplified method are described for determining anefficient course of action to be followed in reducing aset of possible causes of malfunction to the actualcause. The simplified method was proposed as analternative to overcome the cumbersome calculations ofthe complete method. The simplified method uses theratio of mean repair time to probability of failure todetermine the course of action to follow in reducing thelist of possible troubles. Results of the analyses ofmaintenance records were sufficiently encouraging tosuggest that further research of this type be done for avariety of complex devices.


Towne, D.M. (1979) "The Automated Integration of Training andAiding Information for the Operator/Technician." presented to theThird Biennial Maintenance Training and Aiding Conference.Orlando, Fla.

A responsive and intelligent device which can provideboth training information and job performance aiding canbe extremely efficient in utilization of hardware,software, and so-called course-ware. The hardwarecomponents useful for presenting automated instructionare essentially the same as. those required to aid anoperator or technician, viz., a versitile and fastmicroprocessor CPU, ample random access memory, amplebulk storage, and appropriate input-output media. Whilethe software driving such a device may be a collection ofrelatively special-purpose routines designed to satisfyparticular training or aiding needs, a large portion ofthis software may be common utility functions whichrespond to specific requests of the user or serve tomanipulate data about the target equipment or data aboutthe operator/technician. Finally, the data base whichcharacterizes a target equipment can be designed in sucha way that both training and aiding requirements are metwithout duplication or inefficiency of preparationeffort.

Under a contract with Naval Training Equipment Center,Behavioral Technology Laboratories has assembled thehardware elements of such a system, designed the targetequipment, and programmed routines for training noviceoperator/technicians and aiding equipment operation andmaintenance.


r1

' t :'"'

Wescourt, K.T.; Hemphill, L. (1978) "Representing and TeachingKnowledge for Troubleshooting/Debugging." Institute forMathematical Studies in the Social Sciences, Report No. 292,Stanford University, Stanford, CA.

An extensive review of the troubleshooting/debuggingliterature is covered from an artificial intelligenceperspective. The authors were interested in measuringthe effect of training, that is, exposure to specificdebugging heuristics, on problem solving performance.Twelve subjects with relatively little programmingexperience participated. They were currently enrolled ina BASIC programming course. Subjects were randomlyassigned to two groups (aided and unaided) based on acomputer programmer aptitude battery. Aided subjectscould refer to a text tutorial during problem solvingsessions. The chronology of subjects' actions werejudged according to 5 criteria relating to implementationof heuristics contained in the tutorial guidelines andsuccess of attempted solution. Successful debugginggrammars that would capture the subjects' strategies werenot developed. Subjective judgments of participants onthe research team were necessary to identify the extentto which each measure applied to subjects' strategies.Results indicated poor performance of both groups insolving problems. After the experiment large differenceswere found in-post test scores attained on the computerprogrammer aptitude- battery, and thus by chance, subjectsassigned to the non-aided group had become betterprogrammers since completing their course in BASIC. Inconclusion, there were no strong results to show that thetutorial aid affected implementation of heuristics. Itis suggested that a pedagogy is needed for teachingprocedural rather than declarative knowledge.


Ow..

SECONDARY RESEARCH

Czeh, R.S. (1957) "Studies of Complex Behavior and theirRelation to Troubleshooting in Electronic Equipment." Alexandria,VA: Human Resources Research Organization.

This report is a review of the literature concernedeither directly or .indirectly with the troubleshooting ofelectronic equipment. Specifically, this review presentsthe most pertinent findings in the areas of problemsolving and concept formation and indicates the manner inwhich these findings have impinged upon, or could impingeupon thinking about troubleshooting. The literature ontroubleshooting is discussed and the final sectionoutlines a research task designed to develop anefficient, generally applicable procedure fortroubleshooting electronic, and to develop methods forteaching the procedure.


Edwards, J.; Norton, S.; Taylor, S. Weiss, M.; Dusseldorp, R.(November 1975) "How Effective is CAI? A Review of theResearch." Educational Leadership, 33: 147-153.

This article summarizes many research studies thatinvolved Computer Assisted Instruction. The CAI modesdiscussed are: drill and practice, problem-solving,simulation, and tutorial. The areas of CAI researchdiscussed are:

1. CAI as a summplement to traditional instruction2. CAI as a substitute for traditional instruction3. Effectiveness of different modes of CAI4. Comparison of CMI with other nontraditional methods

of instruction5. Compression of time6. Retention of learning with CAI7. Effectiveness according to ability levels

This literature review referred to 9 studies that showedthat the combination of CAI and traditional instructionwas distinctly better than traditional instruction only.As a substitute for traditional instruction, the findingswere mixed with 9 studies finding CAI superior and 8studies finding traditional instruction better. In allcases CAI was effective in reducing training time.


Fattu, N.A. (1956) A Catalog of Troubleshooting Tests. IndianaUniversity: Institute of EducatioinalResearch.

This catalog contains examples of various kinds oftroubleshooting tests. The justification for a catalogof troubleshooting tests is that it summarizes andillustrates materials assembled from widely scatteredsources and makes them available to individuals who needthe information but do not have time to assemble it.This catalog attempts to be representative rather thanexhaustive.

Tests are reported in the following groups:

1. on-the job measures2. performance tests3. simulator tests4. ARC or "tab" tests5. pencil-paper and oral-interview tests.

The major difference between the troubleshooting researchof the fifties and the troubleshooting research of theseventies is the use of computers for data gathering inthe later efforts.


: .; ,I

Fink, C.D.; Shriver, E.L. (July 1978) "Simulators forMaintenance Training." Air Force Human Resources Laboratory,AFHRL-TR-78-27.

This report reviews past and present applications ofsimulation to maintenance training. Emphasis is ondescribing issues, problems and areas for future researchas identified by recent authors. There is littleevidence from the literature regarding the degree towhich simulators promote tradsfer of training to thefield. Thus far, transfer research has demonstrated thatsimulators can increase proficiency during and at the endof resident training. A variety of research has alsodemonstrated that major variations in resident trainingcannot be correlated with job proficiency after a year ormore in the field. It still remains to be shownconclusively that transfer of training affectsperformance during the first weeks and months on the job.If transfer does occur, it should have its most important

and noticeable impact during these weeks. There is aneed, therefore, for studies which examine therelationship between resident training, especially thatsupported by simulators, and on the job proficiencyduring the first one to three months on the job.Allocating the position of simulators and actualequipment systems in maintenance training requiresresearch investigating the optimal sequence of tasks andtraining aids for maintenance training.


I

Foley, J.P., Jr. (1977) "Overview of Advanced Systems DivisionCriterion Research (Maintenance)." Air Force Human ResourcesLaboratory, AFHRL-TR-77-77.

A prime but seldom considered cause of the current highmaintenance cost of DOD hardware, and thus the highownership cost of systems, is the current criteria usedby personnel systems to select, train, assign, andpromote maintenance personnel. The current criteriaemphasize the ability to obtain high scores onpaper-and-pencil theory and job knowledge tests. Thispaper summarizes the many studies which indicate thatsuch tests have little demonstrated relationship toability to perform job tasks. Table 1 summarizes thecurrent research on performance measurement and indicatesfor each citation the low correlations obtained in thoseexperiments for various maintenance tasks andpaper-and-pencil theory tests, job knowledge tests, andschool grades. Based on an experiment conducted in thepast by AFHRL, it was shown that graphic symbolic testswere more promising than symbolic tests in measuring jobtask performance. A maintenance task taxonomy isdiscussed and a hierarchy of dependencies is developedfor electronic maintenance tasks. This conceptualizationis considered necessary before identifying the relevantcriteria in job task performance tests.


King, W. (1978) "New Concepts in Maintenance Training." AviationEngineering and Maintenance, 2 (6): 24-26.

This article discusses the present status of Navymaintenance training in light of recent developments inCAI (Computer Assisted Instruction). The advantages of2-D computer simulators and 3-D real equipment simulatorsfor troubleshooting is the main focus of example. Theauthor presents monetary comparisons between actualequipment trainers (AET) and simulators of both thecomputer and real equipment type. The best example isthe cost comparison for the training of radio roomtechnicians for the Trident submarine. The AET costs 20million dollars compared to a 2 million dollar computersimulator. King believes that the utilization ofcomputer simulations for maintenance training has thepotential to surpass the savings offered by simulatorsfor flight training.

3 pages

Standlee, L.S.; Popham, W.J.; Fattu, N.A. (1956) A Review ofTroubleshooting Research. Indiana University: Tnstitute -3Educational Research.

This report is an annotated bibliography of thetroubleshooting literature of the fifties. It contains abrief description of the work of nearly one hundredauthors. The report is indexed with the following terms:general discussion, proficiency assessment techniques,training equipment or materials, analysis of jobactivities, evaluation of maintenance personnel,experimentation, bibliography. This report is invaluableto persons interested in the early efforts of faultdiagnosis research.


Am

!-1

'?'

};~

*

MANUALS OF ACTUAL FAULT DIAGNOSIS OR TRAINING IN FAULT DIAGNOSIS

AVCO Lycoming. (1975) "AVCO Lycoming Reciprocating EngineTroubleshooting Guide." Williamsport, PA: SSP-475.

A manufacturer published list of problems, causes, andsolutions that represents the more common and recurringproblems encountered by mechanics when troubleshooting.The list is general and refers to turbocharged andnormally aspirated engines. This manual is written forthe professional mechanic and assumes that shop manualsand proper tools are available.

47 pages

Bean, R. (1974) Petersen's Basic Automotive Troubleshooting.Los Angeles,CA: Petersen Publi s11 ng Go.

This manual is a very basic guide for the novice ordo-it-yourself automotive enthusiast. It includes aneasy to follow symptom-cause format with illustrations.The book is divided into such chapters as the following:Troubleshooting the Brakes; Troubleshooting the IgnitionSystem; Troublshooting Automatic Transmissions; EngineOil Pressure Problems; and Troubleshooting with a Vacuumor Compression Gauge. Simple logic trees are includedfor the starting, ignition and fuel systems.

144 pag'es

k4

!J

Chilton Book Co. (1972) "Troubleshooting." Chilton's MustangRepair & Tune-Up Guide. Radnor, PA: Chilton Book Company;

A section of the manual is designed to aid in the rapiddiagnosis of engine problems. The systematic format isused to diagnose problems ranging from engine startingdifficulties to the need for engine overhaul. It isassumed that the user is equipped with basic hand toolsand test equipment (tach-dwell meter, timing light,voltmeter, and ohmmeter). Troubleshooting is dividedinto two sections. The first, General Diagnosis, is usedto locate the problem area. In the second, SpecificDiagnosis, the problem is systematically evaluated. Theformat presents a malfunction system followed by a listof tests and procedures. The proper results of the testare provided in an effort to aid the troubleshooter.

15 pages

Glenn, H.T. (1971) Glenn's Auto Troubleshooting Guide.Philadelphia, PA: Chilton Book Co.

This manual is another example of a popular book writtenfor the automotive enthusiast rather than the experiencedmechanic. The specific areas that it covers are:emergencies, hard starting, engine failure, excessive oilconsumption, engine noises, overheating, fuel systemtrouble, electrical system failure, ignition systemfailure, drive, transmission, and rear axle trouble, andpower brake and power steering failure.

98 pages

Howard W. Sams & Co. (1973) Tune-up Service with Testing,Diagnosis and Troubleshooting Indianapolis, IN Ho d W. ams

& Co., Inc. 2nd Edition.

This manual covers the latest emission control systems,electronic ignition systems, electronic test equipment,and conventional engine analysis equipment. Thetroubleshooting section presents the familiarsymptom/cause format.

72 pages

l , . ... .... .. .. ... ...

Mitchell Manuals, Inc. (1977) Diagnostic Guide. San Diego, CA:Mitchell Manuals, Inc.

The book contains information designed to aid theprofessional mechanic in diagnosis and repair of unusualautomotive problems. Since many malfunctions are causedby incorrect, inoperative, defective or missingcomponents, attention is focused on these problems andconditions. All information presented has been carefullycompiled, primarily from factory service bulletins,pinpointing the cause of specific malfunctions. Thisdiagnostic guide is designed to help the mechanic avoidpotential problems and save time.

The book is arranged for quick reference in findingparticular information. The book's main divisions aredone by vehicle manufacturers: American Motors, Buick,Cadillac, etc. Within each of the divisions there arethree areas of information:

1) troubleshooting and diagnosis2) jacking, hoisting and towing3) hood latch and safety release locations

142 pages

General Motors Corporation. (1978) Diagnosis and Repair Manual -

Easy to Follow Graphic Troubleshooting Procedures. Southfield,MI: General Motors Corporation.

This manual contains graphic mode diagnosis and repaircharts. They are a relatively new approach tosystematic diagnosis and repair. The charts are easy touse since they require minimal reading. The sketches ofexact engine parts allows the mechanic to proceed towork without reading a manual, although on complexrepairs the book emphasizes that a manual should beused.

These diagnosis procedures apply to various automotivesystems in most 1970 thru current GM cars and light-dutytrucks-, with some procedures applying to even earliermodel years. The procedures are based on the fact thatthe vehicle worked right at one time and the problem isdue to time, wear, dirt or other causes.

201 pages

...

MATERIALS INDIRECTLY RELATED TO FAULT DIAGNOSIS

Berner, W.S.; Hamilton, L.A.; Best, W.R. (1974) "A New

Approach to Evaluating Problem-Solving in Medical Students."

Journal of Medical Education. 49:666-672.

This paper describes the development of a new technique

to assess clinical. problem-solving skills in medical

students. The procedure enables each step in the

solution of clinical problems to be evaluated

independently by providing feedback to students at each

decision point. There are several unique features,

including the incorporation of some aspects of the

problem-oriented medical record. Students were required

to read an extensive data base and construct problem

lists, order diagnostic tests, and plan for the

management of the patient. The technique was designed to

(a) be administered to large groups of students, (b) be

easily scorable, and (c) overcome some of the

difficulties inherent in previously developed means ofassessment.


I...

Brown, J.S.; Burton, R.R. (1978) "Diagnostic Models forProcedural Bugs in Basic Mathematical Skills." Cognitive Science.2(2):155-192.

A new diagnostic modeling system for automaticallysynthesizing a deep-structure model of a student'smisconceptions of bugs in his basic mathematical skillsprovides a mechanism for explaining why a student ismaking a mistake as opposed to simply identifying themistake. This report is divided in four sections: Thefirst provides examples of the problems that must behandled by a diagnostic model. It then introducesprocedural networks as a general framework forrepresenting the knowledge underlying a skill. Thechallenge in designing this representation is to find onethat facilitates the discovery of misconceptions of bugsexisting in a particular student's encoding of thisknowledge. The second section discusses some of thepedagogical issues that have emerged from the use ofdiagnostic models within an instructional system. Thisdiscussion is framed in the context of a computer-basedtutoring/gaming system developed to teach students andstudent teachers how to diagnose bugs strategically aswell as how to provide a better understanding of theunderlying structure of arithmetic skills. The thirdsection describes our uses of an executable network as atool for automatically diagnosing student behavior, forautomatically generating "diagnostic" tests, and forjudging the diagnostic quality of a given exam. Icludedin this section is a discussion of the success of thissystem in diagnosing 1300 school students from a database of 20,000 test items. The last section discussesfuture research directions.


Ellery, G.L. (1977) "The EE's Role in the Garage Diagnosis ofVehicle Faults." Proceedings of 1978 IEEE Conference on VehicularTechnology, Denver.

The complexity of modern automobiles, trucks and otherinternal combustion engines demands more sophisticated,computerized diagnostic equipment. The requirement ofhigh fuel economy and low hydrocarbon emissions furtherhighlights this need. The author discusses the fact thatautomobile manufacturers are increasing their involvementin the design and construction of automated testequipment. The R & D for such equipment is the combinedeffort of the EE, ME, and Automotive Service Engineer.

Computer controlled test equipment which relies onmicro-processor technology is already found in manyautomotive repair facilities, but the sophistication ofsuch equipment is rapidly increasing. The author offersexamples of electrical analysis of starter-motor-currentoscillation and analysis of ignition spark amplitude todetermine engine compression.

3 pages

McGuire, C.H.; Babbott, D. (1967) "Simulation Technique in theMeasurement of Problem-Solving Skills." Journal of EducationalMeasurement. 4:1-10.

Employing simulation technique analogous to that utilizedin management games and military exercises, the authorshave directed the development of a series of branchedproblems in patient management requiring sequentialanalysis and decision. These problems are designed tomeasure aspects of behavior'defined by a criterion groupas essential components of clinical competence. Thematerials are suitable to either paper and pencil oron-line application. The validity and reliability ofsimulations is also discussed.


K

McGuire, C.H.; Solomon, L.M.; Bashook, P.G. (1976).Construction and Use of Written Simulation. United States ofAmerica: The Ts-ycho-gic-ai Corporation.

This work is a comprehensive discussion of writtensimulations. The discussion ranges from basicdefinitions through construction, use and evaluation ofsimulations. The book is additionally applicable tocomputer simulations, or real equipment simulations.Sample problems are presented throughout the book.

Murray, T.S.; Cupples, R.W.; Barber, J.H.; Dunn, W.R.; Scott,D.B.; Hannay, D.R. (1977) "Teaching Decision Making to MedicalUndergraduates by Computer-Assisted Learning." Medical Education.11:262-264.

This paper describes a computer-assisted learning (CAL)program that is used on an open-acess basis by studentsat the University of Glasgow medical school. The authorsreport an experiment in which the experimental group(n=12) was presented with on-line case studies in whichthey had to decide on patient treatment. Upon completionof the experiment, the experimental group scored higheron a factual knowledge test and had improved decisionmaking skills based on comparison of their scores tothose of medical experts. The subjects felt that CALenabled them to consider and discuss clinical decisionsin a way that is both more realistic and more instructivethan is possible in the more orthodox clinical teaching.


Swain, A.D. (1961) Factors Affecting Degree of Automation inTest and Checkout Equipment. Stamford, Con-n.: Dunlap anTissocia-e-s, Inc.

This report deals primarily with the human factorsproblems attendant to the increasing automation of weaponsystem test and checkout actions. The purpose of thereport is to identify unfavorable tactical situationsresulting from less-than-optimum allocation of test andcheckout actions, 'from inadequate design of automaticcheckout equipment (ACE), or from particular managementor other practices related to the utilization of ACE.These cases suggest ways of avoiding such situations inthe future. While most of the examples are taken fromAir Force weapon systems, some are taken from Army andNavy weapon systems and equipment; and the principlesand guidelines are generally applicable to all theservices. It is only the application which should bedifferent, as considerable differences often occurbetween services in such areas as training, manpowerselection, environmental conditions, and logisticssupport (to name only a few differences in systemcriteria).

224 pages

¢

Wollmer, R.D.; Bond, N.A., Jr. (1975) "Evaluation of aMarkov-Decision Model for Instructional Sequence Optimization."Behavioral Technology Laboratories, Report No. 76, University ofSouthern California, Los Angeles, CA.

Two CAI programs in electronics and trigonometry werewritten to test the Wollmer Markov Model for optimizinghierarchical learning; calibration samples totalling 110students completed these programs. Since the modelpostulated that transfer effects would be a function ofamount of practice, half the students were required tocomplete one practice problem successfully before movingto the next stage; the other half-had to do two practiceproblems successfully.

All students completed the courses satisfactorily.Practice effects were small; students who had onesuccess in each stage did about as well as those who hadtwo successes. The Wollmer model was thus not suitablefor optimizing instruction, in terms of minimizingoverall time, in these particular courses. Perhaps themain reason for this result was that, as the studentworks up to the top of the hierarchy, the sheer number ofsubskills involved in the final task becomes a majordeterminant of performance time, and number of practicetrials has a relatively minor effect, unless a very largenumber of practice trials is given. To see if a muchlarger amount of practice would affect time to perform,three additional subjects were run with the electronicsproblems under the condition that 5 correct solutionswere required at each of 11 levels. It was found that amuch larger training time was required to get a 2-3minute improvement at the final level. Because of thisrelative inefficiency, the Wdllmer model would notprescribe such extensive additional practice.


IL4

47

AUTHOR INDEX

Avco Lycoming. ......... 38

Babbott, D .. ............ 43Barber, J.H .. .......... 44IBashook, P.G .. ......... 44Bean, R .. .............. 38Bell, A.G .. ........... 5Bergum, B .. ........... 30Berner, W.S .. .......... 41Bernstein, B.B .. ........ 15Best, W.R .. ........... 41Bilinski, C .. .......... 28Bond, N.A., Jr. .. ....... 4, 46Brooke, J.B....... ... . .. .. . . 4 4Birown, J.S.. .......... 5-6 4?

Burton, R.R. .. ......... ,4

Chilton Book Co. .. ....... 39ACrawford, A.M. .......... 7Crawford, K.S .. ........... 7Crooks, W.H .. .......... 8, 12Cupples, R.W. ........... 44Czeh, R.S .. ........... 33

Damrin, D.E. .. ...........13Deutsch, W.E .. ........... 24Duncan, K.D .. .......... 4Dunn, W.R........................44Dusseldorp, R. .. ........ 33

Edwards, J. ........... 33Ellery, G.L .. .......... 43Elliott, T.K .. ......... 9-10, 16Ely, J.H .. ........... 11

Fattu, N.A .. ............ 34, 37Finch, C.R. ............ 11-12Fink, C.D .. ........... 35Foley, J.P., Jr ......... 36Freedy, A .. ........... 8, 12

Gardner, F.M ............ 13General Motors Corporation . . . 40Glaser, R.............13Glass, A.A.o........ .........14Glenn, H.T .. ............ 39Goldbeck, R .A .. ......... 15

* Hall, N.B...............oHamilton, L.A... .......... 41

* Hannay, D.R .. ............ 44Hemphill, L .. ............ 32Hillix, W.A......... ..... 15

~w.,

Hodgson, T .. ............ 30 48Howard W. Sams & Sons............39

Jensen, A .. ........... 18Joyce, R.P .. .......... 10

King, C.A .. ........... 23King, W.. ............. 36Kuppin, M.A. ..............8

Landa, S .. ........... 16

Mallory, W.J .. ......... 16Marshall, E. . .......... 4Marx, M.A. .. ........ .. .. 15Mcguire, C.H ........... 43-44Mills, R.G .............. 17Mitchell Manuals, Inc. .. ...... 40Moore, J.V .............. 27Moran, P.J ............ 23Murray, T.S .. .......... 44

Norton, S .. ........... 33

Osborn, W.C .. .......... 17

Popham, W.J .. .......... 37

Rasmussen, J ........... 18-19Rigney, J.W .. .......... 4, 20-23Robinson, J.E., Jr .. ...... 24Rogers, J.G. ........... 24Rouse, S... ............ 27Rouse, W.B .............. 25-27Rubinstein, R. .......... 6

Saltz, E...............27Saupe, J.L..............28Scott, D.B ............ 44Shriver, E.L ........... 35Silva, J..........................30Solomon, L.M ........... 44Standlee, L.S. .......... 28, 37Steinemann, J.H. ......... 29Stolurow, L.M. .......... 30Swain, A.D .............. 45

Taylor,.................33Towne, D.M .. .......... 21-23, 31

Van Albert, C.E. .. ....... 11

Weiss, M..............33Wescourt, K.T. .......... 32

Wollmer, R.D ........... 46

ea.~.

49

SUBJECT INDEX

Artificial intelligence .... 5-6, 8, 12, 21-22, 32, 42Automated test equipment . ... 43, 45

Complexity ... ........... .. 16, 27Context - automotive ........ .. 11-12, 27, 38-40, 43Context - aviation ......... .. 7, 16, 30, 36, 38Context - computer programming 32-33Context - electronics .... 4-14, 16-18, 23, 27-29, 33, 46Context - mathematics ..... 33, 42, 46Context - medicine ......... .. 41, 43-44Context - process control . . . 4, 19

Heuristics ... ........... .. 8, 15, 17-19, 25-26, 30, 42Human error ..... .......... 24

Manual .... ............. .. 38-40Measures - attitude ........ .. 6, 11-12Measures - cost ... ........ .. 8, 30Measures - errors ......... 4, 6-7, 9, 11, 13-14, 25-26Measures - number of tests . . . 4, 13, 15, 17, 23, 25-26Measures - subjective assessment 6Measures - task time ........ .. 6-7, 11, 14, 23, 25-27Measures - verbal protocol . 18Model ...... ............ .. 4, 25-26, 46

Proceduralization .. ....... .. 9-10

Simulation - computer ...... 4-8, 12, 16, 20-23, 25-27, 31,33, 35-36, 41-42, 44, 46

Simulation - other ......... .. 11, 15, 17Simulation - paper and pencil . 4, 12-14, 16, 35, 43-44Simulation - real equipment . 10-12, 14, 27, 29, 35-36

Task analysis . . . ...... 21, 30, 35

4

51

DISTRIBUTION

ARI Oist'ibution List

4 OAS (M&RAI 2 HOUSACOEC, Ft Ord, ATTN: Library

2 HODA (OAMI-CSZ) 1 HOUSACOEC. Ft Ord. ATTN: ATEC-EX-E-Hum FcI HODA (OAPE-PBRI 2 USASEC, Ft Beniamin Har.on, ATTN: Library1 HODA (DAMA-ARI 1 USAPACDC Ft Benjamin Harrison. AiTN: ATCP-HR1 HODA (OAPE-HRE-POI I USA Comm-Elect Sch. Ft Monm uth. AiN: ATSN-EA1 HODA (SGRO-IO) 1 USAEC. Ft Monmouth. ATrN: AMSEL-CT-H0PI HODA IOAMI-DOT-C) 1 USAEC, Ft Monwmoth. AiN: AMSEL.-PA-1 HODA (OAPC-PMZ-AI 1 IUSAEC. Ft Monmuth. ATIN: AMSEL-SI-CSI HODA (OACH-PPZ-A) 1 USAEC. Ft Monmuth. ATTN: C. Fad ow Br1 HODA (OAPE-HREI I USA Materials Sys Anal Apy, Aberlen. ATTN: AMXSY-PI HODA fDAPE4PO-C) I Edg-wuod Atned. Aberdeen, ATTN: SAREA-O1L-H1 HODA (OAPE-OW) I USA Ord Ctr & Salo. Abserln, ATTN: ATSL-TM-C1 HODA (OAPE-HAL) 2 USA Hn Engr Lob. Aberdeen. ATN: Librrv/Dir1 HODA (OAPE-CPS) I USA Combat Arms Tng Sd. Ft Banning ATTN: Ad Supevisor1H ODA (DAFO-MFA) I USA Infantry Hum Rich Unit, Ft Benning AWTN: ChiefI HODA IOARD-ARS-P 1 USA Infantry Sd. Ft BeninL ATTN: STISC-TE-T1 HODA IOAPC-PAS-A) I USASMA. Ft Slim, AirN: ATSS-LRC1 HODA (OUSA-OR) I USA Air Dof Sefi, Ft Slim. ATTN: ATSA-CT-ME1 HODA IOAMO-ROR) I USA Air DeE Sch. Ft Slim, ATTN: Tecs Lib1 HOOA (OASG) 1 USA Air Oaf Bd. Ft Bliss. ATIN: FILESI HOA (DA 10-PtI I USA Air Dof Sld. Ft Bliss, ATTN: STEBD-POI Chief. Consult Div IDA-OTSG). Adelphi. MO I USA Cmd & General Stf College. Ft Leaanwsmth, AWTN: LibSMil Asst. Hum Res. OODOR&E. OAO (EILS) 1 USA Cand & General St College. Ft Leavenworth. ATTN: ATSW-SE-L

1 HO USARAL. APO Seattle. ATTN: ARAGP-R I USA Cmd & General Stf College. Ft Leavenworth. ATTN: Ed AdvisorI HO First Army, AWN: AFKA-Ol-TI I USA Combined Arms Cmbt Dey Act. Ft Lawwwrth. ATTN: OeCdr2 HO Fifth Army. Ft Sam Houston I USA Combined Arms Cmbt Oev Acts Ft Leavenworth. AWYN: CCS1 Dir. Army Stf Studies Off. ATrN: OAVCSA (DSP) I USA Combined Arm Cmbt 0ev Act. Ft Leweworth. AITN: ATCASA1 Off: Chief of Stf. Studies Oft: I USA Combined Arms Cmbt ow Act. Ft Lomemoush, ATTN: AYCACO-E1 DCSPER. ATYN: CPS/OCP 1 USA Combind Arms Cmbt De Act. Ft Lwnwoet, ArN: ATCACC-CI1 The Army Lib. Pentagon, ATTN: RSS Chief I USAECOM. Night Vision Lab. Ft Belvoir. AIN: AMSEL-NV-SOI The Army Lib. Pentagon, ATTN: ANRAL 3 USA Compuser Sys Cad. Ft Befvoir. ATTN: Treh Library1 Oft, Ant Soct of the Army (R&D) 1 USAMEROC. Ft Belv-ir. ATTN: STSFS-OO1 Tech Support Ofc. OJCS I USA Eng 5ch. Ft Beloir. ATN: Library1 USASA. Arlington. AT'N: IARO-T 1 USA Topographic Lab. Ft Selvoir. ATIN: ETL-TO-SI USA Rsch Oft. Ourham, ATTN: Life Sciences Dir 1 USA Topographic Lab, Ft Selvmr. ATIN: STINFO Center2 USARIEM, Natik. ATTN: SGRO-UE-CA I USA Topographic Lab. Ft Beai ar. ATITN: ETL-GSLI USAT'C. Ft Clayton. ATTN: STETC-MO-A 1 USA Intelligence Cs, & Sch, Ft Huacwca. A"TN: CTO-.MS1 USAIMA. Ft Bragg, ATTN: ATSU-CTO-OM I USA Intelligence Ctr & Slt. Ft Huachuca. ATTN: ATS-CTD-MS1 USAIMA, Ft Bragg, ATrN: Manuat Lib 1 USA Intelligence Ctr & Sch. Ft Huachuca. ATN: ATSI-TE1 US WAC Ctr & Sch. Ft McClellan. ATTN: Lib 1 USA Intelligence Ccr & Sch. Ft Huachuca. ATT'N: ATSI-TEX-GS1 US WAC Ctr & Sch. Ft McClellan, ATTN: Tng Oir I USA Intelligence Ctr & Sch. Ft Huachuce. ATIN: ATSI-CT"-OR1 USA Duarternaster Sth. Ft Lee. ATTN: ATSM.TE 1 USA Intelligence Car & Sah. Ft Huascuca. ATTN: ATSI-CTM-OT1 Intelligence Material oev Oft. EWL. Ft Holabird I USA Intelligence Ct & Sch, Ft Huachuca. AiN: ATSI-CTD-CSI USA SE Signal Sth, Ft Gordon, ATTN: ATSO-EA I USA Intelligence Ctr & Sch. Ft Huehum. ATIN: O*/SRO1 USA ChaWlain Cm' & Sch. Ft Hamilton, ATTN: ATSC-TE-RO I USA Intelligence Ctr & Sch. Ft Huschuce. ATTN: ATSI-TEM1 USATSCH, Ft Eustis. ATTN: Educ Advisor 1 USA Intelligence Car & Sch. Ft Huschuce. ATIN: LibraryI USA War College. Carlisle Barracks. ATrN: Lib I CDR. HO Ft Huachuce. ATrN: Tech Ref Div2 WRAIR. Neuropsychiatry Div 2 CDR. USA Electronic Ism Grd, ATTN: STEEP-MT-S1 DLI. SOA. Monteray I HO. TCATA. ATTN: Tech LibraryI USA Concept Anal Agcy. Belhsda. ATTN: MOCA-MR I HO. TCATA, ATTN: AT CAT-OPO. Ft HoodI USA Concept Anal Apty, Bethesda, ATTN: MOCA-JF 1 USA Reeng Cmd, Ft Sheridan. ATTN: USARCPM.PI USA Arctic Tes Ctr. APO Seattle, ATTN: STEAC-PL-MI 1 Seor Army Ad., USAFAGOOITAC, 1W AF Aux Fld No. 9I USA Arctic Test Ctr. APO Seattle. AIiN: AMSTE-PL-TS I HO. USARPAC. 0CSPER, APO SF 9ML ATTN: GPPE4E1 USA Armament Cmd. Redstone Arsenal. ATTN: ATSK-TEM I Stimwl Lib. Aceo of Health Sciee's, Ft Sam Hoston1 USA Armament Cmd, Rock Island. ATrN: AMSAR.TDC I Marine Corpe Inst.. ATTN: 0ean-MCII FAA.NAFEC. Atlantic City. ATIN: Library I HO. USMC. Cormendant. ATIN: Code MTMT1 FAA-NAFEC. Atlantic City. AT N: Human Engr Or I HO. USMC. Commanlent. ATTN: Code MPI-20-20I FAA Aeronautical Ctr. Oklahoma Citv, ATITN: AAC-440 2 USCG Academy. New London. ATTN: Admiseion2 USA FId Arty Sh. Ft Sill. ATTN: Library 2 USCG Academy. New London. ATIN: LibraryI USA Armor Sh. Ft Knox, ATTN: Library I U9CO Training Ct, NY, ATTN: CO1 USA Armor Sch, Ft Knox. AT1'N: ATSS-01-E I USCO Tralinng Car. NY. ATIN: Educ Svc OffI USA Armor Sch. Ft Knox, ATTN: ATSS-OT-TP 1 USCG. Psat Ra Br. DC, ATTN: GP 1/62I USA Armor Sch. Ft Knox. AITN: ATSB-CD-AO I HO MId-iag Or. MC Det. Quantico AiN: Pil Div

-I,. -

A

52

IUS Marne Corps Liaison Ofc. AMC. Alexandria. ATTN-. AMCGS-F I Def & Civil Instaf Envireo diimeCnada1 USATRAOOC. Ft Moonro. ATTN: ATRO-ED 1 AIR CRESS. Kensington. ATTN: Info Sys Br6 USATRACOC. Ft Monroe. ATTN: ATPR-AO I Milit@gqgykli' TioneMs. Copenhagen1 USATRAOOC. Ft Monroe. ATTN: ATTS-EA 1 Military Attache. Frend Emassy. ATTN: Doc SacI USA Forces Cand. Ft McPherson. ATTN: Library I Misdex Chef. C.E.R.A.-Arunal. Toulon/Navsll France2 USA Aviation Test ad. Ft Rucker. ATTN: STESO-PO 1 Prin Scientific Off, AO Hum Engr Rich Div, Mintry

iUSA Agcv tar Aviation Satety, Ft Rucker. ArTN: Library of Dfense. Now DeWN1 USA Agley for Aviation Safety, Ft Rucker. ATTN: Educ Advisor I Pairs Rath Ofc Library. AKA, Israel Defense Forces1 USA Aviation Sch, Ft Rucer. ATTN: PO Draw 0 1 Ministaris van Defenue. OOOPI'KL Afd SocieelI ISOUSA Aviation Sys Cmd. St Louis. ATTN: AMSAV-ZDR Psyrchologiedie, Zaken. The Hague. Netherlands2 USA Aviation Sys Test Act.. Edwards AFS. ATTN: SAVTE-TI USA Air Oaf Sch. Ft Sis., ATTN* ATSA TEM1 USA Air Mobility Rech & 0ev Lab. Moffatt Fid. ATTN: SAVOL-AS1 USA Aviation Sdi. Rev Trig Mgt, Ft Rucer,. ATTN: ATST-T..RTMI USA Aviation Sci. CO. Ft Rucsker. ATTN: ATST-D-AI MO. OARCOM. Alexand*a ATTN: AMXCO-Ti.I MO. DARCOM. Alesandria. ATTN: CDRI US Military Academy. West Point. ATTN: Sarials UnitI US Military Mademy. West Point. ATTN: Ofic of Milt Ldrahp1 US Military Acaemy. Wort Point. ATTN: MAO.1 USA Standardizebio Go. UK. FP0 NY. A1TM:- MASE-OCI Oft of Naval Rudh. Arlington. ATTN: Cadsi 4123 Oft of Noval Rids. Arigton. ATTN: Cods 4661 Oft of Navel Rids, Arlington. ATTN: Cods 450I Oft of Naval Rsds. Arlington, ATTN: Cods 4411 Naval Aaoope Mad Res Lab. Pensacola. ATTN: Acous Sch Div1 Naval Aeraocp Mod Res Lab. Pensacola, ATTN: Coda LII Naval Aerospc Med Rev Lab. Pensacola. ATTN: Cods LS1 Chief of Never, ATTN: Pon-.OR1 NAVAiRSTA. Norfolk, ATTN: Safety Ctr1 New Oceanographic. DC. ATTN: Cods 6251. Charts & Techt1 Canter of Naval Anal. ATTN: Doc CtrI NavAirSyvom ATTN: AIR-5313CI Nav Su~ed. ATTN: 7 131 Nov~elicptrSubiqua 2. FPO SF 9W60I AFHRL IFT) Williami API1 AFMRL lITT) Lowry AFBI AFMRL 4ASI WPAFB. ON2 AFIIRL (OOUZ) Brooks API1 AFINRL (00*41 Lacktland APII MOUSAF IINYSO)I MOUSAF (DPXXA)1 AFVTG IRDI Randolph API3 AMRL (ME) WPAFI. ON2 AF Int of Tech, WPAFB. OH. ATTN: ENE/SL1 ATC (XPTDI Randolph API1 USAF AeroMad Lib. Brooks API (SUL-41. ATTN: DOC SECI AFOSR 411L1.. ArlingtonI AF Log Cmd. McClellan AFI. ATTN: ALCIOPCRB1 Air Force Academy. CO, ATTN: Dept of Bill Scsi5 HasPars G Dev Ctr. San Diego2 Navy Med Nesuropsychiarric Rach Unit. San Diego1 Nay Electronic Lab. San Diego. ATTN: Rev LabI Nay TrngCon. San Diego. ATTN: Cods 9000-LibI NevPostGreSch, Monterey. ATTN: Cods 5IAa1 NavPostGradch. Montsrey, ATTN: Cods 2124I NavTrngfquipCtr, Orlando. ATTN: Techt Lib1 US Dept of Labor, DC. ATTN: Manpower Admin1 US Dept of Justice. OC. ATTN: Drug Enforce MAdin1 Nat But of Standards. DC. ATTN: Computer Info SectonI Nat Clearing House for MM-Info. Rockiville1 Denver Federal Cir. Lakewood, ATT'N: OLM

12 Defense Documentation Canter4 Dtr Psych. Army Ho. Russell Ofcs. CanberraI Scientific Advsr. Mil 8d, Army Hp, Russell Ofcs. CanoerraI Mil and Air Attache. Austrian EmbassyI Contre do Recherche Des Factsurt. Humaine dis Deferne

Nationals. Brussels2 Canadian Joint Staff Washington1 C'Air Staff. Royal Canadian AF. ATTN: Pori Std Anal St3 Chief. Canadian Daf Rich Staff. ATTN: C:'CRDSlWI

4 Britisti Def Staff. Bintih Embassy, Washington