Project ReportATC-152

Unalerted Air-to-Air Visual Acquisition

J. W. Andrews

26 November 1991

Lincoln Laboratory MASSACHUSETTS INSTITUTE OF TECHNOLOGY

LEXINGTON, MASSACHUSETTS

Prepared for the Federal Aviation Administration, Washington, D.C. 20591

This document is available to the public through

the National Technical Information Service, Springfield, VA 22161

This document is disseminated under the sponsorship of the Department of Transportation in the interest of information exchange. The United States Government assumes no liability for its contents or use thereof.

•

EXECUTIVE SUMMARY

Earlier studies have characterized the ability of pilots to visuallyacquire traffic when aided by the automatic traffic advisories of the .Traffic Alert and Collision Avoidance System (TCAS). In that work. amathematical model was developed that allowed prediction of visual searchperformance for a range of closing speeds and visual conditions.Unfortunately. many encounters occur in today's airspace for which noalerting traffic advisory is provided. The model can be applied to suchcases merely by adjusting model parameters. However. parameter selectionmust be based upon flight test data. To obtain such data. a series offlight tests were flown to measure the air-to-air visual acquisitionperformance of pilots engaged in unalerted visual search.

Twenty-four general aviation pilots participated in the test .Subjects were carefully briefed so that visual search was perceived as onlyone aspect of the pilot techniques of interest to test personnel. Eachsubject. accompanied by a safety pilot. flew the Beech Bonanza aircraft on a45 minute cross-country flight. During this flight. three airborneintercepts were scheduled using a Cessna 421 interceptor. The positions andclosing rates of each aircraft was recorded by radar for later analysis.The time at which the subject saw the intercepting aircraft was recorded bythe safety pilot.

Data analysis revealed that the instantaneous rate of visualacquisition for the subject pilots was

~A

where ~is an empirically derived parameter found to be 17000 steradian/sec.A is the visual area of the target aircraft. and r is the target range. Forexample. for a target of 70 square feet at a range of 1 nmi. theinstantaneous rate of visual acquisition is 3.2 percent per second. Thisrate is approximately eight times lower than the instantaneous acquisitionrate for alerted search as determined from TeAS flight tests.

iii

CONTENTS

EXECUTIVE SUMMARY

1. INTRODUCTION

2. THE VISUAL ACQUISITION MODEL

3. TEST FACILITIES AND PROCEDURES3.1 Subject Pilot Recruitment3.2 Flight Test Format3.3 Subject Pilot Briefings

4. DATA ANALYSIS4.1 Summary of Test Data4.2 Estimation of B for Unalerted Search4.3 Comparison to Alerted Search

5. FURTHER RESULTS5.1 Examination of Pilot Differences5.2 On the Difficulty of Visual Acquisition on Extremely

Clear Days

REFERENCES .

APPENDIX A DATA COLLECTION FORMSA.l Pilot Background QuestionnaireA.2 Safety Pilot In-flight QuestionnaireA.3 Debriefing Questionnaire

v

3

5577

11111116

1717

19

21

A-IA-IA-2A-4

APPENDIX B SUBJECT PILOT DATA BASE B-1

APPENDIXC.lC.2C.3

C SUBJECT PILOT BRIEFINGSProcedural BriefingPre-Flight Briefing on RouteDebriefing

of Flight

C-lC-lC-3C-4

•

APPENDIX D TEST SAFETY PROCEDURES

v

D-1

3-13-23-3

4-1

3-13-2

4-14-2

5-1

B-1B-2

ILLUSTRATIONS

Subject recruitment noticeVFR subject pilot cross country course number 1VFR subject pilot cross country course number 2

Theoretical curves are fit to flight test data to determinean appropriate value of e

TABLES

Summary of subject pilot experience levelsRatings held by subject pilots

Encounter dataVisual acquisition predictions for unalerted search

Linear correlation coefficient of pilot e score withselected variables

Variables used to describe pilotsSubject pilot characteristics

vii

5·89

15

66

1216

18

B-1B-2

1. INTRODUCTION

Flight tests were conducted to determine the ability of general aviationpilots, unalerted to the presence of traffic, to visually acquire otheraircraft approaching on near collision courses. Previous flight tests atLincoln Laboratory had provided data on visual acquisition performance whenpilots are alerted by collision avoidance systems (see Ref. 1 and 2). Amathematical model of visual acquisition was developed to characterize theobserved pilot performance. Although this model was applicable to bothalerted and unalerted search, it could not be applied to unalerted searchbecause of a lack of calibrating flight test data. In order to use the modelto examine see-and-avoid reliability in today's airspace, it was necessary toconduct flight tests in which pilots received no alerting information toassist their search for traffic. This report describes the test proceduresand test results for these flight tests.

1

2. THE VISUAL ACQUISITION MODEL

At any given instant. the likelihood of visual acquisition can bedescribed in terms of the visual acquisition rate. A. defined as follows:

lim~t ~ a

P [ acq in ~t ]

~t(2-1)

...The cumulative probability of visual acqu1s1t1on is obtained by integratingthe acquisition probabilities for each instant as the target aircraftapproaches.

P [ acq by t2 ] Jt2

1.0 - .exp [ - -00 A(t) dt ] (2-2)

This equation describes a nonhomogeneous Poisson process. It is ageneral mathematical formulation that can be used to describe almost anytype of visual search process. In order to make it useful. an expressionfor A(t) must be found. Experimental results (Ref. 1) have shown that forair-to-air visual search, A is proportional to the product of the angularsize of the visual target and its contrast with its background. When theexponential degradation of contrast with visual range is taken into account.the expression for A then becomes

A= p..A... exp [.2.996 L]r2 R

(2-3)

where A is the visual area presented by the target aircraft. r is the rangeof the target. and R is the visual range.

The cumulative probability of visual acquisition in a given situationcan be obtained by inserting the expression for A from equation (2-3) intoequation (2-2). The resulting equation is then evaluated using numericalintegration. In the special case of infinite visual range (perfectly clearatmosphere). the equation simplifies to the following easily-computed form:

P [ aeq by t2 ] -~A1.0 - exp [ -r 2 t2

3

(2-4)

The basic characteristics associated with the visual target (such asclosing rate t target area t or visual range) can be determined from knowledgeof the conditions of search. However t the model parameter e can be determinedonly by observing the performance of pilots in test flights. e can be viewedas a measure of pilot search effectiveness. It reflects the physiological andmental processes underlying pilot performance t and hence can be expected to beconsiderably different under unalerted and alerted search conditions. Thepurpose of the unalerted search flight tests was to determine a value of ethat best describes pilot performance under unalerted search conditions.

4

3. TEST FACILITIES AND PROCEDURES

3.1 Subject Pilot Recruitment

Subject pilots were recruited primarily through notices (see Fig. 3-1)posted at active general aviation airfields in eastern Massachusetts andNew Hampshire. If the subjects seemed suitable after a telephone screening,they were scheduled for a flight test. A total of 24 general aviation pilotsparticipated. Each pilot completed a pilot history questionnaire (seeAppendix A). A summary of selected data from this questionnaire is providedin Tables 3-1 and 3-2.

NOTICE

PILOTS SOUGHT FOR STUDY

The M.I.T. Lincoln Laboratory is seeking subject pilots toparticipate in a study of VFR workload. Each subject will fly one VFRcross-country mission in a Beech Bonanza while workload data iscollected. A test pilot will accompany the subject at all times. As aminimum, subject pilots should possess a private pilot license.

The test aircraft is based at the Lincoln Laboratory FlightFacility at Hanscom Field, Bedford, Massachusetts. Flights will bescheduled Monday through Friday from 0730-1730. Test will be conductedthrough August of 1986. Each participant should allow approximately2 hours for pre-flight briefing and flying.

Interested pilots should call Vic Gagnon at 863-5500 (ext. 812-211)for further information.

6/86

Fig. 3.1. Subject recruitment notice.

5

TABLE 3-1

SUMMARY OF SUBJECT PILOT EXPERIENCE LEVELS

MINIMUM MAXIMUM AVERAGE

age (yr) 21 60 39

Single-engine hours 210 3800 934(non-complex)-

Single-engine hours 0 5100 789(complex)

Multi-engine hours 0 4700 556

Cross-country time (hr) 2 405 60in last 6 mo

-The Beech Bonanza is classified as a complex single-engine aircraft.Pilot single-engine aircraft experience was divided into complex andnon-complex hours.

TABLE 3-2

RATINGS HELD BY SUBJECT PILOTS

RATINGFRACTION OF PILOTS

HOLDING RATING

Commercial 50.0%

Multi-engine 41.7%

Instrument 58.3%

Certified Flight Instructor 16.7%

6

•

,.

3.2 Flight Test Format

Flight tests were conducted between 14 April 1986 and 9 September 1986 atthe M.l.T. Lincoln Laboratory Flight Facility located at L.G. Hanscom Field inBedford, Massachusetts. Each pilot flew a Beech Bonanza aircraft on atriangular cross-country flight of about 45 minutes duration. Two differentcourses were used (see Figs. 3-2 and 3-3). Subject pilots were accompanied atall times by a safety pilot who sat in the right-hand seat. The safety pilotbriefed the subject on the route of flight immediately before leaving thebriefing room. A typical text for this briefing can be found in Appendix C.

Three times during the flight the subject aircraft was deliberatelyintercepted by a Cessna 421 aircraft. All intercepts occurred while theBonanza was established at cruise altitude during a cross-country flightsegment. The Cessna passed at 500 ft altitude separation with as littlehorizontal offset as could be achieved (normally a few tenths of a mile).Range safety was ensured by constantly tracking each aircraft with a groundradar and requiring that the intercept be aborted if the interceptor failed tovisually acquire the subject by 2 nmi range. In addition, the Cessna 421 wasequipped with an experimental version of the Traffic Alert and CollisionAvoidance System (TCAS). The TeAS traffic advisory display providedadditional information to the interceptor regarding the position of thesubject aircraft. Special range safety rules can be found in Appendix D.Data from the ground radar was recorded and used in later analysis.

Two techniques were employed to record the time at which the subject sawthe interceptor. In the first technique, the safety pilot signalled the timeof acquisition by using a remote switch that triggered the SQUAWK IDENT of theaircraft transponder. The time of switch activation could be determined towithin + 3 seconds. In the second technique, the safety pilot used ahand-held stopwatch that had been calibrated with the radar time-of-day clock.Results were checked for reasonableness by comparing radar range at theindicated time of visual acquisition with the safety pilot's estimate of therange at which acquisition occurred.

3.3 Subject Pilot Briefings

A fundamental goal of the testing was to obtain results that closelyapproximated visual search performance under actual non-test flyingconditions. This required great care in the manner in which subject pilotswere briefed and treated during the tests. The lack of traffic advisoryinformation alone would prevent subjects from knowing the time or approachdirection of traffic. But there was concern that if subjects were told thatvisual acquisition performance was of primary interest in the test, theywould devote undue effort to visual search and thus bias test results.Hence, a briefing procedure was developed that, from the subject'sperspective, made it difficult to tell that visual search was any moreimportant to the test than several other aspects of VFR pilotage. It wasemphasized that experimenters wanted to see the individual differences in

7

~ C421c: -=t> aE-33

TO GOM 2920

A~LT 04.&00 fT

M'NUTe~M~,,;NNI---t+----'~o

ftTCt4e UAGr::- -~OG~AONeRr=~~~~---

00

Fig. 3.2 \/Fll sublect pilot eros. country co

ur•

enumber 1.

1015238

--~ C421

C:=~t>BE-33

/.-TO GO,.. 212- If

AU ".500 FT\

MINUTE MAN

R-"'02 0

MOORE

o

Fig. 3.3 VFR subject pilot crOSI country course number 2.

1015228

the way pilots approached VFR workload, so subjects should relax and usetheir own normal flight techniques. Thus, the subjects were given no specialreason to concentrate on visual search as opposed to other cockpit duties.

Subjects were not told that intercepts would be conducted for datacollection purposes. They were merely told that other company (LincolnLaboratory) aircraft might be using the same routes, and that if so they wouldbe aware of our altitude and remain separated from us. The text for thisprocedural briefing is contained in Appendix C. The Cessna 421 interceptorwas taxied away from the Flight Facility before the subject pilot was takenout to the ramp.

Subjects were asked to provide three types of data. First, they answeredperiodic questions asked by the safety pilot (such as "Where is WorcesterAirport from our current location?"). Second, they provided workload ratingson a 1-9 scale when requested. Third, .they called out all sighted trafficseen as soon as they saw it. The requirement to call traffic was described asa way for experimenters to gather insight into the amount of workload beingdevoted to visual search in comparison to tasks inside the cockpit.

Most subjects saw the Cessna on only one or two of the intercepts. Onlytwo of the 24 pilots realized during the test that the intercepting Cessnamust be another test aircraft. Neither came to this realization until afterthe third intercept. Hence, the intercepts themselves did not appear tounduly alter the pilot search behavior reflected in the data.

Although every effort was made to put the subject pilots at ease andencourage normal cockpit behavior, it must be assumed that laxity andinattentiveness were discouraged by the very fact that the subjects wereflying in a test with a safety pilot who they knew would report on the resultsof the flight. Thus, it is possible that pilots performed better than theywould during actual flight. On the other hand, the increased workload causedby unfamiliarity with the aircraft probably degraded performance for somepilots (see discussion in Section 5). It is felt that overall, the level ofpilot visual acquisition performance during the tests approximated that whichcan be readily achieved in actual flight.

10

,

..

4. DATA ANALYSIS

4.1 Summary of Test Data

Data was obtained for 64 encounters. Basic information concerning theseencounters is provided in Table 4-1. Visual acquisition was achieved in 36 ofthese encounters (56 percent of the total). The median acquisition range forthese 36 encounters was 0.99 nmi. The greatest range of visual acquisitionwas 2.9 nmi.

4.2 Estimation of e for Unalerted Search

In any large set of test encounters, there may exist a few anomalousencounters for which the search conditions differ from those intended in theplan for the experiment. These encounters can cause errors in the estimationof e if they result in visual acquisition that occurs much earlier or laterthan normal. The technique for estimating 8 should be relatively insensitiveto the presence of anomalous encounters. Although e can be estimated by amaximum likelihood technique (Ref. 1), this technique is sensitive toanomalous encounters and should be applied only if a confident editing of datacan be carried out. A more robust technique (described below) has beendeveloped that involves a curve-fitting process that is relatively insensitiveto anomalous results.

According to the model, the probability of visual acquisition within anytime interval for which e is constant, can be determined from the timeintegral of the solid angle-contrast product of the target (see equations 2-2and 2-3). Let this integral be represented by Q. Since Q represents theopportunity for visual acquisition that the target has provided the pilot, Qwill be referred to as the "opportunity integral". Then the probability ofvisual acquisition is

where

P[ acq by t] = 1 - Exp[- e*Q(t)] (4-1)

fQ(t)

t

f-""

A

11

-2.996 rExp( -------- ) dt

R(4-2)

TABLE 4-1

ENCOUNTER DATA

Encounter Scenario Visual Closing, Acquisition QID No. Range (nmi) Rate (kt)a Range(nmi)b (ster-v.sec)C

101 1 30 - 94.3 -1.00 24.44102 3 30 -290.4 1.89 8.46103 3 30 -295.0 0.48 29.77201 1 12 -189.4 -1.00 175.54202 2 12 -204.1 1.54 14.13203 3 10 -284.3 -1.00 8.41301 1 6 -227.5 0.67 39.32302 2 8 -280.6 0.47 57.06401 1 10 -163.0 -1.00 194.40402 2 10 -251.8 -1.00 U1.57403 3 8 -282.7 0.37 182.88501 1 10 -204.9 1.74 9.68601 1 20 -170.3 -1.00 156.25602 2 20 -290.1 -1.00 45.35603 3 20 -241.2 0.44 91.04701 1 20 -187.2 -1.00 137.53702 2 20 -257.4 -1.00 98.89703 3 12 -257.6 1.04 16.83801 1 15 -199.6 1.41 23.67802 2 12 -270.3 -1.00 85.55803 3 15 -241.9 -1.00 55.25901 1 20 -182.4 0.74 49.22902 2 20 -270.2 0.37 94.82903 3 20 -290.9 0.99 19.48

1001 1 12 -182.5 0.78 38.041002 2 12 -288.4 -1.00 91.061003 3 15 -285.6 1.56 7.881102 1 15 -291.3 1.01 17.681103 2 15 -278.5 1.08 17.201201 1 8 -153.7 1.50 16.621203 9 8 -232.3 0.85 20.911301 1 8 -166.1 1.68 8.981302 2 15 -264.6 1.23 15.12 "1303 3 12 -201.6 -1.00 21.311401 4 5 -289.8 -1.00 69.641402 5 5 -285.9 -1.00 36.471403 6 5 -251.8 -1.00 87.401501 4 20 -246.1 -1.00 85.201502 5 20 -260.1 1.09 22.941503 6 20 -237.9 -1.00 21.431601 4 15 -302.9 1.57 8.901602 5 15 -260.8 -1.00 28.81

12

TABLE 4-1 (CONT'D)

ENCOUNTER DATA

Encounter Scenario Visual Closing Acq uisiti6n QID No. Range (nmi) Rate (kt)a Range(nmi)b (ster-~sec)C

1603 6 15 -261.2 0.49 49.031701 4 40 -280.0 -1.00 32.141702 5 40 -233.5 2.94 7.171703 6 40 -290.6 -1.00 54.321801 4 30 -294.4 1.80 8.741802 5 20 - 68.7 -1.00 9.131803 6 25 -257.5 1.27 17.601901 4 50 -282.9 -1.00 102.141902 5 50 -206.5 -1.00 62.931903 6 50 -190.7 -1.00 32.212001 4 20 -298.9 -1.00 43.832002 5 20 -279.2 0.32 43.852003 6 24 -298.4 2.30 5.702101 4 15 -293.3 -1.00 25.262102 5 10 -284.1 0.87 26.232103 6 10 -253.8 0.68 44.102202 5 12 -236.3 0.65 51.492203 6 12 -305.7 1.00 112.442301 4 12 -273.2 0.32 107.252303 6 10 -270.5 0.31 7.702401 4 20 -295.3 1.65 8.442402 5 20 -268.4 1.34 18.82

aAverage value

bIf no acquisition, range is entered as -1.00.

clncludes only times prior to range of 0.3 nmi, when target bearing wasbetween 10 and 2 o'clock and target elevation was between -10 degrees and+ 10 degrees.

13

For a given set of experimental data, it is possible to plot theprobability of visual acquisition versus Q. Then the value of a mostappropriate for that data can be determined by fitting equation (4-1) to thecurve. In order to obtain the most robust estimate of a, it is advisable togive heavy weighting to the central portion of the curve (the region centeredat 50 percent probability of acquisition). By emphasizing the centralregion, a natural editing of anomalous data takes place.

In each experimental encounter, Q starts at zero and increases until theencounter terminates. The encounter is terminated whenever visual acquisitionoccurs. It can also be terminated for artificial reasons (e.g., the targetpasses outside the field of view, the encounter is aborted, etc). When anencounter terminates prior to visual acquisition, then that encounter providesdata only for values of Q up to the final value observed. This is because itis not known whether or not acquisition would have occurred if the trial hadcontinued. When an experiment terminates in visual acquisition, then it canbe applied to the entire range of Q values.

In computing Q values for Table 4.1, the integration was terminated if

1) visual acquisition occurred,

2) the intruder passed outside the prime search area (defined as bearingsfrom 10 o'clock to 2 o'clock and elevation angles from -10 to+10 degrees), or

3) the target came within 0.3 nmi of own aircraft.

The first criterion is obvious, but the second two criteria areartificial ones. Criterion 2 limited the a estimate to the angular regionnormally searched by pilots. This is the region most likely to contain threataircraft. Criterion 3 reduced the sensitivity of the estimate to minor errorsin recording the time of visual acquisition (because the size of the aircraftis growing rapidly for ranges less than 0.3 nmi, a slight error in determiningthe time of visual acquisition can result in a large error in Q). Inestimating a values, acquisitions that occurred after the termination of the Qintegration must be ignored. Note that as long as the model is valid, theapplication of arbitrary termination criteria reduces the amount of dataavailable, but does not bias the estimated value of B for targets within theprime search area. It should be noted however, that if an aircraft isapproaching from outside the prime search area, the value of B derived fromTable 4-1 will probably prove to be too great.

,

•

The lower curve in Fig. 4-1 shows theto the unalerted search data in Table 4-1.between experimental results and theory isapproximately 17000/ster-sec.

14

results of fitting equation (4-1)It can be seen that good agreement

obtained for a 6 value of

0- 0

•

:/0

• P-140000/ater-aec0

1 P-17000/ater-aec

": j 00 1c 0

.2- j- .• • t:;0

c:r t00lIl(.

~0 0 i~ f~=. I'J2. FLIGHT TEST DATA.0 fJ2

~0.. f unalerted aearch~~

0

0 alerted aearch (TeAS)t:J.

(II

0

-000

0.0 10.0 100.0 110.0 200.0

., Opportunity Integra. (JLateradlan-aec)

Fig. 4-1. Theoretical curve. are fit to flight te.t data

to determine an appropriate value of /3.

15

Table 4-2 provides model predictions of visual acquisition performanceusing the above result. Here B is set to 17,000/ster-sec and the probabilityof visual acquisition is tabulated in 6 sec increments. It is assumed thatvisual search begins either three minutes before collision or when the size ofthe target exceeds 1 minute of arc (whichever occurs last). The exact time .of search initiation has little impact upon the calculations as long as it isthree or four times the time of evaluation.

TABLE 4-2

VISUAL ACQUISITION PREDICTIONS FOR UNALERTED SEARCH "(Target size = 70 sq. ft.)

Probability of Visual Acquisition byt seconds to collision

Closing Rate (kt) 240 120 240 360 240Visual Range (nmi) 10 20 20 20 300

t

6.00 0.5621 0.9854 0.6148 0.3204 0.669012.00 0.2693 0.8515 0.3281 0.1384 0.396418.00 0.1505 0.6860 0.2003 0.0741 0.263124.00 0.0920 0.5485 0.1320 0.0432 0.186130.00 0.0593 0.4422 0.0909 0.0259 0.136336.00 0.0394 0.3604 0.0641 0.0151 0.101642.00 0.0265 0.2968 0.0457 0.0081 0.076148.00 0.0179 0.2466 0.0324 0.0032 0.056554.00 0.0119 0.2063 0.0226 0.0000 0.041160.00 0.0077 0.1735 0.0151 0.0000 0.0286

4.3 Comparison to Alerted Search

For purposes of comparison, a similar analysis was conducted using datafrom the flight testing (Ref. 2) of the Traffic Alert and Collision AvoidanceSystem (TCAS). The result is shown in the upper curve in Fig. 4.1. It can beseen that a B value of 140,OOO/ster-sec results produces a good fit. Thisindicates that the TeAS traffic advisory increased B by a factor ofapproximately eight (i.e., one second of search with the aid of a TCAS trafficadvisory is as effective as 8 seconds of una1erted search).

16

5. FURTHER RESULTS

5.1 Examination of Pilot Differences

It was not the purpose of the tests to measure or explain the differencesin visual acquisition performance between pilots. The goal of the experimentwas to determine a a value that applied on average to the 24 pilots tested.Because each subject could be exposed to only three encounters, a reliabledetermination of a for an individual pilot was impossible. The random natureof the visual acquisition process leads to individual a values that are farabove and below the value that would be derived from a larger quantity ofdata. However, by grouping pilots together in terms of commoncharacteristics, it is possible to determine if any strong correlations existbetween recorded pilot characteristics (such as age or experience) and visualacquisition performance.

An estimated a value was calculated for each pilot by dividing the numberof visual acquisitions achieved by the sum of the opportunity integrals, Q, inall encounters flown by that pilot. For instance, pilot No. 1 had 2 visualacquisitions in three encounters with Q values of 24.44, 8.46, and 29.77ster-~sec (see Table 4.1 for data). This produces an estimated a value forpilot No. 1 of

62x10

22.44 + 8.46 + 29.77= 31913/ster-sec.

It should be noted that any visual acquisitions that occurred after thecut-off range of 0.3 nmi was reached are not to be counted in the aboveprocess.

The linear correlation coefficient that exists between the a scores ofpilots and the pilot background characteristics (see Appendix B) are shown inTable 5.1. The most positive correlations noted were with instrument rating,eFI rating, and single-engine hours. These correlations indicate that pilotswho were most capable in flying the aircraft also performed better in visualsearch. It could be inferred that the workload involved in merely flying theaircraft detracts from the visual search capability of low-time, inexperiencedpilots. The most negative correlations noted were with age and workloadfactors. (The negative correlation with multi-engine hours is probablyexplained by the fact that a high value of such hours was strongly correlatedwith age).

During the flight, the test pilot estimated the fraction of time thesubject spent looking outside the cockpit. The fraction varied from 0.35 to0.90 with an average value of 0.60. Surprisingly, there was littlecorrelation of a with this fraction (p = -0.004). This may indicate that onlya fraction of the time spent looking outside the cockpit was actually beingspent in a productive search for traffic.

17

In debriefing, subjects were asked whether they thought that the emphasisthey had placed upon visual search during the test had been more or less thanin normal flight. There was little correlation of their answers with theirobserved performance (p = 0.015). This is one indication that visual searchemphasis did not strongly bias test results.

TABLE 5.1

LINEAR CORRELATION COEFFICIENT OF PILOTa SCORE WITH SELECTED VARIABLES

r

CorrelationCoefficient

0.3720.3500.2540.2080.1950.1820.1620.1330.1120.1040.015

-0.004-0.070-0.188-0.216-0.241-0.284-0.349-0.385

Variable

instrument ratingCFI ratingSE hours (non-complex)sky conditionsuse of sunglassescommercial ratingXC time last 6 mo.use of visual landmarksfamiliarity with Beech BonanzaAvg. score on pilotage questionsemphasis on visual search% time looking outside cockpittechniqueSE hours (complex aircraft)ME hoursmilitary flight trainingworkload rating (avg.)ageaircraft impact

To further analyze performance, a stepwise linear regressionwas applied to the data. This analysis indicated that age and SEthe two most important variables in predicting a pilot's a score.equation for predicting a was

a = 59225 - 1211*(age) + 13.58 * (SE hours)

analysishours were

The

This equation produced an R-squared value of 0.276, indicating that B-scorescould not be predicted very precisely from pilot background.

18

5.2 On the Difficulty of Visual Acquisition on Extremely Clear Days

Two experiences during the flight test program suggested that specialvisual acquisition difficulties may be associated with extremely clear days.One day, while a subject pilot flight was being conducted, a mid-air collisionbetween a helicopter and a small fixed-winged aircraft occurred only 15 milesnorth of the test area. It seemed an ironic fact that this accident occurredon an exceptionally clear day. Two weeks later, on another exceptionallyclear day, all three planned intercepts had to be aborted because theintercepting aircraft failed to visually acquire the subject prior to therequired 2 nmi range. These experiences suggested that visual acquisitioncould be more difficult on exceedingly clear days than on days with normalvisibility. The probable explanation for this, and the consequences forvisual performance modeling, are discussed below.

The conspicuousness of a visual target increases as its contrast with itsbackground increases. This contrast is dependent upon the scatteringproperties of the aircraft (paint scheme, surface reflectances, etc.) andenvironmental visual conditions (sunlight illuminance, visual range, clouds,etc.). Scattering that occurs between the eye and the target reduces theinherent contrast of the target. It has been found that if the contrast isreduced to approximately 5 percent, the target disappears into the background.The range at which this occurs for a target that would otherwise have100 percent contrast is used as a measure of the visual range.

Experience shows that for small aircraft on collision courses, mostvisual acquisitions will occur at ranges of one to two miles. As long as thevisual range is two or three times this distance, the scattering that occursbetween the eye and the target has little effect upon visual acquisitioncapability. However, the scattering that occurs behind the target can stillbe significant since it determines the brightness of the background.Normally, there is a milky band of haze near the horizon. This haze issignificantly brighter than the zenith sky. Most aircraft near enough to ownaltitude to pose a collision risk will approach with this haze as background.This is fortunate, since the increased contrast makes aircraft easier to see.On exceedingly clear days, the horizon haze layer disappears and the skybackground is appreciably darker. This reduces the target contrast, makingvisual acquisition more difficult.

Similar comments apply to aircraft that approach with terrain as thebackground. The terrain is significantly darker than the sky. Under hazyconditions, targets that are only slightly below own altitude are seen againstdistant terrain that is receding into the haze. This can provide a brighterbackground that enhances contrast. Haze has the added advantage of obscuringsmall-scale ground features that might make the image of the target difficultto perceive.

The visual acquisition model described in section 2 takes into accountthe degradation in visual acquisition capability due to scattering between theeye and the target. But it does not reflect any degradation that might occurdue to background darkening under exceptionally clear conditions. Since such

19

atmospheric conditions were unusual during the flight tests described in thisdocument, the results are not significantly affected by this limitation of themodel. However, it should be recognized that the results are optimistic whenapplied to situations where extremely clear conditions prevail.

20

REFERENCES

1. Andrews, J.W., "Air-to-air Visual Acquisition Performance with PilotWarning Instruments (PWI)", Project Report ATC-73, Lincoln Laboratory,H.I.T., (25 April 1977), FAA-RD-77-30.

2. Andrews, J.W., "Air-to-air Visual Acquisition Performance with TCAS II",Project ATC-130, Lincoln Laboratory, H.I.T., (27 July 1984),DOT/FAA/PH-84/17.

21

APPENDIX A

DATA COLLECTION FORMS

A.I Pilot Background Questionnaire

CONFIDENTIAL INFORMATION VFR FLIGHT TESTMIT LINCOLN LABORATORY

PILOT BACKGROUND QUESTIONNAIRE

NAME : ----------------ADDRESS

TEL

AGE

RATINGS HELD

YEARS AS PILOT

Any military flight training?

AIRCRAFT EXPERIENCE (HOURS):

___ yes ___ no

Single-engine _

Multi-engine __

Complex _

How much cross-country time in last 6 months?:

Operated at Hanscom Field in last 6 months?: ___ yes

hours

____ no

Aircraft have had most time in (last 6 months): (make/model)

Flown Beech Bonanza in last 6 months?: ___ yes ____ no

Familiar with use of HSI?

Familiar with use of DME?

____ no

___ no

somewhat yes

somewhat __ yes

Have current FAA medical certification? ___ yes no

THANK YOU

A-I

A.2 Safety Pilot In-flight Questionnaire

DATE AM/PM

CONFIDENTIAL INFORMATION

page 1 of 2

VFR FLIGHT TESTS

QII G / F / P QII G / F / P

QII G / F / P WL

WL QII G / F / P

QII G / F / P QII G / F / P

WL QII G / F / P

QII G / F / P WL

QII G / F / P Q# G / F / P

WL

TRAFFIC SEEN (0 = self only, I = subject first, o = self, then subject)

ENCOUNTERS (PTI = prior to intercept)

1

ENC II

2 3

Range of Visual

Workload PTI (1-9)

Distractions PTI (0 = none,1 = possibi1e 2 = definite)

Flight visibility(intercept direction)

Visual background PTI(sky, cloud, haze, terrain)

Earlier intercept affected?

Wind aloft

Other factors:

___ nmi

o / 1/2

nmi---

y / ? / N

A-2

___ nmi

o / 1/2

nmi---

y / ? / N

nmi---

o / 1/2

nmi---

y / ? / N

page 2 of 2

SAFETY PILOT QUESTIONNAIRE (CONT'D)

1. Pilot familiarity with airplane during XC:

inadequate / poor / acceptable / good / excellent

2. How heavily did pilot rely upon visual landmarks to navigate?

not at all/slightly / moderately / heavily

i 3. Time spent in visual search on XC per cent------

4. Search in directions other than 12 Q'clock on XC:

never / rarely / occasionally / regularly

5. Did pilot wear sunglasses on XC? yes no

t

6. Did pilot's overall flight technique appear normal, given his backgroundand experience level?

no / fairly / yes

If not, explain how it differed from normal:

Safety Pilot __

A-3

A.3 Debriefing Questionnaire

DEBRIEFING FORM

VFR FLIGHT TEST

Prior to coming to Lincoln Laboratory, had you discussed this flighttest with any previous subject pilot?

__ yes no

How much did unfamiliarity with the aircraft increase your workloadwhile flying the cross-country course?

i

not at all / slightly / si~nificantly

During the cross-country portion of this flight, did you give more orless attention than you normally would to any of the following aspects offlight. Please give thoughtful and honest consideration to artificialfactors such as your knowledge that you were in a test, presence of thesafety pilot, it wasn't your normal aircraft, etc.:

fuel management

navigation

Somewhat About the SomewhatLess Same More------- -------- -------

----- ----- -----

------ ---- -----

visual search fortraffic

holding altitude

holding course

weather

Any other comments?

THANK YOU!

A-4

}

APPENDIX B

SUBJECT PILOT DATA BASE

Selected background information on the 24 subject pilots was compiledin a computer data base. A list of the variables and the coding used isprovided in Table B-1. The data base itself is provided in Table B-2. Inaddition to data collected from the pilot background questionnaire, thetable contains some descriptive data collected during the flight test.

TABLE B-1

VARIABLES USED TO DESCRIBE PILOTS

Variable

dateage

comm ratingME rating

instr ratingCFI rating

SE hourscomplex hours

ME hoursXC timeBonanza

mil trainingfam w/AC

AC workloadlandmark nav% vis searchsunglasses?

techniq ue normalsky condition

avg workloadvisuals first

visuals second

visuals missedemphasis visual

avg q score

Description

date of flightyearsO=no, l=yesO=no, l=yesO=no, I=yesO=no, I=yestotal hours in non-complex single-engine aircrafttotal hours in complex single-engine aircrafttotal hours in multi-engine aircraftCross-country hours in last 6 monthsO=have not flown Bonanza in last 6 months, 1= haveflown Bonanza in last 6 monthsO=no, 1 = yesO=inadequate, 4=excellentO=not at all, I=slightly, 2=significantlyO=not at all, 3=heavilypercent of time devoted to visual searchO=not worn, I=wornO=no, I=fairly, 2=yesl=clear, 2=scattered, 3=broken, 4=lt. overcast,5=heavy overcast0-90 in tenths of a unittraffic seen first by subjecttraffic seen first by safety pilot, second bysubjecttraffic seen only by safety pilotI=less, 2=same, 3=moreaverage score on in-flight questions (lO=poor,20=fair, 30= good)

B-1

TABLE B-2

SUBJECT PILOT CHARACTERISTICS

Subject No. 1 2 3 4 5 6 7 8 9

date 4/14AM 4/14PM 4/15PM 4/16AM 4/16PM 4/18AM 4/18PM 4/22AM 4/22PMage 38 48 42 32 28 24 25 40 40

comm rating 1 1 1 0 1 0 1 1 0ME rating 1 0 0 0 1 0 0 0 0

instr rating 1 1 1 0 1 0 1 0 0CFI rating 1 1 0 0 1 0 1 0 0

SE hours 675 1450 1350 250 800 240 450 450 450complex hours 200 220 1600 1 850 8 20 8 250

ME hours 25 20 250 0 300 0 2 0 0XC time 5 10 75 60 100 10 3S 3 15Bonanza 0 0 1 0 1 0 0 0 0

mil training 0 0 1 0 0 0 0 0 0fam w/AC 4 2 4 2 4 2 3 1 3

AC workload 1 2 0 1 0 1 1 1 1landmark nav 2 1 1 1 2 1 1 0 2% vis search 80 55 60 40 60 35 60 65 70sunglasses? 1 1 1 0 0 0 0 1 0

techniq ue normal 2 2 2 2 2 2 3 2 2sky condition 4 3 1 3 5 2 1 2 3

avg workload 44 56 34 42 32 60 46 58 40visuals first 1 2 5 1 1 0 3 5 4

visuals second 3 3 2 2 3 2 2 2 3visuals missed 5 3 2 7 1 4 1 0 5

emphasis visual 3 1 3 2 3 2 3 2 2avg q score 26 30 30 26 29 27 29 24 29

(Table continued)

B-2

f

TABLE B-2

SUBJECT PILOT CHARACTERISTICS (CONT'D)

Subject No. 10 11 12 13 14 15 16 17 18

Date S/14PM 5/15PM 5/27PM 5/28PM 5/30PM 6/3PM 6/4PM 6/26PM 6/30PM

age 44 49 26 42 38 43 30 60 21comm rating 0 0 1 0 0 1 0 1 0

ME rating 0 1 1 0 0 1 0 1 1instr rating 0 1 1 0 0 1 0 1 1

CFI rating 0 1 0 0 0 1 0 0 1SE hours 360 3800 650 450 210 1600 300 1000 600

complex hours 200 500 65 60 20 430 50 5000 90ME hours 0 325 56 0 0 12 20 4700 11

XC time 20 25 135 50 85 30 2 10 30Bonanza 0 1 0 0 0 1 0 0 0

mil training 0 0 0 0 0 0 0 1 0fam w/AC 3 1 2 2 3 2 2 2 3

AC workload 1 0 1 1 1 1 0 1 1landmark nav 1 1 1 1 2 1 3 2 2% vis search 50 4S 50 40 70 55 65 60 65

sunglasses? 0 1 1 1 0 1 1 1 1technique normal 2 2 2 1 2 2 2 2 2

sky condition 1 1 0 0 0 0 0 0 0avg workload 42 52 52 58 58 40 56 40 48

visuals first 4 3 1 1 0 2 3 0 2visuals second 1 1 1 1 1 0 0 1 2visuals missed 1 2 2 0 4 1 1 1 4

emphasis visual 3 3 2 2 2 3 3 3 1avg q score 27 25 29 29 30 28 26 29 30

(Table continued)

B-3

TABLE B-2

SUBJECT PILOT CHARACTERISTICS (CONT'D)

Subject No. 19 20 21 22 23 24 AVG

Date 7/lPM 7/8PM 7/9PM 7/22 7/24PM 9/9PMage 48 57 29 48 53 32 39.04

comm rating 0 1 1 0 0 1 0.50ME rating a 1 1 1 0 a 0.42

instr rating 0 1 1 1 0 1 0.58eFI rating 0 1 0 1 0 1 0.42

SE hours 340 2000 2050 900 650 1400 934.37complex hours 30 3000 1200 5IOO 0 40 789.25

ME hours 0 2500 525 4600 0 5 556.29XC time 10 120 405 65 50 100 60.42Bonanza a 1 1 0 0 a 0.25

mil training 0 1 0 1 a 0 0.17fam w/AC 2 4 4 3 1 2 2.54

AC workload 1 0 1 2 2 1 0.92landmark nav 2 1 2 3 1 2 1.50%vis search 90 45 70 75 60 80 60.21sunglasses? 1 1 0 0 0 1 0.58

technique normal 1 2 2 2 2 2 1.96sky condition 0 1.37avg workload 46 44 42 40 70 32 47.17

visuals first 2 1 2 3 0 16 2.58visuals second 2 1 2 0 5 0 1.67visuals missed 1 3 1 1 0 4 2.25

emphasis visual 3 3 1 2 1 2 2.29avg q score 28 27 30 30 27 29 28.08

B-4

,

APPENDIX C

SUBJECT PILOT BRIEFINGS

C.l Procedural Briefing

[Note: This briefing was given when the subject first arrived at thebriefing room. It describes the purpose of the test and the datacollection procedures required of the subject.]

Introduction

This briefing will cover the objectives of the test and our datacollection procedures. Afterwards, our safety pilot will brief you on theroute of flight, the weather, and so forth.

Test Objectives

This study is being carried out by Lincoln Laboratory under thesponsorship of the FAA. Its purpose is to add to our knowledge concerning howVFR pilots actually fly in normal operations (as opposed to a check flight oran emergency situation). We want to determine how you allocate your workloadresources: which tasks you work hard at, which you let slide, which bits ofinformation you have available for immediate recall and which you would haveto look up if you needed them.

We expect to see differences among pilots depending upon training,background, and the individual style of the pilot. It is the range ofvariations in cockpit technique that interests us. Therefore, it is importantthat, insofar as possible in an experimental environment, you relax and flyusing your normal cockpit "style" and level of effort. Our safety pilot willgive you any assistance you need to feel comfortable with the aircraft, butthe flight is basically single-pilot.

Data Collection

Data will be gathered in several ways by the safety pilot who accompaniesyou.

First, there are formal questions. At random intervals, the safety pilotwill say "question" and then ask you some question relative to the flight.You should answer these questions promptly, giving your best guess if you'renot sure of the answer. These questions are designed to tell us what you havebeen paying attention to, what you have ignored, and so forth. You aren'texpected to know the answers to all of the questions (in fact, you may feelthat some of the questions are rather irrelevant to the flight). But youranswers will tell us something about your allocation of attention.

C-l

Secondly, there are workload ratings. The safety pilot will occasionallysay "How's your workload, now?" You should then rate your overall workloadlevel on a 1-9 scale with 5 being average (see hand-out). The exactinterpretation of each number is not so important (it's OK if your "6" is whatsome other pilot would call a "7"). We are most interested in the relativechanges you perceive in the workload during the course of the flight.

As the last item, we want you to callout all sighted traffic as soon as~~ sighted (even if at a great distance and clearly no factor). ~at is-,­point it out even if its an airliner at 30 thousand feet or just a speck onthe horizon. This helps us understand the amount of effort you are devotingto visual search in comparison to tasks inside the cockpit. You should startcalling traffic as soon as you are outside the traffic pattern. Since wewon't prompt you on this (we won't say "Did you see that aircraft that justpassed us?"), its up to you to just get in the habit of pointing out all thetraffic.

POST FLIGHT

After the flight you will be asked to complete a short one-pagequestionnaire about the flight. And that will complete the test.

Naturally, all data we collect is confidential in the sense that nosubject pilot's name will be attached to any data that is released.

SAFETY

Naturally, flight safety comes first and will not be compromised by anytest procedures.

Although the safety pilot will normally provide you with only minimalassistance, he remains the pilot-in-command and should any event bring thesafety of the flight into question he may take control of the aircraft untilthe problem is resolved.

Lincoln Laboratory operates its own radar facility to assist withcompany operations. The radar should be up this afternoon. If so, we willcheck in with them to let them know what we're doing. If there are othercompany operations in the area, they will be kept clear of our altitudes. Wewill not request any radar services from the radar site. As far as you areconcerned, you are on a cross-country VFR flight without any ATe contact.

Here is a summary of the procedures. (See hand-out).

Any questions about the data collection procedures before I turn it overto our safety pilot?

C-2

,

C.2 Pre-flight Briefing on Route of Flight

[Note: This briefing was given by the safety pilot immediately prior toleaving the briefing room for the aircraft. This is a typical text. Theactual text employed was, of course, adapted to the prevailing weatherand the degree of familiarity of the pilot with the area.]

This is a cut-out from a sectional chart showing our route of flight[show hand-out chart]. You can take this with you in the aircraft if youwish. We will depart Hanscom Field on Runway 11, making a right handdeparture to intercept the 292 0 radial to the Gardner VOR. We will climb to4500 ft.

The floor of the Boston TCA extends to 20 nmi at 4000 ft. We will haveto stay under the floor until we reach 20 nmi. We shouldn't have any problemwith that.

The Turner drop zone extends to 3995 at this location [point out]. Weshould be above that altitude.

We will cruise at an airspeed of 150 knots until reaching the GardnerVOR.

The aircraft is equipped with dual comm. One radio we will keep tunedto our company frequency. The other you can use as you desire. Normally, wemonitor Boston approach control during the first leg of the flight.

Upon reaching Gardner, we will then turn to fly the 73 degree radial tothe Manchester VOR, and will descend to 3500 ft for that leg of the flight.

We will be just outside the Manchester airport traffic area [pointout]. Normally, we monitor Manchester approach control while in thevicinity of Manchester.

Out of Manchester we will turn to the 170 degree radial and descend to2500 ft. We will maintain 2500 until we pass the city of Lowell [pointout].

Then we will tune in the Hanscom ATIS and then contact Hanscom tower.If congestion makes it necessary, we can do one or two 360 0 turns here[point out] to delay our arrival at Hanscom.

The weather today is 6000 scattered. Winds are 260 0 at 25 knots. Theseare the current reports from Manchester, Concord, and Hanscom [show pilotweather summaries received by wire].

We can listen to the current ATIS now. We will listen to it again whenwe are in the aircraft. [Listen to Hanscom ATIS on portable radio].

I have some other things to say about the aircraft, but I'll wait untilwe are in the cockpit so that I can point to the guages.

Any questions?

C-3

C.3 Debriefing

[Note: The following material was used in the post-flight debriefing.]

Do you have any suggestions for improving our briefings? Any suggestionsfor making you more comfortable in the aircraft?

Please do not provide potential future subjects with any details on theflight test.

C-4

:

APPENDIX D

TEST SAFETY PROCEDURES

Special Safety Rules for VFR Subject Pilot Missions

The following special safety rules will be observed in VFR subject pilotmissions requiring interception of a subject pilot. They apply in addition tonormal Group 42 safety rules.

1. Subject/interceptor altimeters will be checked in flight if anyavionics modifications or disassembly has occurred that could affect thecalibration. In addition, altimeters will be checked if more than 5 days havepassed since the last visual confirmation of accuracy.

2. For each intercept at least 500 feet altitude separation must bemaintained unless both of the following conditions are true:

~

a) altimetry accuracy was visually confirmed by in-flightobservation earlier in the mission

b) the interceptor has visual contact with the subject

3. An intercept will be aborted if any of the following conditions istrue:

a) MODSEF is unable to provide radar traffic advisories

b) the interceptor fails to establish visual contact by the time theaircraft are reported to be 2 nmi apart

c) visual contact is lost within 2 omi range

d) a non-test aircraft is judged to be a factor that constrains theencounter

4. Whenever test procedures allow, observers and equipment operatorswill assist flight crews in visual search for traffic during encounters.

D-I