controller and pilot evaluation of a datalink-enabled trajectory-based operations concept

7/24/2019 Controller and Pilot Evaluation of a Datalink-Enabled Trajectory-Based Operations Concept

1/10

Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM2011)

CONTROLLER AND PILOT EVALUATION OF A DATALINK-ENABLED

TRAJECTORY-BASED OPERATIONS CONCEPT

Eric Mueller, Dave McNally, Tamika Rentas, Arwa

Aweiss, David Thipphavong, Chester GongNASA Ames Research Center

Moffett Field, CA, USA

[email protected]

Jinn-Hwei Cheng, Joe Walton, John Walker, Chu-Han

Lee, Scott SahlmanUniversity-Affiliated Research Center


Diane Carpenter

Science Applications International Corporation


AbstractResults are presented for a pilot- and controller-in-the-loop evaluation of a near-term datalink-enabled Trajectory-

Based Operations concept with mixed voice and datalink

operations. Eleven recently retired Fort Worth Center

controllers and twelve current commercial pilots evaluated theconcept over 28 hours of simulation time, providing quantitative

metrics on controller workload and trajectory efficiency benefits

and qualitative feedback on the feasibility of the concept. Eight

experimental conditions were tested: four fleet-wide datalink

equipage levels ranging from 0% (voice only) to 80%, along with

two levels of traffic density. Feedback on the feasibility of the

concept was positive from both pilots and controllers, though off-

nominal conditions were not tested. The key objective finding of

the evaluation was that controllers issued significantly more time-

saving flight plan amendments under the datalink conditions

than under the voice-only conditions, amounting to between five

and twelve minutes of flying time savings per hour in the six-

sector area tested. Statistically significant decreases in controller

workload were also measured with increasing datalink equipage

level, but this decrease was far smaller than the variation inworkload across sectors and traffic density levels. The frequency

with which aircraft requests for lateral trajectory changes were

approved did not change with datalink equipage level for voice

requests, but did increase for datalink requests; however, under

all conditions, voice requests were significantly more likely to be

approved than datalink requests.

Keywords-trajectory based operations, datalink, air trafficcontrol and management, human-in-the-loop simulation

I. INTRODUCTION

Air traffic delays in the United States National AirspaceSystem (NAS) will continue to increase as the number of daily

flights increases unless overall capacity is expanded. Estimatesare that the demand will increase up to twofold over the nextfifteen years, with delays increasing much faster than that [1].Currently, uncertainty in the future positions of aircraft leads toincreased air traffic controller workload in maintaining safeseparation between those aircraft, and airspace restrictions areput in place to control the flow of traffic to maintain workloadat a reasonable level. These restrictions result in reducedairspace capacity and longer flight paths with higher fuel burn.A key concept of the NextGen air transportation system is the

use of automation systems to track and predict the futurepositions of aircraft, termed Trajectory-Based Operations(TBO), which should lead to the removal of restrictions andincreases in capacity and efficiency. The concept should also

require minimal changes to flight deck equipage or controllersystems.

Previous modeling and simulation studies of conceptssimilar to datalink TBO suggest that efficiency, capacity andsafety benefits are realizable. Up to half a billion dollars couldbe saved each year through the use of integrated datalink andground automation functions, primarily because capacity couldbe increased in controller workload-limited sectors and therebyreduce delays [2-4]. Trials of datalink in the Miami Air RouteTraffic Control Center (ARTCC) between 2002 and 2003indicated significant reductions in voice frequency congestionwere possible with datalink alone (no TBO required), aproblem that is driving European airspace users to equip with

datalink more quickly than US users [5]. Eurocontrols Link2000+ Program, which is enabling routine transfer ofcommunications and routing instructions via datalink, estimatesan 11% airspace capacity increase when 75% of flights aredatalink equipped.

1Safety can be improved with a datalink

TBO concept by reducing the frequency of operational errors:out of a representative set of 58 errors analyzed in a recentstudy, 20% were the result of pilot-controllermisunderstandings leading to missed altitude level-offs, and52% were from improperly analyzed and issued horizontal orvertical trajectory changes [6]. Instances of errors in eithercategory should be reduced with an integrated datalink-TBOconcept.

While the benefits of datalink usage are generally widely

recognized and datalink is thought to be a key enablingcapability for TBO, translating this perception into aconvincing cost/benefit analysis has been an elusive goal fortwo major reasons. First, the principal direct benefit of datalinkappears to be the reduction of controller workload resultingfrom fewer voice transmissions, most of which are for transferof communications [2, 7, 8]. However, this benefit mechanism

1See Link 2000+ Program website:www.eurocontrol.int/link2000/public/standard_page/overview.html


2/10

is difficult to monetize without broaching the sensitive subjectsof either reducing the number of air traffic controllers orincreasing the allowable number of aircraft in a sector [9]. Thesecond problem with producing a solid cost/benefit analysis fordatalink is that costs to airspace users (in terms of additionalequipage) are quite high under most concepts but the directbenefit to equipped aircraft is no more than that for unequippedaircraft [10]. This creates an incentive for each user to wait toequip as long as possible unless artificial requirements (i.e.regulations) are imposed to provide better service to equippedaircraft. What is needed is a way to equate lower controllerworkload with direct efficiency, capacity and/or safety benefitsfor the system generally and, more preferably, for equippedaircraft.

This paper reports the results of a high fidelity human-in-the-loop (HitL) simulation evaluation of a particular near-termconcept, envisioned as a first step towards en route TBO.Near-term is defined as the time horizon in whichequipped aircraft need nothing new beyond currently-flying(2010) advanced flight deck systems, existing datalink (VDL-2) is used, and existing controller tools are improved, moved tothe R-side, and integrated with datalink. A detailed description

of that concept and a non-HitL simulation analysis of itspotential benefits may be found in [11]. The primary goal ofthe evaluation was to demonstrate the feasibility of TBO in amixed voice and datalink communication modality usingseveral new tools on the controllers R-side display but no newaircraft equipage. The second goal was to explore andobjectively measure benefits provided by these new tools whenintegrated with datalink in terms of controller workload andaircraft flight efficiency. The ultimate goal is to provide resultsto facilitate the design of a cost-effective concept for allowingall aircraft to take maximum advantage of TBO.

II. METHOD

A. Participants

Three teams of commercial pilots and recently retired airtraffic controllers participated in the simulation over fourweeks in September and October 2010. Each team consisted offour test controllers to work traffic, one staff controller to helpcoordinate and train the test controllers, two flight crews (fourpilots) to fly the two high-fidelity cockpit simulators, one staffpilot to coordinate, train and observe the flight crews, and eightpseudo-pilots to fly the lower-fidelity desktop cockpitsimulators.

To ensure feedback on the feasibility of the concept wouldclosely match results from an operational evaluation, a pool ofoperators and users of the air traffic system was recruited. Theeleven air traffic controllers had worked Fort Worth Center

airspace operationally for an average of 25.5 years, and hadbeen retired on average 2.3 years (maximum of 6 years retired).The six flight crewstwelve pilotswho flew the cockpitsimulators averaged 12,900 hours of flight time on a variety ofcommercial transport aircraft and were all currently employedby a major airline. The three crews who flew the 747-400simulator cab were rated on that aircraft and had considerableCPDLC

2 experience; the other three crews flying the research

2Controller-Pilot Data Link Communications

simulator cab were rated on 757 or 767 aircraft and only hadACARS

3datalink experience. This difference was intended to

test the difficulty with which pilots new to CPDLC but familiarwith the Flight Management System (FMS) interface would beable to adapt to the new communication modality. The eightpseudo-pilots who flew the desktop cockpit simulators wereeither currently-rated commercial pilots or retired controllers,and five of the eight had participated in NASA experiments aspseudo-pilots before.

B.Experiment Design

The experiment design was guided by the key assumptionsin the target near-term TBO concept [11], and by two importantvariables relevant to TBO: percentage of aircraft equipped withintegrated datalink/FMS and overall spatial density of traffic.Three key equipage assumptions for the concept were made:ground-based trajectory automation was integrated with theCenter radar (R-Side) controller position; mixed equipagedatalink and voice operations would occur in the sameairspace; and Future Air Navigation System (FANS-1/A)integrated FMS/datalink would be used on equipped aircraft.The values of the experimental variables are shown in Table I.The first team of pilots and controllers evaluated every cell ofthat matrix three times, while the second and third teams hadtime to evaluate each only twice.

Four levels of fleet-wide datalink equipage were tested: 0%,the baseline against which the other conditions are compared;20%, the expected domestic US FANS-1/A equipage levelwhen the datalink infrastructure is enabled in 2016; a 50%equipage level consistent with 2025 projections for NextGen;and an 80% level, a long term estimate of the percentage ofaircraft for which a reasonable business case might be made forupgrading to FANS-1/A.

Two levels of spatial traffic density were tested. The firstlevel represented current traffic density in the southern US. Thesecond traffic level was about 50% higher than this nominaldensity, a value selected because a major benefit of increasedR-side automation and datalink is expected to be reducedcontroller workload. Density in most scenarios tended toincrease with time so even the high density recordings hadperiods of relatively light traffic; because of this within-scenario variation the high and low traffic density conditionsare not compared directly and instead the directly measuredworkload of the controller is correlated with other metrics.

TABLE I. EXPERIMENT MATRIX

Traffic DensityDatalink Equipage

0% 20% 50% 80%

Nominal (1x)

Very Busy (1.5x)

C. Traffic Scenarios

The test airspace included six high-altitude sectors ineastern Fort Worth Center (ZFW); this area was chosen

3Aircraft Communications Addressing and Reporting System


3/10

because of its good mix of arrival and depaDallas-Fort Worth and Houston Internationalover-flight traffic that intersects with thdeparture flows. The six sectors were comoutlined in blue in Fig. 1, with a single controlof the combined sectors. Operationally, whenlow, sectors 90 and 71 are frequently combinefor the other sectors to be combined. Howearea of responsibility provided an additional lwith which to stress the controllers and deterunder the concept. A single ghost controlleoutside the three high-altitude test sectors.

Traffic recordings were generated usingradar track and flight plan data recordings froWorth Air Route Traffic Control Center (A29 and 30, 2010. For the current day (1recordings, the flight plans and initial positionused to initialize the aircraft simulators describscenario started, those simulators calculatedaircraft positions. For the very busy scenarirecordings from the same two days wereindividual flights from other times and dates

traffic count was 50% higher than the currentThese individual flights were time-shifted asthe number of short term conflicts (


4/10

aircraft as is possible on the ground [16]. Thesimilar to that shown in Fig. 3, but usescoupled to a 757-like aerodynamics model aallow modification for research purposes. Pspecific procedures to follow in respondclearances, procedures that were developedearlier experiments in the same facility [17].were used in the 20%, 50% and 80% datalinvoice procedures were used in the 0% datalink

All other aircraft in the scenarios were confour pseudo-pilot stations [18]. Each stationaircraft in a particular sector and wasprofessional pilots. One pilot handled voicewith the sector controller, while the other pilotcontrollers instructions using a simulatedDatalink instructions were automatically acceaccording to a distribution of response time dea previous pilot-in-the-loop experiment [1response delay for a lateral flight plan amseconds (minimum 10, maximum 165) and tfor a vertical amendment was 39 secondmaximum 85). Each pseudo-pilot station g

between 5 and 20 aircraft at a time, the scontroller was managing.

E. Controller Automation Tools

The primary functional improvements ofside system over todays R-side displays weran interactive, rapid-feedback conflict detplanning capability (a form of which is availcontrollers D-side station) [19, 20], a coadvisory function based on the Advanced[21-23], and an integrated datalink capabilityconstruct and transmit the flight plan amendmthe controller using the trial planning capFlight Data Blocks (FDB) on the controllers

in Fig. 4, contained the usual fields found onplus enhancements. Additional fields indicateairport (KDFW), the current sector ownershloss of separation (LOS) in minutes (6), a cadvisory message (L20), and an aircraftindicator (C360) when applicable. Controllroute trial planning function by clicking oairport field, and triggered the altitude trialfrom the altitude field. Clicking the time-to-FDB brought up a graphical depiction of thgeometry, as shown by the red lines in Fig. 5,autoresolution advisory field displayed thconflict resolution maneuver as a trial planyellow line. The datalink message is automat

the controller manipulates the trial planner,the auxiliary waypoint or selecting a differentThe message is transmitted to the aircraft andHost computer database when the coAmend/Uplink button in the trial plan wind6. The trial plan window also contained a listthe trial plan should rejoin the original flighdatalink message to be transmitted, and bamend a clearance without sending a datalinkpilot request, or revert to a previous flight pl

second cockpit is737-style FMS

d is designed toilots were givening to datalink

and refined inhese proceduresconditions, and

conditions.

trolled by one ofmanaged all thestaffed by twocommunicationsimplemented thecockpit display.ted and executedlays measured in]. The medianndment was 56

he median delay(minimum 10,

nerally handled

ame number the

the emulated R- the inclusion of

ction and trialable only on thenflict resolutionirspace Conceptto automaticallynts generated bybility [24]. Thedisplays, shown

n R-side displayd the destinationip (86), time-to-

nflict resolutiondatalink requesters initiated the

the destinationlanning functionOS field in theaircraft conflictand clicking the

e recommended, shown by theically updated as

ither by movingreturn waypoint.

amended in ATCtroller hits thew, shown in Fig.of fixes at whichplan, the actualttons to simply

message, reject aan. The datalink

messages, conforming to the CPformatted to be acceptable to the sior to real FANS-equipped aircraft.

F. Training

Each team participated in the exapproximately 1.5 days were spedesigned scenarios in CTAS to minovel aspects of the interface comfunctional aspects of the R-siprogressing to the data collectiofunctions was reviewed with econsistent use of the automation. Traccomplished considerably fastecontrollers; there were far fewer ch

Figure 4. Flight data blocks with conflict

and datalink request f

Figure 5. Conflict geometry (red) with t

(yellow)

Figure 6. Trial plan and CPDL

DLC standard [25], weremulators in this experiment

periment for four days, andnt training with speciallyimize the degree to which

promised evaluation of thee automation. Beforescenarios a checklist of

ach controller to ensureaining for all the pilots wasr than it was for theanges to current procedures

otification, resolution advisory

om aircraft

rial plan resolution trajectory

C interface window


5/10

for the pilots, and all pilots were able to master the datalinkprocedures in a matter of hours regardless of CPDLCexperience. Training for the pseudo-pilots on procedures andphraseology was conducted for a week before data collection toensure they were capable of managing up to twenty aircraft at atime. Pseudo-pilots were able to attain a level of proficiency inthat first week that was consistently high during the datacollection weeks.

G.

Simulation ProceduresScenarios were presented to the participants in a pseudo-

random order intended to evenly distribute the high and lowworkload scenarios and the datalink-equipage levels, and tominimize any lingering learning-curve effects. In each block offour scenarios two were 1x density and two 1.5x. All fourdatalink levels were represented. Within those constraints theorder was randomized. The order of the scenarios betweenteams was selected to ensure particular scenarios werentoverrepresented at the beginning or end of data collection.

To investigate the effect of datalink-enabled TBO on theapproval of flight deck requests, the pilots in the cab simulatorswere instructed to ask for route efficiency opportunities

whenever they arose. A simulation engineer passed directrouting opportunities to the appropriate pseudo-pilot positionevery 5 minutes. The opportunities were for either voice ordatalink aircraft, and the appropriate modality was then used torelay the request to the controller.

III. RESULTS

The following subsections will describe subjectivefeedback on the feasibility of the TBO concept and objectivebenefits measured during the simulation.

A. Feasibility of TBO Concept

The primary goal of this evaluation was to determinewhether it is feasible to use both datalink and voice

communications in the same airspace to convey routinemessages, simple route, and altitude amendments and complexrouting changes. It was the overwhelming consensus of thepilots and controllers that such mixed-equipage operationswould be appropriate for most of the domestic en routeairspace. Exceptions to this include the very busy northeastregions of the US, particularly Cleveland, Indianapolis,Washington and New York centers.

Controllers provided generally positive comments about theR-side automationfunctionalityadditions, though they did havesuggestions on ways to improve the automation interfaces.Controllers were observed to move from predominantly usingthe trend vectors to predict future positions of aircraft to usingthe trial planner; controllers reported that the trial plannerprovided significantly more information than the trend vectorsbecause of its use of flight plan intent and aircraft performancemodels. Improvements to the vertical trajectory modeling weresuggested, including using the current altitude rate to correctthe predicted rate from the performance model. Theautoresolution advisories frequently corresponded with whatthe controller would have decided to do, but straightforwardimprovements like altitude-for-direction-of-flight conventionsand taking into account the arrival status of aircraft would have

improved its performance. Controllers reported they reliedexclusively on the advisories only when extremely busy, andotherwise it did not lower their workload because they hadformulated a resolution maneuver already. The ability totransmit a datalink message automatically constructed from thetrial planner was frequently cited as a major advantage overcurrent operations, though observation of the controllersactions and their comments suggest that much of the apparentreduction in workload resulting from datalink usage was fromnot having to handle transfer of communications over voice.This finding is consistent with past studies of datalink TBO [2].

Controllers and pilots both expressed concerns stemmingfrom datalink response delays and particularly the lack of animmediate response that at least acknowledges receipt of adatalink message. Uplink messages sent to existing flightdecks take a median of one minute for pilots to load, visualizeor otherwise evaluate, execute, and send a response, whereascontrollers reported that the maximum allowable time (ratherthan the median time) needs to be less than sixty seconds. Thisclosely matches the opinions of controllers using datalink inEuropean trials [26]. This response delay meant thatcontrollers, on average, were comfortable using datalink to

issue resolutions when the conflicts were at least 4.5 minutesaway for altitude changes and 6.4 minutes away for lateral pathchanges. Otherwise, controllers would resort to clearancesissued by voice. Downlink messages requesting a route oraltitude change were displayed in the flight data block, butwere sometimes not noticed by the controller. This could be atraining issue that would not appear in practice; however thepost-run discussions on this topic revealed the desire for somekind of acknowledgement capability through messaging orvoice procedures, or simply faster response times via betterinterfaces or datalink procedural changes.

The pilots found little else to object to in the concept, likelya result of requiring no new flight deck equipage and only

minor modifications of existing datalink procedures. All pilotsquickly learned to use CPDLC regardless of prior experience,calling it simple and straightforward. Certainly the use of anew communication modality like datalink will introduce newconsiderations regarding human-machine interaction errors [17,27, 28], but solutions to these problems have been identified inprior research and the problems are unlikely to be fundamentallimitations that preclude wide use of datalink.

Perhaps the most important qualitative result of thesimulation was that a mix of voice and datalinkcommunications appears to be the best way for controllers andpilots to interact. Voice communications are critical insituations when time-to-LOS is less than several minutes,clarification is needed because several datalink messages have

been exchanged, a complex reroute needs to be negotiated, apilot needs to deviate due to weather or aircraft performance, orin emergency circumstances. Datalink communications aremost effective when the instruction is routine and the expectedresponse is WILCO, or when a more complex instruction canbe automatically loaded and executed by the aircrafts FMS.Based on this studys findings, there is no reason to suppose afuture ATC system must or should attempt to completelyreplace voice with datalink communications.


6/10

B.Benefits of TBO Concept

The following sections will present specific objectivebenefits of the TBO concept. Note that, in comparing thedistributions of ratings using an analysis of variance(ANOVA), p-valuesthe probabilities that two data sets haveequal meansdisplayed at the bottoms of plots are alwaysrelative to the 0% group; unless otherwise noted the 20%, 50%and 80% groups are not compared with each other. The

measure of statistical significance used here is p=0.10, whichis less strict than the usual p=0.05 because data collected in aHitL is necessarily sparse, the presence of other experimentalfactors creates large variations in the variable being tested, andbecause subjective feedback (e.g. in terms of workload versusdatalink percentage) agrees with the statistical tests.

1) Flight Plan AmendmentsOne of the most important proposed benefit mechanisms of

a datalink-enabled TBO concept is that additional automationwould reduce controllers workload for a given level of traffic,which in turn would allow controllers to provide more efficientconflict resolution maneuvers and respond positively to aircraftrequests more frequently. This hypothesis can be testedbroadly by examining the predicted change in flying time forevery flight plan amendment input by the controller. Ahistogram showing the distribution of flying time changes forevery flight plan amendment in the experiment is given in Fig.7. That figure provides context for the types of trajectorychanges used by controllers en route that can impact efficiency,and indicates the potential for time savings or delays from eachtype of amendment. Fig. 7 shows that direct-tos (D2s), lateralamendments in which an aircraft flies direct to a downstreamwaypoint bypassing at least one waypoint on its route of flight,account for the majority of flying time savings for lateral routechanges. Flying time differences for those maneuvers take intoaccount the effect of the wind field, and the controller displayincludes the predicted flying time difference for the D2 route orany route change. Route amendments, in which at least onenew auxiliary waypoint is added to the flight plan of theaircraft, usually add flying time. In both of these cases a flyingtime savings should translate into a fuel savings as well,although the actual fuel savings was not calculated during theevaluation. Vertical trajectory changes showed the largestrange of flying time differences: from ten minutes of savings tosix minutes of delay. This large variation arises because theairspeed is predicted to change with altitude; however the fuelburn will be based on the aircrafts optimum altitude forefficiency rather than simply the flying time.

The key objective finding of this experiment is thatcontrollers issue more time-saving flight plan amendmentswhen more datalink-equipped aircraft are under their control;

the relationship between these variables is shown in Fig. 8. Thedashed lines indicate results for D2 amendments while solidlines are for general route amendments. Blue lines show thetotal flying time differences for all amendments at a particulardatalink percentage while green lines represent the averageflying time difference for a single amendment.

4The data

indicate that at most a weak relationship exists between the

4Flying time difference = new flight plan trajectory flying timeminus old flight plan trajectory flying time

average flying time savings for a single amendment and overalldatalink equipage level (dashed green line), with the averageD2 savings reaching a maximum of 0.88 minutes at 50%equipage compared with 0.78 minutes at 0% and intermediatesavings at 20% and 80%. The key variable that doesdepend ondatalink percentage is the number of flight plan amendments,specifically D2s, that are sent at higher equipage levels:because D2s save flying time the increased number of theseamendments means a greater overall savings at higher datalinkequipage levels (dashed blue line). The savings are significant,rising from 163.5 minutes at 0% to 207 minutes at 20% and80%, and topping out at 248 minutes for 50% equipage, eachover a total of seven hours of scenario time. These represent,respectively, a 27% and 52% increase in savings over the

baseline voice-only condition. These results are based onanalysis of 1104 flight plan amendments (1013 of which areD2s) over 28 hours of simulation and are summarized in TableII. If confirmed in operational trials, this result would indicatean additional five to twelve minutes of flying time savings perhour could be realized in just the eastern portion of ZFW astested here. It should be reiterated that the same trafficrecordings were used for the four datalink equipage levels(though the call signs were changed) so the comparison isdirect, and that the additional savings occurs above what would

Figure 7. Distribution of flying time changes for flight plan amendments

Figure 8. Total and average flying time differences for route and D2

amendments


7/10

be expected in voice-only conditions. An analysis of thepercentage of aircraft that spent time off their flight plansshows that for all conditions it was between 7.43 and 7.95%, soit is not the case that controllers in the voice-only condition stillissued time-saving D2s but did not enter them as flight planamendments. The experimental finding that more time-savingamendments are sent at higher datalink equipage levelsrepresents a significant and direct benefit mechanism forairspace users.

TABLE II. D2FLYING TIME SAVINGS

D2 VariableDatalink Equipage

0% 20% 50% 80%

# Amendments 212 255 282 264

Average t (min) -0.77 -0.81 -0.88 -0.79

Total t (min) -163.5 -206.8 -248.0 -208.0

2) TLX Workload RatingsThe result from the previous section that higher datalink

equipage levels result in more time-saving amendments beingissued raises the question of whether it is a general reduction incontroller workload that accounts for the additionalamendments, whether it is due to the ease with whichamendments can be sent to the Host and to datalink-equippedaircraft with a single mouse click, or something else. End-of-run workload ratings were provided by controllers using theNASA Task Load Index (TLX) scale [29], and those ratingsindicate that at least some of the difference is due to workloadreductions at higher equipage levels.

The TLX workload scale asks participants to rate theiroverall workload during a particular scenario along sixdimensions: mental demand, physical demand, temporaldemand, performance, effort and frustration. These component

ratings can then be examined separately or averaged together togain an overall picture of a participants workload during asingle scenario. In order to capture and analyze the peakworkload during each scenario, participants were asked toprovide ratings corresponding to the busiest periods theyexperienced. The ratings were normalized according to z-scorefor each participant, by which the mean TLX rating for a givenparticipant became a zero z-score, and for every standarddeviation in TLX rating away from a given participants meanrating the z-score was higher or lower by one point. Thistransformation allows a more direct comparison across thedifferent participants: while they were given guidelines on whateach rating meant, inevitably individual controllers will havedifferent average ratings and use more or less of the full scale.

Workload ratings for most scenarios and sectors were low andonly about 17% of the scenarios were given TLX averageratings of 5 or higher.

Fig. 9 shows the average of the six component ratings foreach participant in each scenariolabeled single runbroken down by datalink percentage. A one-way analysis ofvariance was performed between the voice-only condition andeach of the datalink conditions and resulting p-values areincluded in Fig. 9. The figure shows a likelihood of between 9

and 12% that workload at 50% and 80% datalink equipage isequivalent to the workload at 0% (p=0.094 to 0.118).Controllers reported anecdotally that they believed workloadwas much lower when datalink equipage was higher, so therelatively poor p=0.118 confidence level of a measurabledecrease in workload is likely due to the wide variation inworkload ratings for a particular datalink equipage levelnotbecause the difference is due to chance. This variation occursbecause different sectors and different traffic recordings haddifferent traffic densities, but each point in one of the fourequipage levels (e.g. sector 86, 1.5x traffic density, 80%equipage) corresponds to the exact same point in one of theother equipage levels (e.g. sector 86, 1.5x density, 0%equipage). The important result from Fig. 9 is that for the sametraffic densities and sectors, a datalink equipage level of 50%or more provides a measurable reduction in controllers self-rated workload.

The individual workload components of temporal demand,effort and, interestingly, physical demand experienced astatistically significant decrease (at the p=0.10 level) withincreasing datalink equipage. No statistically significant resultswere found for the other three categories (frustration, mental

demand, performance). The ratings for temporal demand onlybecome significant when 80% equipage is reached. The effortratings are significant at 50% equipage (p=0.070) but not at80% (p=0.152). The physical ratings are significant at both50% and 80% equipage (p=0.069 and 0.038 respectively),perhaps because of the very significant reduction in the amountof talking required of the controller (see the final section inResults).

The end-of-run TLX ratings provided by the controllerssuggest, consistent with their verbal feedback, that datalinkequipage levels of 50% or higher result in measurably lowerworkload than the baseline voice-only condition. Togetherwith the results of the previous section it appears that higher

Figure 9. TLX Average workload ratings by fleet datalink percentage


8/10

datalink levels reduce controller workload and allowcontrollers to send more time-saving flight plan amendments toboth voice and datalink aircraft.

3) Realtime Workload RatingsIn addition to end-of-run TLX workload ratings controllers

provided realtime ratings of generic workload (i.e. notcategorized into mental, physical, etc.) every two minutesduring the scenarios. Such a realtime measure is useful because

controllers select a workload rating quickly without over-thinking the metric, it does not rely on controllers memory ofearlier workload, and it allows for comparison of metrics atspecific times within scenarios rather than scenarios as a whole.The realtime workload ratings are normalized into participant-specific z-scores in the same way the TLX scores werenormalized; for reference, the mean of the raw scores was 2.8out of 7, and only 9.7% of the ratings were 5 or higher.

A variance analysis of the realtime workload data given inFig. 10 shows a significant (p=0.066) reduction for the 80%

case compared with the voice-only case, mirroring the resultsfrom TLX, but actually indicate that workload reaches a peakat 20% datalink equipage. This last result was not reflected inthe TLX data and was not consistent with controllerscomments (controllers suggested that the higher the proportionof datalink aircraft the better, and 20% datalink was preferableto voice-only), so it may simply be due to chance. The realtimeworkload data appear to confirm the TLX results that workloaddecreases with the highest datalink equipage levels, but furtherstudy is required to determine whether a small percentage ofdatalink-equipped aircraft will actually increase a controllersworkload.

A slightly different view of the realtime workload ratings isprovided by Fig. 11. That figure shows the cumulativeproportion of workload ratings below a given z-score for eachdatalink percentage. Lines that lie to the left of and aboveother lines represent conditions with lower workload because ahigher percentage of the workload ratings in that condition arebelow the z-score on the x-axis. The take away from this figureis that there is a lack of very high realtime workload ratings atthe 80% datalink level. This may indicate that at low trafficlevels the 80% datalink condition does not have a major effect

on workload, but that at the highest traffic densities workloaddoes decrease if more aircraft are datalink equipped. If borneout by further study, such a finding would suggest datalink-enabled TBO is most useful when large numbers of aircraft aremanaged by a controller and that the maximum number ofcontrollable aircraft may be higher than it is with todayssystems.

4) Aircraft RequestsA hypothesized benefit mechanism of datalink-enabled

TBO is increased acceptance of aircraft (generically, user)requests with higher datalink equipage levels because ofreduced workload and the ease with which a datalink requestcould be evaluated and approved without verbal pilot-controller

exchanges. As specified previously, D2 requests were made bythe pseudo-pilots approximately every five minutes accordingto test procedures, and once or twice per scenario by the high-fidelity cockpit simulators; the total number of requests wasrelatively constant among the datalink equipage conditions(ranging from 55 to 70) and were generally distributed betweenvoice and datalink in the same proportion as the overall fleetequipage percentage.

There is no significant trend towards increased acceptanceof simple route requests with increasing datalink equipagelevels. Fig. 12 shows the trend in overall percentage ofapproved requests, approved voice requests and approveddatalink requests. The overall and voice request approval ratesare roughly 85 to 90% under most conditions, but drop to about

75% approval for the 50% equipage level. No explanation forthis minimum is apparent. In contrast, the approval rate fordatalink requests increases from 63% to 78% as fleet datalinkequipage increases. This is likely partly a result of trainingissues, in which at lower datalink equipage levels controllersare not scanning flight data blocks for aircraft requests as oftenas they are at high equipage levels, and partly a result ofworkload. Because workload tends to decrease with increasingdatalink equipage the controller has more time to look for D2requests when more aircraft are datalink-equipped. The voice

Figure 10. Realtime workload z-scores as a function of fleet datalink

percentage

Figure 11. Cumulative distribution of realtime workload ratings by

fleet datalink percentage


9/10

requests do not show this trend because controllers were forcedto immediately respond to such requests, while they couldintentionally or unintentionally ignore datalink requests.Perhaps the only unequivocal take-away from this plot, and onethat was unanimously confirmed by controller feedback, wasthat voice requests are far more likely to be approved thandatalink requests. This may suggest datalink should be usedfor routine operations that will reduce voice frequency clutter,like transfer of communications, in order to free the voicechannel for aircraft requests and other negotiations that aremore difficult to accomplish with textual datalink. It should beemphasized that these results apply only to simple routerequests that can be made either by voice or datalink; morecomplex route requests not studied here would only be possiblewith datalink. This benefit mechanism has not been quantified.

5) Voice Channel AnalysisThe final set of analyses presented here relates to improved

modeling of controllers voice communications patterns under

the datalink-enabled TBO concept rather than measurement ofa particular benefit mechanism. Fig. 13 shows the relationshipbetween the fleet datalink equipage level and the percentage oftime either the controller or pilot was talking on the voicefrequency in that particular sector. As in the previous resultsdivided by fleet equipage, there is large variability within anysingle condition because different sectors and traffic recordingshave widely differing numbers of aircraft. What is interestingabout Fig. 13 is not the decrease in use of voice with increasingproportion of datalink equipagethat result was inevitablebut that the average talking times for the different conditionsare very nearly linear with fleet datalink equipage. The talkingtimes ranged from 28.5% when all aircraft are voice-equippedto almost exactly zero when all aircraft are datalink-equipped.

The best linear approximation to this relationship is

Mean Talking% = 0.29DL% + 28.5 (1)

IV. CONCLUSIONS

The results of a pilot- and controller-in-the-loop evaluationof a datalink-enabled Trajectory-Based Operations concept arepresented. The concept was designed for near-termimplementation and therefore supported a mixture of voice-

and datalink-equipped aircraft in the same airspace, currentlyflying aircraft Flight Management Systems integrated withdatalink, and a set of new R-side controller automation tools.Four levels of fleet-wide datalink equipage were tested, rangingfrom voice-only operations up to 80% equipage, along withtwo levels of traffic density.

The most important objective finding of the evaluation wasthat controllers issue more time-saving lateral routeamendments under higher datalink equipage levels than they dounder voice-only operations, and the total resulting timesavings is between 27% and 52% more than the baseline case.This translates into between 5 and 12 minutes of flying timesavings per hour in the six-sector area of eastern Fort WorthCenter tested. Realtime measures of controller workload andend-of-scenario evaluations indicated that increased datalinkequipage levels reduce workload, but the reduction was smallin comparison to the variation due to the number of aircraft inthe sector. Pilots reported they were very comfortable using the

datalink procedures provided to request flight plan amendmentsor approve controller clearances. Controllers recommendednumerous improvements to the simulated R-side system, butindicated that the functionalities tested here could significantlyimprove the ease and efficiency with which they managedtraffic once the functionalities were integrated into anoperational system. It was found that simple aircraft routerequests were significantly more likely to be approved whenrequested by voice than by datalink, and that the percentage ofdatalink-equipped aircraft did not have a clear effect on thefrequency with which all requests were approved, though it didhave a positive effect on the frequency with which datalinkrequests were approved.

REFERENCES[1]

Next Generation Air Transportation System Integrated Plan,Department of Transportation, Dec. 12, 2004.

[2] H. Hasan, K. Leiden, S. Mondoloni, D. Kozarsky and S. Green, AnInitial Benefits Assessment of Distributed Air/Ground TrafficManagement Concept Elements, AIAA 2003-6806, AIAA AviationTechnology, Integration and Operations Conference, Denver, CO, Nov.2003.

[3] M. Bennett, D. Knorr and J. Rakas, Economic Benefits of Increased EnRoute Sector Capacity from Controller-Pilot Data Link

Figure 12. D2 request acceptance rateFigure 13. Relationship between fleet datalink percentage and voice usage


10/10

Communications, 83rdTRM Annual Meeting, Transportation ResearchRecord, 2004.

[4]

Computer Sciences Corporation, Life-Cycle Cost/Benefit Assessmentsof DAG-TM Concept Elements 5, 6 and 11: Version 4.0, Final Report,Air Traffic Management System Development and Integration, NASAAmes Research Center, DOC ID CSC/ATMSDI/CTO-08-10/040630,June 2004.

[5]

R. Bolczak, J. Gonda, W. Saumsiegel and R. Tornese, Controller-PilotData Link Communications Build 1 Value-Added Services, DigitalAvionics Systems Conference, Salt Lake City, Oct. 2004.

[6]

R. Paielli and H. Erzberger, Improved Conflict Detection for ReducingOperational Errors in Air Traffic Control, AIAA 2004-6392, AIAAAviation Technology, Integration and Operations Conference, Chicago,IL, Sept. 2003.

[7] R. Bolczak, Operational Evolution of an Integrated URET/CPDLCCapability, 21stDigital Avionics Systems Conference, Irvine, CA, Oct.2002.

[8]

J. Celio, CPDLC Integration with URET A Concept for ImprovingAirspace User Services, 20th Digital Avionics Systems Conference,Daytona Beach, FL, Oct. 2001.

[9] K. Leiden, P. Kopardekar, and S. Green, Controller Workload AnalysisMethodology to Predict Increases in Airspace Capacity, AIAA 2003-6808, AIAA Aviation Technology, Integration and OperationsConference, Denver, CO, Nov. 2003.

[10] T. Prevot, T. Callantine, J. Homola, P. Lee, J. Mercer, E. Palmer and N.Smith, Air/Ground Simulation of Trajectory-Oriented Operations withLimited Delegation, 7thUSA-Europe ATM R&D Seminar, Barcelona,Spain, July 2007.

[11]

D. McNally, E. Mueller, D. Thipphavong, R. Paielli, J. Cheng, C. Lee,S. Sahlman, and J. Walton, A Near-Term Concept for Trajectory-BasedOperations with Air/Ground Data Link Communication, 27 thInternational Congress of the Aeronautical Sciences, Nice, France, Sept.2010.

[12] D. McNally, S. Engelland, R. Bach, W. Chan, C. Brasil, C. Gong, J.Frey and D. Vincent, Operational Evaluation of the Direct-ToController Tool, Proceedings of the 4 th USA/Europe Air TrafficManagement R&D Seminar, Santa Fe, NM, 2001.

[13] H. Swenson, T. Hoang, S. Engelland, D. Vincent, T. Sanders, B. Sanfordand K. Heere, Design and Operational Evaluation of the TrafficManagement Advisor at the Fort Worth Air Route Traffic ControlCenter, Proceeding of the 1st USA/Europe ATM R&D Seminar,Sacalay, France, 1997.

[14] D. McNally and C. Gong, Concept and Laboratory Analysis ofTrajectory-Based Automation for Separation Assurance, Air TrafficControl Quarterly, Vol. 15(1), 2007.

[15]

T. Farley, S. Landry, T. Hoang, M. Nickelson, K. Levin, D. Rowe and J.Welch, Multi-Center Traffic Management Advisor: Operational TestResults, AIAA 2005-7300, AIAA Aviation Technology, Integrationand Operations Conference, Arlington, VA, Sept, 2005.

[16]

B. Sullivan, R. Panda, and P. Soukup, Implementing FANS in theNASA 747-400 Flight Simulator for Airspace Operational Research(Future Air Navigation System), AIAA Flight Simulation TechnologiesConference, Washington, D.C., 1996.

[17]

E. Mueller, and S. Lozito, Flight Deck Procedural Guidelines forDatalink Trajectory Negotiation, AIAA 2008-8901, 26th Congress ofthe International Council of the Aeronautical Congress (ICAS),Anchorage, AK, Sept. 2008.

[18]

T. Prevot, Exploring the Many Perspectives of Distributed Air TrafficManagement: The Multi Aircraft Control System MACS, Proceedingsof the International Conference on Human-Computer Interaction inAeronautics, AAAI Press, Menlo Park, CA, Oct. 2002, pp. 149.

[19]

D. McNally, H. Erzberger, R. Bach and W. Chan, A Controller Tool forTransition Airspace, AIAA-99-4298, AIAA Guidance, Navigation andControl Conference, Portland, OR, Aug. 1999.

[20]

D. McNally, R. Bach and W. Chan, Field Test Evaluation of the CTASConflict Prediction and Trial Planning Capability, AIAA-98-4480,AIAA Guidance, Navigation and Control Conference, Boston, MA,Aug. 1998.

[21] H. Erzberger, T. Lauderdale and Y. Chu, Automated ConflictResolution, Arrival Management and Weather Avoidance for ATM,27thInternational Congress of the Aeronautical Sciences, Nice, France,Sept. 2010.

[22] H. Erzberger, Transforming the NAS: The Next Generation Air TrafficControl System, 24th International Congress of the AeronauticalSciences, Yokohama, Japan, Sept. 2004.

[23]

H. Erzberger, Automated Conflict Resolution for Air Traffic Control,25th International Congress of the Aeronautical Sciences, Hamburg,Germany, Sept. 2006.

[24]

E. Mueller, Experimental Evaluation of an Integrated Datalink andAutomation-Based Strategic Trajectory Concept, AIAA 2007-7777,AIAA Aviation Technology, Integration and Operations Conference,Belfast, Northern Ireland, Sept. 2007.

[25] Interoperability Requirements for ATS Applications using ARINC 622Data Communications (FANS 1/A Interop Standard), DO-258A, RTCASC-189, Washington D.C., 2005.

[26]

PETAL Newsletter, PETAL-to-LINK Programme, Eurocontrol, Vol.4, No. 2, Apr. 2002.

[27]

M. Mackintosh, S. Lozito, A. McGann and E. Logsdon, DesigningProcedures for Controller-Pilot Data Link Communication: Effects ofTextual Data Link on Information Transfer, AIAA 1999-01-5507,AIAA World Aviation Conference, Vol. 1, Washington D.C., Oct. 1999.

[28]

W. Olson and N. Sarter, Management by Consent in Human-MachineSystems: When and Why it Breaks Down, Human Factors: The Journalof the Human Factors and Ergonomics Society, Vol. 43, No. 2, 2001, pp.255-266, DOI: 10.1518/001872001775900904

[29]

S. Hart and L. Staveland, Development of NASA-TLX (Task LoadIndex): Results of Empirical and Theoretical Research, Human MentalWorkload, P.A. Hancock and N. Meshkati (eds.), North Holland Press,Amsterdam, The Netherlands, 1988, pp. 239-350.

AUTHOR BIOGRAPHY

Eric R. Mueller received a bachelors degree in aerospace engineering fromPrinceton University in 2000 and a masters degree in the same field fromStanford University in 2005. He has worked at NASAs Ames Research Centersince 2000 in fields ranging from air traffic automation systems, spacecrafthandling qualities, hybrid rocket propulsion, and unmanned aircraft access tothe National Airspace System.

controller and pilot evaluation of a datalink-enabled trajectory-based operations concept

Documents