may 15-17, 2002 national aeronautical and space administration and volpe national transportation...
TRANSCRIPT
May 15-17, 2002
National Aeronautical and Space Administration and Volpe National Transportation Systems Center, US Dept. of Transportation
Performance Measures Approach for NASA Virtual Airspace Modeling and Simulation (VAMS)
2
Table of Contents
»VAMS Introduction
»Measure Requirements, Message, and Framework
»NASA VAMS Measure Approach
NASA VAMS Performance Measures
Security
VIRTUAL AIRSPACE MODELING AND SIMULATION
Security
ASASV
irtu
al A
irsp
ace
Mo
del
ing
& S
imu
lati
on
- P
roje
ct R
ead
ine
ss R
evie
w
Project Vision
The Virtual Airspace Modeling and Simulation Project provides the technologies and processes for conducting trade-off analyses amongst future air transportation system’s concepts and technologies
Modeling &Simulation Tools
OperationalConcepts
Evaluation Methods& Techniques
ASASV
irtu
al A
irsp
ace
Mo
del
ing
& S
imu
lati
on
- P
roje
ct R
ead
ine
ss R
evie
w
Project Goals & ObjectivesDevelop the capability to model and simulate behavior of air
transportation system concepts and their elements to never-before-achieved levels of fidelity
» Develop a set of analytical and computational models and methods to conduct detailed assessments of candidate operational concepts
» Establish simulation capability that will enable safe investigation of complex advanced air transportation concepts, and develop a deeper understanding of human performance interaction within it
Develop advanced air transportation concepts
» Develop a set of potential operational concepts, concepts of use, and architectures, providing definitions of the future air transportation system and its elements
» Develop technology roadmaps to achieve these concepts
Conduct assessments of advanced air transportation concepts
» Address potential benefits, identify risks and limits, and evaluate performance, safety, operations, and National Airspace System infrastructure and transition challenges
ASASV
irtu
al A
irsp
ace
Mo
del
ing
& S
imu
lati
on
- P
roje
ct R
ead
ine
ss R
evie
w
ASASV
irtu
al A
irsp
ace
Mo
del
ing
& S
imu
lati
on
- P
roje
ct R
ead
ine
ss R
evie
w VAMS Technical Objectives• Develop and validate modeling and simulation tools
providing the multi-objective (safety, capacity, cost) trade space to analyze air traffic management (ATM) concepts to meet forecasted demands of the 2020’s
• Develop operational scenarios, metrics and evaluation methodologies to assess potential operational concepts and technologies to meet the forecasts across the trade space
• Create operational concepts and conceptual architectures that can be used to define the next generation air transportation system, and develop technology roadmaps, to meet long-term Enterprise goals
7
Table of Contents
»VAMS Introduction
»Measure Requirements, Message, and Framework
»NASA VAMS Measure Approach
NASA VAMS Performance Measures
8
Performance Measures Support Decision-Making and Communication
Actionable performance measures for VAMS provide knowledge to support:
Decision-making, e.g.:» Concept is worthwhile to pursue
» Suggest refinements to a concept
Communications, e.g.:» Communicate with FAA about benefits, cost, and feasibility of
concepts
» Tell external stakeholders expected benefits and costs of a concept
» Convey program results or status to top management
» Tell OMB and Congress about program
9
Aviation Performance Measure Problems
Some difficulties in determining aviation measures: Variety of decision-makers and stakeholders Difficult to define a few measures that capture impacts of
interest to all stakeholders Measures are often not easily understandable Measures often do not give direct picture of end-benefits Measures are often not useful for decision-making Often difficult to utilize qualitative measures (e.g., survey
results or anecdotal information) in creditable way
10
Developing Actionable Aviation Performance Measures
Keys to developing actionable performance measures:
Identify requirements for measures to support decision-making and communication
Define the message the measures should tell to support decisions and communication
Develop a narrative framework for the measures to present all necessary information and aid understandability
Include both quantitative and qualitative measures where appropriate
11
Characteristics of Volpe Approach
Identify measure requirements to provide information desired by all decision-makers & stakeholders
Define message performance measure are to convey that is understandable by all decision-makers & stakeholders
Develop framework to» Present direct impact measures, which are usually
calculatable, and relate these to end-user benefits, which are often difficult to calculate
» Combine observed data with estimated data» Use survey or anecdotal data where useful to make the
case (i.e., include quantitative and qualitative measures).
12
Designing Actionable Performance Measures
Develop Actionable
Performance Measures
Identify decisions
Identify information
needed
Top-Down Requirements: What information is needed? Bottom-Up Requirements:
What are the potential impacts and what data is available?
Interactive
ProcessIdentify decision-makers &
stakeholdersIdentity potential project impacts
Identify data sources
Design the measure framework and measures to tell the message
13
Decision-Makers and Stakeholders: Requirements
What are the desired uses of performance measures by each decision-maker and stakeholder? e.g.: NASA: Promising concepts and technologies to pursue FAA: Promising concepts to support Air carriers: Impacts of potential concepts on their operations, revenues, and
costs Manufacturers: Impacts of potential concepts on their products, revenues, and
costs Pilots: Impacts of potential concepts on their tasks Air traffic controllers: Impacts of potential concepts on their tasks General aviation: Impacts of potential concepts on their operations, access to
services, and costs Cargo carriers: Impacts of potential concepts on their operations, revenues,
and costs
(cont’d . . . )
14
Decision-Makers and Stakeholders: Requirements (cont’d)
What are the desired uses of performance measures by each stakeholder? e.g. (cont’d): Cargo carriers: Impacts of potential concepts on their operations, revenues,
and costs U.S. Government
» Executive Branch: Office of Management and Budget: Benefits and costs; feasibility and directions of concepts; relation to related NASA programs
» Congress: Benefits and costs to stakeholders; feasibility and directions of concepts
Other countries: Compatibility impacts Airport operators: Impacts of potential concepts on their operations, revenues,
and costs Flying public: Air travel service, safety, security, & travel costs General public: Noise and air pollution
Capacity Efficiency Predictability Flexibility Environment
Direct aircraft operator costsAATT Economic Measures
AATT Top-LevelPerformanceMeasures
Domain-Specific(Terminal) PerformanceMeasures
Tool-SpecificPerformanceMeasures forPFAST
Example 1: Measure Hierarchy – NASA AATT Project
Total flights flown
Total aircraft travel time
Total aircraft miles
Number of flights more than 15 minutes late from scheduled arrival time
# of user requests honored(measuredin surveys)
Emissions
Noise
Arrivals – (at defined set of airports)
Average number of airport arrivals per hour during peak periods
Total number of airport arrivals per year
Departures – (at defined set of airports)
Average number of airport departures per hour during peak periods
Total number of airport departures per year
Average number of arrivals per hour, per runway, during peak periods
16
Example 2: Measure Flow - FAA Safe Flight 21 Measures
CapabilitiesDirect Output Measures
End-BenefitOutcome Measures
End-Benefits
Direct Impacts
CapabilitiesDirect
Impacts
Display in cockpit of surrounding traffic/equipment
Pilot able to better identify aircraft to follow
Pilot awareness of all proximate traffic positions
Direct Output Measures
Pilot response time for ATC traffic call-out
•Flight time from final approach fix to touchdown
Benefit Impacts
Reduced arrival delays
•Increased predictability of arrival times
End-Benefit Outcome Measures
Safety Accident rate during final approach maneuvers
User Cost Savings/Revenue Enhancement
Arrival rate
FAA Cost Savings· Voice channel occupancy time
17
Table of Contents
»VAMS Introduction
»Measure Requirements, Message, and Framework
»NASA VAMS Measure Approach
NASA VAMS Performance Measures
Operational Scenarios
•Fast-Time Modeling
Concepts
•Real-time Simulations
1. Scope:•issues•NAS Domain•challenges•assumptions
2. Top Level Descriptions:•core ideas•functions
3. Detailed Descriptions:•performance•roles, responsibilities @ humans & machine•human factors•user interfaces
4. NAS infrastructure & technology impacts:•transition planning•architecture•technology requirements
•Empiric Analysis(i.e. expert opinions)
Direct Output
MeasuresEnd-Benefit Outcome
Measures
Stakeholder Viewpoints(questions to be answered)
•Number of traffic events (takeoffs, sector crossings, landings, etc.)•Number of communication events (requests, clearances, directives, etc.)•Throughput (traffic volume)•Delay in phases of flight•Safety incidents (proximity to minimum separation, incursions, encroachments, etc.)•Elapsed flight times•Fuel burn by phase of flight•Personnel workloads•Etc.
Scenario Elements:•NAS Domain•NAS Perturbations(e.g. Wx, Security Incidents)•Origin/Destination Demand•Assumed Technologies•Human/Machine Performance•Defined ATM Procedures•Assumed Equipage•Fleet Mix•Etc.
Stakeholder Viewpoints(questions to be answered)
Average aircraft flight time per air route*•Operational cost per flight mileAverage airport arrival rate during peak periods Average taxi time from pushback to wheels up during peak traffic periods per specific airports or taxi paths within airportsAverage voice channel occupancy time per departure from pushback to take offFuel usageAircraft maintenance costs per flight mileEtc.
* a defined city pair air route
Analysis
Framework for VAMS Evaluation
19
Constraints to Achieving Capacity
To foster a standardized concept evaluation approach, we will develop measures that address common categories of constraints on aviation capacity:» ATC complexity» “Choke” points» Weather» Pilot/Controller/Operator limitations related to performance
and technology» Airport design» User scheduling» Policy on airport/airspace use» Aircraft performance
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
jack perkins:
“To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …”?
20
Potential Evaluation and Measure Approach1. Identify capacity constraint(s) being targeted by concept2. Define concept functionality and map functionality to the
targeted capacity constraint3. Develop/assign measures to concept evaluation
» Direct output measures» End-benefit outcome measures
4. Define operational scenario(s) for the concept evaluation
5. Evaluate concept» Fast-time modeling» Real-time simulation» Empirical assessment
21
Example – Terminal Arrival Maneuvering for Weather Avoidance
Concept functions: Onboard weather detection and route planning equipment
enable flight crew to autonomously navigate around weather cells and maintain separation from terminal area traffic at ATSP discretion
Unequipped following aircraft would be authorized by ATSP to perform self-spacing from an equipped lead aircraft.
Equipped lead aircraft broadcasts trajectory intent information to all local traffic via datalink.
Impacted constraint categories on aviation capacity: Weather
22
Example – Terminal Arrival Maneuvering for Weather Avoidance (cont’d)
Direct Output measures Pilot confidence in equipment in
weather cell avoidance Pilot confidence in equipment for
accuracy in displaying terminal area traffic positions
Average terminal arrival flight time for equipped aircraft compared non-equipped during peak traffic periods
Average number & duration of ATC arrival vectoring callouts to equipped flights during arrival maneuvers impacted by weather cell avoidance
Percentage of flight crew preferred arrival routes made possible during testing (peak arrival periods)
End-Benefit Outcome measures Airport annual arrival rate during
peak periods Total number of airport arrivals per
year Average/Total voice channel
occupancy time per frequency per terminal devoted to communications involving arrival vectoring during peak traffic periods
Total and standard deviation of flight time from 250 miles out to touchdown during peak periods
Total fuel consumption (all flights) per year from 250 miles out to touchdown
23
AddendumExamples of Performance Measures
NASA VAMS Performance Measures
» NASA Advanced Air Transportation Technology Program
» FAA Safe Flight 21 Program
» Federal Railroad Administration Safety R&D Program
» FAA R&D Program
24
Examples of Actionable Performance Measure Narrative Frameworks
Example 1: NASA Advanced Air Transportation Technologies (AATT) Project
Issue: AATT has many projects with different impacts.
Desired Narrative: Show how the AATT projects form a coherent program to support goals.
Solution: Hierarchical measure structure.
(cont’d . . . )
Capacity Efficiency Predictability Flexibility Environment
Direct aircraft operator costsAATT Economic Measures
AATT Top-LevelPerformanceMeasures
Domain-Specific(Terminal) PerformanceMeasures
Tool-SpecificPerformanceMeasures forPFAST
Example 1: Measure Hierarchy – NASA AATT Project
Total flights flown
Total aircraft travel time
Total aircraft miles
Number of flights more than 15 minutes late from scheduled arrival time
# of user requests honored(measuredin surveys)
Emissions
Noise
Arrivals – (at defined set of airports)
Average number of airport arrivals per hour during peak periods
Total number of airport arrivals per year
Departures – (at defined set of airports)
Average number of airport departures per hour during peak periods
Total number of airport departures per year
Average number of arrivals per hour, per runway, during peak periods
26
Examples of Actionable Performance Measure Narrative Frameworks (cont’d)Example 2: FAA Safe Flight 21 Program Issue: Decide which Safe Flight 21 projects provide meaningful
benefits. Desired narrative: Show how the results of operational evaluations,
while not directly measuring end-process benefits, indicate benefits will be achieved.
Solution: Develop flow of measures showing benefit mechanisms.
Describe SF21 Capabilities
Develop Direct Output Measures
Develop End-Benefit Outcome Measures
Identify End-Benefits
Identify Direct Impacts
Measure during operational evaluations to indicate whether benefits are likely to be achieved
Estimate projections
(cont’d . . . )
27
Example 2: Measure Flow - FAA Safe Flight 21 Measures
CapabilitiesDirect Output Measures
End-BenefitOutcome Measures
End-Benefits
Direct Impacts
CapabilitiesDirect
Impacts
Display in cockpit of surrounding traffic/equipment
Pilot able to better identify aircraft to follow
Pilot awareness of all proximate traffic positions
Direct Output Measures
Pilot response time for ATC traffic call-out
•Flight time from final approach fix to touchdown
Benefit Impacts
Reduced arrival delays
•Increased predictability of arrival times
End-Benefit Outcome Measures
Safety Accident rate during final approach maneuvers
User Cost Savings/Revenue Enhancement
Arrival rate
FAA Cost Savings· Voice channel occupancy time
28
Examples of Actionable Performance Measure Narrative Frameworks (cont’d)
Example 3: Federal Railroad Administration (FRA) Safety R&D Program
Issue: Why are projects selected?
Desired narrative: Present rationale for prioritizing and selecting projects.
Solution: Develop measures to reflect decision criteria for selecting projects in the program portfolio. » Present projects and criteria measures in matrix
» Values of criteria measures can be expressed as High, Medium, or Low for quick analysis and easy interpretation.
(cont’d . . . )
29
Example 3: Measure Criteria Matrix - FRA Safety R&D Program
Railroad Safety R&D ProgramsTrack and Components Safety
Project Safety Ratings Project Regulatory Ratings Likelihood of Success R&D Cost
Track Strength & Measurement Ra
il P
as
se
ng
er
Fa
talit
ies
&
Inju
rie
s
Ra
il E
mp
loye
e F
ata
litie
s
& In
juri
es
Tra
in A
cc
ide
nts
Ha
zard
ou
s M
ate
ria
ls
Re
lea
se
s
Off
ice
of
Sa
fety
R
ule
ma
kin
g o
r R
SA
C
Re
vie
w
To
ols
fo
r S
afe
ty
As
su
ran
ce
&
Co
mp
lian
ce
Ind
us
try
Sta
nd
ard
s &
B
es
t P
rac
tic
es
Te
ch
nic
al
Sc
he
du
le
Imp
lem
en
tati
on
FR
A R
&D
Co
sts
($
00
0)
No
n-F
RA
R&
D C
os
ts
($0
00
)
1 Gage Widening & Rail Rollover H M H H H H H H H H 2,000 2,000
2Track Buckling, Lat. Resistance & Long. Forces
H M M M H H H M M H 1,000 1,000
3 Track Vertical Support M M M M M M H M M H 850 1,000
. .
. .
. .
30
Examples of Actionable Performance Measure Narrative Frameworks (cont’d)
Example – FAA R&D Program (Draft) Requirements: Help prioritize and select projects; integrate project view;
show project contributions to goals Message: R&D program has a coherent set of projects that support
goals Measure Framework: Hierarchical framework (shows how activities
relate to goals)
31
AIR TRAFFIC SVCS. AIRCRAFT SAFETYAIRPORTS SECURITY AEROMED R&D MGMTCOMM. SP.
FAA STRATEGICGOALS
FAA SUPPORTINGGOALS
FAA PERF. OBJ.------------------------R&D CHALLENGE
R&DSTRATEGY
R&DPROJECTS
ST
RA
TE
GIC
PL
AN
NA
RP
ENVIRON.
SAFETYSECURITY EFFICIENCY ENV. & ENERGY
Risk Analysis: Reduce aviation system risks
Prevention: Prevent aviation accidents
Mitigation: Mitigate aviation system accidents
Flight Environment
Human Performance
Aircraft Systems
Terminal Area
Commercial SpaceRisk
In-Flight Incidents
Crash Survivability
Post-Crash Response
R&DPROGRAM
Example 4: Measure Hierarchy - FAA R&D Program (Draft)
32
Contact Information
Sandy Lozito
Manager, System Evaluation and Assessment
Virtual Airspace Modeling and Simulation (VAMS) Project
NASA Ames Research Center
M/S 262-4
Moffett Field, CA 94035-1000
650-604-0008
FAX 650-604-3729
James L. PoageOperations Assessment
Division, DTS-43Volpe National Transportation
Systems CenterKendall SquareCambridge, MA 02142617-494-2371FAX [email protected]