probabilistic methods of risk assessment: industry views
TRANSCRIPT
Probabilistic Methods of Risk Assessment: Industry Views
Dr. Allison Ferguson Director, Airspace Research
Summary
1. Introduction o Problem Statement
2. Operational Risk Assessment in Practice
o Pathfinder (Focus Area Two) • EVLOS • Localized BVLOS
o Lessons learned (so far)
3. Moving Forward
o Building the data pool o Engaging with industry
Reduce risk of drone operations
Enable advanced operations such as
BVLOS and package delivery through
technology
Build “backbone” datasets
PrecisionHawk LATAS: Our Goals
1
2
3
Data
Mitigating Risk through Information
• 3D Obstacle Maps
• Buildings
• Trees
• Terrain
• Manned Aircraft
• Radar / ADS-b
• UAS
Surveillance
• Airspace
• Cellular Coverage
Infrastructure
• Cloud services
• Cellular
• Security
• Bandwidth
considerations
Services
• Registration
• Tracking
• Reporting
• Flight planning
• Flight authorization
• Operator validation
But with each new additional component we introduce new failure modes…how to assess
impact on overall system risk?
Mitigating Risk through Information o All risk statements are probabilistic. The problem is getting to those
probabilities: • Evaluating total system risk (target level of safety) inferentially is….probably not a good idea. • There are distributions we CAN access inferentially that have an impact on overall system
risk (e.g. metrics of operator response to manned aircraft in operational area).
o How “good” are the models that link what we can measure with overall system
risk? • Especially in the presence of real-world (i.e. messy) data
o How good do they have to be? • Can we mitigate the risk of using a risk model?
Throwback…Tuesday?
• Weibel, Roland E., and R. John Hansman. "Safety considerations for operation of different classes of
UAVs in the NAS." AIAA 4th Aviation Technology, Integration and Operations Forum, AIAA 3rd
Unmanned Unlimited Technical Conference, Workshop and Exhibit. 2004
Risk Models and “Real” Data
Field Data
Simulation
Historical
o A risk modelling approach isn’t about ignoring “real” sources of information, it’s about collating them in a way that has some logical backing.
• These are problems with lots of variables. The temptation is to assume that all sources of variation are equally important to the final outcome (they are not).
Pathfinder Phase I & II Overview: EVLOS
Phase I Research focused on identifying the
range where an intruder aircraft could be visually identified.
Phase II Research focused on the PIC ability to react to an intruder aircraft within the
EVLOS range.
• EVLOS is an Unmanned Aircraft System (UAS) operation whereby the Unmanned Aircraft (UA) may
be operated marginally beyond the visual range of the remote pilot in command (hereafter referred to
as the PIC) provided he or she maintains uninterrupted visual surveillance for manned aircraft within
the planned UA operating airspace. Within the EVLOS construct, the UA must remain within the preplanned operating area and UA control authority must be maintained by the PIC at all times.
Focus is on collection of quantitative data to support a general operational
risk assessment
Phase I & II Overview: EVLOS
• Industry Impact: Commercial waiver to Part 107 VLOS requirement issued to PrecisionHawk (August 2016). Operational risk assessment based on Phase I/early Phase II results. Full Phase II results to be incorporated into ASTM F38 standard (2017) to support rulemaking efforts.
Pathfinder Phase III: “Localized” BVLOS
EVLOS (2.5 nm)
EVLOS with RVO (5 nm)
• Wide industry applicability. This same picture applies to:
• Emergency response • Insurance • Forestry/Natural Resource
Management
Localized BVLOS
Current operations to understand risk: • Pathfinder, UAS test sites, ASSURE
BUILD ON RESULTS TO DEFINE RISK PROFILE ON NATIONAL SCALE
Drive creation of localized beyond visual line of sight concept of operations in Class G
Towards operating on a National Level
• Shift focus from visual base case to technology assisted case
Phase III: “Localized” BVLOS
Phase III Mission Statement
Evaluate the impact of assistive technology (i.e. LATAS) on operational risk associated with flight of sUAS in BVLOS
applications in the NAS.
What does the technology need to do to achieve an acceptable level of safety? In what ways can it fail?
How do we mitigate the risk of failure?
Quantitative answers to these questions provide regulators with the information they require to define an operational
standard for rulemaking.
Phase III Operational Risk Assessment
• Risks in the technology-assisted case can be loosely grouped into three categories:
Ove
rall
Ris
k
Risks associated with technology not functioning as designed
Risks associated with improper use of the technology (human factors)
Risks inherent in the environment
+ Similarly to the ORA for the EVLOS case, we will ideally define a series of performance-based requirements ANY assistive technology would have to meet.
+ “Environmental” risk is related to conditions that feed into the risk model. For example:
+ Low altitude traffic patterns
+ Population density
Already we see the need for collaboration amongst researchers!
Phase III Operational Risk Assessment
• Risks associated with the technology not functioning as designed (software & hardware):
Tech
no
logy
Ris
k
Phase III Operational Risk Assessment
• Risks associated with the technology not functioning as designed (software & hardware):
Tech
no
logy
Ris
k + From a map of the process we can begin to itemize the potential hazards at each process step.
+ One specific set of hazards is related to GPS & cellular service at altitude:
+ Cellular service (and LTE in particular) is one of the bigger open questions: + Lag? + Intermittency? + How much of either of these is acceptable?
Process Step Hazard
Transmit
telemetry until
operation
complete
i. Device experiences mechanical or electrical failure
ii. GPS issues: GPS service becomes unavailable or intermittently
available during operation, invalid GPS string (improperly
formed), incorrect GPS data (properly formed string)
iii. Cellular issues: No cellular service, poor cellular reception due to
local conditions, intermittent connectivity due to LTE handover
issues
Risk models need to inform the initial
answer here.
Phase III Operational Risk Assessment
• Risks associated with improper use of the technology:
Hu
man
Fac
tors
Ris
k + Important to ask questions about how and when information should be displayed to the operator:
Phase III Operational Risk Assessment
• Risks associated with improper use of the technology:
Hu
man
Fac
tors
Ris
k + MITRE will evaluate a set of proposed assistive technology displays via Human-in-the-Loop (HITL) simulation, followed by field experiment:
Phase III Operational Risk Assessment
• Risks associated with improper use of the technology:
Hu
man
Fac
tors
Ris
k + MITRE will evaluate a set of proposed assistive technology displays via Human-in-the-Loop (HITL) simulation, followed by field experiment:
Well Clear Zone
Blocked Headings
(“Cone of Death”)
Intruder w/ Amber
Highlight
Altitude Meter:
• Blue dot is
your AGL
altitude
• Amber band is
altitude
blocked by
intruder
Relative altitude
and distance
Risk Models and “Real” Data: Part Deux
Field Data
Simulation
Historical
o This process is non-trivial.
• Real data is messy (especially if humans are part of the study).
• It’s not just science…it’s politics. Different CAAs will have different opinions about which variables are important.
o We don’t exactly have a target risk. • We were able to start measuring without
it, but coming up with this limit will be important soon.
o If computational and research
complexity now leads to regulatory and operational simplicity later, it’s worth it.
• But let’s get all the help we can.
Some Views FROM Industry…
o Technology is moving rapidly.
• Gathering flight heritage on rare events to assess risk is problematic. So we need variables we can measure over shorter cycles to assess the risk – conditional probability based risk models are a good way to determine what those variables should be.
o Mitigations need to be palatable. That may ALSO depend on the rate of a particular risk occurring.
• If the mitigation results in total loss of the UAS, whether or not that is acceptable depends on how often that would be likely to occur.
o If computational and research complexity now leads to regulatory and
operational simplicity later, it’s worth it. • But let’s get all the help we can.
Some Views FOR Industry…
o Is your technology is moving too rapidly? Or in the wrong direction?
• Adding “features” doesn’t help (and can overcomplicate the safety case) unless you’re actually solving a key problem at this point. Risk modelling can help identify those blockers.
o We are ALL in a data-starved environment right now. • “We did it that one time and nobody died” is not a sufficient evaluation of long
term risk. • There are opportunities for participation for all sizes of industrial partner (i.e.
ASSURE)
o If computational and research complexity now leads to regulatory and
operational simplicity later, it’s worth it. • But let’s get all the help we can.
