topics and trends in incident reports - atm seminar · topics and trends in incident reports: ......
TRANSCRIPT
TopicsandTrendsinIncidentReports:UsingStructuralTopicModelingtoExploreAviationSafetyReportingSystemData
KennethD.KuhnRANDCorporation2017ATMR&DSeminar
Outline
• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling
• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion
AviationSafety,ATMR&D
• Aviationisremarkablysafeatthemoment,butimprovementmaybepossibleand....• NextGen,SESAR,andotherprojectsareleadingtomajorchangesinairtransportationoperations.• Forexample,theFAAisimplementingWakeRecategorization,“reducingseparationcriteriaformultiplerunwayoperations.”• RecentATMR&Dpapersincludemethodologiesforassessingsafetyimplications.• Fleming,Leveson,andPlacke,2013• FlemingandLeveson,2015
ThisStudy
• ThisstudydescribesanexploratoryanalysisofAviationSafetyReportingSystem(ASRS)data.
• Goal1:evaluatetheusefulnessofanovelmethodforidentifying,evaluatingaviationsafetyissues,trends,etc.
• Goal2(moreambitious):uncoverimportantandpreviouslyunreportedconnectionsorthemesinincidentreports.
AviationSafetyReportingSystem(ASRS)
• ASRSisbasedonconfidentialreportsofsafetyincidentssubmittedbypilots,airtrafficcontrollers,airlinedispatchers,andothers.• TheFAAandNASAdevelopedandmanagetheASRS,inpartto“providedataforplanningandimprovementstothefutureNationalAirspaceSystem.”• Similarsystemsexistelsewhere,includingCHIRPintheUnitedKingdomandREPCONinAustralia.• In2015,theASRSdatabaseincludedover1.3millionrecordsandwasaddingroughly7,500additionalreportseachmonth[NASA].
StatisticsonASRSIncidentReports
• Analystsanonymizesubmissionswhichbecomethe“narrativeportions”ofincidentreports.
• AnalystscodeinotherdataandaddresultstotheASRSdatabase.• flightmission(68%“Passenger,”14%personal,6%cargo,...)• reportingorganization(58%“AirCarrier,”16%government,)• locale(e.g.,“LGA.Airport,”“TUL.TRACON,”...)• month• …
• Researchquestion:How“representative”areASRSdata?
ResearchonASRSData
• BillingsandReynard(1984)combedthroughincidentreportsandsearchedforrelationships.• “Themostcommoncontrollererrorsinvolvefailuretocoordinatetrafficwithotherelementsoftheairtrafficcontrolsystem.”
• Bliss,Freeland,andMillard(1999)studiedtheroleofcockpitalarmsystemsinASRSincidentreports.
• Recentadvancesinthefieldsofmachinelearning,computationallinguistics,naturallanguageprocessing(NLP)allowforfasterandeasierexplorationofASRSdata.
PriorNLPStudiesusingASRSData
• ElGhaoui etal.(2013)describeasuiteofNLPmethodsandtestthemethodsonASRSdata.Themethodswereableto:• “revealcausalandcontributingfactorsinrunwayincursions”• “automaticallydiscoverfourmaintasksthatpilotsperformduringflight”(aviate,navigate,communicate,andmanagesystems)
• Tanguyetal.(2016)providesanoverviewofNLPapplicationstoaviationsafetydata.• “Itappearsthattopicmodellingisverysuitablefor[incidentreport]data”• topicmodelinguncovers“relevantaspectsof[these]documents,ascanbeseenthroughanexpert’sinterpretation”
TopicModeling
• Topicmodelingisamethodthatallowsanalyststoidentify“themainthemesthatpervadealargeandotherwiseunstructuredcollectionofdocuments”(Blei,2012).• ThemostcommonformoftopicmodelingislatentDirichletallocation(LDA).• Documentsandthewordswithinthemarederivedfroma“generativeprobabilisticmodel”(Blei etal.,2003).• ThenumberofwordsN inadocumentisarandomvariabledrawnfromaPoisson(ξ)distribution.
LatentDirichlet Allocation(LDA)Model
• Theparameterθ ofadocumentisrandomvariabledrawnfromaDirichlet(α)distribution.• Thetopicofeachwordzn isarandomvariabledrawnfromamultinomial(θ)distribution.• Eachwordwn isarandomvariablebasedonanotherdrawfromamultinomialdistributiondefiningp(wn|zn,β)terms.
[Blei etal.,2003]
StructuralTopicModeling(STM)
• Structuraltopicmodeling(STM)isanalternativetoLDAwhereparametersdescribingtopicproportions(θ terms)areassumedtobedrawnfromdistributionsthatarebasedoncovariatedata.• Topicsandwordsareassumedtobedrawnfromadistributionspecifictothedocumentbasedonthecovariatedata.• Applicationsinclude:• analysisofMOOCstudentcomments[Reichetal.,2015]• eventdetectionusingtwitterdata[Mishler etal.,2015]• evaluationoflinksbetweencorporatefundingandtheframingofscientificstudiesonclimatechange[Farrell,2016]
Outline
• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling
• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion
FirstSteps,ApplyingSTMtoASRSData
• Thisstudyfirstfocusedonallincidentsreportedfrom1/1/2010to12/31/2015.
• 17topicswereuncoveredwithinincidentreports.
• Togainsomeintuitionregardingtopicmeaning,lookatthewordslinkedtoeachtopic.• Prob - prob.ofwordoccurrenceconditionalontopic• Lift - prob.ofwordoccurrenceconditionalontopicdividedbyprob.ofwordoccurrenceacrosscorpus.• FREX - ratioofwordfrequencyconditionalontopictoword-topicexclusivity.
FurtherAnalysisUsingTopicData
• Wecanlookatcorrelationsamongthetopics.(Recallthatthereisatopicassignedforeachwordpositionineachdocument.)
• Wecanalsolookathowcovariatedataimpactstopicprobabilities.
• Covariatedataconsideredhere:• Phaseofflight• Flightmission• Month
PhaseofFlightandEst.TopicProportion
●
0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14
Estimated Marginal Effect
●When phase of flight = Landing, Topic label = ATC●Landing, Human Factors
●Surface, ATC●Surface, Human Factors
●Takeoff, ATC●Takeoff, Human Factors
●Other, ATC●Other, Human Factors
●Cruise, ATC●Cruise, Human Factors
FlightMissionandEst.TopicProportion
●
0.00 0.05 0.10
Estimated Marginal Effect
●When flight is a passenger flight, Topic label = smoke / fire●passenger, fuel pump / tank / landing gear
●other, smoke / fire●other, fuel pump / tank / landing gear
●cargo, smoke / fire●cargo, fuel pump / tank / landing gear
●private, smoke / fire●private, fuel pump / tank / landing gear
Est.TopicProportionOverTime
0.00
0.05
0.10
0.15
0.20
Time
Expe
cted
Top
ic P
ropo
rtion
ATCfuel pump, tank, landing gearweathermaintenance, fault
1/2011 1/2012 1/2013 1/2014 1/2015
Outline
• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling
• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion
FlightMission,Est.TopicProportionatSFO
●
−0.2 0.0 0.2 0.4 0.6 0.8
Estimated Marginal Effect
●passenger flight, topic label = taxi●passenger flight, topic label = approach
●other, taxi●other, approach
●private flight, taxi●private flight, approach
●cargo flight, taxi●cargo flight, approach
Outline
• Introduction• TheAviationSafetyReportingSystem• StructuralTopicModeling
• Results:AllRecentIncidents• Results:IncidentsfromSFO• Discussion
SpecificFindings
• Initialresultsdemonstrated(andquantified)theimportanceofhumanfactorsandairtrafficcontrol,withtheformerbeingmoreprominentontheairportsurface andthelattermoreprominentduringflight.• Thefrequencyoffuelpump,tank,andlandinggearissuesandthesparsityofsmokeandfireissuesforprivateaircraftwerealsorecorded.• AtSFO,methodshighlightedtheQuietBridgeVisualandTipToeVisualapproachpathsasparticularlyprominentinincidentreports.
MoreGeneralConclusions
• ASRSisusefulresourceforresearchers.• Naturallanguageprocessingtechniquesareusefulhere.• STMisabletoidentifyknownissuesandtouncoversomeissuesthathavenotbeenpreviouslyreported,butdoesnotnecessarilyproduceactionableinsights.• Resultscouldbeusedtosetprioritieswhenplanningfutureaviationsafetyresearch.• Subjectmatterexpertiseisneededtodevelopintuitivemeaningstotopics,tointerpretresults,andtoplanandperformfollow-onwork.