measurements & validation
DESCRIPTION
Measurements & Validation. R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK. What is Validation?. EATMS Validation Strategy (EVAS) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/1.jpg)
11
Measurements & ValidationMeasurements & Validation
R G StampR G Stamp
Head of Department of ATM ResearchHead of Department of ATM Research
National Air Traffic Services Ltd, UKNational Air Traffic Services Ltd, UK
![Page 2: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/2.jpg)
22
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 3: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/3.jpg)
33
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a demonstrated against a pre-defined levelpre-defined level of of functionality, operability and performance”functionality, operability and performance”
![Page 4: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/4.jpg)
44
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a The process through which a desired level of desired level of confidenceconfidence in the ability of a deliverable to in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 5: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/5.jpg)
55
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a operate in a real-life environmentreal-life environment may be may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 6: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/6.jpg)
66
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
Validation builds confidence that the system is Validation builds confidence that the system is fit for purposefit for purpose
![Page 7: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/7.jpg)
77
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
![Page 8: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/8.jpg)
88
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
![Page 9: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/9.jpg)
99
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
![Page 10: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/10.jpg)
1010
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
NATSNATS
DLRDLR
CENACENA
NLRNLR
EECEEC
real-life environmentreal-life environment
![Page 11: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/11.jpg)
1111
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 12: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/12.jpg)
1212
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 13: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/13.jpg)
1313
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 14: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/14.jpg)
1414
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 15: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/15.jpg)
1515
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 16: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/16.jpg)
1616
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
pre-defined level of functionality, operability and performanceof functionality, operability and performance
![Page 17: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/17.jpg)
1717
Measures of MeritMeasures of Merit
MoM’s Chosen to Test HypothesesMoM’s Chosen to Test Hypotheses Controller WorkloadController Workload Airspace CapacityAirspace Capacity Quality of ServiceQuality of Service UsabilityUsability SafetySafety
Measurements Taken:Measurements Taken: e.g. Workload: ISA, TLX, R/T, …..e.g. Workload: ISA, TLX, R/T, ….. Objective; Subjective; ObservedObjective; Subjective; Observed
![Page 18: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/18.jpg)
1818
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 19: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/19.jpg)
1919
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 20: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/20.jpg)
2020
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the I can still handle the aircraft while doing the
crossword!”crossword!”
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 21: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/21.jpg)
2121
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the crossword!”I can still handle the aircraft while doing the crossword!”
desired level of confidencedesired level of confidence
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 22: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/22.jpg)
2222
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Core Validation Group with Representation from all Core Validation Group with Representation from all Participating SitesParticipating Sites
Common Methodology CrucialCommon Methodology Crucial Methodology Improved with ExperienceMethodology Improved with Experience Validation Conducted by Individual SitesValidation Conducted by Individual Sites
Lessons Re-learnt &/or LostLessons Re-learnt &/or Lost Effectiveness DiminishedEffectiveness Diminished
Validation Should be Applied Throughout Project - Validation Should be Applied Throughout Project - Not Just at the EndNot Just at the End
![Page 23: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/23.jpg)
2323
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D
![Page 24: Measurements & Validation](https://reader035.vdocuments.net/reader035/viewer/2022070414/56814db6550346895dbb0a92/html5/thumbnails/24.jpg)
2424
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D