Download - Exercise Evaluation
IAEA Training in Emergency Preparedness and Response
Exercise Evaluation
LectureLecture
Module L-056Module L-056
L-056: Exercise Evaluation 2
Objectives
• Understand the importance of evaluation for the success of the exercise
• Understand performance-based evaluation
• Know the role of evaluators and their procedures
• Be able to develop evaluation criteria
• Know how to grade and report evaluations
L-056: Exercise Evaluation 3
Principles of Evaluation
• Performance required by plans and procedures is being tested
• Evaluations are results-oriented to enhance the plan and the supporting elements, such as proper organization, training, and equipment
L-056: Exercise Evaluation 4
Aim of Evaluations
• The aim of an evaluation is to identify strengths and weakness in the emergency response capability
L-056: Exercise Evaluation 5
What is a Performance-based Evaluation?
• Focus on results, not on process
• Based on response objectives
• It answers the question: was the result achieved?
L-056: Exercise Evaluation 6
How to Evaluate Based on Performance (1)
• Define the response or exercise objective• e.g. Protection of facility personnel
• Define the criteria• e.g. The personnel are promptly alerted
• e.g. The dose to personnel is kept within established limits
L-056: Exercise Evaluation 7
How to Evaluate Based on Performance (2)
• For each performance objective, define the evaluation criteria• e.g. The personnel are promptly alerted
• Was a public address message broadcast immediately upon discovery of an emergency situation?
• Did all personnel hear the message?
• e.g. The dose to personnel is kept within established limits
• Were personnel relocated to safe areas?
• Was the dose rate monitored at personnel assembly areas?
L-056: Exercise Evaluation 8
How to Evaluate Based on Performance (3)
• After the exercise, based on the FACTS, answer the question: “was the performance objective met”?
• It is not absolutely necessary that all criteria be satisfied• The actual response will depend on the situation
L-056: Exercise Evaluation 9
How to Evaluate Based on Performance (4)
• If the performance objective was met• Was the procedure followed?
• Was the result achieved in good faith or by emergency?
• If the performance objective was not met, why?• Was the procedure followed?
• Was training lacking?
• Was equipment lacking?
• Was the exercise at fault?
L-056: Exercise Evaluation 10
Evaluation Team
• Organization• Chief evaluator
• Evaluators for all key groups or locations
L-056: Exercise Evaluation 11
Evaluator Role and Responsibilities
• Chief evaluator• Coordinate the evaluation team
• Instruct evaluators on the schedule, method and criteria
• Ensure consistency between evaluators
• Instruct the evaluators on the format and schedule of the evaluation reports
• Produce the final evaluation report
L-056: Exercise Evaluation 12
Evaluator Role and Responsibilities
• Evaluators• Observe and record
• Do not evaluate during the exercise
• Stick to facts
• Evaluate after the exercise
• Provide a report to the chief evaluator
L-056: Exercise Evaluation 13
Choice of Evaluators
• Evaluators must be qualified for the function they evaluate
• Evaluators must be trained• Evaluation techniques
• Procedures of the groups being evaluated
• Evaluators do not need to know the detailed scenario
L-056: Exercise Evaluation 14
Preparation of the Evaluation Team
• Brief the evaluation team
• Provide clear written instructions
• Organize a site orientation tour
L-056: Exercise Evaluation 15
Evaluator Instructions
• Before the exercise:• Review the scenario timeline and objectives
• Review the procedures used by the participants
• Review the applicable evaluation criteria
• Understand the actions required to demonstrate a performance objective
L-056: Exercise Evaluation 16
Evaluator Instructions (Cont’d)
• During the exercise• Arrive before an event
• Locate a telephone
• Wear Evaluator’s identification
• Do not interfere with the participants
• Never criticize the participants
• Minimize communications
L-056: Exercise Evaluation 17
Evaluator Instructions
• Following the exercise• Attend debriefing and critique
• Observe the effectiveness of the debriefing and critique
• Obtain exercise documents
• Ensure exercise location is in a safe state
• Finalize your chronology notes
L-056: Exercise Evaluation 18
Observation Techniques
• Record arrival times
• Record names and functions
• Record times of major events
• Record repetitive interventions
• Listen to commands and instructions
• Listen and record Controller inputs
• Record exercise observations
L-056: Exercise Evaluation 19
Guide for Evaluators
• After the exercise, based on the FACTS, answer the question: “was the performance objective met”
• It is not absolutely necessary that all criteria be satisfied:• The actual response will depend on the
situation
L-056: Exercise Evaluation 20
Drillmanship
• Drillmanship is the art of exercise delivery
• Record facts on:• Simulation
• The exercise scenario
• Safety
• Participant feedback
L-056: Exercise Evaluation 21
On the Use of Evaluation Checklists...
• Several organizations use checklists for each group being evaluated
• This focuses the evaluation on each group in isolation rather than on the overall coordinated response performance
• Focus on recording important facts rather than “checking” a list
• Evaluate the overall performance on the basis of exercise objectives using all the facts collected by the evaluators
L-056: Exercise Evaluation 22
Debriefings
• Hot Wash-up
• Exercise Team Debriefing
• Executive Debriefing• Brief the management staff
• Give credit to the players for identifying deficiencies and give them a grace period to fix the problems before forcing compliance or imposing penalties
L-056: Exercise Evaluation 23
Safety Responsibilities
• Evaluators are also responsible for monitoring exercise play to ensure a safe environment
L-056: Exercise Evaluation 24
Grading Criteria
• Excellent
• Satisfactory
• Unsatisfactory
L-056: Exercise Evaluation 25
Grading Criteria (cont’d)
• Grading must be supported by facts (if it was not observed, it didn’t happen)
• Trivial mistakes remain trivial even when they are observed
L-056: Exercise Evaluation 26
Classification of Unsatisfactory Grades
• Critical Deficiency
• Major Deficiency
• Minor Deficiency
L-056: Exercise Evaluation 27
Deficiency Classification
• Planning
• Training
• Resources
L-056: Exercise Evaluation 28
Exercise Report
• A useful report is most difficult to write
• Based on facts, not impressions
• Focus on key areas
• Use the exercise objectives as the basis for the report
• Prepare and publish in a timely fashion
• Not everybody will agree with the findings
L-056: Exercise Evaluation 29
Post Exercise Actions
• Report clarification• Ensure there are no inaccuracies in the report
• Critical deficiencies• Take corrective actions as soon as possible, e.g. one
month
• Major deficiencies• Take corrective actions within a reasonable time frame,
e.g. months
• Minor deficiencies• Take corrective actions during the annual revision of the
plan or procedures
L-056: Exercise Evaluation 30
Summary
• The evaluation should be performance-based
• Evaluators and controllers are not the same persons
• An evaluator does NOT evaluate DURING an exercise, but after
• Base the evaluation report on the exercise objectives
• Look for corrective actions