simulation based assessment training the trainer series october, 2014 training the trainer series...
TRANSCRIPT
![Page 1: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/1.jpg)
Simulation based Assessment
Simulation based Assessment
Training the Trainer SeriesOctober, 2014
Training the Trainer SeriesOctober, 2014
![Page 2: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/2.jpg)
Assessment
• Formative
• Summative
![Page 3: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/3.jpg)
Formative Assessment• Assessment for
learning
![Page 4: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/4.jpg)
Summative Assessment
• Assessment of learning
![Page 5: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/5.jpg)
Evaluation Methodology
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483
![Page 6: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/6.jpg)
Methodology:Global Rating Scale
• Overall performance by categories
• Scores (Likert)
• Drawbacks
![Page 7: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/7.jpg)
Global Rating Scale
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
![Page 8: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/8.jpg)
Methodology:Checklists
• Delineated actions
• Weighted, timed, cumulative...
• Adverse events
• Drawbacks
![Page 9: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/9.jpg)
Checklists
Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
![Page 10: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/10.jpg)
Methodology:“Time To”
• Timing of critical actions
• Pace
• Motion analysis
![Page 11: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/11.jpg)
Methodology:Survey Data
• Self assessments
• Psychometrics: Pre and Post Testing
![Page 12: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/12.jpg)
Survey Data
Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356
![Page 13: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/13.jpg)
Methodology:Biometrics
• “Technometrics?”
• Mannequin physiology
![Page 14: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/14.jpg)
Validity
• The Assessment Method
• The Simulation
![Page 15: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/15.jpg)
Validity:“Face”
• General acceptability of the tool used for the task required
• Does this make sense?
![Page 16: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/16.jpg)
Validity:“Content”
• Does the content match what’s to be assessed?
• Task Analysis
• The Experts
![Page 17: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/17.jpg)
The Delphi Method
• BOGSAAT
![Page 18: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/18.jpg)
Validity:“Construct”
• Does this reflect true performance?
• Experts
• Role of experience
• Surveys
![Page 19: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/19.jpg)
Validity:“Construct”
• Statistical Analysis:
• Discrimination: “D Statistic”
• ANOVA
• “Concurrence”
![Page 20: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/20.jpg)
Tool Validation
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
![Page 21: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/21.jpg)
Validity:“Criterion”
• Or “Extrapolation”
• Similar results to other performance measures
• “Predictive”
![Page 22: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/22.jpg)
Consistency:Internal
• Requires realism
• Reflects test quality
• Pilot Testing
![Page 23: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/23.jpg)
Consistency:Statistical Analysis
• Do testers do well or poorly on the same items?
• Generalizability: “G statistic”
![Page 24: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/24.jpg)
Reliability:Inter-Participant
• Set-up
• Simulator
• The Subject
![Page 25: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/25.jpg)
Reliability:The Rater
• Inter-rater reliability (external reliability)
• Statistics: IRR, ICC
![Page 26: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/26.jpg)
Reliability:The Rater
• Rater training
• Rater refreshing
• Scoring
![Page 27: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/27.jpg)
Reliability:The Observation
• Real time
• Recording
• Bias
![Page 28: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/28.jpg)
Validated Tools
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
![Page 29: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/29.jpg)
Take Away
• Rigor in realism, rehearsing and piloting
• Mix Checklists and GRS
• Supplement with survey, psychometric, device data, etc
• More tasks with more (recorded) views
![Page 30: Simulation based Assessment Training the Trainer Series October, 2014 Training the Trainer Series October, 2014](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649d1f5503460f949f3ef1/html5/thumbnails/30.jpg)
Further SourcingSchwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42
Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86:1160-1164
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP:
Testing the raters: Inter-rater reliability of standardized anaesthesia simulator
performance. Can J Anaesth 1997; 44:924–8
Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment
of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18
Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15:1017-1024
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5:226-231
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82:203-206
Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483
McKinley RK, et al. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ 2008; 42:338-349
Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356