Robert Dunfield1

Jaime Riley2

Chris Vaillancourt3

James French4

Jacqueline Fraser5

Paul Atkinson6

Jennifer Woodland7

1. Dalhousie Medicine New Brunswick, Saint John, New Brunswick

2. Department of Emergency Medicine, Saint John Regional Hospital, Saint John, New Brunswick

3. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick

4. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick

5. Department of Emergency Medicine, Saint John Regional Hospital, Saint John, New Brunswick

6. Department of Emergency Medicine, Dalhousie University, Saint John Regional Hospital, Saint John, New Brunswick

7. Department of Research Services, Horizon Health, Saint John Regional Hospital, Saint John, New Brunswick

References1. Greenberger PA, Rotskoff BD, Lifschltz B. Fatal anaphylaxis: postmortem findings and

associated comorbid diseases. AAAI 2007; 98:252-7.

2. Song TT, Worm M, Lieberman P. Anaphylaxis treatment: Current barriers to adrenaline auto-injector use. Allergy 2014; 69:983-991.

3. Simons FER, World Allergy Organization. Epinephrine auto-injectors: first-aid treatment still out of reach for many at risk of anaphylaxis in the community. AAAI 2009; 102:403-409.

4. Ahmed A, Khan FA, Ismail S. Reliability and validity of a tool to assess airway management skills in anesthesia trainees. J Anaesthesiol Clin Pharmacol. 2016 Jul-Sep;32(3):333-8.

ContactEmail: rjdunfield@dal.ca

Cell: 506-434-3289

Conflicts of interestNo conflicts of interest.

Results:

• Overall, there was poor agreement between the two raters.

• Checklist item #1 (contacted EMS) at 0 months had the highest level of

agreement, but did not achieve statistical significance (κ=0.59, moderate

agreement; p<0.05) (Table 3).

• Checklist item #1 at 3 months had the second highest level of agreement (κ=0.482, moderate agreement; p<0.05) (Table 3).

• The remaining checklist items had levels of agreement ranging from poor to slight agreement. None achieved the standard level of agreement of κ ≥ 0.70 (Table 3).

Conclusion: Although microskills checklists have been shown to identify areas where learners and interprofessional teams require deliberate practice, our results support previously published evidence that the use of microskills checklists in the assessment of skills has

poor reproducibility4. Performance will be further assessed in this study using global rating scales, which have shown higher levels of agreement in other studies.

Acknowledgements Sweet Caroline Foundation; Town of Hampton, NB; Dalhousie Emergency Medicine Research Committee.

The devil may not be in the detail: assessment of bystander training to administer publicly available epinephrine using microskills checklists shows low inter-rater reliability.

sjrhem.ca

Methods:

• Prospective, stratified, block randomized study

• 154 participants at 15 sites were block randomized to one of three education interventions:

1. didactic poster teaching; 2. poster with video teaching; and 3. poster, video, and simulation training

• Participants tested at 0-months and at 3-months (post-intervention).

– Assessment involved two blinded raters using a microskills checklist.

– Checklists used fourteen 3-point and 5-point Likert scale questions around anaphylaxis response (Table 1).

– Inter-observer reliability was assessed for each item and expressed as a kappa (κ) value that represented a level of agreement (Table 2).

Background:

• Improving public access and training for epinephrine auto-injectors (EAIs) can reduce time to initial treatment in anaphylaxis and therefore save lives, especially in rural settings.1,2,3

• In southern New Brunswick, this is being addressed with unlocked, alarmed EAI cabinets.

• We investigated the best teaching modality for effective EAI training using a microskillschecklist.

Table 1: Anaphylaxis response microskills checklist

Table 2: Reference κ values for inter-rater agreement.

1. 911/EMS contacted2. Patient positioned properly3. Cabinet properly accessed

4. Both EpiPen’s removed5. Properly handled EpiPen and removed safety guard

6. Correct dosage injected7. EpiPen injected into upper lateral thigh

8. EpiPen held in place for 10 seconds9. Patient encouraged to remain resting post-injection

10. Second dose considered11. Patient monitored until “EMS” arrived

12. Calm demeanour throughout their response13. Clear and effective communication14. Responded in an organized fashion

kappa value (κ) Level of agreement

<0 Poor agreement

0.0-0.20 Slight agreement

0.21-0.40 Fair agreement

0.41-0.60 Moderate agreement

0.61-0.80 Substantial agreement

0.81-1.00 Almost perfect agreement

Assessment Parameter & Measurement Time κ p value Agreement

1. Did the participant contact or instruct someone to contact

emergency medical services (EMS)?

0 months 0.590 <0.05 Moderate

3 months 0.482 <0.05 Moderate

7. Did the participant deliver the EpiPen in proper location (upper

lateral thigh)?

0 months 0.096 0.070 Slight

3 months 0.247 <0.05 Fair

12. Did the participant maintain a calm demeanour throughout the

anaphylactic event?

0 months 0.072 0.085 Slight

3 months -0.103 0.095 Poor

14. Did the participant show the ability to deliver the epinephrine in

an organized fashion?

0 months 0.065 0.112 Slight

3 months 0.067 0.311 Slight

Table 3: Levels of agreement for select checklist items

Generally, a κ ≥ 0.70 is preferred and set as the standard level of agreement.

Use of micro-skills checklists

in the assessment of skills has poor reproducibility

Link to all SJRHEM@CAEP 2019Research