joseph cafazzo, phd peng · phd peng monday, june 28, 2010 ... (uml) 2.0 activity diagrams * from...

Joseph Cafazzo, PhD PEng

Monday, June 28, 2010

“Modern Times”, United Artists 1936


“Science Finds, Industry Applies, Man Conforms”

Slogan from the 1933 Chicago World’s Fair


"People Propose, Science Studies, Technology Conforms"

Don Norman’s person-centered motto for the 21st century.

"Things that make us smart." Addison-Wesley, 1993.


Poor Design

• Images -

• c prompt

• ballot

• friendly fire


“We have a serious training problem that needs to be corrected”

“We need to know how our equipment works: when the battery is changed, it defaults to its own location. We’ve got to make sure our people understand this”

Senior Air Force Official


“Recount”, HBO 2008


You are not alone

Radiation Oncologyand

Patient Safety


How do we prevent error?

Hierarchy of Effectiveness1. Forcing functions & constraints

2. Automation/computerization

3. Simplification/standardization

4. Reminders, checklists, double checks

5. Rules & policies

6. Training & education

tech focus

people focus


Human Factors in Healthcare

• IOM 1999: 44, 000 to 98, 000 US deaths annually due to medical error

• Baker, Norton 2004: 9, 000 to 23, 000 deaths annually due to preventable adverse events


User Centered Design

Analyze userrequirements

Design & Prototype

Evaluate


Design & Prototype


Evaluate• Expert/Heuristic Analysis • Visibility

• Consistency

• Efficiency

• Flexibility

• Autonomy


Evaluation Criteria• Efficiency & Speed

• Task completion time

• Task accuracy & error frequency

• Number of requests for help

• Number of attempts to correct errors

• Impact & Severity of error (patient safety)

• Workload/Difficulty scale (1-7)

• User feedback & preferences


Evaluate

• Usability Testing • Representative users

• Realistic scenarios

• Think-aloud protocol

• Record video & audio

• Qualitative & quantitative


Evaluation of a Functional Prototype


Improving Patient Safety during Radiation Therapy through Human Factors

Alvita ChanPrinciple Investigators: Dr. Mohammad Islam, Dr. Joe Cafazzo

Co-Investigators: Dr. Tony Easty, Dr. David Jaffray


Systematic Literature Review• A systematic review on existing literature was conducted to

understand sources of errors• 7 distinctive studies were found• 40-50% errors occur during treatment delivery• Most frequent errors:

– Incorrect patient position (e.g. wrong site, wrong SSD, wrong couch height)

– Incorrect treatment accessories (i.e. Shielding blocks, bolus)

Yeung TK et al. Radiotherapy and Oncology. 2005, 74:283-291Marks LB et al. International Journal of Radiation Oncology, Biology and Physics. 2007, 69:1579-1586


Yeung TK et al. Radiotherapy and Oncology. 2005, 74:283-291Yeung TK et al. Radiotherapy and Oncology. 2005, 74:283-291

40.4%


Work Environment Analysis• Identified issues and developed

recommendations regarding treatment/control room setup

– Ambient Noise/Interruptions– Furniture/computer features

and placement– Storage– Lighting– Temperature and humidity


Workflow Analysis• Mapped out entire workflow of the treatment delivery

process using the Unified Modeling Language (UML) 2.0 Activity Diagrams

* From 24 observations, 6 for each treatment team


Heuristic Evaluation• Systematic inspection of current user interfaces according to a list of

guidelines/heuristics – E.g. Visibility of system state, minimize memory load, informative

feedback, flexibility and efficiency, prevent errors– Identified 75 usability problems

0

10

20

30

40

High Medium Low Positive

3737

2018

Usability Problems by Severity Ratings

Num

ber o

f Pro

blem

s Id

entif

ied

Severity Ratings

Zhang et al. Journal of Biomedical Informatics. 2003, 36: 23-30


Workflow Analysis • Identified areas that can be improved:

– Heavy reliance on policy• Checking process during patient setup

Hierarchy of effectiveness in preventing errors

1. Forcing functions and constraints

2. Automation / Computerization

3. Simplification / Standardization

4. Checks, double checks, checklists

5. Policy and trainingLeast effective

Institute for Safe Medication Practices. ISMP Medication Safety Alert. 1999, 4 (11)

Hierarchy of Effectiveness


How can the process be improved?

• Focused on checking process during patient setup

• Redesigned components of the user interface:

– To lessen the reliance on policy and procedure– To improve efficiency of the checking process– To display information based on the users’ needs

• Design process: Expert input + 2 focus groups

12


Redesigned InterfaceDRR is displayed visibly to improve

checking compliance

Patient picture is displayed visibly for easy identification


Redesigned Interface

Forcing function/checklist is used to improve checking compliance

New messages and outstanding items are

highlighted

Statuses of approval dates and images are

clearly displayed

DOB given by patient is checked by system

Both SSD and depths are displayed

Notes that required certain actions to be

performed are highlighted


Redesign of Existing System

Current interface:14 mouse clicks

Redesigned interface:4 mouse clicks

Improved efficiency

17


Experimental Evaluation - Usability Testing

• 16 radiation therapy students from PMH were recruited to participate

• An actor played the role of a second therapist• Performed patient setup tasks using both the current and

redesigned interfaces

18


Experimental Design

• 2 interfaces x 4 scenarios

• 3 out of the 4 scenarios were designed with a high potential for common errors to occur:

– Overlooking an important note– Shifting couch incorrectly – Overlooking a change in

approval dates

19


Rate of Errors

P<0.04

P<0.01

20


Task Time

P<0.02

21


User Satisfaction

• Post-test Questionnaire:

22


User Satisfaction

Attributes Current Redesigned Significance

Better at showing the information needed

0% 94% Yes (P<0.01)

Better at drawing attention to important items

0% 100% Yes (P<0.01)

Better for catching an important message, a change in approval date, or an unapproved image

0% 100% Yes (P<0.01)

Enable treatment to be delivered more safely

13% 81% Yes (P<0.01)

Easier to use 25% 56% No

More efficient 19% 63% No

Preferred system 25% 56% No

23


Limitations

• Students vs. experienced therapists

• Mannequins vs. real patients

• Redesigned interface was only a prototype, not all features were functional

• Only the first iteration of the design cycle

24


Acknowledgements

Other: - Catherine Dupuis- Mary Stewart-Hardy- Varuna Prakash- Mark Fan- Radiation therapists at

PMH- Radiation therapist

students

Project team:- Dr. Joe Cafazzo- Dr. Mohammad Islam- Dr. Tony Easty- Dr. David Jaffray- Tara Rosewall

Funding agencies:- NSERC, NPSF, AAPM, CPSI

26


Summary

•Humans are fallible. We will err.

•We can reduce frequency and consequences of errors by designing systems within our limitations

•Solutions should aim towards the top of the Hierarchy of Effectiveness


healthcarehumanfactors.com

Joseph Cafazzo, PhD PEng


joseph cafazzo, phd peng · phd peng monday, june 28, 2010 ... (uml) 2.0 activity diagrams * from...

Documents