onc hit program test results summary for 2014 edition … · test results summary for 2014 edition...

Test Results Summary for 2014 Edition EHR Certification

16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

Part 1: Product and Developer Information1.1 Certified Product Information

1.2 Developer/Vendor InformationPicis Clinical Solutions

Part 2: ONC‐Authorized Certification Body Information2.1 ONC‐Authorized Certification Body Information

Signature and Date

This test results summary is approved for public release by the following ONC‐Authorized Certification Body

Representative:

Adam Hardcastle

ONC‐ACB Authorized Representative Function/Title

San Luis Obispo, CA 93401

Website: www.infogard.com

Email: [email protected]

Phone: (805) 783‐0810

EHR Certification Body Manager

ONC‐ACB Name: InfoGard Laboratories, Inc.

Address: 709 Fiero Lane Suite 25

Email: [email protected]

Phone: (844) 240‐9554

ONC‐ACB Contact: Adam Hardcastle

ONC HIT Certification Program


Product Name: Picis ED PulseCheck

Product Version: 5.4

Domain: Inpatient

Developer/Vendor Contact: Merrily King Turnbull

Rosemont, IL 60018

Website: www.picis.com

Test Type: Modular EHR

Developer/Vendor Name:

Address: 10275 W. Higgins Road Ste. 250

©2016 InfoGard. May be reproduced only in its original entirety, without revision 1

6/28/16


16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

2.2 Gap Certification

(a)(1) (a)(19) (d)(6) (h)(1)

(a)(6) (a)(20) (d)(8) (h)(2)

(a)(7) (b)(5)* (d)(9) (h)(3)

(a)(17) (d)(1) (f)(1)

(a)(18) (d)(5) (f)(7)**

*Gap certification allowed for Inpatient setting only

**Gap certification allowed for Ambulatory setting only

2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification

(a)(19)

(a)(20)

(a)(13)

(c)(2)

(e)(3) Amb. only

No inherited certification

(f)(5) Amb. only

(f)(6) Amb. only

(f)(7) Amb. Only

(h)(1)

(h)(2)

(h)(3)

(a)(11)

(b)(7) (e)(1)

(g)(3)

(a)(12)

(c)(1)

(b)(9)

(a)(18)

(e)(2) Amb. only

(g)(4)

(a)(9)

(b)(5) (d)(8)

(g)(1)

(a)(10)

(b)(6) Inpt. only (d)(9) Optional

(g)(2)

(b)(8)

(d)(6) (a)(8)

(b)(4) (d)(7)

(a)(5)

(b)(1) (d)(4) (a)(6)

(b)(2) (d)(5)

(a)(14)

(c)(3)

(f)(1)

(a)(2)

(a)(15)

(d)(1)

(f)(2)

The following identifies criterion or criteria certified via gap certification

§170.314

No gap certification

§170.314

(a)(3)

(a)(16) Inpt. only

(d)(2)

(f)(3)

(a)(4)

(a)(17) Inpt. only

(d)(3)

(f)(4) Inpt. only

(a)(1)

(a)(7)

(b)(3)



16-2423-R-0022-PRI Vl.O, June 20, 2016

Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: N/A

Test Date(s): N/A

3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Accreditation Number:

Address:

Website:

Email:

Phone:

ATL Contact:

lnfoGard Laboratories, Inc.

NVLAP Lab Code 100432-0

709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

www.infogard.com

[email protected]

(805) 783-0810

Milton Padilla

For more information on scope of accreditation, please reference

http://ts.nist.gov/Sta ndards/scopes/1004320. htm

D

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory

Representative:

Milton Padilla EHR Test Body Manager

ATL Authorized Representative Function/Title

s.~~~4 3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

. . . . . Functionality provided by Add1t1onal Software Applicable Cntena d ft

A ditional So ware

g

D No additional software required

©2016 lnfoGard. May be reproduced only in its original entirety, without revision 3


16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

3.2.2 Test Tools

Version

No test tools required

3.2.3 Test Data

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted

(a)(13)

§170.207(a)(3)

IHTSDO SNOMED CT®

International Release July 2012

and US Extension to SNOMED

CT® March 2012 Release

§170.207(j)

HL7 Version 3 Standard: Clinical

Genomics; Pedigree

The following identifies the standard(s) that has been successfully

tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1)

HL7 Version 3 Implementation

Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide

HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool

HL7 v2 Syndromic Surveillance Reporting Validation Tool

Transport Testing Tool

Direct Certificate Discovery Tool

Alteration (customization) to the test data was necessary and is

described in Appendix [insert appendix letter ]

No alteration (customization) to the test data was necessary

Test Tool

Cypress

ePrescribing Validation Tool

HL7 CDA Cancer Registry Reporting Validation Tool

HL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting

Valdiation Tool

Edge Testing Tool



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

No newer version of a minimum standard was tested

Common MU Data

Set (15)

§170.207(a)(3)

IHTSDO SNOMED CT®




§170.207(b)(2)

The code set specified at 45 CFR

162.1002(a)(5) (HCPCS and CPT‐

4)

The following identifies the newer version of a minimum standard(s) that has

been successfully tested

Newer Version Applicable Criteria

(e)(1)(ii)(A)(2)

§170.210(g)

Network Time Protocol Version 3

(RFC 1305)

§170. 210(g)


(RFC 5905)

(e)(3)(ii) Annex A of the FIPS Publication 140‐2

(b)(7)(i)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for

the indicated conditions

§170.207(a)(3)

IHTSDO SNOMED CT®




(e)(1)(i) Annex A of the FIPS Publication 140‐2

(a)(16)(ii)

§170.210(g)


(RFC 1305)

§170. 210(g)


(RFC 5905)

(b)(2)(i)(A)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for


§170.207(a)(3)

IHTSDO SNOMED CT®




(b)(8)(i)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for


§170.207(a)(3)

IHTSDO SNOMED CT®




(a)(15)(i)

§170.204(b)(1)


Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

3.2.5 Optional Functionality

No optional functionality tested

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD‐

10‐PCS)

(b)(2)(ii)(C)

Transmit health information to a Third Party using

the standards specified at §170.202(b) and (c) (SOAP

Protocols)

(f)(3)

Ambulatory setting only – Create syndrome‐based

public health surveillance information for transmission

using the standard specified at §170.205(d)(3) (urgent

care visit scenario)

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(3) (45 CFR162.1002(a)(4):

Code on Dental Procedures and Nomenclature)

(e)(1)

View, download and transmit data to a third party

using the standard specified at §170.202(d) (Edge

Protocol IG version 1.1)

(f)(7) Ambulatory setting only – transmission to public

health agencies – syndromic surveillance ‐ Create Data

Elements

(b)(1)(i)(B)

Receive summary care record using the standards

specified at §170.202(a) and (b) (Direct and XDM

Validation)

(b)(1)(i)(C) Receive summary care record using the standards

specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B)

Transmit health information to a Third Party using

the standards specified at §170.202(a) and (b) (Direct

and XDM Validation)

Criterion # Optional Functionality Successfully Tested

(a)(4)(iii) Plot and display growth charts



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

3.2.6 2014 Edition Certification Criteria* Successfully Tested

TP** TD*** TP** TD***

*For a list of the 2014 Edition Certification Criteria, please reference

http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)

**Indicates the version number for the Test Procedure (TP)

***Indicates the version number for the Test Data (TD)

(b)(7)

(g)(3)

(c)(1)

(g)(4)

(c)(2)

(b)(3)

(f)(6) Optional & Amb. only

(b)(4)

(b)(5)

(g)(1)

(b)(6) Inpt. only

(g)(2)

(b)(8)

(b)(9)

(f)(7) Amb. only

(h)(1)

(h)(2)

(h)(3)

(a)(16) Inpt. only (f)(3)

(a)(17) Inpt. only (f)(4) Inpt. only

(b)(1)

(f)(5) Optional & Amb. only

(b)(2)

(a)(13) (e)(3) Amb. only

(a)(14) (f)(1)

(a)(15) (f)(2)

(a)(18)

(a)(20)

(a)(19)

(a)(10) (d)(9) Optional

(a)(11) (e)(1)

(a)(12) (e)(2) Amb. only

(a)(7) (d)(6)

(a)(8) (d)(7)

(a)(9) (d)(8)

(a)(4) (d)(3)

(a)(5) (d)(4)

(a)(6) (d)(5)

(a)(1) (c)(3)

(a)(2) (d)(1)

(a)(3) (d)(2)

Criteria #Version

Criteria #Version



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

2 90 136 155

22 117 137 156

50 122 138 157

52 123 139 158

56 124 140 159

61 125 141 160

62 126 142 161

64 127 143 163

65 128 144 164

66 129 145 165

68 130 146 166

69 131 147 167

74 132 148 169

75 133 149 177

77 134 153 179

82 135 154 182

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113

55 104 114

60 105 171

*For a list of the 2014 Clinical Quality Measures, please reference

http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs

Inpatient CQMs

Ambulatory

Inpatient

No CQMs tested



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation was not tested

3.2.9 Attestation

(a)(17) (b)(5)

(a)(9) (a)(17) (b)(5)

(e)(2)

(e)(3)

(a)(6) (a)(15) (b)(3)

Safety‐Enhanced Design* A

(a)(5) (a)(14) (b)(2)

Attestation Forms (as applicable) Appendix

(a)(11) (a)(18)

Automated Numerator Recording Successfully Tested

(a)(1) (a)(11) (a)(18) (b)(6)

(e)(1)

(e)(3)

(a)(9)

(a)(7) (a)(16) (b)(4)

(a)(3) (a)(12) (a)(19) (b)(8)

(a)(4) (a)(13) (a)(20) (b)(9)

(a)(7) (a)(16) (b)(4)

Automated Numerator Recording Successfully Tested

(a)(1) (b)(6)

(a)(5) (a)(14) (b)(2)

Quality Management System** B

Privacy and Security C

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),

(a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)

**Required for every EHR product

(e)(1)

(a)(6) (a)(15) (b)(3)

(a)(3) (a)(12) (a)(19) (b)(8)

(a)(4) (a)(13) (a)(20) (b)(9)

(e)(2)



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

Appendix A: Safety Enhanced Design


10275 West Higgins Road, Suite 250 Rosemont, IL 60018 Tel: 844 240 9554 picis.com

June 16, 2016 Milton Padilla InfoGard, a UL Company 709 Fiero Ln, Suite 25 San Luis Obispo, CA 93401 Dear Mr. Padilla, Per 170.314(g)(3) criteria, this letter serves as attestation of our User-Centered Design practices. Picis PulseCheck follows Solution Architectural Design documents (specifically sections 4.0 – 4.4) for the development of each version, including ED PulseCheck versions 5.4, 5.5, and 5.6. The Solution Architectural Design documents are part of the Picis overall Quality Management System which is designed to meet the requirements of the international standard ISO 13485 Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes Standards as well as the FDA Quality System Regulation 21 CFR Part 820. Additionally, Picis maps our procedures to IEC 62366-1 Application of Usability Engineering to Medical Devices. The procedures were followed for planning, development, testing, and release of each application version and capability for which Picis sought certification, including:

170.314(a)(1) Computerized provider order entry

170.314(a)(2) Drug-drug, drug-allergy interaction checks

170.314(a)(6) Medication list

170.314(a)(7) Medication allergy list

170.314(b)(3) Electronic prescribing Please reference the attached Solution Architectural Design documents for further details and please do not hesitate to contact me with any outstanding questions. Thank you,

Merrily King Turnbull, MBA Director, Product Management Emergency Department Solutions

of 56

EHR Usability Test Report of ED PulseCheck Version 5.4

Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

Optum ED PulseCheck Version 5.4

Date of Usability Test 12/03/2013 – 12/12/2013 Date of Report 01/27/2014 Report Prepared By John Romadka, Senior User Experience Researcher, Optum

(612) 632-2694 [email protected]

Jamie Peffley, User Experience Manager, Optum


Scott Stewart, Senior User Experience Designer, Optum


mailto:[email protected]



of 56

EXECUTIVE SUMMARY ................................................................................................................................................... 4

PERFORMANCE DATA ....................................................................................................................................................... 5 Physicians .................................................................................................................................................................. 6

Major Findings ......................................................................................................................................................................... 7 Areas for Improvement ............................................................................................................................................................ 7

RNs ............................................................................................................................................................................ 8 Major Findings ......................................................................................................................................................................... 8 Areas for Improvement ............................................................................................................................................................ 9

INTRODUCTION ............................................................................................................................................................... 9

METHOD ........................................................................................................................................................................... 9

PARTICIPANTS ................................................................................................................................................................. 9 RNs .......................................................................................................................................................................... 11

STUDY DESIGN .............................................................................................................................................................. 12 TASKS ........................................................................................................................................................................... 12

Physicians ................................................................................................................................................................ 12 RNs .......................................................................................................................................................................... 13

PROCEDURES ................................................................................................................................................................ 14 TEST LOCATION ............................................................................................................................................................. 14 TEST ENVIRONMENT ...................................................................................................................................................... 14 TEST FORMS AND TOOLS ............................................................................................................................................... 15 PARTICIPANT INSTRUCTIONS .......................................................................................................................................... 15 USABILITY METRICS ....................................................................................................................................................... 16

DATA SCORING ............................................................................................................................................................. 17

RESULTS ........................................................................................................................................................................ 18

DATA ANALYSIS AND REPORTING .................................................................................................................................... 18 Physicians ................................................................................................................................................................ 19

Discussion of the Findings ..................................................................................................................................................... 20 Major Findings ....................................................................................................................................................................... 23

RNs .......................................................................................................................................................................... 25 Discussion of the Findings ..................................................................................................................................................... 25 Major Findings ....................................................................................................................................................................... 27

Combined (Physician and RN) Areas for Improvement .......................................................................................... 28

APPENDICES ................................................................................................................................................................. 29

APPENDIX 1: PARTICIPANT DEMOGRAPHICS .................................................................................................................... 30 Physicians ................................................................................................................................................................ 30 RNs .......................................................................................................................................................................... 31

APPENDIX 2: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ................................................................. 32 APPENDIX 3: EXAMPLE MODERATOR’S GUIDE FOR PHYSICIANS AND RNS ........................................................................ 33 APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................................................................ 34 APPENDIX 5: TASKS ....................................................................................................................................................... 35

Physicians ................................................................................................................................................................ 35 Task 0 – Login ....................................................................................................................................................................... 35 Task 1 - Enter a laboratory, radiology, and imaging orders ................................................................................................... 36 Task 2 - Enter medication orders .......................................................................................................................................... 37

of 56

Task 3 – Cancel or hold a medication order .......................................................................................................................... 38 Task 4 – Delete or cancel a medication order ....................................................................................................................... 39 Task 5 – Update a medication order...................................................................................................................................... 40 Task 6 –Update a radiology order ......................................................................................................................................... 41 Task 7 – Approve a medication order with a drug-allergy reaction ........................................................................................ 42 Task 8 – Approve a medication order with a drug-drug interaction ....................................................................................... 43 Task 17 - Configure pharmacy for e-prescription .................................................................................................................. 44 Task 18 - Change pharmacy for e-prescription ..................................................................................................................... 45

RNs .......................................................................................................................................................................... 46 Task 0 – Login ....................................................................................................................................................................... 46 Task 10 - Enter active medications ....................................................................................................................................... 47 Task 11 - Update a medication currently being taken............................................................................................................ 48 Task 12 - Inactivate an active medication .............................................................................................................................. 49 Task 13 – Find active and inactive allergies .......................................................................................................................... 50 Task 14 - Enter active allergies ............................................................................................................................................. 51 Task 15 - Update the effects of an allergy ............................................................................................................................. 52 Task 16 - Inactivate an allergy ............................................................................................................................................... 53

APPENDIX 6: DEMOGRAPHICS QUESTIONNAIRE ............................................................................................................... 54 APPENDIX 7: DATA SCORING DETAILS ............................................................................................................................. 56

of 56

Executive Summary A summative usability test of ED PulseCheck version 5.4 was conducted 12/03/2013 – 12/12/2013 by the

Optum User Experience Center of Excellence. The test was conducted remotely using a web-based testing

tool, UserZoom, and WebEx. The purpose of this test was to provide evidence of usability in the EHR Under

Test (EHRUT) and identify areas for incremental usability improvement.

Ten physicians and 13 RNs matching the target demographic criteria participated in the usability test, using

the EHRUT in simulated but representative tasks.

During the 60-minute one-on-one usability sessions, each participant was greeted by the administrator and

asked to review and sign a non-disclosure/consent form (see Appendix 2). Participants were instructed that

they could withdraw at any time. All participants had prior experience with the EHRUT.

The administrator introduced the test and instructed participants to complete a series of tasks (presented

randomly one at a time) using the EHRUT. During the testing, the testing tool (UserZoom) recorded user

performance data electronically. A Data logger also recorded user performance. The administrator did not

give the participant assistance in how to complete the task (and if it became necessary, that task was

marked as failed).

Physicians performed the following tasks:

Area of Certification

Tasks

§170.314(a)(1)

Computerized

provider order

entry

1 Enter laboratory and radiology/imaging orders

2 Enter medication orders

3 Cancel or hold a medication order

4 Delete or cancel a medication order

5 Update a medication order

6 Update a radiology order

§170.314(a)(2)

Drug-drug, drug-

allergy

interaction

checks

7 Approve a medication order with a drug-allergy reaction

8 Approve a medication order with a drug-drug interaction

§170.314(b)(3)

Electronic

prescribing

17 Configure pharmacy for e-prescription

18 Change pharmacy for e-prescription

of 56

RNs performed the following tasks:


Tasks

§170.314(a)(6)

Medication list

10 Enter active medications

11 Update a medication currently being taken

12 Inactivate an active medication

§170.314(a)(7)

Medication

allergy list

13 Find a list of active and inactive allergies

14 Enter active allergies

15 Update the effects of an allergy

16 Inactivate an allergy

Each participant’s interactions with the EHRUT were recorded for subsequent analysis. The following types

of data were collected:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations

• Participant’s satisfaction ratings of the tasks and system

All participant data was de-identified to avoid linking the identity of the participant to the data collected.

Following the conclusion of the testing, participants were asked to complete a post-test questionnaire.

Physicians and RNs were compensated $250 and $150 respectively for their time.

Performance Data Various recommended metrics, in accordance with the examples set forth in the NISTIR Guide to the

Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the

usability of the EHRUT. Following is a summary of the performance and rating data collected on the

EHRUT.

of 56

Physicians

Measure

N Task

Success Path

Deviation Task Time Errors

Task Ratings (5=Easy)

Task # Mean (SD)

Deviations (Observed /

Optimal)

Mean (SD)


Optimal)

Mean (SD)

Mean (SD)

1. Enter laboratory and

radiology/imaging orders 10

70%

(±48%) 1.7

99

(±54) 179%

40%

(±70%)

3.9

(±74%)

2. Enter medication orders 10 50%

(±53%) 2.1

180

(±50) 279%

70%

(±95%)

3.8

(±63%)

3. Cancel or hold a

medication order 10

90%

(±32%) 0.9

44

(±22) 98%

10%

(±32%)

4.6

(±84%)

4. Delete or cancel a

medication order 10

80%

(±42%) 1.5

46

(±19) 133%

20%

(±42%)

4.7

(±67%)

5. Update a medication

order 10

90%

(±32%) 1.4

98

(±40) 121%

10%

(±32%)

4.3

(±67%)

6. Update a radiology order 10 90%

(±32%) 1.8

78

(±29) 144%

10%

(±32%)

4.0

(±105%)

7. Approve a medication

order with a drug-allergy

reaction

10 80%

(±42%) 3.0

75

(±30) 123%

20%

(±42%)

4.9

(±32%)

8. Approve a medication

order with a drug-drug

interaction

10 80%

(±42%) 2.3

76

(±21) 148%

20%

(±42%)

4.5

(±53%)

17. Configure pharmacy for

e-prescription 10

50%

(±53%) 2.4

167

(±56) 207%

50%

(±53%)

3.4

(±97%)

18. Change pharmacy for

e-prescription 10

50%

(±53%) 1.9

184

(±142) 196%

80%

(±103%)

3.7

(±82%)

The System Usability Scale was used to score participants’ subjective satisfaction based on their overall

experience with the EHRUT. The overall score was 86 (±10).1

In addition to the performance data, we recorded the following qualitative observations.

1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

of 56

Major Findings Major findings include:

• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness

• Participants indicated an above-average level of product satisfaction

• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants

used a variety of the methods available (in some cases, this resulted in slower task times)

• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT

As well as issues with:

• E-prescription search

• Data loss when users entered free text

• Incomplete record entries/updates (e.g., without schedules)

• Multiple action buttons too close to each other

• Dropdown lists (number of items and inconsistencies in position)

Areas for Improvement Recommendations for improvement include the following:

• Improve flexibility in search entries and results presentation for e-Prescription searches

• Alert the user when necessary data (e.g., medication schedule) are skipped to avoid incomplete

entries/updates

• Help users find the correct action (e.g., visually distinguishing the button required to complete the

action)

• Recognize free text data entries (e.g., medical abbreviations) to avoid data loss

• Consistent dropdown list display (location and length)

of 56

RNs

Measure N

Task Success

Path Deviation

Task Time Errors Task

Ratings (5=Easy)

Task # Mean (SD)


Optimal)

Mean (SD)


Optimal)

Mean (SD)

Mean (SD)

10. Enter active medications 13 60%

(±52%) 2.3

172 (±29)

235% 80%

(±104%) 4.6

(±70%)


currently being taken 13

69% (±48%)

1.8 120

(±45) 187%

54% (±88%)

3.9 (±126%)

12. Inactivate an active

medication 13

85% (±38%)

2.0 114

(±30) 165%

15% (±38%)

4.4 (±96%)

13. Find a list of active and

inactive allergies 13

46% (±52%)

1.4 59

(±25) 198%

62% (±51%)

4.3 (±125%)

14. Enter active allergies 13 92%

(±28%) 1.4

113 (±33)

145% 31%

(±85%) 4.5

(±78%)

15. Update the effects of an

allergy 13

77% (±44%)

1.4 82

(±57) 133%

31% (±48%)

4.4 (±119%)

16. Inactivate an allergy 13 69%

(±48%) 1.2

98 (±32)

128% 23%

(±44%) 4.7

(±48%)

The System Usability Scale was used to score participants’ subjective satisfaction with the system based on

their overall experience with the EHRUT. The overall score was 87(±17).2

In addition to the performance data, we recorded the following qualitative observations.

Major Findings Major findings include:

• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness

• Participants’ indicated an above-average level of product satisfaction

• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants

used a variety of the methods available (in some cases, this resulted in slower task times)

• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT)

As well as issues with:

• Difficulty finding “Show inactive” allergies

• Multiple action buttons too close to each other

2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

of 56

• Difficulty finding how to update the effects of an allergy

• Consistent dropdown list display (location and length)

Areas for Improvement Recommendations for improvement include the following:

• Make “Show inactive” more visually identifiable

• Help users find the correct action buttons

• Don’t rely on a hyperlinked line with no label or text for navigation

• Consistent dropdown list (location and length)

Introduction The EHRUT tested for this study was ED PulseCheck version 5.4. Designed to present medical information

to healthcare providers in an emergency department, the EHRUT consists of a tracking board, patient

charts, medication ordering system, lab ordering system, and e-prescription system. The usability testing

represented realistic tasks.

The purpose of this test was to provide evidence of usability in the EHRUT and identify areas for

incremental usability improvement. To this end, measures of effectiveness, efficiency, and user satisfaction

(such as task completion, time on task, and task efficiency) were captured during the usability testing.

Method

Participants A total of 23 participants, consisting of 10 physicians and 13 RNs, were tested on the EHRUT. All

participants were given the same orientation instructions. Physicians and RNs were compensated $250 and

$150 respectively for their time.

Participants were recruited by Leede Research. Participants had no direct connection to the development of

or organization producing the EHRUT. Participants were not from the testing or supplier organization. All

participants currently use the EHRUT in a professional workplace and were recruited based on job title,

gender, and age.

The following table summarizes the characteristics of the physicians who participated in the usability test,

including demographics, professional experience, computing experience and use of assistive technology.

Participant names were replaced with Participant IDs so that data cannot be tied back to identities.

of 56

Part ID

Part ID

Gender Age Education Occupation Professional Experience

Computer Experience

Product Experience

Assistive Technologies

1 P4 Male 30-39 Professional

degree (e.g.

MD)

Physician 7-9 years 20-29

years

1-3 years No


degree (e.g.

MD)

Physician 20-29 years 7-9 years 4-6 years Yes, Dragon


degree (e.g.

MD)


years

7-9 years No


degree (e.g.

MD)


years

4-6 years No

5 P25 Female 50-59 Professional

degree (e.g.

MD)

Physician 20-29 years 7-9 years 4-6 years No


degree (e.g.

MD)


years

1-3 years No


degree (e.g.

MD)

Physician 20-29 years 7-9 years Less than

1 year

No


degree (e.g.

MD)


years

4-6 years No


degree (e.g.

MD)



degree (e.g.

MD)


Fourteen physicians were originally recruited, including two pilots.

• Three sessions were eliminated from results because of an inconsistency between the script and the

test environment in task 2

• One participant was found not to meet the requirements

• There were two no-shows

Two additional physicians were recruited to provide test results for 10 physicians.

of 56

Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each

session for debrief by the administrator and data logger and to reset systems to proper test conditions. A

spreadsheet was used to track the participant schedule.

RNs

The following table summarizes the characteristics of the RNs who participated in the usability test,

including demographics, professional experience, computing experience, and use of assistive technology.

Participant names were replaced with Participant IDs so that data cannot be tied back to identities.

Part ID

Part ID


Computer Experience

Product Experience


1 P2 Female 30-39 Associate’s

degree

Registered

Nurse

10-19

years

10-19 years 7-9 years No

2 P3 Female 40-49 Doctoral

degree

Nurse

Practitioner

10-19

years


3 P5 Male 50-59 Bachelor’s

degree

Registered

Nurse

7-9 years 10-19 years 7-9 years No


degree

Registered

Nurse

10-19

years


5 P8 Female 30-39 Bachelor’s

degree

Registered

Nurse



degree

Registered

Nurse

4-6 years 1-3 years Less than 1

year

Yes, Dragon


degree

Registered

Nurse

20-29

years


8 P19 Male 60-69 Associate’s

degree

Registered

Nurse

20-29

years



degree

Registered

Nurse

10-19

years



degree

Registered

Nurse

4-6 years 10-19 years 4-6 years Yes, Dragon


degree

Registered

Nurse

20-29

years



degree

Registered

Nurse



degree

Registered

Nurse

10-19

years


of 56

A total of 14 RNs were recruited for the test. One participant failed to show for the study. Test results for 13

RNs are included in this report.

Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each

session for debrief by the administrator and data logger and to reset systems to proper test conditions. A

spreadsheet was used to track the participant schedule.

Study Design

Overall, the objective of this test was to measure how well the application performed with regard to

effectiveness, efficiency, and user satisfaction; to identify areas for improvement; and to recommend

improvements. Additionally, the data from this test will serve as a baseline for future tests for comparison

with updated versions of the EHR and/or other EHRs. In short, this testing serves as a means to benchmark

current usability and to identify areas where improvements can be made.

All participants were given the same orientation instructions and accessed the EHRUT using the same

remote testing environment. Two task sets were defined—one for physicians and one for RNs. All

participants used the same test data. The EHRUT was evaluated using the following measures:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations (comments)

• Participant’s satisfaction ratings of the tasks and system

Additional information about the various measures can be found in the Usability Metrics section of this

document.

Tasks Tasks were selected based on how frequently they are used, how critical they are, and how likely they are

to be troublesome for users. Task details were defined to be realistic and representative of the kinds of

activities users might perform with the EHRUT.

Physicians

Physicians performed the following tasks:


Tasks

§170.314(a)(1)

Computerized

provider order

1 Enter laboratory and radiology/imaging orders

2 Enter medication orders

of 56


Tasks

entry 3 Cancel or hold a medication order

4 Delete or cancel a medication order

5 Update a medication order

6 Update a radiology order

§170.314(a)(2)

Drug-drug, drug-

allergy

interaction

checks

7 Approve a medication order with a drug-allergy reaction

8 Approve a medication order with a drug-drug interaction

§170.314(b)(3)

Electronic

prescribing

17 Configure pharmacy for e-prescription

18 Change pharmacy for e-prescription

RNs

RNs performed the following tasks:


Tasks

§170.314(a)(6)

Medication list

10 Enter active medications

11 Update a medication currently being taken

12 Inactivate an active medication

§170.314(a)(7)

Medication

allergy list

13 Find a list of active and inactive allergies

14 Enter active allergies

15 Update the effects of an allergy

16 Inactivate an allergy

of 56

Procedures Upon arrival, participants were greeted and their identity was verified and matched with the participant

schedule. Participants were then assigned a participant ID.3 Each participant digitally signed a non-

disclosure and informed consent form. See Appendix 2: Non-Disclosure Agreement and Informed Consent

Form for an example of the form.

The test was conducted by two staff members, the administrator, and the data logger. The individuals

conducting the test were usability practitioners with over 15 years of experience and educational

backgrounds in Human Factors.

The administrator moderated the session, delivered initial instructions, and took notes on participant

comments, path deviations, and errors. The data logger also took notes on task success, path deviations,

errors, and comments. The testing tool (UserZoom) led participants through the tasks. UserZoom recorded

task times, post-task rating data, post-test satisfaction data, and demographic data.

Participants were instructed to perform the tasks (see the Participant Instructions below):

• As quickly as possible making as few errors and deviations as possible

• Without assistance

• Without commenting until they had completed each task (if possible)

UserZoom provided participants with instructions before and during each task. Tasks were presented to

each participant in a randomized order to avoid learning bias. Task timing began once the participant clicked

the “Start Task” button in UserZoom. Task timing ended once the participant clicked the “Completed Task”

button in UserZoom. Scoring is discussed in the Data Scoring section.

Following the session, UserZoom displayed the post-test survey, including a System Usability Scale (SUS)

questionnaire and demographic questions. See Appendix 4: System Usability Scale Questionnaire for

additional information on SUS questionnaires. The administrator told participants to expect to receive

compensation in 2-3 weeks.

Test Location

Tests were conducted remotely, with each individual at their own location. Only the participant,

administrator, and data logger were present.

Test Environment The EHRUT typically is used in an emergency department. Tests were conducted remotely, using the

administrator’s computer to run WebEx, Windows Remote Desktop Connection (RDC), and TechSmith

Camtasia (for recording). RDC was used to connect remotely to the EHRUT test environment via a Virtual

Private Network (VPN) using a broadband cable connection with a minimum throughput of 10Mbps

download and 3Mbps upload. Participants used their own PCs during studies. They were given control of

the administrator’s computer to access the application during the session, but used their mouse, keyboard,

and display. (Note, many participants indicated they were not using the computer setup they normally use

with the EHRUT.) Participants PC information was not collected.

3 All participant data has been de-identified and will be kept confidential.

of 56

Additionally, participants were told to maximize the WebEx sharing window to full screen and not to change

any default system settings (such as control of font size). They also were told to close any applications

other than the WebEx session.

The administrator’s computer is an HP EliteBook 8570w laptop with a Core i7 Processor, 180GB SSD

hard drive, 16GB RAM, running the Windows 7 Basic Theme. The display resolution was set to

1440x900. This computer ran WebEx screen sharing, the VPN broadband connection, the RDC to the

EHR test environment, and Camtasia.

The EHR test environment on the RDC used Internet Explorer 8.0 (zoom level of 125%). The RDC

displayed at a resolution of 1920x1080 in full screen mode with 24-bit True Color and was set for high-

speed broadband (2-10Mbps) with persistent bitmap caching.

Test Forms and Tools During the usability test, UserZoom presented various documents, including:

1. Informed Consent

Participants were emailed the Informed Consent prior to session so they could review it. At the

beginning of session, they were asked to indicate in UserZoom if they agreed or disagreed with the

terms.

2. Test instructions

3. Post-task Questionnaire

4. Post-test Questionnaire

5. Participant Demographics Questionnaire

Examples of these documents can be found in Appendix 2: Non-Disclosure Agreement and Informed

Consent Form, Appendix 3: Example Moderator’s Guide, and Appendix 4: System Usability Scale

Questionnaire. The Moderator’s Guide was designed to enable the data logger to easily capture data.

The script for participant orientation follows:

Participant Instructions

Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at

Optum, makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and they’ll

be helping take notes.

Before we begin, I’d like to take a moment to describe the study we’re conducting.

We’re collecting feedback on a software product called ED PulseCheck.

I know someone already asked you this, but I just wanted to verify that you currently use ED

PulseCheck to work with patient records.

Has not used ED PulseCheck Has used ED PulseCheck

We’re going to ask you to perform a series of tasks. You should know that the tasks were created to

be independent of each other and some may sound redundant. You will be asked to complete these

tasks on your own, trying to do them as quickly as possible with the fewest possible errors or

of 56

deviations. Please do not do anything more than the task asks.

You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the

product is to use, so just do your best to complete them as you typically would. We didn’t have

anything to do with designing the product, and your comments are entirely anonymous, so please

feel free speak candidly. However, please save your comments until the end of a task, or the end of

the session as a whole, when we can then discuss them freely.

We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please

read the task information out loud, so we can follow along.

I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my

computer, but I’ll give you control of my computer’s mouse and keyboard for the duration of the

study. Because we can’t always control the bandwidth between our two locations, you may notice a

tiny lag or latency when you’re in control of the mouse and keyboard. However, we don’t anticipate

this will affect your ability to use ED PulseCheck.

Before I do that, please quit or close any other applications that are currently running on your

computer (email and Instant Messaging), with the exception of your browser and WebEx. Also,

please set any mobile devices to silent during the study.

Participants were then presented with tasks one at a time, randomly assigned by UserZoom. See Tasks for

a complete list of tasks.

Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for

users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this

end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing.

The goals of the test were to assess:

1. Effectiveness of the EHRUT (ED PulseCheck) by measuring participant success rates and errors

2. Efficiency of the EHRUT by measuring the average task time and path deviations

3. Satisfaction with the EHRUT by measuring ease-of-use ratings

of 56

Data Scoring The following table details how the testing team scored the tasks, evaluated the errors, and analyzed the time

data.4

Measures Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct

outcome, without assistance, within the time allotted.

The total number of successes were calculated for each task, and then the results were

divided by the total number of times that task was attempted. The results are provided as a

percentage.

The total Task times were recorded for successes. Observed task times were divided by the

optimal performance time for each task to provide a measure of optimal efficiency. The

results are provided as a percentage mean.

Optimal task performance time was benchmarked when constructing tasks by using

Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and task time to obtain

an expected time based on the optimal path. (KLM was used rather than expert time

because the product includes many paths to accomplish the tasks, and experts frequently

didn’t follow optimal paths.) Additional time, based on test environment measurements, was

added to the KLM results to account for the following:

1. User’s need to review information while interacting with the user interface (medical

professionals double-check their work routinely to avoid error)

2. System response times

See Appendix 7: Data Scoring Detail for additional information regarding how the KLM

was calculated.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer, or performed it

incorrectly, the task was counted as a “Failure.”

The total number of errors was calculated for each task, and the results were divided by the

total number of times the task was attempted. This was also expressed as the mean

number of failed tasks per participant.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations

occurred if the participant went to a wrong screen, clicked an incorrect menu item, followed

an incorrect link, or interacted incorrectly with an on-screen control. Each participant’s path

was compared to the optimal path. The number of steps in the observed path was divided

by the number of optimal steps to provide a ratio of path deviation.

4 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman. Also see

www.measuringusability.com.

http://www.measuringusability.com/

of 56

Deviations were defined as:

• Task Recognition: failure to recognize task or completion of a different task

• Data Entry: failure to enter complete data or entry of incorrect data

Efficiency:

Task Time

Each task was timed from when the participant clicked the “Start Task” button until the

participant clicked the “Task Complete” button. Only completion times for successfully

completed tasks were included in the average task time analysis. Average times per task

and variance measures (standard deviation and standard error) were calculated for each task.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease-of-use of the application was measured by

administering a post-task question and a post-session questionnaire.

After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1

(Very Difficult) to 5 (Very Easy). These data are averaged across participants.5 Common

convention is that average ratings for systems judged easy to use should be 3.3 or above.

To measure participants’ confidence in and satisfaction with the EHRUT overall, the testing

team administered the System Usability Scale (SUS) post-test questionnaire. Questions

included, “I think I would like to use this system frequently”, “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.”

See Appendix 4: System Usability Scale Questionnaire for a complete list of SUS

questions.6

Results

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the preceding Usability

Metrics section. We excluded data for participants who failed to follow session and task instructions.

We conducted four sessions with physicians and discovered an issue with one task. The original script

asked the participant to administer the medication nitro “as needed,” but there was no “as needed” option in

the application test environment. We removed “as needed” from the script to avoid confusion. Sessions

impacted by this were eliminated from the results. We recruited additional physicians so that 10 physician

results could be included in the analysis.

The usability testing results for the EHRUT are detailed below. The results are presented in the context of

the objectives and goals outlined in Study Design. The report includes actionable results that, if corrected,

should yield a material, positive impact on user performance.

The following table presents data for physicians.

5 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco, D. & Tullis, T. (2006)

A comparison of methods for eliciting post-task subjective ratings in usability testing. Usability Professionals

association Conference, June 12 – 16, Broomfield, CO. 6 The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the

system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other

iterations of the system.

of 56

Physicians

Measure N

Task Success

Path Deviation

Task Time Errors Task

Ratings (5=Easy)

Task # Mean (SD)


Optimal)

Mean (SD)


Optimal)

Mean (SD)

Mean (SD)

1. Enter laboratory and

radiology/imaging orders 10

70%

(±48%) 1.7

99

(±54) 179%

40%

(±70%)

3.9

(±74%)

2. Enter medication orders 10 50%

(±53%) 2.1

180

(±50) 279%

70%

(±95%)

3.8

(±63%)

3. Cancel or hold a medication

order 10

90%

(±32%) 0.9

44

(±22) 98%

10%

(±32%)

4.6

(±84%)

4. Delete or cancel a

medication order 10

80%

(±42%) 1.5

46

(±19) 133%

20%

(±42%)

4.7

(±67%)

5. Update a medication order 10 90%

(±32%) 1.4

98

(±40) 121%

10%

(±32%)

4.3

(±67%)

6. Update a radiology order 10 90%

(±32%) 1.8

78

(±29) 144%

10%

(±32%)

4.0

(±105%)

7. Approve a medication order

with a drug-allergy reaction 10

80%

(±42%) 3.0

75

(±30) 123%

20%

(±42%)

4.9

(±32%)

8. Approve a medication order

with a drug-drug interaction 10

80%

(±42%) 2.3

76

(±21) 148%

20%

(±42%)

4.5

(±53%)

17. Configure pharmacy for

e-prescription 10

50%

(±53%) 2.4

167

(±56) 207%

50%

(±53%)

3.4

(±97%)

18. Change pharmacy for

e-prescription 10

50%

(±53%) 1.9

184

(±142) 196%

80%

(±103%)

3.7

(±82%)

The SUS scored participants’ subjective satisfaction with the system based on their overall experience with

the EHRUT. The overall score was 86 (±10). Broadly interpreted, scores under 60 represent systems with

poor usability; scores over 80 would be considered above average.7

7 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).

of 56

Discussion of the Findings

Findings for each major area’s research question follow.

§170.314(a) (1) Computerized provider order entry

Can a user order medication, laboratory, and radiology / imaging orders? (Task 1-2)

• Participants were more successful entering lab/radiation/imaging orders (70%) than entering a

medication order (50%). Entering medication orders accounted for some of the highest incidences of

data entry errors (70%), primarily because participants frequently didn’t use the clinical pathways or

chose the incorrect clinical pathway (e.g., Data set 1). These two tasks also accounted for some of

the lowest task satisfaction ratings.

Can a user change an order? (Task 3-6)

• Overall, participants had the most success completing these three tasks, the lowest number of

errors, and some of the highest task satisfaction ratings.

§170.314(a) (2) Drug-drug, drug-allergy interaction checks

Can a user order a medication with a drug-allergy interaction? (Task 7)

• While the task success was comparatively high (80%), participants exhibited more path deviations

than with any other task. Compared to the expected path, participants needed three times more

clicks to complete the task, mainly because they were unfamiliar with the schedule and route

dropdown lists and had to search more. (The applications they use in their ERs have customized

dropdown lists.) However, this task had the highest task satisfaction.

• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication

schedule, were unaware that the application didn’t accept the entry, and submitted the order without

a schedule.

Can a user order a medication with a drug-drug interaction? (Task 8)

• The task success (80%) and satisfaction (4.5) was comparatively high, and errors (20%) were

relatively low.

§170.314(b) (3) Electronic prescribing

Can a user issue e-prescriptions? (Task 17-18)

• These two tasks represented the lowest task success ratios (50%) and had comparatively high error

rates. These tasks have the lowest task satisfaction scores primarily because search results were

very difficult to obtain, and the results display was inconsistent. Participants entered correct data, but

if they included apostrophes or capital letters, no results appeared. Also, sometimes results

appeared “below the fold” of the pop-up window, and participants didn’t realize that they needed to

scroll down.

of 56

Effectiveness

Task 1

• 20% of participants entered the incorrect Gas & Carbon panel.

Task 2

• 20% of participants didn’t enter the schedule for one (or both) of the medications.

• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication

schedule, were unaware the application didn’t accept the entry, and submitted the order without a

schedule.

Tasks 3-6

• These four tasks represented the highest task success ratios.

o Task 3: 90%

o Task 4: 80%

o Task 5: 90%

o Task 6: 90%

Task 17-18

• 50% of participants failed to complete these two tasks. This was because: (1) search did not accept

entries that included capitalization or apostrophes and (2) results occasionally were “below the fold”

of the pop-up window where participants didn’t notice to would have needed to scroll down.

Half of the participants who successfully completed the task made multiple attempts entering search

criteria.

Efficiency

In general, participants who were the most efficient understood that they could type the first letter or two into

dropdown lists to narrow their results. The user interface does not indicate that this is possible.

The greatest variability in the actual task time vs. expected task time was due to the following:

• Use of Search instead of Quicklists for medications

• Unfamiliarity with dropdown list items and length

• Limited use of navigational shortcuts (e.g., GO button)

Task 1

• Average task time was 79% longer than expected because participants either didn’t use the

predefined clinical pathway (Abdominal and chest pain) or selected the “Data Set 1” or “Data Set 2”

clinical pathway. Many opted to search for the orders. This was likely a result of the participants’

unfamiliarity with the test environment data and customizations.

of 56

Task 2

• 70% of participants searched for medications, rather than using Quicklists. As a result, the average

task time was 179% longer than expected, higher than any other task, and they had more than twice

the path deviations than the optimal path. Not using Groups accounted for a 70% error rate,

primarily because the participants then had to enter the medication schedules manually. Fifty per

cent of participants either didn’t enter a medication schedule or entered a medical abbreviation (e.g.,

PO or one time stat) which the EHRUT did not accept. The EHRUT did not alert the participants,

who thought they had submitted the medication order correctly.

Task 3-6

• These four tasks represented the lowest average error rates. Participants were generally able to

successfully complete the task at, or very close to, the expected task time.

o Task 3:

Errors: 10% (Only an average of 10% of participants made errors)

Task Time: -2% (Participants, on average, performed this task about the same, but

slightly faster than, expected)

o Task 4:

Errors: 20%

Task Time: 33% longer than expected, primarily because some participants used the

“cancel” link rather than “delete.” This added an additional step (and click)

o Task 5:

Errors: 10%

Task Time: 21% longer than expected as a result of inefficiencies typing the search

entry for penicillin. Some participants either misspelled or didn’t type enough letters

using “filter/find as you type” to easily find the item in the dropdown list

o Task 6:

Errors: 10%

Task Time: 44% longer than expected. When performing a search for “wrist,” some

participants either misspelled or didn’t type enough letters using “filter/find as you

type” to easily find the item in the dropdown list

Task 7

While participants’ task times were only slightly longer (23%) than expected, there were three times

the path deviations than was optimal. This was a direct result of inefficiencies typing the search entry

for “Vicodin ES.” Most participants either misspelled or didn’t type enough letters using “filter/find as

you type” to easily find the item in the dropdown list

of 56

Task 8

• The average task time was 48% longer than expected, because most participants used “Search”

rather than “Groups” to enter the medication. This required them to manually enter the dose, unit,

and route.

Task 17-18

• Because participants had difficulty finding the Search link, which was below the fold, the task times

for task 17 and 18 were higher than any other task. Participants took 107% (for task 17) and 96%

(for task 18) longer than the expected task times. Successful participants attempted the search

several times by editing the search parameters until they achieved a result.

• This was the first time any participants had used the e-prescription search, so they weren’t familiar

with its location. These task times should improve with greater familiarity and improved flexibility in

search entries.

Satisfaction

Task 17-18

• These two tasks represented the lowest satisfaction score, which is likely due to participants’

frustration with search results not being returned as expected.

Major Findings

1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness

2. Participants consistently reported above average levels of product satisfaction. Many reported that

of all the similar applications they’ve used, this was “the most intuitive and easy to use.” Other

comments were:

"Initially, I was skeptical about (ED PulseCheck) but a month after using it I found I was faster than

(hand-written notes and dictated orders).”

“I’ve grown fond of the product even though I “came to the dance” reluctantly.”

3. Participants often used mixed methods to navigate. Participants who used the GO button would

occasionally also use the top menu. Using the GO button is more efficient.

4. Multiple and often less efficient navigational paths are used to accomplish tasks, increasing users’

cognitive load.

5. Only 50% of participants successfully completed Tasks 17 and 18 (relating to e-prescription search)

because the search results were not provided or were not visible. Those participants that were

unsuccessful made multiple attempts to change the search entry before they gave up.

E-prescription search did not accept search entries that included capitalization or apostrophes. For

example, an entry of “Walgreen’s”, which is correct, did not yield results, while an entry of

“walgreens” did yield results. Results sometimes displayed “below the fold” of the pop-up window. It

wasn’t evident to participants that they needed to scroll down to view them.

Participants were very encouraged with the prospect of being able to enter e-prescriptions.

of 56

6. Data was lost and record updates were incomplete. This occurred when participants entered

medical notations (e.g., PO or one time stat) for the medication routes or schedule, were unaware

the application didn’t accept it, and submitted the order without the entry.

7. Because action buttons are too close to each other, participants accidently clicked (or hovered over

and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of

various buttons (e.g., Print Triage, Enter/Task, Enter/Med, Repeat) conflicts with the Enter button.

8. Dropdown lists displayed an inconsistent number of items depending on where they were on the

page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom

of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed

approximately 25 items.

9. Regarding medication orders, one participant commented, "I checked 'bulk' (medication for one of

the tasks). It's the same mistake I make at work. Sometimes those differentiations don't mean

anything to a physician. I just delete the medication, go back, and search again."

10. In general, task times and path deviations that were higher than anticipated may have been because

participants normally have localized customizations (e.g., predefined pathways, drug lists, and

schedules) but those familiar customizations weren’t available in the test environment. (They are

specific to each ER.) Using an EHR with the customizations participants are familiar with would likely

improve task times and reduce path deviations to more closely match the expected task times and

optimal paths.

One participant said “90-95% of the meds I use are on my Quicklist and 70-75% of the time they're

on my summary page. It's the same with orders. It's a wonderful shortcut."

11. One participant suggested a feature request: “I want to be able to draw on a picture [to show] big

lacerations on a body.”

12. Three participants suggested that customized order set personalization (by a user) would be very

useful and more efficient.

of 56

RNs

Measure

N Task

Success Path Deviation Task Time Errors

Task Ratings (5=Easy)

Task # Mean (SD)


Optimal)

Mean (SD)


Optimal)

Mean (SD)

Mean (SD)

10. Enter active medications 13 60%

(±52%) 2.3

172 (±29)

235% 80%

(±104%) 4.6

(±70%)


currently being taken 13

69% (±48%)

1.8 120

(±45) 187%

54% (±88%)

3.9 (±126%)

12. Inactivate an active

medication 13

85% (±38%)

2.0 114

(±30) 165%

15% (±38%)

4.4 (±96%)

13. Find a list of active and

inactive allergies 13

46% (±52%)

1.4 59

(±25) 198%

62% (±51%)

4.3 (±125%)

14. Enter active allergies 13 92%

(±28%) 1.4

113 (±33)

145% 31%

(±85%) 4.5

(±78%)

15. Update the effects of an

allergy 13

77% (±44%)

1.4 82

(±57) 133%

31% (±48%)

4.4 (±119%)

16. Inactivate an allergy 13 69%

(±48%) 1.2

98 (±32)

128% 23%

(±44%) 4.7

(±48%)

The SUS scored participants’ subjective satisfaction with the system based on their overall experience with

the EHRUT. The overall score was 87 (±17). Broadly interpreted, scores under 60 represent systems with

poor usability; scores over 80 would be considered above average.8

Discussion of the Findings

Findings for each major area’s research questions follow.

§170.314(a)(6) Medication list

Can a user enter active medications? (Task 10)

• This task resulted in the highest error rate (80%) and over twice the number of path deviations than

the optimal path, primarily due to incorrect medication being updated. Task satisfaction was the

second highest (4.6), likely due to the ease with which the EHRUT allows participants to efficiently

and effectively enter active medications.

8 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).

of 56

Can a user update active medications? (Task 11-12)

• 69% of participants successfully completed Task 11 and did so in nearly the same as the expected

task time. However, this task had the lowest task satisfaction (3.9), primarily because participants

had difficulty finding the control needed to update the medication schedule. This is a line with no text

that is a hyperlink; its functionality is not obvious.

• 85% of participants successfully completed Task 12 and had the fewest errors (15%). However,

there were two times more path deviations (clicks) than the optimal path, primarily because

participants used search rather than the Quicklist to find the medication.

§170.314(a)(7) Medication allergy list

Can a user find a list of active and inactive allergies? (Task 13)

• 54% could not complete the task, primarily because participants couldn’t find the “Show Inactive” link

to display the list of active and inactive allergies. This looks like a column heading rather than a link

or button, and the instructional test is not close to the control.

Can a user enter active allergies? (Task 14)

• 92% of participants successfully completed this task.

Can a user update active allergies? (Task 15-16)

• 77% of participants successfully completed Task 15.

• 69% successfully completed Task 16. However, it took 28% longer than expected, primarily because

several participants used Search to find the medication rather than the Quicklist.

Effectiveness

Task 10

• Low task success (60%) was primarily due to data entry errors (80%). Participants:

o Added notes to the wrong medication

o Selected an extra medication

o Recorded a schedule in the notes field

o Did not enter a schedule or entered it incorrectly.

Task 11

• Errors were made by 54% of participants which included:

o Choosing a medication unit "mcg" rather than "mg"

o Clicked on "Make Inactive'

o Clicked Print Triage Report rather than Enter. This resulted in the entries disappearing. The

EHRUT did not alert the user.

of 56

Task 13

• 54% of participants couldn't find the "Show Inactive" link and gave up without completing the task.

Task 14

• 92% of participants successfully completed this task.

Efficiency

• In general, the most efficient participants understood they could type the first letter or two into

dropdown lists to narrow their results. The user interface does not indicate that this functionality is

available.

• The greatest variability in the actual task time vs. expected task time was due to:

o Use of Search instead of Quicklists for medications

o Unfamiliarity with the dropdown lists items and length

o Limited use of navigational shortcuts (e.g., GO button)

• Longer than expected task times for Tasks 10, 11, 12, and 16 were primarily due to participants’ use

of Search rather than the Quicklist. Participants didn't notice that the medications they needed to

enter were in the Quicklist.

o In task 10, some participants had to navigate extremely long lists in the schedule dropdown.

If the participant didn't "type ahead" to narrow the list, they had to scan as many as 36

choices.

o In task 11, several participants had difficulty finding the hyperlink line, causing longer than

expected task times. One participant couldn’t figure out how to modify the current

medication, so they inactivated it and created a new entry.

• 46% of participants weren't familiar with or couldn't find the "Show Inactive" link in Task 13. Those

that were successful, but unfamiliar with it, eventually found it after reading the instructional note

below the table.

• The higher number of path deviations (compared to optimal) for Tasks 10, 12, and 16 was primarily

due to participants’ use of Search rather than the Quicklist. Most participants didn't notice that the

medications they needed to enter were in the Quicklist.

Satisfaction

• Task 11: The lower satisfaction rating was due to participants’ difficulty finding the control, which

was a hyperlinked line with no text.

Major Findings 1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness.

2. Participants expressed above average level of product satisfaction.

3. Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT).

of 56

4. The EHRUT has multiple navigation paths that participants can use to complete tasks; participants

used a variety of the methods available (in some cases, this resulted in slower task times). For

example, some participants clicked the patient radio button (which is unnecessary) before they used

the GO button.

5. Participants often used mixed methods to navigate. Participants who used the GO button would

occasionally also use the top menu. Using the GO button is more efficient.

6. Participants had difficulty finding “Show inactive” allergies.

7. Because action buttons are too close to each other, participants accidently clicked (or hovered over

and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of

the Print Triage button conflicts with the Enter button.

8. Participants had difficulty identifying the control (hyperlinked line) for editing an allergy. Participants

frequently clicked on the drug name link, expecting to be able to edit. This displayed a drug

information help window.

9. Dropdown lists displayed an inconsistent number of items depending on where they were on the

page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom

of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed

approximately 25 items.

10. Pharmacy search parameters were too restrictive (case-sensitive and not permitting punctuation).

Results sometimes required scrolling to view.

11. Users are not made aware when the system deletes unacceptable entries, resulting in data loss.

12. Users are unaware of “filter/find as you type” that would increase their efficiency.

Combined (Physician and RN) Areas for Improvement

1. Continue to emphasize to users that the GO button will improve their efficiency.9

2. Make “Show inactive” more visually identifiable.

3. Help users distinguish and click on the desired action button; consider the proximity and order of

importance.

4. Don’t rely on hyperlinked lines for navigation; use text.

5. Provide consistent behavior regarding how many items a dropdown list will display regardless of

where the list is clicked on in the page.

6. Provide more flexibility in the e-prescription search entry; allow for punctuation and capital letters.

Consistently present the results where users can see them without needing to scroll.

7. Allow entry of medical notations (e.g., PO or one time stat) for the medication routes or schedules to

avoid data loss, or alert users before loss of data.

8. Provide hints for “filter/find as you type” functionality.

9 Cognitive modeling indicates that users will save an estimated .5/sec. by using the GO button vs. the top-level menu.

When .5/sec. is extrapolated over the number of times a single user interacts with patient clinical documents using the GO button, over the course of a year, the efficiency improvement can be significant.

of 56

Appendices The following appendices include supplemental data for this usability test report. Following is a list of the

appendices provided:

1. Participant demographics

2. Non-Disclosure Agreement (NDA) and Informed Consent Form

3. Example Moderator’s Guide

4. System Usability Scale Questionnaire

5. Tasks

6. Demographics Questionnaire

of 56

Appendix 1: Participant Demographics Following is a high-level overview of the participants in this study.

Physicians

The following table summarizes the characteristics of the physicians that participated in the usability test,

including demographics, professional experience, computing experience and use of assistive technology.

Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to

individual identities.

Part ID

Part. ID


Computer Experience

Product Experience



degree (e.g.

MD)


years

1-3 years No


degree (e.g.

MD)



degree (e.g.

MD)


years

7-9 years No


degree (e.g.

MD)


years

4-6 years No

5 P25 Female 50-59 Professional

degree (e.g.

MD)



degree (e.g.

MD)


years

1-3 years No


degree (e.g.

MD)

Physician 20-29 years 7-9 years Less than 1

year

No


degree (e.g.

MD)


years

4-6 years No


degree (e.g.

MD)


of 56


degree (e.g.

MD)


RNs

The following table summarizes the characteristics of the RNs that participated in the usability test, including

demographics, professional experience, computing experience and use of assistive technology. Participant

names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual

identities.

Part

ID

Part.

ID Gender Age Education Occupation

Professional

Experience

Computer

Experience

Product

Experience

Assistive

Technologies


degree

Registered

Nurse 10-19 years 10-19 years 7-9 years No

2 P3 Female 40-49 Doctoral

degree

Nurse

Practitioner 10-19 years 10-19 years 1-3 years No


degree

Registered



degree

Registered



degree

Registered



degree

Registered

Nurse 4-6 years 1-3 years

Less than 1

year Yes, Dragon


degree

Registered


8 P19 Male 60-69 Associate’s

degree

Registered



degree

Registered



degree

Registered

Nurse 4-6 years 10-19 years 4-6 years Yes, Dragon


degree

Registered



degree

Registered



degree

Registered


of 56

Appendix 2: Non-Disclosure Agreement and Informed Consent Form Understanding Your Participation

I agree to participate in this study conducted by Optum.

I understand Optum will record my participation for later review. By signing/acknowledging this form, I give

Optum permission to use the data gathered and my comments for the purposes of evaluating the product in

the study and showing the results of these evaluations, combined with other participants, to the product

team. Optum will not include my name or any identifying characteristics in the results.

Any information I acquire about the product in the study is confidential and proprietary and is being

disclosed only so that I can participate in the evaluation. By signing this form, I agree not to talk about this

product to anyone. I may only tell others that I helped to evaluate the software. I understand that if I disclose

my health information in my comments, I have disclosed it voluntarily, and Optum may use it anonymously

for research and evaluation purposes only.

I understand that I can stop participating in this study at any time.

I agree to raise any concerns or areas of discomfort with the study moderator.

Name (print): __________________________________________________

Signature: __________________________________________________

Date: __________________________________________________

of 56

Appendix 3: Example Moderator’s Guide for Physicians and RNs

Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at Optum,

makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and she’ll be helping take

notes.

Before we begin, I’d like to take a moment to describe the study we’re conducting.

We’re collecting feedback on a software product called ED PulseCheck.

I know someone already asked you this but I just wanted to verify that you currently use ED PulseCheck to

work with patient records.

Has not used ED PulseCheck Has used ED PulseCheck

We’re going to ask you to perform a series of tasks. You should know that the tasks were created to be

independent of each other and some may sound redundant. You will be asked to complete these tasks on

your own, trying to do them as quickly as possible with the fewest possible errors or deviations. Please do

not do anything more than the tasks asks.

You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the product is

to use, so just do your best to complete them as you typically would. We didn’t have anything to do with

designing the product, and your comments are entirely anonymous, so please feel free speak candidly.

However, please save your comments until the end of a task, or the end of the session as a whole, when

we can then discuss them freely.

We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please read this

task information out loud, so we can follow along.

I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my computer, but

I’ll give you control of my computers mouse and keyboard for the duration of the study. Because we can’t

always control the bandwidth between our two locations, you may notice a tiny lag or latency when you’re in

control of the mouse and keyboard. However, we don’t anticipate this will affect your ability to use ED

PulseCheck.

Before I do that, please quit or close any other applications that are currently running on your computer

(email and Instant Messaging), with the exception of your browser and WebEx running. Also, please set any

mobile devices to silent during the study.

of 56

Appendix 4: System Usability Scale Questionnaire In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems

usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have

elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper,

at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).

16 Brooke, J.: SUS: A “quick and dirty” usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (eds.) Usability Evaluation in Industry pp. 189--194. Taylor & Francis, London, UK (1996). SUS is copyrighted to Digital Equipment Corporation, 1986.

Lewis, J R & Sauro, J. (2009) "The Factor Structure Of The System Usability Scale." in Proceedings of the Human Computer Interaction International Conference (HCII 2009), San Diego CA, USA

http://www.usabilitynet.org/trump/documents/Suschapt.doc

of 56

Appendix 5: Tasks

Physicians

There are multiple correct paths to successfully complete the tasks; however, the “optimal path” is indicated

in bold.

Task 0 – Login Role: Physician

Starting URL: http://localhost/ibex23.mpex

For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency

department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming

patients. Your ED PulseCheck information is below.

Username: chuffman

Password: 123

Log in to ED PulseCheck and view the tracking board.

When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.

No time tracking required.

No completion required.

of 56

Task 1 - Enter a laboratory, radiology, and imaging orders

Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient

record has already been created for Robert in ED PulseCheck and has been assigned to you.

Enter the following lab, radiology, and imaging orders:

Complete blood count (CBC)

Hemoglobin A1c/hemoglobin total

Gas & carbon monoxide panel

CT abdomen & pelvis w/ contrast material

Myocardial image PET perfus single study rest/stress


1. Navigate to the Orders page.

a. Directly from the Tracking Board page

i. Select the patient > hover over All on the menu bar > click Orders

ii. Select the patient > click Order on the menu bar

iii. Select the patient > click Go > Orders on the patient row

b. Via the My Patients page (selecting the patient first is not necessary)

i. Click All on the menu bar > My Patients > select patient if necessary > Orders

ii. Click My Patients on the menu bar > select patient if necessary > Orders

2. Enter orders.

i. Select orders from Abdominal and chest pain clinical pathway

ii. Search for individual order by name

Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering the orders.

Completed

Yes No/Assisted

Overall this task was

1 2 3 4 5

of 56

Task 2 - Enter medication orders


record has already been created for Robert in ED PulseCheck and assigned to you.

Task: Enter the following medication orders:

Digoxin 0.25 mg tablet by mouth once daily

Morphine 2 mg IV push x1 for pain , every 4 hours as needed

Nitro-bid oral 2.5 mg oral, as needed


1. Navigate to the Medication Services page.

a. Directly from the Tracking Board page:

i. Select the patient > hover over All on the menu bar > click Med SVC

ii. Select the patient > click Med SVC on the menu bar

iii. Select the patient > click Go > Med SVC on the patient row

b. Via the My Patients page (selecting the patient first is not necessary):

i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services

ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services

c. Via the Orders page:

i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab

ii. Select the patient > click Orders on the menu bar > click the Medication Services tab

iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab

iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab

v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab

2. Enter orders.

a. Click Group>Abdominal & chest pain; select medications > Enter

b. Search for individual medications by name > Enter

Successful Completion Criteria: User clicks Enter after the orders are entered.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 3 – Cancel or hold a medication order


record was created for Robert in ED PulseCheck and assigned to you. You ordered Aspir-81 for Robert

earlier but he is refusing the medication.

Task: Update Robert’s medication orders to reflect the fact that he is refusing the medication. Enter a note

stating, “Patient refused.”














iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab

v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab

2. Cancel medication

a. Locate medication > click Cancel link > check Medication refused > clicks checkbox “Patent refused”

>click Cancel button

b. Locate medication > click Hold link > check Medication refused > enter note >click Hold button

Successful Completion Criteria: User clicks Cancel or Hold button after entering a note.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 4 – Delete or cancel a medication order Role: Physician w/ experience entering medication orders

Monica Webster was brought to the emergency department for a laceration to her left hand. It has come to

your attention that you accidently ordered Norcuron 10 mg IVP for her instead of your other patient.

Task: Update Monica’s medication orders to reflect the fact that Norcuron 10 mg IVP is not needed.














iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the

Medication Services tab

v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the


2. Delete medication

i. Locate medication > click Delete link

ii. Locate medication > click Cancel link > click Cancel button

Successful Completion: User clicks Delete link or Cancel button.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 5 – Update a medication order Role: Physician w/ experience entering medication orders

Javier Knight was brought to the emergency department for a sore throat. It has come to your attention that

Penicillin V Potassium 125mg/5ml oral was inadvertently ordered for Javier instead of Penicillin G

Potassium 1 million units, IM.

Task: Update Javier’s medication orders to reflect the fact that Penicillin G Potassium 1 million units, IM

should be given.









Click My Patients on the menu bar > select patient if necessary > click Medication Services









2. Cancel, delete, or hold medication.

a. Cancel: Locate medication > click Cancel > check Medication refused > enter note

b. Delete: Locate medication > click Delete

c. Hold: Locate medication > click Hold > check Medication refused > enter note

3. Enter order.

a. Search for medication (Penicillin) by name.

Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering medication order with

correct dosage.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 6 –Update a radiology order Role: Physician w/ experience entering lab and imaging orders

John Smith was brought to the emergency department for an injury to his right wrist. A patient record was

created for John in ED PulseCheck and assigned to you. John needs a 2 view wrist x-ray but a 2 view

chest x-ray was accidently ordered for him.

Task: Update John’s record to reflect the fact that he needs a 2 view wrist x-ray rather than chest x-rays.


Navigate to the My Tasks page.


i. Select the patient > hover over All on the menu bar > click Orders

ii. Select the patient > click Order on the menu bar

iii. Select the patient > click Go > Orders on the patient row

iv. Click the Orders icon


i. Click All on the menu bar > My Patients > select patient if necessary > Orders

ii. Click My Patients on the menu bar > select patient if necessary > Orders

2. Cancel the chest x-ray.

i. Click Cancel

3. Click Orders

4. Order the wrist x-ray.

i. Search by SVC Name > type wrist > select wrist or wrist (2 views) from list

Successful Completion: Depends on order in which user completes tasks. User can click Enter after entering the wrist

x-ray or click Cancel after selecting the chest x-ray.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 7 – Approve a medication order with a drug-allergy reaction Role: Physician w/ experience entering medication orders

John Smith was brought to the emergency department for an injury to his right wrist. A patient record has

already been created for John in ED PulseCheck and assigned to you. John is allergic to Tylenol but has

been treated with Vicodin in the past and has responded well with no side effects.

Task: Enter an order for one tablet of Vicodin ES for John.


















2. From the Medication Services page, search for medication by name.

3. In Allergy Reaction alert, check Patient tolerated same medication(s) in the past.

4. Click Enter.

Successful Completion: Interaction alert has been acknowledged and user clicks Enter.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 8 – Approve a medication order with a drug-drug interaction Role: Physician w/ experience entering medication orders

Zackary A Pearson was brought to the emergency department for a fever. A patient record has already

been created for Zackary in ED PulseCheck and assigned to you. Zackary is currently taking HCTZ. You

believe the benefits of taking lithium carbonate outweigh the risks associated with drug interactions.

Task: Order two lithium carbonate 150mg capsules for Zackary.


















2. Enter orders.

i. Use Groups to enter medications

ii. Search for individual medications by name

iii. In Severe Interaction alert for HCTZ and lithium carbonate, check Benefits outweigh risks.

iv. Click Enter

Successful Completion: User clicks Enter after the orders are entered. Completed

Yes No/Assisted


1 2 3 4 5

http://10.164.161.18/ibex4w.ibx?m=Q&p=20131009082216



of 56

Task 17 - Configure pharmacy for e-prescription Roles: Physician

Janet Kikura was brought to the emergency department for a severe migraine headache. A patient record

has already been created for Janet in ED PulseCheck and assigned to you. You have learned that you can

use the newest version of ED PulseCheck to send an electronic prescription to Janet’s pharmacy so it will

be available for her to pick up on the way home.

Task: Use ED PulseCheck to write a prescription for Ibuprofen 800mg, 1 tablet taken orally every 8 hours

as needed, no refills, 20 tablets to be dispensed. The closest pharmacy to Janet’s home is Walgreen’s,

located at 930 Elk Grove Town Ctr. Elk Grove Village, IL.

**Formulary information is not currently available. You can ignore any message regarding formulary

information.**


1. Navigate to the Prescriptions page.

a. To access the Prescriptions page from the Tracking Board page:

i. Select the patient > hover over All on the menu bar > click Rx

ii. Select the patient > click Rx on the menu bar

iii. Select the patient > click Go > Rx on the patient row

b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):

i. Select patient (if necessary) > hover over All on the menu bar > click Rx

ii. Select patient (if necessary) > click Rx on the menu bar

iii. Select patient (if necessary) > click Prescriptions in the right panel

c. To access the Rx page from the Disposition page:

i. Scroll to bottom > click Rx button

2. Add medication.

a. Find medication

i. Click Search All ii. Type medication name

iii. Select medication from list

b. Enter dose, schedule, and dispense instructions.

3. Send to pharmacy.

i. Click Pharmacy

ii. From the Pharmacy popup, enter pharmacy name

iii. Select pharmacy from list

iv. Click OK

v. Click Enter Successful Completion: User clicks Enter after selecting medication and pharmacy.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 18 - Change pharmacy for e-prescription Role: Physician


been created for Zackary in ED PulseCheck and assigned to you. You have learned that you can use the

newest version of ED PulseCheck to send an electronic prescription to Zackary’s pharmacy so it will be

available for him to pick up on the way home.

Task: Zackary usually fills prescriptions at Osco Drug, located on 6200 W. Higgins Avenue Chicago IL,

which is close to his home. He is planning on staying with a family member when he is discharged and

would like to fill his prescription at another pharmacy instead. Use ED PulseCheck to write a prescription

for Amoxicillin (SUSPENSION, RECONSTITUTED, ORAL (ML) : 125 mg/5 mL ) ; 5mls every 6 hours ;

dispense 1 bottle and send the prescription to Costco Pharmacy located at 505 W. Army Trail Road

Bloomingdale IL.

**Formulary information is not currently available. You can ignore any message regarding formulary

information.**


1. Navigate to the Prescriptions page.

a. To access the Prescriptions page from the Tracking Board page:

i. Select the patient > hover over All on the menu bar > click Rx

ii. Select the patient > click Rx on the menu bar

iii. Select the patient > click Go > Rx on the patient row

b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):

i. Select patient (if necessary) > hover over All on the menu bar > click Rx

ii. Select patient (if necessary) > click Rx on the menu bar

iii. Select patient (if necessary) > click Prescriptions in the right panel

c. To access the Rx page from the Disposition page:

i. Scroll to bottom > click Rx button

2. Send to pharmacy.

i. Click Pharmacy

ii. From the Pharmacy popup, enter pharmacy name

iii. Select pharmacy from list

iv. Click OK

v. Click Enter

Successful Completion: User clicks Enter after selecting medication and pharmacy.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

RNs

There are multiple correct paths to successfully complete the tasks; however the “optimal path” is indicated in bold.

Task 0 – Login Starting URL: http://localhost/ibex23.mpex

For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency

department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming

patients. Your ED PulseCheck information is below.

Username: chuffman

Password: 123

Log in to ED PulseCheck and view the tracking board.


No time tracking required.

No completion required.

of 56

Task 10 - Enter active medications

Susan McCoy was brought to the emergency department for stomach pain that was previously diagnosed

as gastro esophageal reflux disease (GERD). A patient record has already been created for Susan in ED

PulseCheck and assigned to you. Susan neglected to mention that she takes several over-the-counter

medications regularly for GERD.

Task: Update Susan’s record to reflect the medications she takes.

Pepcid AC 10mg chewables, 2 daily

TUMS E-X 300mg chewables, 4 daily

Zegerid OTC 20mg capsule, 1 daily


1. Navigate to the Current Medications page.


i. Select the patient > hover over All on the menu bar > click Current Meds

ii. Select the patient > click Current Meds on the menu bar

iii. Select the patient > click Go > Current Meds on the patient row


i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds

ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds

c. Via the patient’s Charting page:

i. Click the patient’s name > click Current Medications (only valid if current medications have not been

previously entered)

d. Via the Allergies page:

i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab

ii. Select the patient > click Allergies on the menu bar > click Medications tab

iii. Select the patient > click Go > Allergies on the patient row > click Medications tab

iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click

Medications tab

v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click Medications

tab

vi. Click the patient’s name > click Allergies (only valid if current medications have not been previously

entered) > click Medications tab

2. Enter medications.

a. Use the QuickList to add the medications > click Enter

Successful Completion Criteria: User clicks Enter or Enter/Allergies after entering three medications.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 11 - Update a medication currently being taken Role: Nurse

Zackary A Pearson was brought to the emergency department for fever. A patient record has already been

created for Zackary in ED PulseCheck and assigned to you. Zachary said he took digoxin but neglected to

say how often he took it.

Task: Update Zachary’s record to reflect the fact that he takes digoxin, orally, 0.25 mg once a day (in the

morning).











i. Click the patient’s name > click Current Medications (only valid if current medications have not

been previously entered)






Medications tab

v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click

Medications tab

vi. Click the patient’s name > click Allergies (only valid if current medications have not been

previously entered) > click Medications tab

2. Add dosage information.

i. Click on the hyperlinked line to the right of the medication name

ii. In the Schedule field, select once a day

iii. Click Update

Successful Completion: User clicks Update after entering dosage.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 12 - Inactivate an active medication Role: Nurse


been created for Zackary in ED PulseCheck and assigned to you. Earlier, Zackary said he took Zegerid

OTC once a day. A family member has arrived at the hospital with his medications and you learn that

Zackary is taking Tums EX, not Zegerid OTC.

Task: Update Zackary’s record to reflect the fact he is taking Tums EX 300 mg calcium (750 mg), 1 tab

oral, once a day, not Zegerid OTC 20 mg-1.1 gram 1 tab oral once a day.











i. Click the patient’s name > click Current Medications (only valid if current medications have not

been previously entered)






Medications tab

v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click

Medications tab

vi. Click the patient’s name > click Allergies (only valid if current medications have not been

previously entered) > click Medications tab

2. Inactivate medication.

i. Click on Make inactive to the right of Zegerid

3. Add medication.

i. Type medication name in the Quick List tab > enter dosage and schedule information >

click Enter ii. Type medication name in the Search tab > enter dosage and schedule information > click

Enter Successful Completion: User clicks Enter after adding medication.

Completed

Yes No/Assisted


1 2 3 4 5

http://10.164.161.18/ibex2w.ibx?n=495776&t=CA

of 56

Task 13 – Find active and inactive allergies Role: Nurse


been created for Zackary in ED PulseCheck and assigned to you. Zackary and his family have provided

conflicting lists of allergies for Zackary. You want to read through a list of allergies that have been

mentioned so you can verify which ones affect him.

Task: Display a complete list of Zackary’s active and inactive allergies that you would use to verify each

allergy with him and his family.


1. Navigate to the Known Allergies page.


i. Select the patient > hover over All on the menu bar > click Allergies

ii. Select the patient > click Allergies on the menu bar

iii. Select the patient > click Go > Allergies on the patient row


i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies

ii. Click My Patients on the menu bar > select patient if necessary > click Allergies


i. Click the patient’s name > click Known Allergies

2. Display active and inactive allergies.

i. Click Show Inactive to include inactive allergies in the list of allergies

Successful Completion: User clicks Show Inactive.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 14 - Enter active allergies Role: Nurse

Susan McCoy was brought to the emergency department for gastro esophageal reflux disease (GERD). A

patient record has already been created for Susan in ED PulseCheck and assigned to you. Susan says

she is allergic to several medications.

Allergies: Aspirin

Bactrim

Cipro

Nausea

Hives

Difficulty breathing

Mild

Moderate

Severe

Task: Update Susan’s record to reflect her allergies.












2. Add allergies.

i. From the Known Allergies page, type medication names in the Quick List tab > Click

checkbox, enter reaction, severity and source > click Enter

ii. From the Known Allergies page, type medication names in the Search tab > enter reaction,

severity and source > click Enter

Successful Completion: User clicks Enter.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 15 - Update the effects of an allergy Role: Nurse

Richard Bird was brought to the emergency department with a broken arm. A patient record has already

been created for Richard in ED PulseCheck and assigned to you. Richard said he was allergic to penicillin

but the severity and nature of his allergy was not noted. A family member mentioned that Richard had

difficulty breathing due to a swollen throat when he’d been given penicillin in the past.

Task: Update Richard’s record to reflect the fact that he is severely allergic to penicillin.












2. Update the allergy.

i. Click on the hyperlinked line to the right of the allergy name

ii. Enter reaction, severity and source

iii. Click Update

Successful Completion: User clicks Update after entering reaction details have been added to allergy.

Completed

Yes No/Assisted


1 2 3 4 5

of 56

Task 16 - Inactivate an allergy Role: Nurse

John Smith was brought to the emergency department for right wrist pain. A patient record has already

been created for John in ED PulseCheck and assigned to you. Earlier, a family member said John was

allergic to Tylenol. After talking with John, you learn that he is allergic to Lortab ASA , not Tylenol.

Task: Update John’s record to reflect the fact that he is allergic (mild hives) to Lortab ASA, not Tylenol.












2. Inactivate allergy.

i. Click on Make inactive to the right of Tylenol

3. Add allergy.

i. From the Known Allergies page, type medication names in the Quick List tab > enter

reaction, severity and source > click Enter

ii. From the Known Allergies page, type medication names in the Search tab > enter reaction,

severity and source > click Enter

Successful Completion: User clicks Enter after entering reaction details. Completed

Yes No/Assisted


1 2 3 4 5

of 56

Appendix 6: Demographics Questionnaire 1. Gender

O Female

O Male

2. Age

O 18-29 years

O 30-39 years

O 40-49 years

O 50-59 years

O 60-69 years

O 70 or more years

3. What is the highest degree or level of school you have completed?

O High School, GED, or equivalent

O Some college, no degree

O Technical certification

O Associate degree

O Bachelor degree

O Master degree

O Professional degree

O Doctorate degree

O Other: (please specify) _____________________________

4. What title best describes your current occupation?

O Emergency Medical Technician

O Health Information System Administrator

O Nurse Practitioner

O Office Administrator

O Pharmacist

O Physician

O Physician’s Assistant

O Psychologist, Counselor, or Social Worker

O Registered Nurse

O Other: (please specify) _____________________________

of 56

5. How long have you held your current title?

O Less than 1 year

O 1-3 years

O 4-6 years

O 7-9 years

O 10-19 years

O 20-29 years

O 30-39 years

O 40-49 years

O 50 or more years

6. How long have you used a computer in a professional workplace?

O Less than 1 year

O 1-3 years

O 4-6 years

O 7-9 years

O 10-19 years

O 20-29 years

O 30-39 years

O 40-49 years

O 50 or more years

7. How long have you used ED PulseCheck?

O Less than 1 year

O 1-3 years

O 4-6 years

O 7-9 years

O 10 or more years

8. Do you use assistive technology; such as an enhanced visual display, physically adapted pointing device, or text-

to-speech engine; in the workplace?

O Yes

O No

If yes, please describe __________________________________

of 56

Appendix 7: Data Scoring Details

As was described in the Data Scoring – Effectiveness, Task Success section, optimal task performance time was

benchmarked when constructing tasks by using Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and

task time to obtain an expected time based on the optimal path. (KLM was used rather than expert time because the

product includes many paths to accomplish tasks, and experts frequently didn’t follow optimal paths.)

To quantify the additional time added to the KLM calculations, three expert performances of three tasks with varying

complexity were reviewed (Physician tasks: 2, 4, 18; RN tasks: 10, 13, 16) based on the rules described below. An

average time for each rule was calculated and then consistently applied throughout the KLM calculations.

The rules below detail how this additional time, based on test environment measurements, was added to the KLM

results to account for the following:

1. User’s need to review information while interacting with the user interface (medical professionals double-check their

work routinely to avoid error):

• 5 seconds: Review data entry before next step

• 3 seconds: Look at information displayed (e.g., table data)

• 3 seconds: Find patient in list of Tracking Board

2. System response times:

• 7 seconds: (Tasks 17-18) After Search button is clicked until the search window appears

• 4 seconds: For next page to appear after clicking Enter button

Or

2 seconds: (If there was no Enter button) System responds and displays the next page


16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

Appendix B: Quality Management System



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

Appendix C: Privacy and Security



16‐2423‐R‐0022‐PRI V1.0, June 20, 2016

Test Results Summary Document History Version

V1.0

END OF DOCUMENT

Description of Change

Initial release

Date

June 20, 2016


onc hit program test results summary for 2014 edition … · test results summary for 2014 edition...

Documents