onc hit certification program test results summary for ...€¦ · email:...

Test Results Summary for 2014 Edition EHR Certification Version EHR-Test-144 Rev 07-Aug-2015

of 12

ONC HIT Certification Program

Test Results Summary for 2014 Edition EHR Certification

Part 1: Product and Developer Information

1.1 Certified Product Information

Product Name: Practice Partner® Product Version: 11.0ss Domain: Ambulatory Test Type: Complete EHR

1.2 Developer/Vendor Information

Developer/Vendor Name: McKesson Address: 5995 Windward Parkway Alpharetta GA 30005 Website: http://www.mckesson.com/bps Email: [email protected] Phone: 563-585-4773 Developer/Vendor Contact: Tom Reinecke


of 12

Part 2: ONC-Authorized Certification Body Information

2.1 ONC-Authorized Certification Body Information

ONC-ACB Name: Drummond Group

Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750

Website: www.drummondgroup.com

Email: [email protected]

Phone: 817-294-7339

ONC-ACB Contact: Bill Smith

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Bill Smith

Certification Body Manager ONC-ACB Authorized Representative Function/Title

9/14/2015

Signature and Date

2.2 Gap Certification The following identifies criterion or criteria certified via gap certification

§170.314

(a)(1) (a)(19) (d)(6) (h)(1)

(a)(6) (a)(20) (d)(8) (h)(2)

(a)(7) (b)(5)* (d)(9) (h)(3)

(a)(17) (d)(1) (f)(1)

(a)(18) (d)(5) (f)(7)** *Gap certification allowed for Inpatient setting only **Gap certification allowed for Ambulatory setting only

x No gap certification

http://www.drummondgroup.com/

mailto:[email protected]


of 12

2.3 Inherited Certification

The following identifies criterion or criteria certified via inherited certification

§170.314

x (a)(1) (a)(16) Inpt. only x (c)(2) x (f)(2)

x (a)(2) (a)(17) Inpt. only x (c)(3) x (f)(3)

x (a)(3) (a)(18) x (d)(1) (f)(4) Inpt. only

x (a)(4) (a)(19) x (d)(2) (f)(5) Amb. only

x (a)(5) (a)(20) x (d)(3)

x (a)(6) (b)(1) x (d)(4) (f)(6) Amb. only

x (a)(7) (b)(2) x (d)(5)

x (a)(8) x (b)(3) x (d)(6) (f)(7)

x (a)(9) x (b)(4) x (d)(7) (g)(1)

x (a)(10) x (b)(5) x (d)(8) x (g)(2)

x (a)(11) (b)(6) Inpt. only (d)(9) Optional x (g)(3)

x (a)(12) x (b)(7) (e)(1) x (g)(4)

x (a)(13) (b)(8) x (e)(2) Amb. only (h)(1)

x (a)(14) (b)(9) x (e)(3) Amb. only (h)(2)

x (a)(15) x (c)(1) x (f)(1) (h)(3)

No inherited certification


of 12

Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: TEB-09092015-1988-1

Test Date(s): 10/9/2013, 10/16/2013, 11/1/2013, 11/15/2013, 3/4/2014, 9/9/2015

3.1 NVLAP-Accredited Testing Laboratory Information

ATL Name: Drummond Group EHR Test Lab

Accreditation Number: NVLAP Lab Code 200979-0

Address: 13359 North Hwy 183, Ste B-406-238, Austin, TX 78750

Website: www.drummondgroup.com

Email: [email protected]

Phone: 817-709-1627

ATL Contact: Kyle Meadors

For more information on scope of accreditation, please reference NVLAP site.

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:

Timothy Bennett

Test Proctor ATL Authorized Representative Function/Title

9/14/2015

Nashville, TN Signature and Date Location Where Test Conducted

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

Additional Software Applicable Criteria Functionality provided by Additional Software

Surescripts Network for Clinical Interoperability

b.1, b.2, e.1

Direct messaging

No additional software required

http://www.drummondgroup.com/

mailto:[email protected]

http://www.nist.gov/nvlap/


of 12

3.2.2 Test Tools

Test Tool Version

x Cypress 2.4.1 x ePrescribing Validation Tool 1.03

HL7 CDA Cancer Registry Reporting Validation Tool 1.0.3 HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.8.2

x HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.7.1

x HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7 x HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7 x Transport Testing Tool 181 x Direct Certificate Discovery Tool 3.0.4

Edge Testing Tool 0.0.5

No test tools required

3.2.3 Test Data

Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter]

x No alteration (customization) to the test data was necessary

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

x §170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(13)

x §170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree


of 12


(a)(15)(i)

x §170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(16)(ii) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)

x §170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions

§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

(b)(7)(i)

x §170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(b)(8)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(e)(1)(i)

Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] AES-256 SHA-1

(e)(1)(ii)(A)(2) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

x §170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(e)(3)(ii)

Annex A of the FIPS Publication 140-2 [list encryption and hashing algorithms] AES-256 SHA-1

Common MU Data Set (15)


x §170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)


of 12


None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested

Newer Version Applicable Criteria

No newer version of a minimum standard was tested

3.2.5 Optional Functionality

Criterion # Optional Functionality Successfully Tested

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(e)(1) View, download and transmit data to a third party utilizing the Edge Protocol IG version 1.1

(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)

(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements


Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)


Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)

x No optional functionality tested


of 12

3.2.6 2014 Edition Certification Criteria* Successfully Tested

Criteria # Version

Criteria # Version

TP** TD*** TP TD

(a)(1) 1.3 1.5 (c)(3) 1.11 1.11

(a)(2) 1.2 (d)(1) 1.2

(a)(3) 1.2 1.4 (d)(2) 1.6 (a)(4) 1.4 1.3 (d)(3) 1.3 (a)(5) 1.4 1.3 (d)(4) 1.3 (a)(6) 1.3 1.4 (d)(5) 1.2 (a)(7) 1.3 1.3 (d)(6) 1.2 (a)(8) 1.3 (d)(7) 1.2 (a)(9) 1.3 1.3 (d)(8) 1.2 (a)(10) 1.2 1.4 (d)(9) Optional 1.2 (a)(11) 1.3

x (e)(1) 1.11 1.5 (a)(12) 1.3 (e)(2) Amb. only 1.2 1.6 (a)(13) 1.2 (e)(3) Amb. only 1.3

(a)(14) 1.2 (f)(1) 1.2 1.2 (a)(15) 1.5 (f)(2) 1.3 1.3

(a)(16) Inpt. only 1.3 1.2 (f)(3) 1.3 1.3

(a)(17) Inpt. only 1.2 (f)(4) Inpt. only 1.3 1.3

(a)(18) 1.1 1.5 (f)(5) Amb. only 1.2 1.2

(a)(19) 1.1 1.5

(a)(20) 1.1 1.5 (f)(6) Amb. only 1.4 1.4

x (b)(1) 1.7 1.4

x (b)(2) 1.4 1.6 (f)(7) Amb. only 1.1

(b)(3) 1.4 1.4 (g)(1) 2.0 2.0 (b)(4) 1.3 1.4 (g)(2) 2.0 2.0 (b)(5) 1.4 1.2 (g)(3) 1.4

(b)(6) Inpt. only 1.3 1.3 (g)(4) 1.2

(b)(7) 1.4 1.7 (h)(1) 1.1

(b)(8) 1.2 1.2 (h)(2) 1.1

(b)(9) 1.1 1.1 (h)(3) 1.1 (c)(1) 1.11 1.11

(c)(2) 1.11 1.11


of 12

Criteria # Version

Criteria # Version

TP** TD*** TP TD

No criteria tested *For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)

http://www.healthit.gov/certification


of 12

3.2.7 2014 Clinical Quality Measures*

Type of Clinical Quality Measures Successfully Tested:

x Ambulatory

Inpatient

No CQMs tested

*For a list of the 2014 Clinical Quality Measures, please the CMS eCQM Library (Navigation: June 2014 and April 2014 Updates)

Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

x 2 v4 x 90 v4 x 136 v4 x 155 v3

22 x 117 v3 x 137 v3 x 156 v3

x 50 v3 x 122 v3 x 138 v3 157

52 x 123 v3 x 139 v3 158

x 56 v3 x 124 v3 x 140 v3 159

61 x 125 v3 x 141 v4 160

62 x 126 v3 x 142 v3 161

64 x 127 v3 x 143 v3 x 163 v3

65 x 128 v3 x 144 v3 x 164 v3

x 66 v3 x 129 v4 x 145 v3 x 165 v3

x 68 v4 x 130 v3 x 146 v3 x 166 v4

x 69 v3 x 131 v3 x 147 v4 x 167 v3

74 132 148 169

x 75 v3 133 149 177

77 x 134 v3 x 153 v3 179

82 x 135 v3 x 154 v3 x 182 v4

Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113 55 104 114

60 105 171

http://cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/eCQM_Library.html


of 12

3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording Successfully Tested

(a)(1) (a)(11) (a)(18) (b)(6)

(a)(3) (a)(12) (a)(19) (b)(8)

(a)(4) (a)(13) (a)(20) (b)(9)

(a)(5) (a)(14) (b)(2) (e)(1)

(a)(6) (a)(15) (b)(3) (e)(2)

(a)(7) (a)(16) (b)(4) (e)(3)

(a)(9) (a)(17) (b)(5)

x Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation Successfully Tested

x (a)(1) x (a)(11) (a)(18) (b)(6)

x (a)(3) x (a)(12) (a)(19) (b)(8)

x (a)(4) x (a)(13) (a)(20) (b)(9)

x (a)(5) x (a)(14) x (b)(2) x (e)(1)

x (a)(6) x (a)(15) x (b)(3) x (e)(2)

x (a)(7) (a)(16) x (b)(4) x (e)(3)

x (a)(9) (a)(17) x (b)(5)

Automated Measure Calculation was not tested

3.2.9 Attestation

Attestation Forms (as applicable) Appendix

x Safety-Enhanced Design* A

x Quality Management System** B

x Privacy and Security C

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9). **Required for every EHR product

3.3 Appendices

Attached below.


of 12

Test Results Summary Change History

Test Report ID Description of Change Date

2014 Edition Test Report Summary

October 4, 2013

Drummond Group Inc

13359 North Hwy 183, Ste B-406-238

Austin, TX 78750

Submission of Documented User Centered Design Process and Usability Test Results

This letter and attachments serve as a formal submission of the Documented User Centered Design Process and Usability Test Results on behalf of McKesson Corporation (“McKesson”) for Practice Partner® Ver. 11 for Test Criteria: 170.314.g.3 – Safety Enhanced Design.

We thank you for your consideration.

Regards,

James Reynolds

Executive Director, Practice Partner

McKesson

Practice Partner® User Centered Design Process Documentation

User Centered Design Standard Alignment

The Practice Partner® Ver. 11 User Centered Design process aligns with the McKesson User Centered

Design (UCD) process.

The McKesson UCD process is based on information contained in NISTIR 7741 (Schumacher & Lowry,

2010) and the UCD visual map found on the Usability.gov website

(http://www.usability.gov/methods/process.html). Each of the following criteria have been developed

or iterated (in the case of legacy systems) according to this process:

§ 170.314(a)(1): Computerized Provider Order Entry (CPOE);

§ 170.314(a)(2): Drug-Drug and Drug-Allergy Interaction Checks;

§ 170.314(a)(6): Medication List;

§ 170.314(a)(7): Medication Allergy List;

§ 170.314(a)(8): Clinical Decision Support;

§ 170.314(b)(3): Electronic Prescribing; and

§ 170.314(b)(4): Clinical Information Reconciliation.

McKesson User Centered Design Process The McKesson UCD process consists of various activities carried out in 5 phases: Planning, Research,

Analysis, Design and Evaluation. The individual activities are listed and described in the table below.

Phase Activity ID Activity Description

Planning 1

Develop plan for user-centered design activities

This activity occurs at the beginning of the project in conjunction with the overall project planning activities. The plan identifies the purpose, design strategy, resources, scope of the project, timelines, planned UCD activities, deliverables, and constraints and risks.

Research 2 Site visits/field research

Site visits provide empirical evidence for the analytical processes of user research and task analysis (see below). This type of field research is usually conducted at the beginning of a larger development effort to understand the context of use; that is the environment the task is conducted in, the users, or user groups, and the existing workflows that the new product or feature will support. In the most common form of this research, a user or a few users are shadowed and the researcher takes notes to document relevant aspects of the task environment. Other forms of observation may focus on a physical location at which users interact for handoffs or for information exchange, or also on documenting the flow of information from a patient’s perspective. Interviews may also be conducted to understand workflow, tasks, and needs. These interviews may be structured or unstructured and conducted with direct users or indirect beneficiaries of the technology, such as patients. During any of these types of observational research, artifacts are collected (paper forms, any written form of information exchange) to document current practices and user thinking about the task or the workflow under development. Transcripts and artifacts from field visits can be re-used across multiple design projects, but teams should be aware of differences that might exist between the current design project and possible previous environments and observations. After some time, previous practices may change or have been altered through the introduction of technology, so that new field research might be necessary.

Analysis 3

Analyze information about users

This activity, once complete, can be reused across projects and tasks (but see same caution as above). Habits and knowledge of user groups may change. This activity requires the analysis and synthesis of information about users gathered through field visits or other forms of empirical research. The purpose is to understand the types of users interacting with the system, as well as characteristics of those users. The output from this activity is used during the design process. The activity results in user profiles and, optionally, personas. The data may contain information related to the physical and social contexts in which the user performs tasks.

Analysis 4

Analyze the task/activity that the product supports

This activity occurs in conjunction with user research and before design begins. Its purpose is to learn about the user's goals and identify tasks performed with the system. It can have several levels of detail. It often starts with an overall goal of the system (which includes the user) and decomposes it hierarchically into tasks. The analysis should include the context in which the tasks are performed, determine if the tasks differ based on different roles, or if multiple personas share the same task flow and end goals. Task analysis helps in the discovery of what tasks and workflows the system must support and the appropriate content and scope.

Analysis 5 Evaluate current application/feature

This activity is optional. It occurs in the context of existing system features. The purpose is to create a benchmark from which to compare a new system or enhancement. The evaluation includes, but is not limited to, gathering customer complaints, observing user interactions with the system, and interviewing existing customers. The objective is to identify unnecessary steps in the workflow, redundant or misleading information, or other items that negatively impact the user's ability to effectively interact with the system.


Analysis 6 Create Usability Objectives

Usability objectives are defined quality objectives for a system which includes the user interactions with the system.. Typically, usability objectives are based on the following measures, depending upon the workflow/tasks and the importance or prioritization of goals.

Error counts/types

Time to complete the task/workflow

Time required to learn the steps necessary to perform the task

Successful completion of the task

Satisfaction with the steps necessary to perform the task as well as the user interfaces and system performance.

The objectives are compared with the actual measures collected when a user is interacting with the system. When used in conjunction with usability tests, the measures help product management determine whether further design iterations are necessary.

Analysis 7 Write Scenarios

This activity results in scenarios created from the information gathered through user research and task analysis. The scenarios include fictional narratives of typical sets of steps that a user takes to accomplish a task or workflow. The scenarios should cover as many real world situations as possible, including the 'happy paths' and 'alternate paths'. The scenarios are fed into the design and can be used to identify alternate workflows and potential use errors. The scenarios can also be used to document the steps a participant needs to take to complete a test script related to formative or summative testing.

Analysis 8 Determine user requirements

User requirements are created from a user's point of view and describe the expected functionality necessary to solve the user's goal. User requirements are inputs to the UI designs and are particular to a feature. They should not include details about how the user performs the task but, rather, describe what the user is trying to accomplish. They are testable statements of expected functionality and should be traceable to original business requirements. They can be used to create test scripts for usability testing.

Design 9 Create UI design

The information gathered about users, requirements, and tasks is translated into a UI design. Information Architecture (IA) process should be used to classify and organize the system content and navigation to improve user access to and comprehension of the information. The UI design is usually documented in a UI specification which details out all the interactions, characteristics and visual aspects of the design. It is important that the UI design requirements be traced back to user requirements and the original design strategy defined in the UCD plan.

Design 10 Develop a prototype

This activity can occur in parallel with task analysis, user requirements definition, and IA activities. Prototypes are models of the new feature or enhancement. Prototypes can be created at different levels of fidelity. Which level is employed will depend on the goal of the prototype. A prototype can be used to evaluate the design or demonstrate to the project team how users will interact with the system. A prototype may not be reusable by the project.


Evaluation 11 Summative testing

This activity occurs when the product is ready for QA and before GA. A group of 10 - 20 users is recruited to perform a set of tasks on one or more interfaces. Objective measurement data is collected, statistically analyzed and compared against the usability quality objectives. The findings may be used to adjust the finished product prior to a GA release.

Evaluation 12 Heuristics evaluations

This activity is optional. It occurs before the development phase of the project. A group of 3 - 5 experts perform separate analyses based upon general design principles. The goal is to identify usability issues early in the design phase. The evaluation can be performed on paper mockups or screen shots. The analysis can focus on performing high priority tasks, or meeting user goals. The output of the evaluators is collated and a list of usability issues is created that can be fed back into the next design iteration.

Evaluation 13 Formative testing

This activity occurs prior to the start of developing the system. It can continue iteratively throughout the development process. A set of 3 - 7 users are given a set of tasks and a prototype. They are asked to perform the tasks to ensure the overall design and navigation meets their needs. This type of testing results in qualitative data that is turned into a list of recommendations to improve the design.

References Schumacher, R.M. & Lowry, S.Z. (2010). (NISTIR 7741) NIST guide to the processes approach for

improving the usability of electronic health records [NIST Interagency/Internal Report (NISTIR) - 7741].

Retrieved from http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907313.

U.S. Department of Health & Human Services. Step-by-Step Usability Guide. Retrieved from

http://www.usability.gov/methods/process.html.

EHR Usability Test Report Part I

EHR: Practice Partner® Version 11

User Role: Physician

Setting: Ambulatory

Applications Tested

Practice Partner® VER. 11

Dates

Usability Test: September 9-12, 2013

Report: September 25, 2013

Prepared by

Marita Franzke, Ph.D., Senior HFE, McKesson Provider Technologies

Gergana Pavlova-Grozeva, Business Analyst, McKesson Provider Technologies

Contact Marita Franzke

McKesson Provider Technologies 11000 Westmoor Cr., Suite 125

Westminster, CO 80021 (720) 356-7711

[email protected]

Report based on ISO/IEC 25062:2006

Common Industry Format for Usability Test Reports

of 77

EHR: Practice Partner VER. 11, Part I – Physician – Ambulatory September 25, 2013

Contents

Tables ............................................................................................................................................................... 4

Executive Summary ........................................................................................................................................... 6

Major Findings and Areas for Improvement .................................................................................................. 12

Introduction ...................................................................................................................................................... 16

Intended Users............................................................................................................................................. 16

Method ............................................................................................................................................................ 16

Participants .................................................................................................................................................. 16

Study Design ............................................................................................................................................... 17

Procedures .................................................................................................................................................. 19

Test Locations.............................................................................................................................................. 20

Test Environment ......................................................................................................................................... 20

Test Forms and Tools .................................................................................................................................. 21

Usability Metrics ........................................................................................................................................... 21

Data Scoring ................................................................................................................................................ 22

Results ............................................................................................................................................................ 24

Data Analysis and Reporting ........................................................................................................................ 24

Discussion of the Findings ............................................................................................................................... 29

170.314(a)(1) – CPOE ................................................................................................................................. 29

Test contexts ............................................................................................................................................ 29

Task 3 – Place a new lab order ................................................................................................................ 29

Task 4 – Look up and change existing lab order ....................................................................................... 32

Task 5 – Look up and change existing image order .................................................................................. 34

Task 6 – Place new image order .............................................................................................................. 37

170.314(a)(2) – Drug-drug and Drug-Allergy Interaction Checks .................................................................. 39

Test contexts ............................................................................................................................................ 39

Task 8 and 9 – Order new prescription, receive interaction warning, and cancel ...................................... 39

170.314(a)(6) – Medication List, also 170.314(a)(1) – CPOE - Medication Orders ................................... 42

Test contexts ............................................................................................................................................ 42

Task 7 – Access current prescription/order/medication list item and change entry.................................... 42

of 77


Task 10 and 11 – Access historical prescription/order/medication list item and change entry ................... 44

Task 8 and 9 – Enter new medications (receive interaction alert and cancel) ........................................... 46

170.314(b)(3) – Electronic Prescribing ......................................................................................................... 49

Test context .............................................................................................................................................. 49

Task 11 – Change med list and e-prescribe.............................................................................................. 49

170.314(a)(8) – Clinical Decision Support .................................................................................................... 51

Test context .............................................................................................................................................. 51

Task 1 – Select patient and review health maintenance alerts .................................................................. 51

Task 2 – Use the InfoButton to review treatment options for patient’s problem. ........................................ 53

List of Appendices ........................................................................................................................................... 57

Appendix A: Demographic Questionnaire ..................................................................................................... 58

Appendix B: Administrator Script and General Instructions .......................................................................... 60

Appendix C: Informed Consent Form ........................................................................................................... 62

Appendix D: Usability Test Instructions ........................................................................................................ 63

Appendix E: Experimenters’ Short Biographies ............................................................................................ 66

Appendix F: Participant Instructions for Each Task ...................................................................................... 67

Appendix G: End-of-Task Questionnaires .................................................................................................... 70

Appendix H: System Usability Scale (SUS) Questionnaire ........................................................................... 75

Appendix I: System and Patient Scenario for Physicians’ Tasks .................................................................. 76

of 77


Tables

Table 1. Physicians’ Tasks: Meaningful Use Criteria, Risk, and Application Tested ........................................... 6

Table 2. Result Summary by Task ..................................................................................................................... 8

Table 3. Result Summary by Meaningful Use Criterion .................................................................................... 11

Table 4. Participant Demographics .................................................................................................................. 17

Table 5. Physicians’ Tasks – same as Table 1, repeated here for easier reference ......................................... 18

Table 6. Test Environment, Technical Data ..................................................................................................... 20

Table 7. Data Scoring: Measures & Rationale ................................................................................................. 22

Table 8. Usability Data by Task and Subtask - same as Table 2, repeated here for easier reference .............. 24

Table 9. Usability Data by Meaningful Use Criteria – same as Table 3, repeated here for easier reference ..... 27

Table 10. Usability Data for Task 3 – Order new lab ....................................................................................... 30

Table 11. Participant Errors for Task 3 – Order new lab ................................................................................. 30

Table 12. Participant Comments for Task 3 – Order new lab .......................................................................... 31

Table 13. Usability Data for Task 4 – Look up and change existing lab order ................................................. 32

Table 14. Participant Errors for Task 4 – Look up and change existing lab order ............................................ 33

Table 15. Participant Comments for Task 4 – Look up and change existing lab order .................................... 33

Table 16. Usability Data for Task 5 – Look up and change existing image order ............................................ 34

Table 17. Participant Errors for Task 5 – Look up and change existing image order ....................................... 35

Table 18. Participant Comments for Task 5 – Look up and change existing image order ............................... 36

Table 19. Usability Data for Task 6 – Place new image order ......................................................................... 37

Table 20. Participant Errors for Task 6 – Place new image order ................................................................... 38

Table 21. Participant Comments for Task 6 – Place new image order ............................................................ 38

Table 22. Usability Data for Tasks 8 and 9 – Interaction Warnings ................................................................. 40

Table 23. Participant Errors for Tasks 8 and 9 – Interaction Warnings ............................................................ 40

Table 24. Participant Comments for Tasks 8 and 9 – Interaction Warnings .................................................... 41

Table 25. Usability Data for Task 7 – Lookup medication and change ............................................................ 42

Table 26. Participant Errors for Task 7 - Lookup medication and change ....................................................... 43

Table 27. Participant Comments for Task 7 - Lookup medication and change ................................................ 44

Table 28. Usability Data for Task 10 and 11 – Lookup historical medication and change ............................... 45

Table 29. Participant Errors for Task 10 and 11 – Lookup historical medication and change .......................... 45

of 77


Table 30. Participant Comments for Tasks 10 and 11 – Lookup historical medication and change ................. 46

Table 31. Usability Data for Tasks 8 and 9 – Add new medication order ........................................................ 47

Table 32. Participant Errors for Task 8 and 9 – Add new medication order ..................................................... 47

Table 33. Participant Comments for Task 8 and 9 – Add new medication order ............................................. 48

Table 34. Usability Data for Task 11 – Fill prescription and e-prescribe .......................................................... 50

Table 35. Participant Errors for Task 11 – Fill prescription and e-prescribe .................................................... 50

Table 36. Participant Comments for Task 11 – Fill prescription and e-prescribe ............................................. 51

Table 37. Usability Data for Task 1 – Select patient and review health maintenance alerts ............................ 52

Table 38. Participant Errors for Task 1 – Select patient and review health maintenance alerts ....................... 52

Table 39. Participant Comments for Task 1 – Select patient and review health maintenance alerts ............... 53

Table 40. Usability Data for Task 2 – Use the InfoButton ................................................................................ 54

Table 41. Participant Errors for Task 2 – Use the InfoButton .......................................................................... 54

Table 42. Participant Comments for Task 2 – Use the InfoButton ................................................................... 55

of 77


Executive Summary A usability test of Practice Partner® VER. 11 was conducted with 22 physician end users of Practice

Partner during the week of September 9-12, 2013 by Marita Franzke and Gergana Pavlova of the

McKesson Provider Technologies. The testing was conducted on two different customer sites in York,

ME and Pensacola FL. The purpose of this test was to test and validate the usability of five Meaningful

Use Criteria as implemented in the functions of the Practice Partner® VER. 11. The Office of the

National Coordinator for Health Information Technology (ONC) requires vendors to test 7 Meaningful

Criteria outlined in the test procedures for §170.314(g)(3) Safety-enhanced design1 for ambulatory

products. The criteria focused on the performance of nursing and administrator tasks will be discussed

in two other reports. Those reports detail the results of usability studies involving nurses/medical

assistants, and administrators, as those clinical tasks are typically conducted by those user roles.

During the usability test, 22 physicians served as participants; they were all familiar with previous

releases of the Practice Partner EHR. In this study, they used this software in 11 simulated, but

representative tasks. More specifics on the sample demographics and participant experience with the

EHR will be detailed below in the Method section.

Below is a list of all tasks employed, which were chosen to evaluate the usability of 5 of the 7

Meaningful Criteria outlined in the test procedures for §170.314(g)(3) Safety-enhanced design for

ambulatory use. Table 1 below lists the physicians' tasks in order of performance, the linked

Meaningful Use criteria, and a risk rating for performing each task.

Table 1. Physicians’ Tasks: Meaningful Use Criteria, Risk, and Application Tested

Task Meaningful Use Criterion Tested

Application Risk

1 Select Patient and Review Health Maintenance Alerts

170.314(a)(8) – Clinical decision support

Practice Partner

Medium

2 Use the InfoButton Search to research problem/treatment information


Practice Partner/

Third Party

Medium

3 Place a new lab order 170.314(a)(1) – CPOE Practice Partner

Medium

1 Test Procedure for §170.314(g)(3) Safety-enhanced design. Approved Test Procedure Version 1.2. December 14, 2012; and Health Information Technology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information Technology, Final Rule.

of 77


4 Look up and change existing lab order

170.314(a)(1) – CPOE Practice Partner

Medium

5 Look up and change existing image order


Medium

6 Place a new image order 170.314(a)(1) – CPOE Practice Partner

Medium

7 Look up medication order/prescription and change (make inactive)

170.314(a)(1) – CPOE

170.314(a)(6) – Medication list

Practice Partner

High

8 Order/prescribe new medication/ receive drug-allergy interaction alert/cancel

170.314(a)(1) – CPOE


170.314(a)(2) – Drug-Drug and Drug-Allergy interaction checks

Practice Partner

High

9 Order/prescribe new medication/ receive drug-drug interaction alert/cancel

170.314(a)(1) – CPOE



Practice Partner

High

10 Look up medication history 170.314(a)(6) – Medication list

Practice Partner

High

11 Change med list (activate inactive medication) and e-prescribe

170.314(a)(1) – CPOE


170.314(b)(3) – Electronic prescribing

Practice Partner

High

During the 45 minute, one-on-one usability test, each participant was greeted by an administrator when

they entered the conference room. Each participant had prior experience with a previous version of the

Practice Partner ® EHR. The participants were asked to review a self-guided training module that

reviewed the new InfoButton feature, and presented instructions for the usability study and using the

testing software. The administrator introduced the test, and instructed participants to complete a series

of tasks (given one at a time) using the EHR Under Test (EHRUT). During the testing, data logging

software timed the test and recorded user screen captures and audio. These recordings were later

used to score action steps, errors and task completion rates. The test administrator did not assist the

participant in completing the tasks.

The following types of data were collected for each participant:

Number of tasks successfully completed without assistance

Time to complete each task

Number and types of errors

of 77


Path deviations

Participant’s verbalizations and written comments

Participant’s satisfaction ratings of the system

All participant data was de-identified so that no correspondence can be made from the identity of the

participant to the data collected. At the end of the test session, each participant completed a post-test

questionnaire.

The results from the System Usability Scale scored the subjective satisfaction with the system based

on performance with these tasks to be: 71.76.

According to Tullis and Albert (2008), raw scores under 60 represent systems with poor usability;

scores over 80 would be considered above average.2 The details of our results are summarized in

Tables 2 and 3 below. Usability judgments were also collected at the end of each major task. The

averages obtained from those task-based usability scores were fairly high, and varied between 4.12

and 4.94 even for the ordering tasks that the participants were not familiar with. The NISTIR 7742

states that “common convention is that average ratings for systems judged easy to use should be 3.3.,

or above.” One single task presented a notable difficulty to the participants and scored only at 2. We

will give special attention to the analysis of this task in the Findings and Recommendation section

below.

Various additional recommended metrics, in accordance with the examples set forth in the NIST Guide

to the Processes Approach for Improving the Usability of Electronic Health Records, were used to

evaluate the usability of the EHRUT. Following is a summary of the performance and rating data

collected on the EHRUT.

Table 2. Result Summary by Task

Tasks and subtasks

N Task Success

Path Deviation

Task Time (in seconds)

Errors Task Ratings 5=Easy

# Percentage

Deviations (Observed / Optimal)

Mean Observed

(SD)

Optimal Deviations (Observed / Optimal)

Mean (SD)

Mean (SD)

1

Select Patient and Review Health Maintenance Alerts

100% 1.38 50.70

(15.52) 33.01 1.54

0.71

(0.59)

4.65

(0.61)

Select Patient 17 100% 1.68

34.65

(15.71) 20.01 1.73

0.71

(0.59)

4.82

(0.39)

Review Alerts 17 100% 0.62

16.05

(6.68) 13.00 1.24 0

4.44

(.73)

2 Use the InfoButton Search to research problem/treatmen

17 82% 1.25

79

(47.97)

26.93 3.98 .43

(.76)

4.36

(.84)

2 See Tullis, T. & Albert, W (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

t information

Navigation 17 82% 1.45

42.15

(43.49) 10.58 2.25

.36

(.74)

4.43

(.94)

InfoButton Lookup

17 82% .95 36.85

(23.2) 16.35 2.93

.07

(.27)

4.43

(.84)

3 Place a new lab order

17 100% 1.40 61.64

(27.43) 45.03 1.37

1.17

(1.18)

4.12

(.86)


25.02

(19.39) 32.57 2.01

.35

(.7)

4.12

(.93)

Place the order

17 100% 1.10 36.61

(13.02) 12.46 1.12

.82

(.88)

4.12

(.86)


17 88.24% 1.4 28.32

(30.53) 9.91 2.86

.13

(.35)

4.67

(.62)

Find order 17 88.24% 1.43 13.35

(19.59) 3.85 3.47

.13

(.35)

4.67

(.82)

Change Order 17 88.24% 1.36 11.28

(9.9) 6.06 1.86 0

4.73

(.62)

5 Look up and Cancel existing image order

17 5.88% 1.87 102.46

(NA)* NA NA 2 2

Find existing image order

17 94.12% 2 4.79

(1.228) 3.33 1.44 0

4.31

(1.14)

Cancel Order 17 5.88% 1.86 95.22

(NA) NA NA 2 2

6 Place new image order

17 100% 1.61 34.23

(14.60) 19.7 1.73

.41

(.71)

4.41

(.51)

Navigate to Order Screen

17 100% 1.82 7.66

(6.57) 4.61 1.66 0

4.61

(.49)

Order Image 17 100% 1.56 26.58

(15.82) 15.17 1.75

.41

(.71)

4.41

(.51)


17 94.12% 1.29 19.01

(6.02) 13.36 1.42

.31

(.47)

4.94

(.25)

Find prescription

17 94.12% 1.5 6.84

(3.31) 4.23 1.62

.06

(.25)

4.88

(.33)

Inactivate prescription/order

17 94.12% 1.19 12.18

(4.97) 9.12 1.33

.25

(.45)

4.94

(.25)

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

8 Order new prescription/ review and cancel

17 100% 1.38 72.91

(34.14) 35.1 2.08

.411

(.71)

4.41

(.87)

Write prescription

17 100% 1.57 58.78

(30.38) 27.70 2.12

.06

(.24)

4.41

(.94)

Acknowledge drug-allergy

interactions and cancel order

17 100% 1 14.13

(7.09) 7.39 1.91

.35

(.7)

4.47

(1.01)

9 Order new prescription review and cancel

17 94.12% 1.60

56.64

(22.35)

30.62 1.85 .62

(.72)

4.31

(.7)

Write prescription

17 94.12% 1.7 38.33

(23.4) 17.23 2.23

.13

(.34)

4.44

(.63)

Acknowledge drug-drug


17 94.12% 1.44 18.31

(8.03) 13.40 1.37

.5

(.73)

4.25

(.86)

10 Look up medication history

17 100% 1.64 28.28

(16.13) 13.25 2.13

.87

(1.15)

4.47

(.74)

Find medication

17 100% 1.33 20.42

(5.25) 3.25 2.14 0

4.47

(.64)

Find details 17 100% 1.8

21.34

(13.6) 10.00

2.13

.87

(1.13)

4.53

(.64)

11 Fill prescription and e-prescribe

16 80% 1.65 40.28

(23.84) 28.41 1.41

.23

(.43)

4.5

(.76)

Change med list (make active)

16 80% 1.32 10.38

(8.48) 4.99 2.08

.18

(.4)

4.5

(.76)

e-prescribe 16 80% 2.17 34.13

(19.63) 23.42 1.46

.08

(.29)

4.5

(.76)

The usability result data from Table 2 is grouped by Meaningful Use Criteria in Table 3 below, to allow

for better overview of the usability scores for each criterion.

of 77


Table 3. Result Summary by Meaningful Use Criterion

Meaningful Use Criteria and Tasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

170.314(a)(1) – CPOE

Ordering …

labs 17 100% 1.10

36.61

(13.02) 1.12

.82

(.88)

4.12

(.86)

images 17 100% 1.56

26.58

(15.82) 1.75

.41

(.71)

4.41

(.51)

medications 17 100% 1.57

58.78

(30.38) 2.12

.06

(.24)

4.41

(.94)

medications 17 94.12% 1.7

38.33

(23.4) 2.23

.13

(.34)

4.44

(.63)

Finding existing …

Lab orders 17 88.24% 1.43

13.35

(19.59) 3.47

.13

(.35)

4.67

(.82)

Image orders 17 94.12% 2

4.79

(1.228) 1.44 0

4.31

(1.14)

medications 17 94.12% 1.5

6.84

(3.31) 1.62

.06

(.25)

4.88

(.33)

Medications

(historical) 17 100% 1.33

20.42

(5.25) 2.14 0

4.47

(.64)

Changing existing …

Lab orders 17 88.24% 1.36

11.28

(9.9) 3.47

.13

(.35)

4.67

(.82)

Image orders

(cancel) 17 5.88% 1.86

95.22

(NA) NA 2 2

Medications

(discontinue) 17 94.12% 1.19

12.18

(4.97) 1.33

.25

(.45)

4.94

(.25)

Medications

(activate/renew) 16 80% 1.32

10.38

(8.48) 2.08

.18

(.4)

4.5

(.76)

170.314(a)(2) – Drug-Drug and Drug-Allergy Interaction Checks

Acknowledging drug-allergy interaction and cancel

17 100% 1 14.13

(7.09) 1.91

.35

(.7)

4.47

(1.01)

Acknowledging drug-drug interaction and

cancel

17 94.12% 1.44 18.31

(8.03) 1.37

.5

(.73)

4.25

(.86)


Find current medication

17 94.12% 1.5 6.84

(3.31) 1.62

.06

(.25)

4.88

(.33)

Find medication history

17 100% 1.33 20.42

(5.25) 2.14 0

4.47

(.64)

Change medication 16 80% 1.32 10.38

(8.48) 2.08

.18

(.4)

4.5

(.76)

of 77



N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

(activate)

Change medication

(discontinue) 17 94.12% 1.19

12.18

(4.97) 1.33

.25

(.45)

4.94

(.25)

Record new medication

17 100% 1.57 58.78

(30.38) 2.12

.06

(.24)

4.41

(.94)


17 94.12% 1.7 38.33

(23.4) 1.37

.5

(.73)

4.25

(.86)


Acknowledge Health Maintenance alert

17 100% 0.62 16.05

(6.68) 1.24 0

4.44

(.73)

Use InfoButton 17 82% 1.25 79

(47.97) 2.93

.43

(.76)

4.36

(.84)


e-prescribe, transmit electronically to pharmacy

16 80% 2.17 34.13

(19.63) 1.46

.08

(.29)

4.5

(.76)

In addition to the performance data, qualitative observations were made. A more detailed discussion of

result details, tasks, and findings is provided later in the document (in Discussion of the Findings).

Major Findings and Areas for Improvement

This section summarizes the findings and areas for improvement at a high level. For additional detail, please

see section "

of 77


Discussion of the Findings”.

Finding Recommendation

CPOE, Labs and Images

The order interface of Practice Partner generally

proved to be easy to learn and use –none of the

participants had used order entry before. There

were a few smaller and one more notable

exception:

- If users entered the order function

through the Order shortcut button, they

could have a difficult time finding the list

of existing orders or verifying that an

order had been submitted.

- The ‘cancel’ function is hidden so well

that users simply could not find it – it also

became confused with a more

accessible ‘delete’ function.

- It is possible that an associated hospital

instructed their end users to not use

right-click functions (this was mentioned

by one of our participants).

- Some users struggled with the search

function, expected the search string to

match any word, not just the first word in

the target set.

1. Training: When introducing CPOE make

users enter it through the order folder which

leads to the order list first. Teach the use of

the Order button only as a shortcut after

users are well aware of the function of the

order list.

2. Design: Consider always flowing to the

order list after the completion of an order –

even if the order was started from the Order

button. Users want confirmation that the

order has indeed been placed – and this

saves them an extra step.

3. Training: When introducing CPOE, pay

special attention to the right-click context

menu functions, and practice these – they

are difficult to discover and difficult to

remember if they have been included in

training.

4. Design: Consider making the ‘cancel’

function easier to find – possibly exchange

the ‘delete’ button with a ‘cancel’ button.

Canceling orders is possibly a more

frequently used function than deleting.

5. Training: During training, familiarize users

with the search algorithm so users know

exactly what to expect and how to use this

search function.

6. Design: Consider using a search algorithm

that matches the search string in any word,

not just the first one.

Drug-Drug and Drug-Allergy Interaction

Checks

The warnings were well written and easy to

understand.

- Users had some difficulty with selecting

the correct function keys which resulted

from an apparent confusion between

‘accepting’ the warning and ‘canceling’

1. Design: Make the button labels more direct.

Consider using ‘Override’ instead of ‘OK’

and “Cancel Order’ instead of ‘Cancel’. If

‘Cancel Order’ is selected, eliminate the

extra step of having to cancel the order in

the order window.

of 77


the prescription.

Medication List (also CPOE for Medication

Orders/Prescriptions)

In general, the med list functions were well

practiced and appeared easy to use. Testing

discovered a few smaller issues.

- When filling prescriptions, some users

shied from using templates (which can

save many steps), because they were

not sure that changes to a template

would carry forward.

- A small glitch in the search dialog makes

it possible that a user selects a

medication instead of initiating a search

when using the ‘enter’ key.

1. Training: Emphasize that templates can

save many clicks and that any fields in a

template can be changed for each particular

prescription once it has been applied.

2. Design: Consider indicating that a template

has been modified through some visual cue

(italicizing, adding an asterisk, etc. – some

marker that can be used consistently for

this purpose).

3. Design: In the search dialog, when the user

places focus in the search field by placing

the cursor, immediately make the ‘search’

button the default function when the ‘enter’

key is selected.

Clinical Decision Support

Health maintenance alerts exhibited no usability

issues. Access to clinical information via the

InfoButton was appreciated as a new function by

a number of participants.

- Access using the right-click context

menu presented a small stumbling block

… many participants did not remember

this method despite the training they had

received/

1. Design: If the right-click method is going to

be used more frequently, make its function

consistent across all Practice Partner

windows and functional areas.

Alternatively, consider a small icon or link to

indicate that Clinical content is available.

2. Training: The right-click function needs to

receive some emphasis and practice during

training, so that a habit can be formed,

otherwise it will be easily forgotten.

E-prescribing

In general this was easy to do once the

transmission mode indicator had been set.

- A few participants had a difficult time

finding the transmission mode indicator.

1. Training: During training, introduce the list

of transmission choices, found under the

‘Print’ label.

2. Design: Consider relabeling this function to

‘transmission’, ‘transmission mode’, or

similar.

of 77


General Observation:

While the participants seemed to be well

oriented within Practice Partner and seemed to

know to navigate it effortlessly, there was one

small inconsistency in the use of its controls that

led to some struggle:

In some contexts, double-clicking on a list item

leads to more choices, in other similar contexts

it doesn’t. In other contexts the right-click

context menu is used to introduce more choices.

1. Review Practice Partner with respect to this

control method. Try to apply it consistently

independent of immediate context. Double-

click was a method tried many times by

many participants, they expected to find

details and additional functions. Right-click

was never tried. If introducing this as a new

and quick way to access functions, it needs

to be introduced across all lists in a similar

manner, and training should accompany

this introduction especially training of super

users to promote.

of 77


Introduction The McKesson Practice Partner® EHR tested for this study was accessible to the participants in an

integrated environment, as would be typical in a provider implementation. The entire environment was

software version VER. 11. The usability testing environment and tasks attempted to simulate realistic

implementations, workflows and conditions. The purpose of the study was to test and validate the

usability of the current user interface, and provide evidence of usability in the EHR Under Test

(EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction were captured during

the usability testing.

Intended Users

Intended physician users of the EHRUT are mid-level providers in ambulatory practices or clinics, such

as nurse practitioners, physician’s assistants, and physicians of any specialty. Users may have varying

levels of experience in the clinical setting and may have varying years of experience performing clinical

functions with the EHRUT. They can be expected to have received training on the EHRUT, and to be

frequent users (weekly or daily use). Depending on site or specialty, they may have varying levels of

experience with the specific MU functions and tasks.

Method

Participants

A total of 22 participants were tested on the EHRUT for the physician role in the ambulatory setting.

Participants were associated with clinics in one of two customer sites (nine were associated with site A

and the remaining 13 with site B). The participants had been recruited by IT staff in their organizations.

None of the participants had a direct connection to the development of the product. Eighteen

Participants were MDs or ODs, and four were PA’s or NPs.

All participants had experience using the EHRUT, ranging from 1 to 20 years. All participants use the

system for prescribing medications and writing physicians’ notes on a daily basis. However, most

participants were not, or not very familiar with the ordering functions (CPOE), as both sites had only

started implementing this feature in their sites recently. None of the participants had used the

InfoButton feature which was an enhancement with VER. 11. A demographics questionnaire was filled

out before the session began; the questionnaire is included as Appendix A: Demographic

Questionnaire.

Six of the participants were excluded from the analysis because of technical difficulties encountered

during the first day of testing.

Table 4 lists all 17 participants included in the analysis by demographic characteristics, such as

demographics, education, and EHR experience. Participant names were replaced with Participant IDs

so that an individual’s data cannot be tied back to individual identities.

of 77


Table 4. Participant Demographics

ID Gender Age Education Specialty Years in practice

Years using Practice Partner

Frequency of use (filling prescriptions)

1 f 31-40 MD Pediatrics 2 2 Many times a day

2 m > 61 MD Internal Medicine 33 5 Many times a day

3 f 41-50 MD Internal Medicine 15 1 Many times a day

4 f 31-40 PA Dermatology 10 1 Many times a day

5 f 21-30 PA GI 1 1 Many times a day

6 f 31-40 MD Ophthalmology 9 7 Many times a day

7 m 41-50 MD Geriatrics 21 15 Many times a day

8 m 41-50 MD Gastroenterology 14 3 Many times a day

9 m 51-60 MD Internal Medicine 23 19 Many times a day


11 m 51-60 MD OB-Gyn 29 1.5 Many times a week


13 m 31-40 MD Otolaryngology/ENT 1 1 Many times a day

14 m 31-40 MD PM&R 5 5 Less than once a month

15 f 31-40 ARNP Dermatology 7 1 Many times a day

16 m 51-60 MD Plastic Surgery 24 20 Many times a day

17 f 51-60 MD Neurology/Sleep Disorders

20 15 Many times a day

A short, self-paced training PowerPoint presentation was provided to participants before the study, to

review the new InfoButton functionality. This training module was similar to what end-users would

usually receive when introduced to new functionality. No training or review of the order CPOE

functionality was included in the training, because the expectation was that participants would have

been familiar with this function. The training also included some explanation of the software used to

capture the session data.

Participants were scheduled for 45 minute sessions which provided time between sessions for debrief

by the administrator, and to get the system ready for the next participant. A spreadsheet was used to

keep track of the participant schedule, and the participant demographics.

Study Design

Overall, the objective of this test was to uncover areas where the application performed well — that is,

effectively, efficiently, and with satisfaction — and areas where the application failed to meet the needs

of the participants. The data from this test may serve as a baseline for future tests with an updated

version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In

of 77


short, this testing serves as both a means to record or benchmark current usability, but also to identify

areas where improvements must be made. Specifically, the purpose of this test was to test the usability

of 5 of the 7 Meaningful Use Criteria that are defined in 45 CFR Part 170, RIN 0991-AB823 for

ambulatory use. The current study focused on the 5 MU criteria used typically by physicians. Two

separate reports detail the findings from tasks associated with the MU criteria typically conducted by

nurses and administrators. Table 5 below cross-tabulates which task in the study of physicians tasks is

associated with which MU criterion and lists the risk associated with that task.

Table 5. Physicians’ Tasks – same as Table 1, repeated here for easier reference


Application Risk

1 Select Patient and Review Health Maintenance Alerts


Practice Partner

Medium



Practice Partner/

Third Party

Medium

3 Place a new lab order 170.314(a)(1) – CPOE Practice Partner

Medium



Medium

5 Look up and change existing image order


Medium

6 Place a new image order 170.314(a)(1) – CPOE Practice Partner

Medium


170.314(a)(1) – CPOE


Practice Partner

High

8 Order/prescribe new medication/ receive drug-allergy interaction alert/cancel

170.314(a)(1) – CPOE



Practice Partner

High

3 45 CFR Part 170, RIN 0991-AB82. Health Information Technology: Standards, Implementation Specifications, and

Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information Technology.

of 77


9 Order/prescribe new medication/ receive drug-drug interaction alert/cancel

170.314(a)(1) – CPOE



Practice Partner

High

10 Look up medication history 170.314(a)(6) – Medication list

Practice Partner

High

11 Change med list (activate inactive medication) and e-prescribe

170.314(a)(1) – CPOE



Practice Partner

High

During the usability test, participants interacted with the above listed functions within the McKesson

PracticePartner® VER. 11. Each participant used the system in the exact same configuration, and was

provided with the same instructions. The system was evaluated for effectiveness, efficiency and

satisfaction as defined by measures collected and analyzed for each participant:

o Number of tasks successfully completed without assistance

o Time to complete tasks

o Number of errors

o Path deviations

o Participant verbalizations (comments)

o Participant’s satisfaction ratings of the system

Procedures

The test participant was greeted by the facilitator upon entering the conference room. Once the

participant was seated, he or she was verbally instructed and oriented to the test situation (see

Appendix B: Administrator Script and General Instructions). Each participant was asked to sign an

informed consent form (Appendix C: Informed Consent Form), and then filled out the demographics

questionnaire (Appendix A: Demographic Questionnaire). Then the participant was asked to step

through a set of self-paced training slides. Participants spent between 5-10 minutes reviewing this

presentation that introduced the InfoButton functionality. The presentation also included usability test

instructions and an introduction to the Morae recording software (Appendix D: Usability Test

Instructions). During this presentation participants had a chance to ask questions and discuss the new

feature and any issues concerning the study itself.

One test administrator per participant ran the test. Typically two sessions were happening in the same

room, but enough space was provided so that these two sessions did not interrupt or distract each

other. In some cases, additional persons (hospital officials and a product director) were present to

observe, but they did not interact with the participants. Data logging was achieved automatically by the

of 77


Morae recording software. Detailed data analysis and scoring of the recorded protocols was done after

the test procedure by coding the recording with more fine grained task, action and error markers.

Two facilitators conducted the test: a senior usability professional (Ph.D. Cognitive Psychology with a

Human Factors emphasis.), with over 20 years of experience in the field of User Experience Design

and Research, and a Business Analyst (see Appendix E: Experimenters’ Short Biograph).

For each task, the testing software provided an on-screen description of the task to be performed. See

"Appendix F: Participant Instructions for Each Task" for the complete text of each on-screen task

prompt.

Task timing began when the participant clicked the Start Task button. Task time was stopped when the

participant clicked the End Task button. Participants were instructed to click End Task when they had

successfully completed the task or when they could proceed no further.

After each task the participant was given a short satisfaction questionnaire related to that task

(Appendix G: End-of-Task Questionnaires). After all tasks were completed, the participant was given a

System Usability Scale (SUS) Questionnaire (Appendix H: System Usability Scale (SUS)

Questionnaire).

Following the test, the administrators provided time for the participants to informally communicate

impressions about the usability of the EHRUT and about the testing process.

Test Locations

Participants were tested in their own facilities. In facility A (York, ME), an IT training room was used on

one day, and a conference room on another day. In facility B (Pensacola, FL) the boardroom of the

hospital building was used for all sessions. While all sites tried to create a quiet space for the sake of

testing, their busy professions did not always allow for this. On occasion there was background

conversation and other distractions associated with busy hospital environments. Given the usually

hectic context of clinical practices and the use of EMRs, this represented fair and realistic conditions

under which to test the software.

Test Environment

The participants performed the test tasks on the facilitator’s laptops which were both running

independent copies of the EHR server, client and interaction engine. All of these were Practice Partner

VER. 11. The specifications of the two facilitator machines are as follows: (1) an HP Elitebook Folio

9470m laptop, DPI setting normal (96 DPI), Highest Color Quality (32 bit), 1366 X 768 pixel screen

resolution, and a 60 Hertz screen refresh rate, and (2) a Dell Latitude E6410 laptop, DPI setting normal

(96 DPI), Highest Color Quality (32 bit), 1366 X 768 pixel screen resolution, and a 60 Hertz screen

refresh rate. Audio was recorded through each system’s internal microphone. Participants interacted

with the remote software through their keyboard and mouse or touchpad.

Table 6. Test Environment, Technical Data

of 77


Connection to the InfoButton Content

Site A: WAN. Dial-in into McKesson VPN, connection into the vendors website for content.

Site B: LAN. Dial-in into McKesson VPN, connection into the vendors website for content.

PracticePartner®

VER. 11 Usability Test

Environment

Two special usability test environments were created using the generic data configuration used for the PracticePartner Environment. The environments simulates the complete EHR including all functions and backend support software that a real implementation would provide, including data integration with external systems allowing for data flow from pharmacy solutions, integration with electronic data transfer to pharmacies, and allowing integration with the InfoButton reference vendor. No other user groups had access to this environment, ensuring a pristine test environment with full control of the configuration of the test data.

It is important to note that while all efforts were made to create a fully functioning test environment, the configurations of menus and menu choices, applications, alerts, etc. were similar but not the same as what any particular provider/customer would use. All of the customized functions are different from site to site, depending on hospital policies and implementation.

Our test environment is configured to show what the system is capable of and therefore presents more alerts, confirmation screens, list items etc., than many facilities would choose to implement.

Generally, the system performance (i.e. response time) was representative to what actual users would experience in a field implementation. Data from tasks or trials where system response time slowed perceivably were excluded from the analysis.

EHR software and version

PracticePartner® Version 11

Testing software Morae Recorder Version 3.3.2 (participant)

Morae Manager Version 3.3.2 (data analysis)

Test Forms and Tools

During the usability test, various documents and instruments were used, including:

o Informed Consent

o Demographic Questionnaire

o Experimenter’s Verbal Instructions

o Training slides

o Observer’s Task Sheets

o Post-task surveys (presented through Morae)

o Post-test survey (System Usability Scale, SUS, presented through Morae)

o Most of these materials are included in the appendices.

Usability Metrics

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records, EHRs should support a process that provides a high level of usability for all users. The goal is

of 77


for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction.

To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability

testing. The goals of the test were to assess:

o Effectiveness of the EHRUT by measuring participant success rates and errors.

o Efficiency of the EHRUT by measuring the average task time and path deviations.

o Satisfaction with the EHRUT by measuring ease of use ratings.

Data Scoring

The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Table 7. Data Scoring: Measures & Rationale

Measures Rationale and Scoring

Effectiveness:

Task Success

A task is counted as a “Success” if the participant was able to achieve the correct outcome, without user interface directed assistance. Direction was occasionally provided with understanding task instructions, but never in how a task was to be accomplished.

A task is counted as a “Failure” if the participant was never able to find the correct action, or ended the task believing that they had accomplished all tasks without having done so.

The total number of successes are calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times are recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by the 3 users who finished the task without errors and deviations and in the shortest amount of time, was calculated by averaging the times of these 3 users. In cases where more than 2 participants failed a task, the optimal task performance time was calculated using the 2 users who finished the task without errors and deviations in the shortest amount of time. No adjustments were made to the optimal time. Average group times were divided by the optimal performance time for a measure of how close to optimal efficiency each task was performed. It was decided not to define time allotments, because it was felt that different adjustments would have to be made for new tasks and for existing task flows; and we could not find a generally accepted empirical or principled way to determine meaningful adjustments. All participants were allowed to finish the tasks, and the time ratios provide a measure of how much longer an average user would take to perform these tasks.

We felt this was a cleaner, and less biased measure than to define task allotment times based on arbitrary time adjustments. Using the same or no adjustment for a number of studies would also allow better comparison across studies, rather than trying to compare rates or ratios that had all been derived by different adjustments.

Effectiveness:

Task Failures and

Errors

If the participant abandons the task, does not reach the correct answer or performs it incorrectly the task is counted as a “Failure.” For each task the percentage of participants who successfully completed it are reported. The failures can be calculated by subtracting the successful completions from 100.

The total number of errors is calculated for each task and then divided by the total number of times that task was attempted. Not all deviations are counted as errors.

Errors are selections of wrong menu items, wrong list items, or entering wrong data formats. If an error results in a number of corrective steps, only the first actions, resulting in the error is counted as error. All corrective steps are counted as deviations.

On a qualitative level, an enumeration of errors and error types is collected.

Efficiency: The participant’s path (i.e., steps) through the application is recorded. Deviations occur if

of 77



Task Deviations the participant, for example, chooses a path to task completion which requires more steps than the optimal, fills out additional fields, or performs any steps to correct a previous error. This path is compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency:

Task Time

Each task is timed from when the participant clicks the Start Task button until the participant clicks the End Task button. Only task times for tasks that are successfully completed are included in the average task time analysis. Average time per task is calculated for each task. Variance measures (standard deviation) are also calculated.

Satisfaction:

Task Rating

Each participant’s subjective impression of the ease of use of the application is measured

by administering both a simple post-task questionnaire as well as a post-session

questionnaire. After each task, the participant is asked to rate “Overall, this task was: on a

scale of 1 (Very Difficult) to 5 (Very Easy)”. If the task included a number of subtasks, a

question addressing the ease of each subtask was included (see Appendix G: End-of-Task

Questionnaires).

These data are averaged across participants. Common convention is that average ratings

for systems judged easy to use should be 3.3 or above.

To measure participants’ confidence in and likeability of the [EHRUT] overall, the testing team administers the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See the full System Usability Scale questionnaire in "Appendix H: System Usability Scale (SUS) Questionnaire".

of 77


Results

Data Analysis and Reporting

The results of the usability test were calculated according to the methods specified in the Usability

Metrics section above.

Table 8. Usability Data by Task and Subtask - same as Table 2, repeated here for easier reference

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1


100% 1.38 50.70

(15.52) 33.01 1.54

0.71

(0.59)

4.65

(0.61)

Select Patient 17 100% 1.68

34.65

(15.71) 20.01 1.73

0.71

(0.59)

4.82

(0.39)

Review Alerts 17 100% 0.62

16.05

(6.68) 13.00 1.24 0

4.44

(.73)


17 82% 1.25

79

(47.97)

26.93 2.93 .43

(.76)

4.36

(.84)


42.15

(43.49) 10.58 3.98

.36

(.74)

4.43

(.94)

InfoButton Lookup

17 82% .95 36.85

(23.2) 16.35 2.25

.07

(.27)

4.43

(.84)


17 100% 1.40 61.64

(27.43) 45.03 1.37

1.17

(1.18)

4.12

(.86)


25.02

(19.39) 32.57 2.01

.35

(.7)

4.12

(.93)

Place the order

17 100% 1.10 36.61

(13.02) 12.46 1.12

.82

(.88)

4.12

(.86)


17 88.24% 1.4 28.32

(30.53) 9.91 2.86

.13

(.35)

4.67

(.62)

Find order 17 88.24% 1.43 13.35

(19.59) 3.85 3.47

.13

(.35)

4.67

(.82)

Change Order 17 88.24% 1.36 11.28

(9.9) 6.06 1.86 0

4.73

(.62)

5 Look up and Cancel existing image order

17 5.88% 1.87 102.46

(NA)* NA NA 2 2

Find existing 17 94.12% 2 4.79

(1.228) 3.33 1.44 0

4.31

(1.14)

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

image order

Cancel Order 17 5.88% 1.86 95.22

(NA) NA NA 2 2


17 100% 1.61 34.23

(14.60) 19.7 1.73

.41

(.71)

4.41

(.51)


17 100% 1.82 7.66

(6.57) 4.61 1.66 0

4.61

(.49)

Order Image 17 100% 1.56 26.58

(15.82) 15.17 1.75

.41

(.71)

4.41

(.51)


17 94.12% 1.29 19.01

(6.02) 13.36 1.42

.31

(.47)

4.94

(.25)

Find prescription

17 94.12% 1.5 6.84

(3.31) 4.23 1.62

.06

(.25)

4.88

(.33)


17 94.12% 1.19 12.18

(4.97) 9.12 1.33

.25

(.45)

4.94

(.25)

8 Order new prescription/ review and cancel

17 100% 1.38 72.91

(34.14) 35.1 2.08

.411

(.71)

4.41

(.87)

Write prescription

17 100% 1.57 58.78

(30.38) 27.70 2.12

.06

(.24)

4.41

(.94)

Acknowledge drug-allergy


17 100% 1 14.13

(7.09) 7.39 1.91

.35

(.7)

4.47

(1.01)

9 Order new prescription review and cancel

17 94.12% 1.60

56.64

(22.35)

30.62 1.85 .62

(.72)

4.31

(.7)

Write prescription

17 94.12% 1.7 38.33

(23.4) 17.23 2.23

.13

(.34)

4.44

(.63)

Acknowledge drug-drug


17 94.12% 1.44 18.31

(8.03) 13.40 1.37

.5

(.73)

4.25

(.86)

10 Look up 17 100% 1.64

28.28

(16.13) 13.25 2.13

.87

(1.15)

4.47

(.74)

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

medication history

Find medication

17 100% 1.33 20.42

(5.25) 3.25 2.14 0

4.47

(.64)


21.34

(13.6) 10.00

2.13

.87

(1.13)

4.53

(.64)


16 80% 1.65 40.28

(23.84) 28.41 1.41

.23

(.43)

4.5

(.76)


16 80% 1.32 10.38

(8.48) 4.99 2.08

.18

(.4)

4.5

(.76)

e-prescribe 16 80% 2.17 34.13

(19.63) 23.42 1.46

.08

(.29)

4.5

(.76)






averages obtained from those task-based usability scores were fairly high, and varied between 4.12

and 4.94 even for the ordering tasks that the participants were not familiar with. The NISTIR 7742

states that “common convention is that average ratings for systems judged easy to use should be 3.3.,

or above.” One single task presented a notable difficulty to the participants and scored only at 2. We

will give special attention to the analysis of this task in the Findings and Recommendation section

below.





In the following table, Table 9, the usability metrics are reported grouped out by MU criterion. Please

note that the MU criteria related to clinician tasks have been tested with two other user groups (nurses

and administrators). Those results are summarized in 2 separate reports.


of 77


Table 9. Usability Data by Meaningful Use Criteria – same as Table 3, repeated here for easier reference


N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

170.314(a)(1) – CPOE

Lab orders

Place a new lab order 17 100% 1.40

61.64

(27.43) 1.37

1.17

(1.18)

4.12

(.86)


25.02

(19.39) 2.01

.35

(.7)

4.12

(.93)

Place the order 17 100% 1.10

36.61

(13.02) 1.12

.82

(.88)

4.12

(.86)

Look up and change existing lab order

17 88.24% 1.4 28.32

(30.53) 2.86

.13

(.35)

4.67

(.62)

Access order 17 88.24% 1.43 13.35

(19.59) 3.47

.13

(.35)

4.67

(.82)

Change Order 17 88.24% 1.36 11.28

(9.9) 1.86 0

4.73

(.62)

Image Orders

Place new image order

17 100% 1.61 34.23

(14.60) 1.73

.41

(.71)

4.41

(.51)


17 100% 1.82 7.66

(6.57) 1.66 0

4.61

(.49)

Order Image 17 100% 1.56 26.58

(15.82) 1.75

.41

(.71)

4.41

(.51)

Look up and Cancel existing image order

17 5.88% 1.87 102.46

(NA)* NA NA 2


17 94.12% 2 4.79

(1.228) 3.33 1.44 0

Cancel Order 17 5.88% 1.86 95.22

(NA) NA NA 2

Medication Orders (prescribing)

Look up medication order/prescription and change (make inactive)

17 94.12% 1.29 19.01

(6.02) 13.36 1.42

.31

(.47)

Order new prescription/ review and cancel

17 100% 1.38 72.91

(34.14) 2.08

.411

(.71)

4.41

(.87)

Order new prescription review and cancel

17 94.12% 1.60

56.64

(22.35)

1.85 .62

(.72)

4.31

(.7)


of 77



N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

Acknowledging drug-allergy interaction

and cancel

17 100% 1 14.13

(7.09) 1.91

.35

(.7)

4.47

(1.01)

Acknowledging drug-drug interaction and

cancel

17 94.12% 1.44 18.31

(8.03) 1.37

.5

(.73)

4.25

(.86)


Find current medication

17 94.12% 1.5 6.84

(3.31) 1.62

.06

(.25)

4.88

(.33)

Find medication history

17 100% 1.33 20.42

(5.25) 2.14 0

4.47

(.64)

Change medication

(activate) 16 80% 1.32

10.38

(8.48) 2.08

.18

(.4)

4.5

(.76)

Change medication

(discontinue) 17 94.12% 1.19

12.18

(4.97) 1.33

.25

(.45)

4.94

(.25)


17 100% 1.57 58.78

(30.38) 2.12

.06

(.24)

4.41

(.94)


17 94.12% 1.7 38.33

(23.4) 1.37

.5

(.73)

4.25

(.86)


Acknowledge Health Maintenance alert

17 100% 0.62 16.05

(6.68) 1.24 0

4.44

(.73)

Use InfoButton 17 82% 1.25 79

(47.97) 2.93

.43

(.76)

4.36

(.84)


e-prescribe, transmit electronically to pharmacy

16 80% 2.17 34.13

(19.63) 1.46

.08

(.29)

4.5

(.76)

of 77


Discussion of the Findings This section presents the study findings, organized by Meaningful Use Criteria.

170.314(a)(1) – CPOE

Test contexts

CPOE tasks were tested in two different contexts:

Labs and images are frequently ordered to be performed by different entities, and a classic ordering

function exists in Practice Partner. CPOE in this functionality was a new function to the participants in

both sites, but training was not provided on this functionality. The result from these tasks (tasks 3-6)

should therefore be seen as an indication of how easy it would be for a novice to learn ordering through

exploration in Practice Partner.

Writing a medication order in an ambulatory context is equivalent to writing a prescription. A number

of tasks were included that tested the interaction with the medication list, from which prescriptions can

be written, and they will be reviewed in the Medication list and e-prescription sections below. Because

of the way this functionality is implemented in Practice Partner these tasks 7-11 are associated with

CPOE, Medication list, Drug-drug, drug-allergy interactions, and e-prescribing. These tasks were all

well-practiced for the participants in our study.

Task 3 – Place a new lab order

Analysis Notes

Linked Meaningful Use Criteria

170.314(a)(1) – CPOE

Task Description

This task asked participants to fill out a STAT order for a urine analysis. This is the first time the

participants had to fill an order. They had not received instructions, so we can expect lots of

exploration and small navigational errors in this task.

Risk Analysis

Medium. While getting timely and accurate labs drawn for patients is crucial for medical decision

making, it does not run the risk of causing immediate harm.

Scoring Notes

All participants followed instructions as expected. There were no difficulties resulting in exclusions.

of 77


Exclusions

None

Quantitative Summary

Table 10. Usability Data for Task 3 – Order new lab

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


17 100% 1.40 61.64

(27.43) 45.03 1.37

1.17

(1.18)

4.12

(.86)


25.02

(19.39) 32.57 2.01

.35

(.7)

4.12

(.93)

Place the order

17 100% 1.10 36.61

(13.02) 12.46 1.12

.82

(.88)

4.12

(.86)

Participant Errors

Table 11. Participant Errors for Task 3 – Order new lab

Participant Error Severity Error Description

4, 6, 20,

15, 15 Minor Navigation exploration

4 Minor Wrong order type selected, immediately corrected by selecting the correct one

4, 21, 21,

12, 13, 14,

15, 18, 7,

7, 8, 9

Minor Control exploration

Effectiveness

All participants performed this task successfully. Several participants commented that they were not

familiar with this task and appeared a little hesitant to try it. As the list of errors shows, errors fall mostly

into the category of navigation errors (accessing the wrong tab or menu to find orders), and control

exploration errors (understanding which parts of the order entry screen can be clicked, or double-

clicked to achieve the desired result). But even though navigation mistakes were made, 14/17

participants found the order entry screen immediately, and even though many minor mistakes were

of 77


made in filling the order, the order entry screen was forgiving enough and self-explanatory enough, so

that all participants eventually filled and submitted their first order successfully.

Efficiency

Given that this was the participants’ first interaction with the order functionality, the step deviation ratio

(2.29 for navigation, and 1.10 for placing the order) were very small, and the time deviations varied

accordingly (2.01 for navigation and 1.12 for placing the order). Overall, it took the participants a little

over a minute to place their first order. It is important to note that they were able to cut this time in half

by the second new order (task 6, place image order), which also demonstrates that the interface

supported learning very well.

Comments

Comments are listed only if a participant provided a written comment.

Table 12. Participant Comments for Task 3 – Order new lab

Participant Do you have any additional comments?

20 for no instructions at all not too bad

21 A keyboard shortcut and/or better tabs for switching between windows in PracticePartner would be very useful in general.

13 fairly easy and user friendly

15 First time I used the order menu button- Again, I think should be easy after using it more

9 When the culture returns, I would like to be notified in practice partner. This way, I can review results quicker.

Satisfaction

The participants’ comments show that the users were surprised at how easy ordering seemed given

that they had never used it and had not received instructions, the usability score mirrors this

observation (4.12).

Major Findings

Most users found one of two ways of entering orders immediately. We noticed later that the shortcut to

the order entry screen, even though it was especially efficient, produced the unwanted side effect that

users did not see their orders in the order list after the order sequence and were not sure if it had

actually been placed. The conceptually more meaningful way to enter orders appears to be through the

list of orders because it presents the context of existing orders, both before starting the order and after

it has been placed. It might be good to point this out during training. One could also consider always

displaying the list of orders at the end of the order sequence, even if the user has started the order

through clicking the shortcut button. Other observations: (1) some users struggled with the search

algorithm (they expected their entered search term to match any part of the string they were looking for

of 77


instead of only the first word), (2) some users thought they could click the order status in the table on

the order form to change the status (from routine to stat).

Task 4 – Look up and change existing lab order

Analysis Notes


170.314(a)(1) – CPOE

Task Description

This task asked participants to change an existing blood glucose order from routine to stat. This is the

second order task, but the first time participants have to find/access an existing order.

Risk Analysis


making, it does not run the risk of causing immediate harm or impact patient safety directly.

Scoring Notes


Exclusions

None


Table 13. Usability Data for Task 4 – Look up and change existing lab order

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


17 88.24% 1.4 28.32

(30.53) 9.91 2.86

.13

(.35)

4.67

(.62)

Find order 17 88.24% 1.43

13.35

(19.59) 3.85 3.47

.13

(.35)

4.67

(.82)

Change Order

17 88.24% 1.36 11.28

(9.9) 6.06 1.86 0

4.73

(.62)

of 77


Errors Table 14. Participant Errors for Task 4 – Look up and change existing lab order


4, 4, 5, 7,

20 Minor navigation

Effectiveness

All but 2 participants finished this task successfully. The error table shows that the exploration of

navigation and the function of the controls in the order window allowed them to perform this task more

effectively with much fewer mistakes. All of the navigation mistakes resulted from participants who

clicked on the order shortcut button first. To find the list of orders, the order tab in the patient record

has to be selected. One participant failed this task because she could never find that order tab – she

was trying to access the existing orders through the order window.

Efficiency

Most participants found the existing order quickly and with few extra steps (step deviation score was

1.43). However a few participants looked at the orders screen first, and took a little longer to find the

order tab that links to the list of orders, leading to a longer time deviation score (3.47). Once the order

screen was found most participants quickly made their change, however one participant could not find

out how to change the order status. Step and time deviations for making the change are fairly small

though (1.36 and 1.86, respectively).

Comments


Table 15. Participant Comments for Task 4 – Look up and change existing lab order


4 no prior training

5 Never done before but easy to do.

20 again no idea where existing order would be

9 Next to the order, I would like one of the columns to mention routine vs stat. Also, will a PIN be required to order these like it would be required for a medication.

of 77


Satisfaction

The satisfaction ratings were high, 4.67 across the whole task, reflecting that participants started to feel

confident that they would know how to use the order functionality.

Major Findings

The observations from Task 3 apply here as well. In this task, where existing orders need to be

accessed, the order shortcut button presents a bit of a trap. If participants see this as the main way to

access the order functionality they might not notice that access to the list of existing orders is through a

tab in the patient tab – emphasize this in training when introducing the order functionality. Changing the

status indicator is generally easy once it has been found.

Task 5 – Look up and change existing image order

Analysis Notes


170.314(a)(1) – CPOE

Task Description

This task asked participants to find an existing CT and cancel it. This is the third order task, but the first

image order and the first and only time participants have to cancel an order.

Risk Analysis



Scoring Notes


Exclusions

None


Table 16. Usability Data for Task 5 – Look up and change existing image order

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

5 Look up and 17 5.88% 1.87 102.46 NA NA 2 2

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

Cancel existing image order

(NA)*


17 94.12% 2 4.79

(1.228) 3.33 1.44 0

4.31

(1.14)

Cancel Order

17 5.88% 1.86 95.22

(NA) NA NA 2 2

Errors Table 17. Participant Errors for Task 5 – Look up and change existing image order


18, 9, 7, 6,

5, 17, 16,

15, 14, 13,

12, 21, 20

Severe Uses delete instead of cancel function because they can’t find a cancel option.

9, 7, 7, 6,

6, 5, 5, 15,

15, 14, 14,

14, 13, 12,

21, 20, 20,

20, 20, 20,

6, 4, 4

Minor

Navigation, participant is looking for ‘cancel’ option but cannot find Each count in the leftmost column represents the start of another unsuccessful navigational exploration, so some participants started a few before selecting the ‘delete’ button.

Effectiveness

Only 1 participant finished this task successfully. Most other participants searched the order window

and the order list window for a ‘cancel’ functionality, and finally gave up and used the readily available

‘delete’ function, even though many were quite certain that this was not what they had been asked to

do. Clearly, the problem was not with finding the existing order – this was done in less than 5 seconds,

and by nearly 95 % of the participants. Image orders are just as easy to find as lab orders.

Efficiency

The one successful participant searched for 95 seconds before finding the ‘cancel’ function, but many

participants searched for at least that long before giving up or selecting the wrong function. With only

one successful participant, deviation scores and mean error scores could not be meaningfully derived.

of 77


Comments


Table 18. Participant Comments for Task 5 – Look up and change existing image order


5 As long as deleting an order cancels it that is a fast way to do it.

6 Not clear if deleting an order cancels the order.

20 had to be told where existing orders were

13 slightly confusing with "cancel" button vs. deleting item from list

15 Just not familiar with the order menue.

7 not clear about deleting vs cancelling orders

9 Is there any record of this order existing. Maybe it should state beside the order that it was cancelled be a certain provider.

Satisfaction

Finding the order was perceived as easy (the average satisfaction rating was 4.31). Only one

participant found the ‘cancel’ function, and he found this subtask very difficult (with a rating of 2). The

comments above demonstrate that participants were confused about the difference between canceling

and deleting, and simply could not find the right-click menu functions.

Major Findings

When evaluating this task, one has to keep in mind that none of the participants had received an

instruction, and none of them had ever performed this task before. However, the function of canceling

an order (which was hidden behind a right-click context menu) all but escaped discovery – which is a

startling contrast to all other order actions which had been generally easy to discover. The use of right-

click context menus is not implemented consistently throughout Practice Partner (although it does exist

in other parts of the application), and the tested users (except for one) clearly didn’t ever think about

using it. [What added to this issue may have been a policy at an associated hospital ‘to not use any of

the right-click menu’s’, that was brought our attention by one of our participants. We don’t know for

sure, but this may have played into the willingness of our participants to try right-clicking as an

exploratory action.] However, many, if not all tried double-clicking the order to see if they could access

more functions in this manner. To make matters worse, a ‘delete’ button was clearly visible on the

bottom of the order list. Although many participants suspected rightly that this was not the correct

action (as they wanted to preserve a note about who canceled the order and why), many ended up

choosing this as the only alternative they could find. Suggestions are to promote right-click context

menus through training and super user education, to review whether right-click vs. double click

of 77


methods might need to be implemented more consistently throughout Practice Partner, and to consider

turning the ‘delete’ button which is clearly visible into the ‘cancel’ action which is clinically more relevant

and probably a more frequent user action. The ‘delete’ action could be moved into the context menu.

As our ‘info-button’ task shows (see details below), even if instructions on using right-click context

menus are provided, many users forget – it is not an instruction that is easily remembered from one

single training session, because it is based on pure recall vs. recognition of a cue in the display.

Task 6 – Place new image order

Analysis Notes


170.314(a)(1) – CPOE

Task Description

This task asked participants to place an order for a stat renal ultrasound. This is the second time that

the participants have to enter a new order. Except for the difference in the particulars of the order, they

should start feeling familiar with this task at this point.

Risk Analysis

Medium. While getting timely and accurate images taken for patients is crucial for medical decision


Scoring Notes


Exclusions

None


Table 19. Usability Data for Task 6 – Place new image order

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


17 100% 1.61 34.23

(14.60) 19.7 1.73

.41

(.71)

4.41

(.51)


17 100% 1.82 7.66

(6.57) 4.61 1.66 0

4.61

(.49)

Order Image 17 100% 1.56

26.58

(15.82) 15.17 1.75

.41

(.71)

4.41

(.51)

of 77


Errors Table 20. Participant Errors for Task 6 – Place new image order


4, 5, 15,

16, 16 Minor Exploring control

5, 5 Minor Misleading search term used

Effectiveness

All participants finished this task successfully. The error and comments tables show that ordering has

become easier, although some smaller struggles still exist. Two participants tried to search for the

order type and ran into difficulties because the search engine only looks for matches in the first word.

This led two participants to conclude that the order type did not exist (they entered ‘ultrasound’, and

since it looks for matches with the first word, it only shows ‘ultrasound therapy’ but not ‘renal

ultrasound’). If the facilitator would not have asked them to look in other ways, they would have given

up their search under normal circumstances. Participants also still struggled with expanding the search

tree, but those issues were easily overcome.

Efficiency

Overall, participants found the order screen quickly at this point, and also were able to place their order

quickly. They were able to cut the time they needed to fill the order form in half when compared with

their first order task (task 4). On average the order was placed in close to 30 seconds.

Comments


Table 21. Participant Comments for Task 6 – Place new image order


5 Ultrasound of the kidney was not in the lookup group, but there in the double click of the radiology group. I would very rarely order that test, so would not have found it without coaching.

20 now that i know !

7 practice makes things easier

of 77


Satisfaction

The greater familiarity was evidenced in two of the comments (see participant 20 and 7 above), and in

relatively high satisfaction scores (4.41 across the whole task).

Major Findings

The main finding from this task was that the search algorithm for this particular search can be

misleading. Participants expect a ‘partial’ match, no matter what position that match is in (first or

second search term), and can make the wrong conclusion that an item does not exist if the desired

order item does not display in the list.

170.314(a)(2) – Drug-drug and Drug-Allergy Interaction Checks

Test contexts

Tasks 8 and 9 were included to test whether participants would be able to understand drug-drug (task

8) and drug-allergy (task 9) warnings and would be able to cancel orders successfully after receiving

them. These tasks were part of a longer sequence of tasks dealing with finding a good treatment plan

for the patient. This started with discontinuing an existing prescription (or medication ‘order’, task 7),

trying to place two orders or prescriptions that both resulted in interaction alerts (Tasks 8 and 9),

looking up a historical medication and activating that prescription (Tasks 10 and 11).

Task 8 and 9 – Order new prescription, receive interaction warning, and cancel

Analysis Notes



Task Description

In task 8 participants are asked to order/prescribe amoxicillin which results in a drug-allergy warning for

the patient. In task 9 an order/prescription for Cipro results in a drug-drug interaction warning. Here we

will analyze the reaction to the interaction warnings.

Risk Analysis

High. Creating alerts can help clinicians to recognize potential interaction issues for patients and allow

them to address those issues. However, alerts must be used with care as too many can lead to alert

fatigue, resulting in clinicians missing important alerts due to the possibility of false alarms.

Scoring Notes


of 77


Exclusions

None


Table 22. Usability Data for Tasks 8 and 9 – Interaction Warnings

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

8 Acknowledge drug-allergy


17 100% 1 14.13

(7.09) 7.39 1.91

.35

(.7)

4.47

(1.01)

9 Acknowledge drug-drug


17 94.12% 1.44 18.31

(8.03) 13.40 1.37

.5

(.73)

4.25

(.86)

Errors Table 23. Participant Errors for Tasks 8 and 9 – Interaction Warnings


Task 8

6, 6, 5, 5, 06 Medium

Participant misinterprets the logic of the controls in the warning window. “OK” in the window means ‘okay to order anyway = override’, but clearly many participants think it simply means ‘ok’ to having seen the alert window. After making the initial mistake they correctly say ‘no’ to a secondary window that asks whether it is okay to place the order despite the interaction warning. All of these participant appear stunned by this order of events, and one mistakenly says ‘yes’ to the second window. Some of them make the same mistake a number of times in a row, and again in the next task which is structurally similar to the first one.

Task 9

15, 15, 20,

9, 13, 06,

06, 6, 7, 15,

Medium

Participant misinterprets the logic of the controls in the warning window. “OK” in the window means ‘okay to order anyway = override’, but clearly many participants think it simply means ‘ok’ to having seen the alert. After making the initial mistake they correctly say ‘no’ to a secondary window that asks whether it is okay to place the order despite the interaction warning. All of these participant appear stunned by this order of events, and one mistakenly says ‘yes’ to the second window. Some of them make the same mistake a number of times in a row, and again in the next task which is structurally similar to the first one.

14 Severe Overrides warning by mistake

of 77


Effectiveness

All participants performed task 8 successfully, during task 9 one participant gets tricked into overriding

the interaction alert. As noted above in the error analysis, many participants are seen stumbling across

the logic of the button controls in the alert window. To click “OK” in this window means “Override”, and

“Cancel” means “Cancel Order/Prescription”. However, many participants interpret these to be

standard “OK” and “Cancel” buttons as seen in many pop-up windows that simply allow to acknowledge

the warning. After clicking “OK” a second window asks whether ordering the drug is the intended

action, with “Yes” being the default choice. Many of the participants were stunned by this sequence

and repeatedly went through it without quite understanding what they needed to do to cancel (‘cancel’

the order on the order screen.) However, in all but one case, the orders eventually were cancelled

correctly. One participant got confused enough to override the warning. Furthermore, experience with

this task did not seem to help matters. In fact, more mistakes were made during the second pass

through the interaction alert than the first one (see error table above).

Efficiency

These tasks were performed quite efficiently. Step deviation ratios were 1 and 1.44 and time deviation

ratios were 1.91 and 1.37.

Participant Comments


Table 24. Participant Comments for Tasks 8 and 9 – Interaction Warnings


6 "Cancel" on warning should cancel entire Rx, instead of going back to Rx to cancel.

Satisfaction

Although participants stumbled over the glitch in the logic of the warning window, they found these two

tasks easy to perform (task 8 = 4.47, and task 9 = 4.29).

Major Findings

In general, participants had little difficulty with this task, however, given that the whole purpose of the

alerts is to present a clear and simple warning, as a mistake in this context can be associated with a

high patient safety risk, one might want to consider changing the flow of the alert windows, or at least

the name of the buttons to more clearly communicate their meaning. OK = Override, Cancel = Cancel

Order. If Override is selected, then OK would be the default on the next warning message. If Cancel it

selected, then the order should be canceled without having the user do this step (as it is currently

implemented).

of 77


170.314(a)(6) – Medication List, also 170.314(a)(1) – CPOE - Medication Orders

Test contexts

These tasks were part of a longer sequence of tasks dealing with finding a good treatment plan for the

patient. This started with discontinuing an existing prescription (or medication ‘order’, task 7), trying to

place two orders or prescriptions that both resulted in interaction alerts (Tasks 8 and 9), looking up a

historical medication and activating that prescription (Tasks 10 and 11). All of these tasks also support

the maintenance of the patient’s medication list, by accessing, adding and changing items on the list.

Task 7 – Access current prescription/order/medication list item and change entry

Analysis Notes


170.314(a)(1) – CPOE

170.314(a)(6) – Medication List

Task Description

In task 7 participants were asked to find an existing prescription for Levaquin and discontinue it.

Risk Analysis

High. Maintaining an accurate medication list and prescriptions has a direct impact on the medications

taken by a patient. Inaccurately prescribed medications could have a high patient safety risk.

Scoring Notes


Exclusions

None


Table 25. Usability Data for Task 7 – Lookup medication and change

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

of 77


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


17 94.12% 1.29 19.01

(6.02) 13.36 1.42

.31

(.47)

4.94

(.25)

Find prescription

17 94.12% 1.5 6.84

(3.31) 4.23 1.62

.06

(.25)

4.88

(.33)


17 94.12% 1.19 12.18

(4.97) 9.12 1.33

.25

(.45)

4.94

(.25)

Errors Table 26. Participant Errors for Task 7 - Lookup medication and change


20, 15, 17 Minor Navigation

20 Severe Delete instead of inactivate

12 Medium Mistakenly indicated that medication was effective (it should have been labeled as ineffective)

13, 8 Minor Explore control

Effectiveness

All but one participant performed this task successfully. The one participant who failed indicated in her

comments that she was not performing this general function anymore, but had her staff do it for her. All

the other participants are very familiar with this task. The only effectiveness issue with this task is with

filling out the DC reason. If the participant fills out the reason, they are asked again in another window

whether the medication was effective (this second response is what gets logged in the database).

Because of this, several of the participants had learned to disregard filling out the field in the first dialog,

and simply answer the question with ‘no’ in the second.

Efficiency

Since this is a well-practiced task, the participant performed it with few extra steps (overall step

deviation ratio was 1.29), and with little extra time (overall time deviation ration was 1.42).

of 77




Table 27. Participant Comments for Task 7 - Lookup medication and change


5 Done plenty of these

20 Have not used this aspect of the program since e prescribing

9 I accidentally clicked end task before completing it. I will perform this task next.

Satisfaction

The high satisfaction ratings reflect the familiarity with this task (4.94).

Major Findings

In general, participants had no difficulty with this task. They knew how to navigate to the medication

list, look up existing medications and knew how to discontinue it. Additionally, some users had found a

way to make the double-dare of indicating the ineffectiveness of the medication more efficient.

Task 10 and 11 – Access historical prescription/order/medication list item and change

entry

Analysis Notes


170.314(a)(1) – CPOE


Task Description

In task 11 participants were asked to find a previous prescription for Nitrofurantoin and re-activate it.

Risk Analysis



Scoring Notes


of 77


Exclusions

None


Table 28. Usability Data for Task 10 and 11 – Lookup historical medication and change

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

10 Look up medication history

17 100% 1.64 28.28

(16.13) 13.25 2.13

.87

(1.15)

4.47

(.74)

Find medication

17 100% 1.33 20.42

(5.25) 3.25 2.14 0

4.47

(.64)


21.34

(13.6) 10.00 2.13

.87

(1.13)

4.53

(.64)

11 Change med list (make active)

16 80% 1.32 10.38

(8.48) 4.99 2.08

.18

(.4)

4.5

(.76)

Errors

Table 29. Participant Errors for Task 10 and 11 – Lookup historical medication and change


Task 10 Lookup historical med and details

16, 14, 12,

8, 15, 17, 5,

21, 05

Minor Control exploration = these participants were looking for the ‘details’ of the prescription and tried double-clicking the prescription list entry.

Effectiveness

All participants found the list of historical medications easily and without navigation errors. Finding the

details of the prescription was slightly more difficult. Most participants seemed to expect that double-

clicking the list entry would open up a window with more details and options (see error table and

comment 21 below). After some exploration all participants found a way to look at the details. All

of 77


participants easily figured out how to renew (or re-activate) the prescription, but some eventually failed

this task because they could not figure out how to electronically submit (see discussion of e-prescribing

below).

Efficiency

Finding the historical medication was done quickly (step deviation ratio 1.3, time deviation ratio 2.14),

finding the details was slightly less efficient (step deviation ratio 1.8), but making the change was also

done quickly and easily (step deviation ratio 1.32).



Table 30. Participant Comments for Tasks 10 and 11 – Lookup historical medication and change


5 This was my first time to look at a prior prescription and details. It wasn't really hard to do. With just a year or so of Practice Partner I haven't run into the need, but will in the future.

21 I would think double clicking the previous prescription would pull up the detail screen.

12 would be easy to have a botton with rapid access instead of having to have to edit the prescription

Satisfaction

Satisfaction ratings were fairly high (4.47, 4.53, and 4.5), and several participants who had not used

this function before verbally expressed that they liked finding past medication history in this manner

(see comment 5 above).

Major Findings

Participants had little difficulty with this task. The only slight stumble was the expectation to double-

click for opening more details about the medication. As mentioned before, it might be a good idea to

review the use of double-clicking and right-clicking for consistency across all functions of Practice

Partner. Our participants seemed often confused about when one of the methods would or would not

work and what their results would be. Generally, double-clicking was tried many times, right-clicking

was difficult to discover.

Task 8 and 9 – Enter new medications (receive interaction alert and cancel)

Analysis Notes


170.314(a)(1) – CPOE

of 77



Task Description

In task 8 and 9 participants were asked to enter two new medications. After receiving interaction alerts

they were supposed to cancel these prescription orders. Here only the first step, entering the

medications will be evaluated. The Alerts are evaluated above in the section dealing with drug-drug

and drug-allergy interaction alerts.

Risk Analysis



Scoring Notes


Exclusions

None


Table 31. Usability Data for Tasks 8 and 9 – Add new medication order

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

8 Write prescription

17 100% 1.57 58.78

(30.38) 27.70 2.12

.06

(.24)

4.41

(.94)

9 Write prescription

17 94.12% 1.7 38.33

(23.4) 17.23 2.23

.13

(.34)

4.44

(.63)

Errors

Table 32. Participant Errors for Task 8 and 9 – Add new medication order


18, Task 8 Medium Picks wrong medication by accident

20, Task 9 Medium Misspells name of drug

of 77


Effectiveness

The medication order is filled by all participants successfully in task 8 and all but one participant in task

9. During task 8, four participants fill out the order by hand (rather than using a template); and during

task 9 two participants use the manual method. When asked about this later, the participants

commented that they were not sure that any changes to the template (Frequency, for example) were

saved, because the name of the template did not appear changed (see also user comment 9 below).

Using the template can save participants many steps and also helps with invoking the alert checks.

Triggering the alerts sometimes fails with manually entered drugs that don’t trigger an NDC number.

The one task failure came from the participant who does not use the medication list (she has assistants

helping her with that). Another issue appeared when users misspell the names of medications. When

they decide to change the search term and use the keyboard shortcut to start the search (hit enter), a

previous selection in the previous result list becomes selected. This could lead to mistaken selections

of medications. It happened once in our case.

Efficiency

Medication order entry is a well-practiced task and most participants were able to fill the prescriptions

without error and quickly. Step deviation ratios and time deviation ratios were relatively small.



Table 33. Participant Comments for Task 8 and 9 – Add new medication order


Task 8

4 can program remember trends for individual users

5 No problems.

6 Rx field busy. Layout does not flow well. NDC?

20 we are now e scribing

21 The NDC number did not purport in when I put in the medication. Therefore, I could not send in the medication and did not receive any allergy warning.

Task 9

5

When putting in a template for a drug and modifying the template to reflect a different dosage schedule, the top of the template remains unchanged. To me this is a bit confusing for the user, and I think it would be good to know the dosing has changed on the whole template.

9 This time I searched for the medication and the NDC field was filled out.

of 77


Satisfaction

The relatively high satisfaction ratings reflect the familiarity with this task (4.41 and 4.91).

Major Findings

In general, participants had little difficulty with this task. There seems to be some uncertainty about the

result of selecting prescription templates and modifying them. An indication in the template name,

showing that this application of the template has been modified, might help address this issue. If it

were addressed, users could fill prescriptions more efficiently (fewer clicks), and interaction alerts would

get triggered with more accuracy. The other issue is with the search screen for templates and

medication names. When the search field gets put in focus a second time, the default ‘enter’ action

should be the ‘Search’ button, rather than the ‘OK’ button for the list item. Making this mistake could be

a safety concern, because a user could inadvertently select the wrong medication for a patient (even

though this would not be very likely, because they would review the medication on the next screen.

This problem should be addressed though.

170.314(b)(3) – Electronic Prescribing

Test context

Electronic Prescribing was tested as part of task 11. After renewing the Nitrofurantoin prescription,

participants were asked to electronically submit it to the patient’s default Pharmacy.

Task 11 – Change med list and e-prescribe

Analysis Notes


170.314(a)(1) – CPOE


170.314(b)(3) – Electronic Prescribing

Task Description

After renewing the Nitrofurantoin prescription, participants were asked to electronically submit it to the

patient’s default Pharmacy.

of 77


Risk Analysis

Medium. Being able to transmit prescriptions electronically rules out some misinterpretation and can

speed up the process of getting prescriptions but does not directly lead to patient safety issues.

Scoring Notes

One participant had to be excluded from scoring because he could not finish the task due to long

system delays.

Exclusions

Participant 6 – long system delays


Table 34. Usability Data for Task 11 – Fill prescription and e-prescribe

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


16 80% 1.65 40.28

(23.84) 28.41 1.41

.23

(.43)

4.5

(.76)


16 80% 1.32 10.38

(8.48) 4.99 2.08

.18

(.4)

4.5

(.76)

e-prescribe 16 80% 2.17

34.13

(19.63) 23.42 1.46

.08

(.29)

4.5

(.76)

Errors Table 35. Participant Errors for Task 11 – Fill prescription and e-prescribe


21, 14, 17,

20 Medium Print instead of transmit

Effectiveness

Four participants did not successfully complete this task. They overlooked the control for setting the

transmit method and kept it in print mode. While this does not preclude the prescription from being

executed, it ultimately did not meet the goal set in this task.

of 77


Efficiency

The participants who did set the transmit mode, did so fairly efficiently (overall step deviation ratio was

2.17), and with little extra time (overall time deviation ration was 1.46).



Table 36. Participant Comments for Task 11 – Fill prescription and e-prescribe


5 Not bad …

Satisfaction

The participants who finished this task successfully were fairly satisfied with it (satisfaction rating for e-

prescribing was 4.5).

Major Findings

The label for the field that allows to set the transmit method might be misleading, which may have led to

the somewhat low success rate. Changing it to ‘transmit’ might help to point users in the right direction.

170.314(a)(8) – Clinical Decision Support

Test context

Clinical decision supports were tested in two different tasks. Task 1 looked at the usability of the health

maintenance alerts. Task 2 looked at the functionality of the new InfoButton.

Task 1 – Select patient and review health maintenance alerts

Analysis Notes



Task Description

Participants were asked to select their patient and review any health maintenance alerts.

of 77


Risk Analysis

Medium. While health maintenance alerts are a great tool for preventative medicine, they do not

usually present information that is relevant to immediate patient safety.

Scoring Notes


Exclusions

None.

Table 37. Usability Data for Task 1 – Select patient and review health maintenance alerts

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1


17 100% 1.38 50.70

(15.52) 33.01 1.54

0.71

(0.59)

4.65

(0.61)

Select Patient

17 100% 1.68 34.65

(15.71) 20.01 1.73

0.71

(0.59)

4.82

(0.39)

Review Alerts

17 100% 0.62 16.05

(6.68) 13.00 1.24 0

4.44

(.73)

Errors Table 38. Participant Errors for Task 1 – Select patient and review health maintenance alerts


NA NA NA

Effectiveness

All participants successfully completed this task, and no errors were recorded.

Efficiency

Participants did this task quickly (overall step deviation ratio was 1.38, and overall time deviation ratio

was 1.54). They took a little time to look at the alerts, but many specialists voiced that they would not

take the time to read these alerts as they perceive them to be more helpful to primary physicians.

of 77




Table 39. Participant Comments for Task 1 – Select patient and review health maintenance alerts


5 For an acute visit I wouldn't normally look at these, but it may be a good thing to see the reminder to work toward an annual visit if it is overdue.

20 primary care uses alerts Specialists may not

12 To much information in the alert box. Needs to be more selective

7 great reminder

Satisfaction

The satisfaction rating for reviewing the alerts was 4.4. It was perceived to be easy, however some

participants mentioned that it was too much information, or that they would not take the time to read

alerts like this.

Major Findings

This was an easy task; however some participants questioned the utility of such alerts, while others

found them interesting and useful.

Task 2 – Use the InfoButton to review treatment options for patient’s problem.

Analysis Notes



Task Description

Participants were asked to use the Infobutton to research treatment options for the patient’s main

problem.

Risk Analysis

Medium. While updated clinical information is a great tool for providers, not having access to it, or

being able to find it, does not usually present information that is relevant to immediate patient safety.

Scoring Notes


of 77


Exclusions

None.

Table 40. Usability Data for Task 2 – Use the InfoButton

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


17 82% 1.25 79

(47.97) 26.93 2.93

.43

(.76)

4.36

(.84)


42.15

(43.49) 10.58 3.98

.36

(.74)

4.43

(.94)

InfoButton Lookup

17 82% .95 36.85

(23.2) 16.35 2.25

.07

(.27)

4.43

(.84)

Errors Table 41. Participant Errors for Task 2 – Use the InfoButton


15, 8, 8 Minor Navigation error

20, 20, 7, 5, Minor Double click or single click instead of right-click

Effectiveness

Three participants failed to complete this task because they could not figure out on their own how to

navigate to the InfoButton and gave up. The training powerpoint had introduced them to using the

right-click to access the context menu, but many participants had forgotten about this by the time they

got to the task (some referred back to the training material to help them remember – this was scored as

success as this represents typical user behavior when new tasks are learned). As the error table

shows below, discovering the right-click option made this task error prone. However, once this had

been discovered, several participants found this option useful and easy to use.

Efficiency

As the time deviation ratio shows (3.98), navigating to the Infobutton content, was difficult to discover –

even after having received training on it. Navigating the vendor site also indicated some issues (some

of 77


participants had a hard time discovering how the shortcuts on the third party vendor site worked, but

analysis of third party vendor content is beyond the scope of this report.)



Table 42. Participant Comments for Task 2 – Use the InfoButton


4 Would you have access to this from the note?

5 Just initially remembering the right click to get there was the only issue. Once I've done it a time or two it will be easy. Good information readily found.

20 like UpToDate

21 UpToDate is a little less useful for my specialty (ENT) than it is for others.

13 very helpful and easy to use

15 just will take time to become habit

18 Intuitve

5 Once I knew how to find the infosearch button, it was easy.

7 great ooption

9 Only able to be used with up to date subscription.

Satisfaction

The satisfaction rating for the overall task was 4.36. It was perceived to be easy, and a number of

participants welcomed this integration of information lookup. A few participants commented that the

content at the site did not correspond with their needs.

Major Findings

Right-click context menus can be an efficient way to navigate, but they are difficult to remember, and

even more difficult to discover. Consider embedding a link or an icon that can be clicked to navigate to

the InfoButton content, instead.

of 77


of 77


List of Appendices

The following appendices include supplemental data for this usability test report.

Appendix A: Demographic Questionnaire

Appendix B: Administrator Script and General Instructions

Appendix C: Informed Consent Form

Appendix D: Usability Test Instructions

Appendix E: Experimenters’ Short Biograph

Appendix F: Participant Instructions for Each Task

Appendix G: End-of-Task Questionnaires

Appendix H: System Usability Scale (SUS) Questionnaire

of 77



of 77



Before participant arrives:

o Start remote desktop application and log into test environment

o Wait with logging into applications until Participant is ready (would time out)

o Start Morae Recorder, but don’t click the red button yet

o Open the training presentation; leave in presentation mode

When participant arrives:

o Welcome Participant

o Make sure participant has filled out Informed Consent sheet, make sure address is legible.

o Make sure participant has filled out Demographics sheet

o Get Participant started on training presentation

Verbal Instructions:

- “Thank you for participating in this usability study today. During this study will be doing

some tasks that are very familiar to you. But you will also perform some new tasks. To

familiarize yourself with these new functions, please walk through this short PowerPoint.

- You will notice that our system is similar to the one you are using, but it is not the same.

Some of the configurations will be slightly different from what you are used to, simply

because this is a somewhat limited test system.

- The PowerPoint will also tell you more about participating in a usability study, and show

you how to interact with the recording software.

- You can always ask me questions during these training slides.”

When participant is done with training presentation:

o Start Morae (press red button)

o Log into applications


- Remember that this system may be configured slightly different from what you are used

to.

- You start by clicking this button here. You can also move this window around, so that

you can see the system better. If you like, you can use these printed instructions and

these Navigation Instructions to follow along.

of 77


- When you have started, I won’t be able to answer any more questions. Do you have any

questions before you start?”

of 77



of 77



Introduction to the Study

• Thank you for participating in this usability study.

• Your participation is very valuable and critical for certification.

• Usability testing is required by ARRA 2; vendors cannot certify their EHR’s unless they have been

evaluated by our end-users

• Your feedback will help us improve the usability of Pratice Partner.

• During the study, you will be performing a few routine tasks using Pratice Partner, most of which will be

very familiar to you.

• This presentation provides a review of the routine tasks.

• The following screens present an overview of features in self-guided presentation mode.

• Please take 10 minutes to familiarize yourself with these new functions.

• You can use the page up and down functions to navigate through the presentation.

• If you want to see a part again, you can always back up to previous screens by using the Page Up keys

on your keyboard.

Functionality Review

Study Instructions

• Thank you for participating in this study. Your input is very important.

• Our session today will last about 30 minutes. During that time you will use Pratice Partner Version 11 to

perform tasks.

Objective of the Study

• The purpose of the study is to uncover:

o Areas where the application performs well; that is, effectively, efficiently, and with satisfaction

o Areas where the application does not fully meet your needs.

• Please note that we are not testing you, we are testing the application. If you have difficulty, this means

only that something needs to be improved in the application.

• The study administrator will be here in case you need specific help with the procedures, but the

administrator will not provide help in how to use the application.

Starting the Study

• The system will show an instruction window. The window displays over the EHR screens you will be

using to complete the task.

of 77


• Click the Start button to start the study.

• Within the instruction window, the system provides instructions for completing the first task.

• After reading the on-screen instructions, click the Start Task button.

• The instruction window will collapse vertically so that it will not obscure the application.

• You can click the Show Instructions button in the instruction window at any time to expand the

window and read the instructions again.

Performing the Task

• Using the EHR screens, begin performing the task. Please try to complete the tasks on your own

following the on-screen instructions very closely.

• You can also move and reposition the instruction window.

• If you place the mouse cursor focus in the instruction window, reposition the mouse cursor back in the

application window before continuing the task.

Indicating Task Completion

• When you are satisfied that you have completed the task, or when you have completed as much as you

can, click the End Task button.

• The system will open a short survey with questions about the task you have just completed.

• After answering the Participant Survey questions for the task, click the Done button.

• The system will present instructions for the next task.

• Between some tasks, the study administrator may adjust the application to start the next task at the

right location.

Completing the Study

• After you complete the Participant Task Evaluation for the final task, the system will open a System

Usability Scale Questionnaire to enable you to record your overall impressions of the application

of 77


including all the tasks you performed. Note that you will need to scroll vertically to view and answer all

of the questions.

• After completing the questionnaire, click the Done button to complete the study.

• The instruction window shows a message indicating that the study is complete. Tell the study

administrator that you have completed the study.

• The administrator will click the OK button and save your work.

of 77


Appendix E: Experimenters’ Short Biographies

Marita Franzke received her Ph.D. in Cognitive Psychology after completing a dissertation on Exploration-

based Learning of Display Based User Interfaces at the University of Colorado. She has worked in the field of

Human Computer Interaction since 1991, conducting research on usability and adoption of new end user

technologies in the fields of telecommunication (U S WEST Advanced Technologies, Media One Labs, and

AT&T Broadband Labs), Educational Software (Pearson Knowledge Technologies) and EMR Technologies

(McKesson Provider Technologies). She has 21 years of experience in the field of user research and one year

experience in the field of EMR usability.

Gergana Pavlova joined McKesson Practice Partner over 5 years ago as a technical writer. A year ago she

joined the Practice Partner Product Management team to help with the functional design requirements and

user interface designs of the Meaningful Use Stage 2 Incentive Program capabilities. She has an MS in

English and certificates in User-Centered Design in Business Analysis from the University of Washington.

of 77



Introduction

You will be treating a 43 year old female patient today.

Some of her data are already in the system. She has been showing signs of a UTI since last week. Her current prescription does not seem to be effective.

You will be asked to perform 11 short tasks using the EHR provided to you.

For each task, we will provide you with specific instructions.

When you are ready, please click the START button.

Task 1

Task Instructions:

Your patient comes to the office with symptoms of a UTI. To get started ...

1) Find your patient in the patient list.

2) Select your patient.

3) Read the patient alerts.

4) Close the patient alert window.

Task 2

Task Instructions:

Your patient has been diagnosed with Urinary Tract Infection (UTI). You want to get a

quick glimpse at the newest treatment options for the disease.

1) Find the information about treatment options for cystitis.

2) You are done when you have found this information.

Task 3

The patient also complains of mild mid back pain. You need to check if the patient has

not developed a kidney infection.

1) Place a STAT order for URIN CULTURE.

of 77


Task 4

Task Instructions:

You also notice that the patient already has a repeated order for a blood glucose check

coming up. You decide to change that to STAT to make sure her infection has not

adversely affected her blood glucose levels.

1) Find the order “Glucose, Fasting".

2) Change the date to STAT.

Task 5

Task Instructions:

You want to order an image of her kidneys. You see there is an existing order for an

image.

1) Find the existing order for CT of Abdomen.

2) Then cancel it.

Task 6

Task Instructions:

You think that an ultrasound is a better way to image her kidneys.

1) Order a STAT Renal Ultrasound.

Task 7

Task Instructions:

You get ready to come up with a treatment plan. Review the existing antibiotics

prescription.

1) Find the prescription for Levaquin.

2) Cancel the prescription because it does not seem to be working.

Task 8

Task Instructions:

You will now order a different antibiotic to treat your patients’ UTI. The prescription will be

sent electronically to the patient’s pharmacy (Walgreens).

1) Enter an order for Amoxicillin, 875 mg, BID 7 days.

of 77


2) If you get a warning, cancel the order.

Task 9

Task Instructions:

Now try to order a different antibiotic. The prescription will be sent electronically to the patient’s pharmacy (Walgreens).

1) Enter an order for Cipro, 250 mg, oral, once a day for 7 days.

2) If you get a warning, cancel the order.

Task 10

Task Instructions:

The patient notices that you can’t find an antibiotic that works, so she tells you that she

took some kind of antibiotic that started with ‘nitro’ in the past, without any problems.

1) Find out which antibiotic was prescribed in the past without incident, then ...

2) Review the details about this prescription.

3) When you have reviewed the details you are done with this task.

Task 11

Task Instructions:

You decide to prescribe NITROFURANTOIN because it has worked for this patient in the past.

1) Activate the discontinued NITROFURANTOIN prescription, then ...

2) Send the prescription electronically to the patient’s pharmacy (Walgreens).

of 77



1 - Select Patient and Alerts

1. Finding and selecting the patient was ...

very difficult 1 2 3 4 5 very easy

2. Reading the Health Maintenance alerts was ...


3. Overall this task was ...


4. Do you have any additional comments about Health Maintenance alerts?

Comments:

2 - InfoButton

1. Finding the InfoButton control was ...


2. Finding the Treatment Options was ...




4. Do you have any additional comments about the InfoButton?

Comments:

3 - Order Lab

1. Finding the correct screen to accomplish this task was ...


2. Ordering the Lab was ...


of 77




4. Do you have any additional comments?

Comments:

4 - Access and Modify Lab Order



2. Finding the existing lab order was ...


3. Changing the order to STAT was ...


4. Overall, this task was ...


5. Do you have any additional comments about this task?

Comments:

5 - Access and Cancel Image Order



2. Finding the existing image order was ...


3. Canceling the order was ...




of 77



Comments:

6 - Place Image Order



2. Placing a new image order was ...





Comments:

7 - Access and Inactivate Med Order



2. Finding the existing prescription was ...


3. Inactivating the prescription was ...





Comments:

8 - Order Med (Drug-Allergy)



of 77


2. Entering the prescription was ...


3. The interaction warning was easy to understand ...

disagree 1 2 3 4 5 agree




Comments:

9 - Order Med (Drug-Drug)










Comments:

9 - Order Med (Drug-Drug)





of 77







Comments:

10- Medication History

1. Finding the screen to accomplish this task was ...


2. Finding the prescription details was ...





Comments:

11 - Change Med List

1. Activating the prescription was ...





Comments:

of 77



This questionnaire was presented to the participant after the completion of all tasks and task-based

questionnaires.

Thank you for participating in this study. The following questions address your experience with all the tasks in this study.

1. I think that I would like to use this system frequently

Strongly Disagree 1 2 3 4 5 Strongly Agree

2. I found the system unnecessarily complex


3. I thought that the system was easy to use


4. I think that I would need the support of a technical person to be able to use this system


5. I found the various functions in this system were well integrated


6. I thought there was too much inconsistency in this system


7. I would imagine that most people would learn to use this system very quickly


8. I found the system very cumbersome to use


9. I felt very confident using the system


10. I needed to learn a lot of things before I could get going with this system


of 77


Appendix I: System and Patient Scenario for Physicians’ Tasks

Setting: Ambulatory

Participant Group: Physician/NursePractitioner

PATIENT SETUP

Name: XXX Blue

Age:43

DOB: 02/03/1970

Gender: female

123 XXX, Denver, CO 80238

Telephone Number: 303-329-6802

Usual provider: Physician

Preferred Pharmacy: Walgreens, NPI number 1003086380

In patient’s chart:

Allergy List:

Penicillin Potassium

moderate

skin rash

onset date: 01/01/1995

Problem List:

Asthma:

o onset date: 1995

o entered on the Other Problems tab

o apply HM template per patient after creating the proble m

Diabetes:

o onset date: 2000

o entered on the other problems tab

o apply HM template per patient after creating the proble m

Acute UTI

o current date (predate to a date next week)

o listed as a Major Problem

Med List:

Levaquin 5007D

o Date 09/07/2013

Azmacort, MDI 2PUFFS Q6H X 30 D

o Date 01/01/2000

of 77


Inactive on the Med History:

Nitrofurantoin 50 MG

o date 01/01/ 2010,

o discontinue 02/02/2010,

o therapy completed, effective

Orders on the current electronic chart:

Glucose, Fasting,

o every 30 days,

o start date some time back

CT of Abdomen, STAT, predate (or do within 7 days)

SYSTEM SETUP

Order Names

Create a new order name (“Renal Ultrasound”) and add it to an Order Tree

Health Maintenance Template(s)

Apply a HM template for 40-49 Year Old Females, do I need to add a special procedure?

Apply a HM template for patients with Asthma, do I need to set a special procedure?

Select Enable Active Health Maintenance Reminders check box on Special Features Records 6 tab

EHR Usability Test Report Part II

EHR: Practice Partner® Ver. 11

User Role: Nurse

Setting: Ambulatory

Applications Tested

Practice Partner® Ver. 11

Dates

Usability Test: September 9-12, 2013

Report: October 2, 2013

Prepared by


Gergana Pavlova, Business Systems Analyst, McKesson Provider Technologies



Westminster, CO 80021 (720) 356-7711

[email protected]



of 54

EHR: Practice Partner Ver. 11, Part II – Nurse – Ambulatory October 2, 2013

Contents

Tables ............................................................................................................................................................... 4


Major Findings and Areas for Improvement .................................................................................................... 8

Introduction ...................................................................................................................................................... 11

Intended Users............................................................................................................................................. 11

Method ............................................................................................................................................................ 11

Participants .................................................................................................................................................. 11

Study Design ............................................................................................................................................... 13

Procedures .................................................................................................................................................. 14

Test Locations.............................................................................................................................................. 15




Data Scoring ................................................................................................................................................ 16

Results ............................................................................................................................................................ 19



170.314(b)(4) – Clinical information reconciliation ........................................................................................ 21

Task 1 – Open incoming CMA .................................................................................................................. 21

Task 2 – Perform Medication Reconciliation ............................................................................................. 24

Task 3 – Perform Allergy Reconciliation ................................................................................................... 26

Task 4 – Perform Problem Reconciliation ................................................................................................. 28

170.314(a)(7) – Medication allergy list ......................................................................................................... 30

Test contexts ............................................................................................................................................ 30

Task 5 – Look up and edit existing allergy list entry .................................................................................. 30

Task 6 – Create a new allergy list entry .................................................................................................... 33

Task 7 – Confirm review of allergy list ...................................................................................................... 34



of 54





Appendix E: Experimenters’ Short Biographies ............................................................................................ 46




Appendix I: System and Patient Scenario for Nurse’s Tasks ........................................................................ 53

of 54


Tables

Table 1. Nurses' Tasks: Meaningful Use Criteria, Risk, and Application Tested ................................................. 5

Table 2. Result Summary by Meaningful Use Criterion and Task ...................................................................... 7


Table 4. Nurses’ Tasks – same as Table 1, repeated here for easier reference .............................................. 13



Table 7. Result Summary by Meaningful Use Criterion and Task same as Table 2, repeated here for easier

reference ......................................................................................................................................................... 19

Table 8. Usability Data for Task 1 – Open incoming CMA .............................................................................. 22

Table 9. Participant Errors for Task 1 – Open incoming CMA ......................................................................... 22

Table 10. Participant Comments for Task 1 – Open incoming CMA ................................................................ 23

Table 11. Usability Data for Task 2 – Perform medication reconciliation ......................................................... 24

Table 12. Participant Errors for Task 2 – Perform medication reconciliation ................................................... 24

Table 13. Participant Comments for Task 2 – Perform medication reconciliation ............................................ 25

Table 14. Usability Data for Task 3 – Perform allergy reconciliation................................................................ 27

Table 15. Participant Errors for Task 3 – Perform allergy reconciliation .......................................................... 27

Table 16. Participant Comments for Task 3 – Perform allergy reconciliation ................................................... 27

Table 17. Usability Data for Task 4 – Perform problem reconciliation ............................................................. 29

Table 18. Participant Errors for Task 4 – Perform problem reconciliation ........................................................ 29

Table 19. Participant Comments for Task 4 – Perform problem reconciliation ................................................ 29

Table 20. Usability Data for Task 5 – Lookup and edit existing allergy list entry ............................................. 31

Table 21. Participant Errors for Task 5 – Lookup and edit existing allergy list entry ........................................ 31

Table 22. Participant Comments for Task 5 – Lookup and edit existing allergy list entry ................................. 32

Table 23. Usability Data for Task 6 – Add an allergy to the allergy list ............................................................ 33

Table 24. Participant Errors for Task 6 – Add an allergy to the allergy list....................................................... 33

Table 25. Participant Comments for Task 6 – Add an allergy to the allergy list ............................................... 34

Table 26. Usability Data for Task 7 – Confirm review of allergy list ................................................................. 35

Table 27. Participant Errors for Task 7 – Confirm review of allergy list ........................................................... 35

Table 28. Participant Comments for Task 7 – Confirm review of allergy list .................................................... 35

of 54


Executive Summary A usability test of Practice Partner® VER. 11 was conducted with 24 nurse/medical assistant end users

of Practice Partner during the week of September 9-12, 2013 by Marita Franzke and Gergana Pavlova

of the McKesson Provider Technologies. The testing was conducted on two different customer sites in

York, ME and Pensacola FL. The purpose of this test was to test and validate the usability of five

Meaningful Use Criteria as implemented in the functions of the Practice Partner® VER. 11. The Office

of the National Coordinator for Health Information Technology (ONC) requires vendors to test 7

Meaningful Criteria outlined in the test procedures for §170.314(g)(3) Safety-enhanced design1 for

ambulatory products. The criteria focused on the performance of physician and administrator tasks will

be discussed in two other reports. Those reports detail the results of usability studies involving

physicians and system administrators, as those clinical tasks are typically conducted by those user

roles.

During the usability test, 24 nurses and clinical staff served as participants; they were all familiar with

previous releases of the Practice Partner EHR. In this study, they used this software in 7 simulated, but



Below is a list of all tasks employed, which were chosen to evaluate the usability of 2 of the 7

Meaningful Criteria outlined in the test procedures for §170.314(g)(3) Safety-enhanced design for

ambulatory use. Table 1 below lists the nurses’ tasks in order of performance, the linked Meaningful

Use criteria, and a risk rating for performing each task.

Table 1. Nurses' Tasks: Meaningful Use Criteria, Risk, and Application Tested


Application Risk

1 Open incoming CMA § 170.314(b)(4) Clinical information reconciliation

Practice Partner

High

2 Perform Medication Reconciliation § 170.314(b)(4) Clinical information reconciliation

Practice Partner/

Third Party

High

3 Perform Allergy Reconciliation § 170.314(b)(4) Clinical information reconciliation

Practice Partner

High


of 54


4 Perform Problem Reconciliation and finish import of CMA

§ 170.314(b)(4) Clinical information reconciliation

Practice Partner

High

5 Look up and change existing allergy in the allergy list

§ 170.314(a)(7) Medication allergy list

Practice Partner

High

6 Add an allergy to the allergy list § 170.314(a)(7) Medication allergy list

Practice Partner

High

7 Confirm review of allergy list § 170.314(a)(7) Medication allergy list

Practice Partner

High

During the 45 minute, one-on-one usability test, each participant was greeted by an administrator when

they entered the conference room. Each participant had prior experience with a previous version of the

Practice Partner ® EHR. The participants were asked to review a self-guided training module that

reviewed the new Clinical Data Reconciliation feature, and presented instructions for the usability study

and using the testing software. The administrator introduced the test, and instructed participants to

complete a series of tasks (given one at a time) using the EHR Under Test (EHRUT). During the

testing, data logging software timed the test and recorded user screen captures and audio. These

recordings were later used to score action steps, errors and task completion rates. The test

administrator did not assist the participant in completing the tasks.





Path deviations





questionnaire.







of 54


averages obtained from those task-based usability scores varied between 4.0 and 4.93, quite a bit

above the recommendation put forward in NISTIR3 7742. The NISTIR 7742 states that “common

convention is that average ratings for systems judged easy to use should be 3.3., or above.”





Table 2. Result Summary by Meaningful Use Criterion and Task

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

§ 170.314(b)(4) -- Clinical information reconciliation

1

Open incoming CMA

15 92.86 2.0 119.75

(66.38) 52.07 2.30

1.36

(.84)

4.14

(.77)

Navigate to function

15 100 2.13 43.10

(43.68) 23.13 1.86

.05

(.76)

4.0

(1.11)

Locate and open CMA 15 100 2.16

63.62

(42.83) 19.63 3.24

.86

(.36)

4.5

(.85)

Open existing patient chart 15 92.86 1.27

13.02

(7.13) 9.32 1.40 0

4.14

(.77)

2 Perform Medication Reconciliation

14 85.71 1.75 52.6

(37.75) 19.72 2.67

1.33

(.98)

4

(1.28)

3 Perform Allergy Reconciliation

10 100 1.15 23.47

(22.12) 11.08 2.11

1.2

(.63)

4.8

(.42)


9 88.89 1.25 36.96

(23.11) 11.52 3.21

.5

(.53)

4.63

(.74)

§ 170.314(a)(7) -- Medication allergy list

5 Look up and edit existing allergy list entry

14 92.86 1.46 72.72

(47.74) 34.21 2.13

1.75

(1.87)

4.71

(.47)

Find existing allergy list item

14 92.86 1.07 43.99

(25.32) 24.66 1.78

.33

(.65)

4.57

(.51)

Change allergy list item

14 92.86 2.51 32.55

(42.46) 9.55 3.41

1.69

(2.14)

4.71

(.47)

3 Schumacher, Robert M., and Lowry, Svetlana Z. (2010), Customized Common Industry Format Template for Electronic

Health Record Usability Testing. National Institute of Standards and Technology, NISTIR 7742.

of 54


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

6 Add an allergy to the allergy list

15 100 1.46 58.26

(22.59) 34.76 1.68

.07

(.26)

4.6

(.63)

7 Confirm review of allergy list

15 100 2.4 17.22

(10.68) 8.71 1.98

.07

(.26)

4.93

(.26)




This section summarizes the findings and areas for improvement at a high level. For additional detail,

please see the section "Discussion of the Findings”.

Findings Recommendations

Clinical Reconciliation

Clinical reconciliation (the import of external

medical summaries and reconciliation with

existing patient charts) is conceptually difficult

and complex in terms of the number of decisions

involved.

Two issues made this task especially difficult for

our participants: (1) Finding the xml document

with the external summary was challenging

because participants believed they had to enter

the file name rather than browsing to it.

(2) The reconciliation process was often cut

short because participants clicked the ‘Import’

button before they had reviewed all data – and

this action could not be reversed.

A few smaller issues having to do with

reconciling each list are discussed in detail in

the “Discussion of the Findings” section of the

report.

1. Training: When introducing this

functionality it should be preceded by

training that not only provides hands-on

examples and allows users to understand

the user interface, but also by a conceptual

explanation of data exchange between

practices and EHRs. Users need to

understand where external records may

result from, how to obtain and store them –

organizational policies and resulting

methods and procedures will have to be

disseminated to end users. This training

should also include training on how to

navigate within the windows file structure

(to find files for import), the effects of

merging external records with the existing

data, as well as how to use the Import

functionality.

2. Design: When launching the import

process, it may be possible to eliminate the

first dialog and immediately launch the

windows browse dialog. Users don’t need a

function to enter the file name by hand – it

of 54


will always be safer and easier to browse to

existing files. If a password needs to be

entered, intersect a simple password dialog.

3. Design: Consider disabling the ‘Import’

button until all data categories have been

reconciled.

4. Design: Launch a dialog after clicking the

‘Import’ button that asks the users to verify

that they are ready to import and merge

both data files, and gives them a chance to

cancel, in case they are not.

5. Design: Given the conceptual difficulties

with this task, one might consider

implementing this function through a

‘wizard’ like functionality, that steps users

through each decision point one-by-one,

and presents the reconciled lists at the end.

Medication Allergy List

Generally, the participants were able to find the

allergy list quickly and add items to the list.

We identified one issue concerning access to

existing allergy items. The medication allergy

list and medication list are presented in the

same window. However the buttons at the

bottom of this window mostly pertain to the

functions of the medication list. Participants

were confused about this and took a while to

discover that double-clicking opens allergies for

editing (rather than any of the buttons on the

bottom of the window.)

1. Training: Make sure that users are aware of

hidden access methods, such as double-

clicking and the right-click menu. Knowing

that these two methods are used in Practice

Partner will make it much more likely that

users will discover hidden functionality.

Specifically make sure that super users are

aware of this, so that they can disseminate

these hints to other users.

2. Design: Consider expanding the ‘Allergy’

button in the dialog into several options

“edit’ and ‘new’. Also consider grouping

buttons affecting the medication list, and

creating a divider between them and the

allergy buttons to clearly communicate

which button/function pertains to which list.

General Issues

Several of the usability issues came about

because of inconsistencies in how list items can

be edited, deleted or otherwise acted upon.

Lists that appeared in the same dialog box had

different interaction methods (click + button,

double-click + form, right-click, or checkboxes).

This created confusion in users, because it

makes it difficult to distinguish when and where

1. Review Practice Partner with respect to

inconsistencies in interaction methods on

list items and attempt to standardize.

of 54


a certain method works, and when not.

Inconsistencies like this can lead to inefficient

user behavior in the best case and failure to

discover important functions in the worst case.

of 54


Introduction The McKesson Practice Partner® EHR functions tested for this study were all accessible to the

participants in an integrated environment, as would be typical in a provider implementation. The entire

environment was software version VER. 11. The usability testing environment and tasks attempted to

simulate realistic implementations, workflows and conditions. The purpose of the study was to test and

validate the usability of the current user interface, and provide evidence of usability in the EHR Under

Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction were captured

during the usability testing.

Intended Users

Intended users of the EHRUT are mid-level clinical staff in ambulatory practices or clinics, such as

registered nurses, licensed practical nurses, or medical assistants. Users may have varying levels of

experience in the clinical setting and may have varying years of experience performing clinical functions

with the EHRUT. They can be expected to have received training on the EHRUT, and to be frequent

users (weekly or daily use). Depending on site or specialty, they may have varying levels of experience

with the tasks.

Method

Participants

A total of 24 participants were tested on the EHRUT for the nurse role in the ambulatory setting.

Participants were associated with clinics in one of two customer sites (nine were associated with site A

and the remaining 15 with site B). The participants had been recruited by IT staff in their organizations.

None of the participants had a direct connection to the development of the product. Seventeen were

medical/ophthalmic assistants, four were licensed practicing nurses (LPN), two were accredited nurses

anesthesia (COA), and one was a registered nurse (RN).

All participants had experience using the EHRUT, ranging from 9 months to 19 years. All participants

use the system for entering problem, medication, or allergy data on a daily basis. However, most

participants were not familiar with the Clinical Data Reconciliation feature which was an enhancement

with ER11. A demographics questionnaire was filled out before the session began; the questionnaire is

included as Appendix A: Demographic Questionnaire.

Nine of the participants (all from site A) were excluded from the analysis because the initial training

materials to introduce the clinical reconciliation function did not contain any conceptual explanations of

this functionality. It became clear that users needed more information about data information exchange

between practices, and methods for achieving this. It was also necessary to set expectations about how

quickly these practices might come into effect. While testing at site A, the testing sessions became

interrupted many times because the participants had too many questions about these topics. After

some discussion, the facilitators made a special attempt at addressing these issues in depth and giving

of 54


participants time to explore all their questions during the training session, which led to much better

testing sessions at site B.

Table 3 lists all 15 participants included in the analysis by demographic characteristics, such as

demographics, education, and EHR experience. Participant names were replaced with Participant IDs

so that an individual’s data cannot be tied back to individual identities.


ID Gender Age Education/Job Title Years in practice

Years using Practice Partner

Frequency of use (entering medication data)

5 f 20-30 MA 3 years 3 years Many times a day

6 f 51-60 MZ 1.5 years

1.5 years Many times a day

7 m 31-40 LPN 7 + years

3 years Many times a day

8 f 41-50 LPN 20 years 6 years Many times a day

9 f 20-30 MA 7 years 7 years Many times a week



12 f 31-40 Certified Ophtalmic Assistant

7 years 7 years Many times a day

13 f 41-50 OA 7 years 7 years Many times a day

14 f 31-40 RMA 9 years 13 years Many times a day

15 f 31-40 EMR Order Entry Specialist

18 years 15 years A few times a month

16 f 31-40 Ophtalmic surgical assistant, scribe

13 years 5 years Many times a week

17 f 31-40 RMA 12 years 4 years Missing

18 f 41-50 Medical Assitant/Team Leader

20 years 18 years Many times a day

19 f 31-40 MA 2 years 2 years Missing

A short, self-paced training PowerPoint presentation was provided to participants before the study to

review the new Clinical Data Reconciliation functionality. This training module was similar to what end-

users would usually receive when introduced to new functionality. The training also included some

explanation of the software used to capture the session data.

Participants were scheduled for 45 minute sessions which provided time between sessions for debrief

by the administrator, and to get the system ready for the next participant. A spreadsheet was used to

keep track of the participant schedule, and the participant demographics.

of 54


Study Design








ambulatory use. The current study focused on the 2 MU criteria used typically by nurses. Two separate

reports detail the findings from tasks associated with the MU criteria typically conducted by physicians

and administrators. Table 4 below cross-tabulates which task in the study of nurse/medical assistants’

tasks is associated with which MU criterion and lists the risk associated with that task.

Table 4. Nurses’ Tasks – same as Table 1, repeated here for easier reference


Application Risk

1 Open incoming CMA § 170.314(b)(4) Clinical information reconciliation

Practice Partner

High

2 Perform Medication Reconciliation § 170.314(b)(4) Clinical information reconciliation

Practice Partner/

Third Party

High

3 Perform Allergy Reconciliation § 170.314(b)(4) Clinical information reconciliation

Practice Partner

High


§ 170.314(b)(4) Clinical information reconciliation

Practice Partner

High

5 Look up and change existing allergy in the allergy list

§ 170.314(a)(7) Medication allergy list

Practice Partner

High

6 Add an allergy to the allergy list § 170.314(a)(7) Medication allergy list

Practice Partner

High

7 Confirm review of allergy list § 170.314(a)(7) Medication allergy list

Practice Partner

High



of 54



Practice Partner® VER. 11. Each participant used the system in the exact same configuration, and was

provided with the same instructions. The system was evaluated for effectiveness, efficiency and

satisfaction as defined by measures collected and analyzed for each participant:



o Number of errors

o Path deviations



Procedures

The test participant was greeted by the facilitator upon entering the conference room. Once the

participant was seated, he or she was verbally instructed and oriented to the test situation (see

Appendix B: Administrator Script and General Instructions). Each participant was asked to sign an

informed consent form (Appendix C: Informed Consent Form), and then filled out the demographics

questionnaire (Appendix A: Demographic Questionnaire). Then the participant was asked to step

through a set of self-paced training slides. Participants spent between 5-10 minutes reviewing this

presentation that introduced the Clinical Data Reconciliation functionality, the facilitator explained the

purpose for this functionality and solicited and answered any additional questions that participants had

about exchanging data between practices and EHRs. The presentation also included usability test

instructions and an introduction to the Morae recording software (Appendix D: Usability Test

Instructions). During this presentation participants had a chance to ask questions and discuss the new

feature and any issues concerning the study itself.

One test administrator per participant ran the test. Typically two sessions were happening in the same

room, but enough space was provided so that these two sessions did not interrupt or distract each

other. In some cases, additional persons (hospital officials and a product director) were present to

observe, but they did not interact with the participants. Data logging was achieved automatically by the

Morae recording software. Detailed data analysis and scoring of the recorded protocols was done after

the test procedure by coding the recording with more fine grained task, action and error markers.

Two facilitators conducted the test: a senior usability professional (Ph.D. Cognitive Psychology with a

Human Factors emphasis.), with over 20 years of experience in the field of User Experience Design

and Research, and a business systems analyst with over 5 years experience documenting and

designing the user interfaces of Practice Partner (see Appendix E: Experimenters’ Short Biograph).



prompt.

of 54








Questionnaire).

Following the test, the administrators provided time for the participants to informally communicate


Test Locations

Participants were tested in their own facilities. In facility A, an IT training room was used on one day,

and a conference room on another day. In facility B the boardroom of the hospital building was used

for all sessions. While all sites tried to create a quiet space for the sake of testing, their busy

professions did not always allow for this. On occasion there was background conversation and other

distractions associated with busy hospital environments. Given the usually hectic context of clinical

practices and the use of EMRs, this represented fair and realistic conditions under which to test the

software.

Test Environment

The participants performed the test tasks on the facilitator’s laptops which were both running

independent copies of the EHR server and client. All of these were Practice Partner Ver. 11. The

specifications of the two facilitator machines are as follows: (1) an HP Elitebook Folio 9470m laptop,

DPI setting normal (96 DPI), Highest Color Quality (32 bit), 1366 X 768 pixel screen resolution, and a

60 Hertz screen refresh rate, and (2) a Dell Latitude E6410 laptop, DPI setting normal (96 DPI), Highest

Color Quality (32 bit), 1366 X 768 pixel screen resolution, and a 60 Hertz screen refresh rate. Audio

was recorded through each systems' internal microphone. Participants interacted with the remote

software through their keyboard and mouse or touchpad.


Practice Partner®


Environment

Two special usability test environments were created using the generic data configuration used for the Practice Partner Environment. The environments simulates the complete EHR including all functions and backend support software that a real implementation would provide, including data integration with external systems allowing for data flow from pharmacy solutions, integration with electronic data transfer to pharmacies. No other user groups had access to this environment, ensuring a pristine test environment with full control of the configuration of the test data.


Our test environment is configured to show what the system is capable of and therefore presents more alerts, confirmation screens, list items etc., than many facilities would

of 54


choose to implement.

Generally, the system performance (i.e. response time) was representative to what actual users would experience in a field implementation. Data from tasks or trials where system response time slowed perceivably were excluded from the analysis.


Practice Partner® Version 11





o Informed Consent

o Demographic Questionnaire

o Experimenter’s Verbal Instructions

o Training slides

o Observer’s Task Sheets

o Post-task surveys (presented through Morae)

o Post-test survey (System Usability Scale, SUS, presented through Morae)

Most of these materials are included in the appendices.

Usability Metrics






o Effectiveness of the EHRUT by measuring participant success rates and errors.

o Efficiency of the EHRUT by measuring the average task time and path deviations.

o Satisfaction with the EHRUT by measuring ease of use ratings.

Data Scoring


of 54




Effectiveness:

Task Success





Optimal task performance time, as benchmarked by the 3 users who finished the task without errors and deviations and in the shortest amount of time, were calculated by averaging the times of these 3 users. In cases where more than 2 participants failed a task, the optimal task performance time was calculated using the 2 users who finished the task without errors and deviations in the shortest amount of time. No adjustments were made to the optimal time. Average group times were divided by the optimal performance time for a measure of how close to optimal efficiency each task was performed. It was decided not to define time allotments, because it was felt that different adjustments would have to be made for new tasks and for existing task flows, and we could not find a generally accepted empirical or principled way to determine meaningful adjustments. All participants were allowed to finish the tasks, and the time ratios provide a measure of how much longer an average user would take to perform these tasks.


Effectiveness:

Task Failures and

Errors





Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application is recorded. Deviations occur if the participant, for example, chooses a path to task completion which requires more steps than the optimal, fills out additional fields, or performs any steps to correct a previous error. This path is compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency:

Task Time


Satisfaction:

Task Rating


by administering both a simple post-task questionnaire as well as a post-session

questionnaire. After each task, the participant is asked to rate “Overall, this task was on a

scale of 1 (Very Difficult) to 5 (Very Easy)”. If the task included a number of subtasks, a

question addressing the ease of each subtask was included (see Appendix G: End-of-Task

of 54



Questionnaires).




of 54


Results




Table 7. Result Summary by Meaningful Use Criterion and Task same as Table 2, repeated here for easier reference

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)

Optimal Deviations (Observed

/ Optimal)

Mean (SD)

Mean (SD)

§ 170.314(b)(4) -- Clinical information reconciliation

1

Open incoming CMA

15 92.86 2.0 119.75

(66.38) 52.07 2.30

1.36

(.84)

4.14

(.77)


15 100 2.13 43.10

(43.68) 23.13 1.86

.05

(.76)

4.0

(1.11)

Locate and open CMA 15 100 2.16

63.62

(42.83) 19.63 3.24

.86

(.36)

4.5

(.85)

Open existing patient chart 15 92.86 1.27

13.02

(7.13) 9.32 1.40 0

4.14

(.77)


14 85.71 1.75 52.6

(37.75) 19.72 2.67

1.33

(.98)

4

(1.28)


10 100 1.15 23.47

(22.12) 11.08 2.11

1.2

(.63)

4.8

(.42)


9 88.89 1.25 36.96

(23.11) 11.52 3.21

.5

(.53)

4.63

(.74)

§ 170.314(a)(7) -- Medication allergy list


14 92.86 1.46 72.72

(47.74) 34.21 2.13

1.75

(1.87)

4.71

(.47)


14 92.86 1.07 43.99

(25.32) 24.66 1.78

.33

(.65)

4.57

(.51)


14 92.86 2.51 32.55

(42.46) 9.55 3.41

1.69

(2.14)

4.71

(.47)


15 100 1.46 58.26

(22.59) 34.76 1.68

.07

(.26)

4.6

(.63)

7 Confirm review of 15 100 2.4

17.22

(10.68) 8.71 1.98

.07

(.26)

4.93

(.26)

of 54


Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

allergy list






averages obtained from those task-based usability scores varied between 4.0 and 4.93, quite a bit

above the recommendation put forward in NISTIR6 7742. The NISTIR 7742 states that “common

convention is that average ratings for systems judged easy to use should be 3.3., or above.”







evaluate the usability of the EHRUT.




of 54



170.314(b)(4) – Clinical information reconciliation

Clinical information reconciliation was new functionality developed for Practice Partner Ver. 11. It is a

complex task for which training materials were developed. After training and testing at our first test site,

it became clear that the training needed to include a broader conceptual introduction because the idea

of an electronic information exchange was not a familiar concept to our user base. Because of this, the

data collected from the first 9 participants at site A were excluded from the analysis. The overall task of

opening the data files, performing medication, allergy and problem reconciliation and finishing the

import were broken into smaller tasks for this test, because specific instructions needed to be provided

to a complete novice user.

Task 1 – Open incoming CMA

Analysis Notes



Task Description

In this task, participants were asked to open the CMA for a particular patient.

Risk Analysis

High. Finding the correct file and connecting it with the matching patient is crucial, any confusions

could have devastating effects.

Scoring Notes


Exclusions

None


of 54


Table 8. Usability Data for Task 1 – Open incoming CMA

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1

Open incoming CMA

15 92.86 2.0 119.75

(66.38) 52.07 2.30

1.36

(.84)

4.14

(.77)


15 100 2.13 43.10

(43.68) 23.13 1.86

.05

(.76)

4.0

(1.11)

Locate and open CMA

15 100 2.16 63.62

(42.83) 19.63 3.24

.86

(.36)

4.5

(.85)

Open existing patient chart

15 92.86 1.27 13.02

(7.13) 9.32 1.40 0

4.14

(.77)

Participant Errors Table 9. Participant Errors for Task 1 – Open incoming CMA


5, 17, 17,

12, 7, 11 Minor

During navigation subtask: participants expected to open the patient’s chart first, rather than opening the CMA first

7 Minor Navigation Error, participant explores the wrong tab

5, 6, 7, 9,

10, 11, 13,

14 ,15 ,16,

17, 18, 19

Minor When opening the CMA, participants expected to type the file name instead of using the browse function. Not one participant included a path name.

16, 16 Medium Participant used the ‘View’ button and never found a way to open the CMA for import.

Effectiveness

All but one participant performed this task successfully. Five participants thought that they had to open

up the patient’s chart first, which was scored as a minor error. 13 out of 15 participants struggled with

the browse dialog by attempting to type the file name by hand – and none asked for or tried to add a

path name to the file name. While this is fairly minor mistake it shows that participants struggled

conceptually with opening an external file, and most took many tries at spelling the file name until they

noticed that they could use the browse function. Users had no difficulties with finding and selecting the

correct patient chart to connect the CMA with, no errors were made during this subtask, and the step

deviation score was low (1.27).

of 54


Efficiency

The ‘locate and open CMA’ subtask stands out in terms of time deviation (3.24) – the cause of this was

the participants’ struggle with finding the browse functionality. This was the first time that participants

were using this functionality, so these times are indicative of how difficult it was to explore and learn the

correct use of this control. It does not measure long term efficiency. Once the CMA had been located,

users connected it with the matching patient chart very quickly and without mistakes (time deviation

score 1.40).

Comments


Table 10. Participant Comments for Task 1 – Open incoming CMA


6 None

9 Na

10 No

11 I had trouble knowing whether to click the "browse" button or the"ok" button when typing in the file name.

16 I think just getting use to the system and it will be great

Satisfaction

Relatively speaking, this task received fairly low satisfaction scores, between 4.0 and 4.5.

Major Findings

It appears that users think of this task as adding a CMA to an already open patient chart instead of the

other way around – so letting them access the functionality from the patient chart (via a button) might

be another option to start this workflow. The browse dialog presented a major hurdle. All but 2 users

struggled with this. It might be possible to skip this step altogether by launching the Windows browse

functionality immediately when this workflow is invoked. If a password protects this functionality, it

might be possible to just ask for the password at this point and then launch into the Windows browse

functionality. This would save the user an unnecessary step and completely avoid this confusion.

of 54


Task 2 – Perform Medication Reconciliation

Analysis Notes



Task Description

Participants were asked to reconcile the medications on the patient chart with the medications on the

CMA. To do this, they had to un-select a medication from the import list and discontinue the same

medication on the patient chart.

Risk Analysis

High. Rendering an accurate medication list is necessary to allow for creating a drug-treatment plan

that takes all currently taken medications and possible drug-drug interactions into account.

Scoring Notes

One participant could not complete this task, because they had mistakenly clicked the import button

immediately after opening the CMA – this rendered this patient’s chart changed, so that tasks 2-4 could

not be attempted.

Exclusions

Participant 16 – Import executed too early.


Table 11. Usability Data for Task 2 – Perform medication reconciliation

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


14 85.71 1.75 52.6

(37.75) 19.72 2.67

1.33

(.98)

4

(1.28)

Errors Table 12. Participant Errors for Task 2 – Perform medication reconciliation


9, 9, 10,

10, 15, 15,

Minor Participants struggled with discovering the right-click control for existing medications or the un-check control for incoming medications.

of 54


18

7, 11, 12,

13, 14, 15,

17, 19

Medium Participants did not discontinue the medication on the existing medication list.

7, 12, 14, 1 Severe Participant selected the import button before finishing the clinical reconciliation process (allergies and problems).

Effectiveness

Four participants did not complete this task successfully because they clicked the import button before

finishing the reconciliation. Many participants would have clicked this button if the experimenter would

not have prevented them from doing so (we decided to intervene, because it was impossible to conduct

the next two tasks after importing the data into the patient chart). As the enumeration of errors shows,

there were also issues with operating the list controls: participants either struggled with discovering the

right-click menu, or simply didn’t discontinue the existing medication. After selecting ‘discontinue’ some

participants expected the item to disappear from the list rather than being highlighted yellow.

Efficiency

Because of the struggles with the list item controls, the step deviation ratio (1.75) and time deviation

ratio (2.67) were fairly large. This indicates problems with exploring the functionality of this new

feature, not necessarily the long-term efficiency of this design.

Comments


Table 13. Participant Comments for Task 2 – Perform medication reconciliation


6 None

9 Na

10 No

13 i would have expected to see the imported medication in the "new" clinical area immediately

Satisfaction

The satisfaction ratings were relatively low, 4 across the whole task.

Major Findings

There were several issues with this task: While the two lists looked similar, their functions were either

visible (incoming list, checkboxes) or hidden (context menu: existing med list). This led in the worst

of 54


case to the (wrong) conclusion that the discontinue function was not available in the existing list. In the

best case, it took participants a while to figure out how to access this function. Participants also

seemed a little confused about the notion of ‘un-selecting’ items from being imported. Several

mentioned that a positive action, such as ‘selecting’ items for import, would have been more intuitive.

Furthermore, participants seemed to want to see a result of their action of ‘selecting’ or importing. One

participant mentioned in the comments and others verbally, that they wanted to ‘move’ items from the

list on the left to the list on the right. In their attempt to make something happen, almost every

participant wanted to click the ‘Import’ button at the bottom of this window. This button instantly imports

the data (even the categories that have not been reviewed) and makes it impossible for the participant

to reverse their action. There are two simple methods for preventing this from happening: (a) disable

the button until all categories have been reconciled, and/or (b) present a warning dialog that allows the

user to cancel out of this action. One might also consider graphically separating the ‘import’ button from

the lists by a divider, to communicate that this function operates on the whole of the reconciliation

process rather than the specific list in view.

Task 3 – Perform Allergy Reconciliation

Analysis Notes



Task Description

Participants were asked to reconcile the allergies on the patient chart with the medications on the CMA

– and to make sure all new allergies were being imported. To do this, they had to check the

‘reconciled’ check box at the bottom of the window (the new allergy was already selected for import by

default).

Risk Analysis

High. Rendering an accurate allergy list is necessary to allow for creating a drug-treatment plan that

takes all currently taken medications and possible drug-allergy interactions into account.

Scoring Notes

Five participants could not complete this task, because they had mistakenly clicked the import button

already – this rendered this patient’s chart changed, so that tasks 3-4 could not be attempted.

Exclusions

Participants 7, 12, 14, 16, 17 – Import executed too early.


of 54


Table 14. Usability Data for Task 3 – Perform allergy reconciliation

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


10 100 1.15 23.47

(22.12) 11.08 2.11

1.2

(.63)

4.8

(.42)

Errors Table 15. Participant Errors for Task 3 – Perform allergy reconciliation


6, 16 Minor Control exploration, participants tried to change the state of the incoming allergy list

10, 11, 13,

9, 8, 15, 5,

18, 19

Medium Participants did not select the checkbox that indicates completed allergy reconciliation

15 Severe Participant selected the Import button before finishing all reconciliation steps.

Effectiveness

While all participants found and selected the ‘Allergy’ tab without any problems only one actually

selected the checkbox that indicated that they had reconciled the two allergy lists. Currently, this

checkbox does not have an actual function (other than a mental checkmark for the user that this

function has been performed). If the user does make changes to either list that checkmark becomes

checked automatically. While some users looked and noticed the checkmarks, they did not seem to

understand their function and wanted to click the ‘Import’ button instead.

Efficiency

There were few actions to be done in this subtask, navigation to the allergy list was done without

issues, so the step deviation ratio (1.15) and the time deviation ratio (2.11) were both fairly low.

Comments


Table 16. Participant Comments for Task 3 – Perform allergy reconciliation


6 None

9 Na

of 54


10 No

13 would like to have an area to add new allergies.

Satisfaction

The satisfaction ratings were relatively higher than in the preceding task, 4.8 – there were few steps

involved so this task was perceived as easy.

Major Findings

This subtask revealed that the participants did not really understand the function of the ‘reconciled’

checkboxes. Part of the problem seems to be that they perceived them to be set by the system as a

result of a different action (changes in list item status) rather than user controls. A button that sets a

control might be more effective – since users were clearly looking for a button to click.

Task 4 – Perform Problem Reconciliation

Analysis Notes



Task Description

Participants were asked to reconcile the problems on the patient chart with the medications on the CMA

– and to make sure no existing problems were being imported. Then they were to finalize the import.

Since all the problems were already on the existing chart, they had to unselect three problems, then

select the import button.

Risk Analysis

High. Rendering an accurate allergy list is necessary to allow for creating a drug-treatment plan that

takes all currently taken medications and possible drug-allergy interactions into account.

Scoring Notes

Six participants could not complete this task, because they had mistakenly clicked the import button

already – this rendered this patient’s chart changed, so that tasks 4 could not be attempted.

Exclusions

Participants 7, 12, 14, 15, 16, 17 – Import executed too early.

of 54



Table 17. Usability Data for Task 4 – Perform problem reconciliation

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


9 88.89 1.25 36.96

(23.11) 11.52 3.21

.5

(.53)

4.63

(.74)

Errors Table 18. Participant Errors for Task 4 – Perform problem reconciliation


8 Minor Confused about which list to work on (right or left)

5, 11, 18,

19 Medium Participants did not select the ‘Import’ button.

Effectiveness

One participant did not make any selections to the problem list (un-selections) and was scored as

unsuccessful. Other participants ended up not selecting the ‘Import’ at the end of this tasks – but this

probably an artifact of having been told to NOT select this button in the previous 3 tasks. One

participant still displayed uncertainty about which list to treat as the ‘import’ list vs. the ‘chart’ list.

Efficiency

If participants did this task successfully they exhibited few extra steps (step deviation ratio was 1.25)

but some uncertainty expressed in a high time deviation ratio (3.21). Even when performing the third

step in the sequence of reconciliations, participants were still not sure about the correct actions.

Comments


Table 19. Participant Comments for Task 4 – Perform problem reconciliation


6 Whenever recociling the patient's chart if it has not been completed allergies, problems, and medications it should stop you until it has been done.

of 54


9 Na

10 No

Satisfaction

The satisfaction ratings were relatively high, 4.63 – but one participant noted that it would be good to be

stopped from prematurely importing the new record.

Major Findings

No new problems were revealed during this last step in the reconciliation sequence. The uncertainty

about what the function of the ‘import’ buttons was or when to use it led to several participants avoiding

it during this last step.

170.314(a)(7) – Medication allergy list

Test contexts

Usability of the medication allergy list was tested in three tasks, tasks 5-7. Unlike the medication

reconciliation tasks, this functionality was familiar to the participants in our study. During Task 5,

participants were asked to find an existing allergy in the medication allergy list and modify its severity

level. During the Task 6, they were asked to add an allergy to the list, and during Task 7, they were

asked to verify that they had reviewed all allergies.

Task 5 – Look up and edit existing allergy list entry

Analysis Notes


170.314(a)(7) – Medication allergy list

Task Description

This task asked participants to find an existing allergy entry and change the severity from mild to

severe.

Risk Analysis

High. An accurate and updated allergy list is important as it alerts physicians to potentially harmful

drug-allergy interactions when creating a treatment plan for patients.

of 54


Scoring Notes

One participant could not execute this task because she mistakenly had forwarded to the next task

instructions without performing the task.

Exclusions

Participant 17 – mistakenly clicked ‘end task’ in the Morae instruction window.


Table 20. Usability Data for Task 5 – Lookup and edit existing allergy list entry

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


14 92.86 1.46 72.72

(47.74) 34.21 2.13

1.75

(1.87)

4.71

(.47)


14 92.86 1.07 43.99

(25.32) 24.66 1.78

.33

(.65)

4.57

(.51)


14 92.86 2.51 32.55

(42.46) 9.55 3.41

1.69

(2.14)

4.71

(.47)

Errors Table 21. Participant Errors for Task 5 – Lookup and edit existing allergy list entry


5, 6, 7, 7, 7,

7, 7, 9, 9, 9,

10, 12, 12,

13, 14, 16,

16, 16 ,16,

16, 16, 19

minor Participants tried to figure out how to enter ‘edit’ mode for an existing allergy entry.

11, 14, 13 minor Navigation mistakes

10, 14 minor Search form mistake when looking up patient

of 54


Effectiveness

All but one participant performed this task successfully. All participants found the allergy entry without

any problem, but 10/14 participants struggled with finding a way to edit it, leading to a number of errors

of opening the wrong dialogs for achieving this goal. All but one participant eventually used ‘double-

click’ to open the edit window and once they had found it, they finished the task correctly without further

issues.

Efficiency

Navigation to the allergy list was performed efficiently. Step and time deviation ratios were small for this

subtask (1.07, and 1.78 respectively). The struggle with opening the edit window led to longer

deviation ratios for the second sub task (2.51 for steps and 3.41 for time).



Table 22. Participant Comments for Task 5 – Lookup and edit existing allergy list entry


6 None …

9 Na

10 No

16 It was easy

Satisfaction

Satisfaction ratings for this task were fairly high, the relatively lower score for ‘finding the allergy’

probably reflects the struggle with finding a way to open the edit window, not finding the allergy itself

which was quickly located.

Major Findings

The difficulties during this task were caused by inconsistencies in the access methods for opening

existing medication entries and allergy entries that are both presented in the same window. A number

of buttons address functions for the medication list, but there is only one allergy button which allows

creating a new allergy, rather than editing an existing one. Furthermore, ‘double-click’ is not used as an

access method for items on the medication list, so it took participants a while to discover that they could

use it on the allergy list. List functions should be reviewed for consistency and consistent methods for

common actions, such as editing, adding and discontinuing/deleting items should be established and

applied across these common lists.

of 54


Task 6 – Create a new allergy list entry

Analysis Notes


170.314(a)(7) – Medication Allergy List

Task Description

Participants were asked to enter a new allergy into the allergy list.

Risk Analysis



Scoring Notes


Exclusions

None


Table 23. Usability Data for Task 6 – Add an allergy to the allergy list

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


15 100 1.46 58.26

(22.59) 34.76 1.68

.07

(.26)

4.6

(.63)

Error Analysis

Table 24. Participant Errors for Task 6 – Add an allergy to the allergy list


16 minor Participant mistakenly selects the wrong list item, but immediately notices and changes the selection.

of 54




Table 25. Participant Comments for Task 6 – Add an allergy to the allergy list


6 none

9 NA

10 no

16 No …

Effectiveness

All of the participants successfully completed this task. One minor mistake in selecting a list item was

quickly corrected.

Efficiency

Participants were well used to adding allergies to the allergy list and quickly stepped through the

necessary actions. Step and time deviation ratios were small (1.46 and 1.68).

Satisfaction

Satisfaction scores were high, and there were no comments.

Major Findings

There were no additional findings from this task. Adding an allergy is done through a clearly visible

button and all participants knew how to find this button and filled out the necessary fields with no

problems.

Task 7 – Confirm review of allergy list

Analysis Notes


170.314(a)(7) – Medication Allergy List

Task Description

Participants were asked to confirm that they had reviewed the allergy list.

of 54


Risk Analysis



Scoring Notes


Exclusions

None


Table 26. Usability Data for Task 7 – Confirm review of allergy list

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

7 Confirm review of allergy list

15 100 2.4 17.22

(10.68) 8.71 1.98

.07

(.26)

4.93

(.26)

Error Analysis

Table 27. Participant Errors for Task 7 – Confirm review of allergy list


19 minor Reconcile instead of review (same functionality)



Table 28. Participant Comments for Task 7 – Confirm review of allergy list


6 none

9 NA

of 54


10 no

16 No …

Effectiveness

All of the participants successfully completed this task. One participant clicked the ‘reconcile’ button

instead of the review button, but both buttons have the same function.

Efficiency

Participants were used to reviewing allergies. They were slightly startled by a new reconciled button

that duplicates the function of reviewing and clicked both buttons which explains the slightly elevated

step deviation ratio (2.4) and time deviation ratio (1.98).

Satisfaction

Satisfaction scores were high (4.93), and there were no comments.

Major Findings

This task was completed without issues. It might make sense to remove the old verification button.

of 54


List of Appendices






Appendix E: Experimenters’ Short Biograph




of 54



of 54
















familiarize yourself with these new functions, please walk through this short PowerPoint.












to.




of 54


- When you have started, I won’t be able to answer any more questions. Do you have any


of 54



of 54








• Your feedback will help us improve the usability of Practice Partner.

• During the study, you will be performing a few routine tasks using Practice Partner, most of which will

be very familiar to you.

• This presentation provides a review of some of the new features of Practice Partner.

• The following screens present an overview of new features in self-guided presentation mode.




on your keyboard.


Study Instructions


• Our session today will last about 30 minutes. During that time you will use Practice Partner Version 11

to perform tasks.









Starting the Study



of 54








Performing the Task













right location.




of 54



of the questions.





of 54


Appendix E: Experimenters’ Short Biographies












of 54



Introduction

You will be treating a 43 year old female patient today.

She comes to the office with symptoms of a UTI.

Please conduct the following 7 tasks using the EHR provided to you.

When you are ready, please click the START button.

Task 1

Task Instructions:

Your patient has recently been seen in another clinic. You have received the Medical

Summary from that clinic and would like to reconcile the data. The import file is located in

the 'Medical Summaries' folder on your desktop.

1) Import the Medical Summary file into the patient chart. The file name and your

patient's name are provided to you on paper.

2) Stop when the Clinical Data Lists screen is displayed. You will reconcile the clinical

information in the next tasks.

Task 2

Task Instructions:

Perform Medication Reconciliation.

1) The patient tells you that she has stopped taking the Cipro, so discontinue it and don’t

select it for import.

2) Select all other new medications for import.

3) Stop when you have reconciled all medications. You will work on problems and

allergies in the upcoming tasks.

Task 3

Task Instructions:

Perform Allergy Reconciliation.

1) Make sure all new allergies will be imported.

of 54


Task 4

Task Instructions:

Perform Problem Reconciliation

1) Make sure all new major problems will be imported.

2) Don't import existing problems.

3) Finalize and import all selected data.

Task 5

Task Instructions:

The patient tells you that her Penicillin allergy has gotten much worse.

1) Change the severity from mild to severe.

Task 6

Document a Wheat Allergy with:

today’s date,

severity level “Moderate”, and

reaction “Nausea/Vomiting/Diarrhea” for the patient.

Task 7

Task Instructions:

1) Review all other allergies.

2) Indicate that you have reviewed the patient’s allergies today.

of 54



1 - Import CMA

1. Finding the screen to do this task was ...


2. Finding the Medical Summary file to import was ...


3. Matching the file with the correct patient was ...





Comments:

2 - Medication Reconciliation

1. Discontinuing Cipro was ...


2. Selecting Medications for Import was ...





Comments:

of 54


3 - Allergy Reconciliation



2. Selecting allergies to import was ...





Comments:

4 - Problem Reconciliation



2. Selecting problems to import was ...


3. Importing all data was ...





Comments:

5 - Change Allergy



of 54


2. Changing the allergy severity was ...





Comments:

6 - Add Allergy



2. Adding a new allergy was ...





Comments:

7 - Review Allergy List



2. Marking the allergies as reviewed was ...





of 54




questionnaires.






















of 54


Appendix I: System and Patient Scenario for Nurse’s Tasks

Setting: Ambulatory

Participant Group: Nurse

PATIENT SETUP

Name: XXX Green

Age:43

DOB: 02/03/1970

Gender: female

123 XXX, Denver, CO 80238

Telephone Number: 303-329-6802

Usual provider: Nurse

In patient’s chart:

Allergy List:

Penicillin Potassium

mild

skin rash

onset date: 01/01/1995

Problem List:

Asthma:

o onset date: 1995

o entered on the Major Problems tab

Diabetes:

o onset date: 2000

o entered on the Major problems tab

Med List:

Azmacort, MDI 2PUFFS Q6H X 30 D

o Date 01/01/2000

Metformin, 500 MG BID X 30 D

o Date 01/01/2000

Ciprofloxacin (Cipro oral)

o 250mg twice a day orally, for 7 days,

o start date: 09/05/2013

of 54


Information on CMA (xml)

Medications:

Will be discontinued and not imported: Ciprofloxacin (Cipro oral)

o 250mg twice a day orally, for 7 days,


Will be continued: Phenazopiridine HCL

o 200 mg TID X 3D

o Start date: 09/05/2013

Allergies:

Egg allergy,

o severe, shock


Problems

UTI - acute

o start date 09/05/2013

o listed as a Major Problem

Asthma:

o onset date: 1995

o entered on the Major Problems tab

Diabetes:

o onset date: 2000

o entered on the Major problems tab

EHR Usability Test Report Part III

EHR: Practice Partner® VER. 11

User Role: Administrator

Setting: Ambulatory

Applications Tested

Practice Partner® VER. 11

Dates

Usability Test: August 12-29, 2013

Report: September 2, 2013

Prepared by


Gergana Pavlova-Grozeva, Senior BA, McKesson Provider Technologies



Westminster, CO 80021 (720) 356-7711

[email protected]



of 48

EHR: Practice Partner VER. 11, Part III – Administrator – Ambulatory September 2, 2013

Contents

Tables ............................................................................................................................................................... 4


Major Findings and Areas for Improvement .................................................................................................... 8

Introduction ...................................................................................................................................................... 11

Intended Users............................................................................................................................................. 11

Method ............................................................................................................................................................ 11

Participants .................................................................................................................................................. 11

Study Design ............................................................................................................................................... 13

Procedures .................................................................................................................................................. 14

Test Locations.............................................................................................................................................. 15




Data Scoring ................................................................................................................................................ 16

Results ............................................................................................................................................................ 19



170.314(a)(2) – Drug-drug, drug allergy interaction checks .......................................................................... 22

Test contexts ............................................................................................................................................ 22

Task 1 – Configure drug-drug interaction severity level ............................................................................ 22

Task 2 – Change drug-allergy warning interventions ................................................................................ 25

170.314(a)(8) – Clinical decision support ..................................................................................................... 26

Test contexts ............................................................................................................................................ 26

Task 3 – Create and activate demographics-based decision support (Health Maintenance alert) ............. 27

Task 4 – Create and activate complex decision support ........................................................................... 29





of 48



Appendix E: Experimenter's Short Biography ............................................................................................... 42




of 48


Tables

Table 1. Administrators' Tasks: Meaningful Use Criteria, Risk, and Application Tested ..................................... 5

Table 2. Result Summary by Task ..................................................................................................................... 7

Table 3. Result Summary by Meaningful Use Criterion ...................................................................................... 8


Table 5. Administrators' Tasks – same as Table 1, repeated here for easier reference ................................... 13



Table 8. Usability Data by Task and Subtask - same as Table 2, repeated here for easier reference .............. 19

Table 9. Usability Data by Meaningful Use Criteria – same as Table 3, repeated here for easier reference ..... 20

Table 10. Usability Data for Task 1 – Configure drug-drug interaction ............................................................ 23

Table 11. Participant Errors for Task 1 – Configure drug-drug interaction ....................................................... 23

Table 12. Participant Comments for Task 1 – Configure drug-drug interaction ............................................... 24

Table 13. Usability Data for Task 2 – Configure drug-allergy warning intervention .......................................... 25

Table 14. Participant Errors for Task 2 – Configure drug-allergy warning intervention .................................... 25

Table 15. Participant Comments for Task 2 – Configure drug-allergy warning intervention ............................. 26

Table 16. Usability Data for Task 3 – Create and activate alert ....................................................................... 27

Table 17. Participant Errors for Task 3 – Create and activate alert ................................................................. 28

Table 18. Participant Comments for Task 3 – Create and activate alert .......................................................... 28

Table 19. Usability Data for Task 4 – Create complex alert rule ...................................................................... 30

Table 20. Participant Errors for Task 4 – Causes for overall task failures ....................................................... 31

Table 21. Participant Comments for Task 4 – Create complex alert rule ......................................................... 31

of 48


Executive Summary A usability test of Practice Partner® VER. 11 was conducted August 12-29, 2013 remotely by Marita

Franzke and Gergana Pavlova-Grozeva of the McKesson Provider Technologies. The testing was

conducted remotely due to the difficulty in finding more than a few administrators at any individual

customer site. The purpose of this test was to test and validate the usability of two Meaningful Use

Criteria as implemented in the functions of the Practice Partner® VER. 11. The Office of the National

Coordinator for Health Information Technology (ONC) requires vendors to test 7 Meaningful Criteria

outlined in the test procedures for §170.314(g)(3) Safety-enhanced design1 for ambulatory products.

The criteria focused on the performance of clinical tasks will be discussed in two other reports that

detail the results of usability studies involving nurses/medical assistants, and physicians, as those

clinical tasks are typically conducted by those user roles.

During the usability test, 9 administrators served as participants; they were all familiar with previous

releases of the Practice Partner EHR. In this study, they used this software in 4 simulated, but



This study collected performance data on 4 tasks typically conducted by administrators who are

configuring an EHR or user roles for interacting with an EHR. Below is a list of all tasks employed,

which were chosen to evaluate the usability of two of the eight Meaningful Criteria outlined in the test

procedures for §170.314(g)(3) Safety-enhanced design. Table 1 below lists the administrators' tasks in

order of performance, the linked Meaningful Use criteria, and a risk rating for performing each task.

Table 1. Administrators' Tasks: Meaningful Use Criteria, Risk, and Application Tested


Application Risk

1 Configure drug-drug interaction severity level


Practice Partner

High

2 Change drug-drug warning severity level


Practice Partner

High

3 Create and activate simple demographics-based decision support (Health Maintenance alert)


Practice Partner

High


of 48


4 Create and activate complex demographics and drug-based decision support (Health Maintenance alert)


Practice Partner

High

During the one hour, one-on-one usability test, each participant was greeted by an administrator when

he/she called into the conference line. Each participant had prior experience with a previous version of

the Practice Partner ® EHR. The participants were asked to review a self-guided training module that

reviewed creating complex Health Maintenance rules, as this task is not practiced frequently, and

introduced instructions for the usability study and using the testing software. The administrator

introduced the test, and instructed participants to complete a series of tasks (given one at a time) using

the EHR Under Test (EHRUT). During the testing, data logging software timed the test and recorded

user screen captures and audio. These recordings were later used to score action steps, errors and

task completion rates. The test administrator did not assist the participant in completing the task.





Path deviations





questionnaire.


on performance with these tasks to be: 68.




averages obtained from those task-based usability scores varied between 3.11 and 4.89. The first

three tasks all scored over 4.4 with all tasks scoring over 4, quite a bit above the recommendation put

forward in NISTIR3 7742. The NISTIR 7742 states that “common convention is that average ratings for

systems judged easy to use should be 3.3., or above.” The final task presented a real difficulty to all

participants; in fact only two participants completed this task successfully. This task was rated as 3.11.

We will give special attention to the analysis of this task, which asked the participants not only to

implement a complex health maintenance rule, but it also asked to write a file with conditional logic, and

this appeared to be outside of what any of the participants saw as their normal job duties. The struggle




of 48


with this final task noticeable influenced how the participants felt about the system, and may have had

an impact on their final assessment, as expressed in the SUS score.





Table 2. Result Summary by Task

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1

Configure drug-drug interaction severity level

9 100 1.93 85.26

(94.29) 28.25 3.02 .78

4.56

(1.01)

Navigation 9 100 2.08

52.93

(80.54) 14.67 3.60 .56

4.44

(0.53)

Configuration 9 100 1.61

32.32

(17.63) 13.58 2.38 .22

4.78

(0.44)

2 Change drug-drug interaction severity level

9 100 1.07 30.53

(15.23) 18.85 1.62 0

4.89

(0.33)

Navigation 9 100 1

12.94

(7.48) 9.32 1.39 0

4.89

(0.33)


17.6

(8.73) 9.52 1.85 0

4.89

(0.33)

3 Create and activate demographics-based decision support (Health Maintenance alert)

9 100 1.36 113.43

(43.66) 75.12 1.5 .55

4.44

(.73)


20.21

(13.23) 15.38 1.31 .11

4.67

(0.5)

Activation of rule

9 100 1.67 28.24

(7.34) 25.57 1.11 0

4.33

(0.71)

Create specific overdue dates

9 100 1.24 64.99

(37.3) 35.17 1.85 .44

4.44

(.73)


9 22 NA, see

Discussion of the Findings

12 minutes,

38 seconds

NA, see

Discussion of

the Findin

gs

NA, see Discussion of the Findings

NA, see

Discussion of

the Finding

s

3.00

(1.41)

of 48


The usability result data from Table 2 is grouped by Meaningful Use Criteria in Table 3 below, to allow

for better overview of the usability scores for each criterion.

Table 3. Result Summary by Meaningful Use Criterion


N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)



9 100 1.93 85.26

(94.29) 3.02 .78

4.56

(1.01)

Change drug-drug interaction severity level

9 100 1.07 30.53

(15.23) 1.62 0

4.89

(0.33)


Create and activate simple demographics-based decision support (Health Maintenance alert)

9 100 1.36 113.43

(43.66) 1.5 .55

4.44

(.73)

Create and activate complex demographics and drug-based decision support (Health Maintenance alert)

9 22 NA, see


12 minutes,

38 seconds

NA, see Discussion

of the Findings

NA, see Discussion

of the Findings

3.00

(1.41)




This section summarizes the findings and areas for improvement at a high level. For additional detail, please

please see section "

of 48


Discussion of the Findings”.

Finding Areas for Improvement

Finding the Drug-drug Interaction Screen

For users who were rusty or unfamiliar with this

function, the menu labels provided little

guidance. The help file eventually revealed the

navigation path, but the participant had to

search for it even there.

1. Design: Revise the help file to provide more

direct pointers to the navigation path for

someone looking to set ‘drug-drug

interactions’, ‘drug drug’ ‘interaction alerts’,

‘drug alerts’, ‘drug-drug alerts’ or drug-allergy

alerts’. Optimize the search terms and provide

direct links to instructions for the navigation

path. ‘Prescription defaults’ was not a helpful

key word or menu label.

2. Menu labels. I may not be advisable to

change the menu labels since users quickly

remembered the navigation path after

discovering it the first time (and it might

confuse existing users).

Drug Interaction Settings Tab

1. As expressed in the participant

comments and the behavioral

observations, participants were slightly

confused about the severity levels

expressed in the drop-down controls.

2. Participants did not understand a second

control having to do with displaying the

depth of the documentation.

Misunderstanding of this control could

lead to erroneous and possibly harmful

settings of this control.

1. Design: Consider a more direct way of

labeling the drop down and drop down

instructions, use verbal labels next to each

number. Apply this to the severity and

documentation controls.

2. Design: Provide more detailed help

explaining the documentation control, and

consider including this function in the

training.

3. Design: Consider making this function

visually more different from the severity

control so that users don’t get misled in

thinking that these two fields have to be set

together.

Activating Decision Supports (Health

Maintenance Alerts)

1. The user interface provided good

guidance when creating simple alerts

based on one type of trigger alone.

2. When a combined trigger had to be

created, participants had to leave the

user interface and create a rule file, then

attach that rule file to an existing rule.

While this allows for a great amount of

1. Training: If the creation of complex

decision rules is expected to be part of the

administrators’ job, train on logic, syntax

and efficient way of connecting rule files to

existing rules. Provide plenty of

opportunities for practice.

2. Design: At a minimum, remove the grey

shading of the rule field in the procedure

form, to make it appear more editable and

easier to find.

3. Design: Better yet, change the filed into a

‘browse’ functionality so that the users can

navigate to the rule file they want to upload.

of 48


flexibility in the types of alerts that can be

built, this task proved to be extremely

difficult for the participants.

This eliminates the frustration of having to

remember the file name; it will also solve

the problem of entering the wrong path

name.

4. Design: Consider providing more active

supports around writing rule files.

Suggestions:

a. A syntax checker

b. A form-based rule editor

c. Find out the types of rules most

commonly created. From the help

files, consider linking to examples of

rule elements (based on this

analysis above) that could be used

to copy frequently used elements

into files, where they can be edited.

d. Consider providing a rule test

facility.

of 48


Introduction The McKesson Practice Partner® EHR tested for this study was accessible to the participants in an

integrated environment, as would be typical in a provider implementation. The entire environment was

software version VER. 11. The usability testing environment and tasks attempted to simulate realistic

implementations, workflows and conditions. The purpose of the study was to test and validate the

usability of the current user interface, and provide evidence of usability in the EHR Under Test

(EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction were captured during

the usability testing.

Intended Users

Intended administrative users of the EHRUT are medical directors, system administrators, or

implementation and support team members working in an environment supporting ambulatory patient

care. In these functions the may be working for the vendor or Value Added Resellers (VARs),

supporting smaller clinics who don’t have their own system administrators, or they may be working for

provider organizations directly. Users may have varying levels of experience in the clinical setting and

may have varying years of experience performing administrative functions with the EHRUT. They can

be expected to have received training on the EHRUT, and to be frequent users (weekly or daily use).

Specifically for the administrators' tasks, setting and changing the drug interaction warnings and

activating Health Maintenance rules is part of implementing the EHR and customizing it for a particular

site. It can be expected that these functions may be used infrequently by the majority of users on

customer provider sites as setups do not often change and the majority of the setup is done when the

software is implemented in the facility. Since Practice Partner is a product that support relatively small

Medical Practices, most often the task of implementing, training, and configuring/customizing goes to

the vendor, VARs or third-party support organizations. The participants reflect these groups of users.

Method

Participants

A total of 9 participants were tested on the EHRUT for the administrator role in the ambulatory setting.

Participants were associated with the McKesson Implementation and Support team, four different VARs

and two customer sites. Ten participants had been recruited by technical difficulties with the test

administrators PC prohibited data collection with one participant. All participants were recruited through

an email sent out to various customers. Participants had no direct connection to the development of

the product.

Participants were recruited through an open email to several customers, so there was no control over

responses to the email. All participants had experience configuring previous versions of Practice

Partner, training users on the application and creating various templates. The experience with the

application ranged from 3.5 years to 7 years. A demographics questionnaire was sent out in an email

of 48


and had to be received by the facilitator before the session began; the questionnaire is included as

Appendix A: Demographic Questionnaire. Table 4 lists all 9 participants included in the analysis by

demographic characteristics, such as demographics, education, and EMR experience. Participant

names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual

identities.


ID Gender Age Education Title Years experience with Practice Partner

Percentage Time spent configuring system

Percent time spent building templates

1 M 31-40 Some college

Account Mgr. missing 10 20

2 M 31-40 BS Tier 2 Technical Support

6.5 80 0

3 F 51-60 MA Technical Course Developer

3.5 25 0

4 M 41-50 Associate Senior Support Analyst

6 15 10

5 M 41-50 MD Supervisor, optimization, training & process mgmt

3.5 15 15

6 M 51-60 BS Technical Consultant

3.5 2 2

7 M 51-60 BS Implementation Specialist

7 30 30

8 F 51-60 HS Sr. Practice Mgr & EMR Trainer

6 25 35

9 F 31-40 BA EMR Analyst 6 25 25

All participants indicated that they rarely created conditional logic files as needed for the complex

decision support task. A short, self-paced training PowerPoint presentation was provided to

participants before the study, to review this functionality. These training modules were similar to what

end-users would usually receive when introduced to new functionality. The training also included some

explanation of the software used to capture the session data.

Participants were scheduled for one hour sessions which provided time between sessions for debrief by

the administrator, and to get the system ready for the next participant. A spreadsheet was used to keep

track of the participant schedule, and the participant demographics.

of 48


Study Design








ambulatory use. The current study focused on the 2 MU criteria used typically by administrators. Two

separate reports detail the findings from tasks associated with 7 MU criteria typically conducted by

clinicians. Table 5 below cross-tabulates which task in the study of administrator tasks is associated

with which MU criterion and lists the risk associated with that task.

Table 5. Administrators' Tasks – same as Table 1, repeated here for easier reference


Application Risk

1 Configure drug-drug interaction severity level


Practice Partner

High

2 Change drug-drug warning severity level


Practice Partner

High

3 Create and activate simple demographics-based decision support (Health Maintenance alert)


Practice Partner

High



Practice Partner

High


PracticePartner® VER. 11. Each participant used the system remotely in the exact same configuration,

and was provided with the same instructions. The system was evaluated for effectiveness, efficiency

and satisfaction as defined by measures collected and analyzed for each participant:





of 48


o Number of errors

o Path deviations



Procedures

Because the testing was conducted remotely, the participant and facilitator had to enter a Webex

meeting and connect to the conference call. Once the participant and facilitator were both in the

session, the participant was verbally instructed and oriented to the test situation (see Appendix B:

Administrator Script and General Instructions). Each participant had been sent an informed consent

form (Appendix C: Informed Consent Form) through email several days before the session and asked

to return the completed documents before the session. After being greeted and receiving the verbal

instructions and orientation, the demographics questionnaire was administered (Appendix A:

Demographic Questionnaire). Then the facilitator gave control of the screens to the participant. The

participant was asked to take control of the screens by clicking their mouse, which allowed him/her to

step through a set of self-paced training slides. Participants spent between 5-10 minutes reviewing this

presentation of the software. The presentation also included usability test instructions and an

introduction to the Morae recording software (Appendix D: Usability Test Instructions).

One test administrator per participant ran the test; another test administrator attended the testing

sessions remotely, but remained silent. Data logging was achieved automatically by the Morae

recording software. Detailed data analysis and scoring of the recorded protocols was done after the

test procedure by coding the recording with more fine grained task, action and error markers.

The usability testing facilitators conducting the test was a senior experienced usability practitioner

(Ph.D. Cognitive Psychology with a Human Factors emphasis.), with over 20 years of experience in the

field of User Experience Design and Research, and 1.5 years of experience developing EHRs (see

Appendix E: Experimenter's Short Biography).



prompt.







Questionnaire).

Following the test, the administrator provided time for the participant to informally communicate


of 48


Test Locations

Participants were tested in their own facilities through a remote connection. While all participants tried

to create a quiet space for the sake of testing, their busy professions did not always allow for this.

Background noise in these test situations varied from completely quiet, to busy background

conversations. Given the usually hectic context of clinical practices and the use of EMRs, this

represented fair and realistic conditions under which to test the software.

Test Environment

The participants performed the test tasks on their own computer systems. However, the presentation

of the information was done through the test administrator's system. The details of that system are: (1)

an HP Elitebook Folio 9470m laptop, DPI setting normal (96 DPI), Highest Color Quality (32 bit), 1366

X 768 pixel screen resolution, and a 60 Hertz screen refresh rate. Audio was recorded through the

computer’s internal microphone. Participants interacted with the remote software through their

keyboard and mouse or touchpad.


Connection WAN. Dial-in into McKesson VPN, connection into a remote login server running the Horizon Ambulatory Care® ER 12 and HCCT ER 12 usability test environment.

PracticePartner®


Environment

A special usability test environment was created using the generic data configuration used for the PracticePartner Environment. The environment simulates the complete EHR including all functions and backend support software that a real implementation would provide, including data integration with external systems allowing for data flow from pharmacy solutions, integration with electronic data transfer to pharmacies, and allowing integration with the InfoButton reference vendor. No other user groups had access to this environment, ensuring a pristine test environment with full control of the configuration of the test data.


Our test environment is configured to show what the system is capable of and therefore presents more alerts, confirmation screens, list items etc., than many facilities would choose to implement.

Generally, the system performance (i.e. response time) was representative to what actual users would experience in a field implementation; however, testing remotely introduced a lag time. Data from tasks or trials where system response time slowed perceivably were excluded from the analysis. Additionally, participants were instructed not to change any of the default settings (such as font size).


PracticePartner® Version 11



of 48




Informed Consent

Demographic Questionnaire

Experimenter’s Verbal Instructions

Training slides

Observer’s Task Sheets

Post-task surveys (presented through Morae)

Post-test survey (System Usability Scale, SUS, presented through Morae)

Most of these materials are included in the appendices.

Usability Metrics






Effectiveness of the EHRUT by measuring participant success rates and errors.

Efficiency of the EHRUT by measuring the average task time and path deviations.

Satisfaction with the EHRUT by measuring ease of use ratings.

Data Scoring


of 48




Effectiveness:

Task Success





Optimal task performance time, as benchmarked by the 3 users who finished the task without errors and deviations and in the shortest amount of time, was calculated by averaging the times of these 3 users. In cases where more than 2 participants failed a task, the optimal task performance time was calculated using the 2 users who finished the task without errors and deviations in the shortest amount of time. No adjustments were made to the optimal time. Average group times were divided by the optimal performance time for a measure of how close to optimal efficiency each task was performed. It was decided not to define time allotments, because it was felt that different adjustments would have to be made for new tasks and for existing task flows, and we could not find a generally accepted empirical or principled way to determine meaningful adjustments. All participants were allowed to finish the tasks, and the time ratios provide a measure of how much longer an average user would take to perform these tasks.


Effectiveness:

Task Failures and

Errors





Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application is recorded. Deviations occur if the participant, for example, chooses a path to task completion which requires more steps than the optimal, fills out additional fields, or performs any steps to correct a previous error. This path is compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency:

Task Time


Satisfaction:

Task Rating


by administering both a simple post-task question as well as a post-session questionnaire.

After each task, the participant is asked to rate “Overall, this task was:” on a scale of 1

(Very Difficult) to 5 (Very Easy). If the task included a number of subtasks, a question

addressing the ease of each subtask was included (see Appendix G: End-of-Task

of 48



Questionnaires).




of 48


Results




Table 8. Usability Data by Task and Subtask - same as Table 2, repeated here for easier reference

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1


9 100 1.93 85.26

(94.29) 28.25 3.02 .78

4.56

(1.01)


52.93

(80.54) 14.67 3.60 .56

4.44

(0.53)


32.32

(17.63) 13.58 2.38 .22

4.78

(0.44)


9 100 1.07 30.53

(15.23) 18.85 1.62 0

4.89

(0.33)

Navigation 9 100 1

12.94

(7.48) 9.32 1.39 0

4.89

(0.33)


17.6

(8.73) 9.52 1.85 0

4.89

(0.33)

3 Create and activate demographics-based decision support (Health Maintenance alert)

9 100 1.36 113.43

(43.66) 75.12 1.5 .55

4.44

(.73)


20.21

(13.23) 15.38 1.31 .11

4.67

(0.5)

Activation of rule

9 100 1.67 28.24

(7.34) 25.57 1.11 0

4.33

(0.71)

Create specific overdue dates

9 100 1.24 64.99

(37.3) 35.17 1.85 .44

4.44

(.73)


9 22 NA, see


12 minutes,

38 seconds

NA, see

Discussion of

the Findin

gs

NA, see Discussion of the Findings

NA, see

Discussion of

the Finding

s

3.00

(1.41)

of 48



on performance with these tasks to be: 68.




averages obtained from those task-based usability scores varied between 3.11 and 4.89. The first

three tasks all scored over 4.4 with all tasks scoring over 4, quite a bit above the recommendation put

forward in NISTIR6 7742. The NISTIR 7742 states that “common convention is that average ratings for

systems judged easy to use should be 3.3., or above.” The final task presented a real difficulty to all

participants; in fact only two participants completed this task successfully. This task was rated as 3.11.

We will give special attention to the analysis of this task, which asked the participants not only to

implement a complex health maintenance rule, but it also asked to write a file with conditional logic, and

this appeared to be outside of what any of the participants saw as their normal job duties. The struggle

with this final task noticeable influenced how the participants felt about the system, and may have had

an impact on their final assessment, as expressed in the SUS score.





In the following table, Table 9, the usability metrics are reported grouped out by MU criterion. Please

note that the MU criteria related to clinician tasks have been tested with two other user groups (nurses

and physicians). Those results are summarized in 2 separate reports.

Table 9. Usability Data by Meaningful Use Criteria – same as Table 3, repeated here for easier reference


N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)



9 100 1.93 85.26

(94.29) 3.02 .78

4.56

(1.01)

Change drug-drug interaction severity level

9 100 1.07 30.53

(15.23) 1.62 0

4.89

(0.33)


Create and activate 9 100 1.36 113.43 1.5 .55 4.44




of 48



N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

simple demographics-based decision support (Health Maintenance alert)

(43.66) (.73)

Create and activate complex demographics and drug-based decision support (Health Maintenance alert)

9 22 NA, see


12 minutes,

38 seconds

NA, see Discussion

of the Findings

NA, see Discussion

of the Findings

3.00

(1.41)

of 48



170.314(a)(2) – Drug-drug, drug allergy interaction checks

Test contexts

Drug-drug, drug-allergy interaction configurations were tested twice. In Task 1 the participants were

asked to turn the configurations on and set the alerts to the most sensitive level. Task 2 tests the

usability of changing drug-allergy alerts to appear only for the highest level of drug-drug interactions.

Task 1 – Configure drug-drug interaction severity level

Analysis Notes


170.314(a)(2) – Drug-drug, drug-allergy interaction checks

Task Description

This task asked participants to configure drug-drug interaction alerts such that all alerts will show, even

alerts of the least severity.

Risk Analysis

High. Configuring alerts for drug-drug interactions based on severity allows clinicians to adjust

prescribed medications based on identified issues. Depending upon the severity level for which the

alert is configured, too many alerts may appear for the clinician, resulting in alert fatigue. On the other

hand, if the alert don’t show important interactions they remain inefficient. Depending on the situation

administrators should be able to configure this setting without error and without confusing the levels.

Scoring Notes


Exclusions

None

of 48



Table 10. Usability Data for Task 1 – Configure drug-drug interaction

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

1

Configure drug-drug interaction

severity level

9 100 1.93 85.26

(94.29) 28.25 3.02 .78

4.56

(1.01)

Navigation 9 100 2.08 52.93

(80.54) 14.67 3.60 .56

4.44

(0.53)

Configuration 9 100 1.61 32.32

(17.63) 13.58 2.38 .22

4.78

(0.44)

Participant Errors

Table 11. Participant Errors for Task 1 – Configure drug-drug interaction


3 Minor Navigation Error, participant explores menu options

6 Minor Navigation Error, participant explores the wrong menu

3 Minor Navigation Error, participant explores the wrong tab in correct screen

8 Medium

Select the wrong severity level. Selected the second to lowest. This participant never opened the drop down and seemed to assume that ‘1’ was the lowest level (in fact, ‘0’ is the lowest level). This would still display most interaction warnings, but it is not the lowest level, and so in a situation where a clinic wanted to display everything this would present a medium amount of risk.

9 Minor Navigation Error. This participant had never used this function before and explored many menus, resorting to the help files to eventually find the correct menu, screen and controls.

6 Medium

This participant selected another control and set it to high. This will increase the amount of documentation shown – something we didn’t ask for. When asked later, this participant did not understand this function. Changing this function did not appear to create a safety risk. However, ‘superstitious’ use of controls can always create unwanted states and therefore is associated with some amount of risk.

9 Minor Navigation Error. Another sequence of exploring menus by the participant unfamiliar with this function.

of 48


Effectiveness

All participants performed this task successfully. All except for one participant seemed very familiar

with this task. The variances in steps (deviations) and task times (deviations), were mostly due to this

participants’ long navigation sequences. Two other participants also accessed the wrong menus or

tabs briefly but found the functions quite quickly after that.

Efficiency

Eight of the nine participants quickly found the prescription screen and made the necessary

adjustments. One participant struggled with finding the correct menu; however she used the provided

online help files to eventually locate the controls she was looking for. It is mostly due to this last

participant’s long exploration sequence that the navigation step deviation ratio was 2.08 and the time

deviation was 3.02. While every participant set the severity level to a low level, one participant kept the

second to lowest (which is a medium mistake), but eight participants also set a second control to the

lowest level – which was not necessary. This second drop-down list allows to control the level of

documentation to display. When asked after the end of the study why they set this control, participants

seemed to be confused about the actual function of this control, and they appeared to want to ‘make

sure’ that they alert really displayed.

Comments


Table 12. Participant Comments for Task 1 – Configure drug-drug interaction


5 Would be helpful if the description for the levels in the "Rx Defaults" tab is more descriptive (eg- 0- Lowest- All interactions- minor & major, ...... )

6 Instead of a scale of 1 - 5 how about low severety, medium severity and high severity as drop downs (or similar)

Satisfaction

This familiar task was perceived to be easy for users (4.56).

Major Findings

Users familiar with the application had no difficulty finding it in the menu structure. However for a user

unfamiliar with this function, the menu labels provided little guidance. The help file eventually revealed

the navigation path, but the participant had to search for the path even there. I might be good to review

the help text and make the pointers to the navigation path more obvious. The task revealed two more

significant issues. As expressed in the participant comments and some of the observations, the

participants were slightly confused about the severity levels expressed in the drop down controls. Most

importantly, a second control on the same screen (Level of documentation to display) is poorly

understood and poorly labeled.

of 48


Task 2 – Change drug-allergy warning interventions

Analysis Notes


170.314(a)(2) – Drug-drug, drug-allergy interaction checks

Task Description

This task asked participants to change the configuration for displaying drug-allergy alerts based on

severity of the allergy.

Risk Analysis

High. Configuring alerts for drug-allergy interactions based on allergy severity allows clinicians to

adjust prescribed medications based on identified issues. Depending upon the severity level for which

the alert is configured, too many alerts may appear for the clinician, resulting in alert fatigue.

Scoring Notes


Exclusions

None


Table 13. Usability Data for Task 2 – Configure drug-allergy warning intervention

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


9 100 1.07 30.53

(15.23) 18.85 1.62 0

4.89

(0.33)

Navigation 9 100 1

12.94

(7.48) 9.32 1.39 0

4.89

(0.33)


17.6

(8.73) 9.52 1.85 0

4.89

(0.33)

Errors Table 14. Participant Errors for Task 2 – Configure drug-allergy warning intervention

Participant Error Severity Error Desctiption

- NA Participants did not perform any errors during this task.

of 48


Effectiveness

All participants performed this task successfully. After the navigation path had been discovered during

Task 1, no navigation errors were committed. This task also did not lead to any errors in changing the

severity to the desired level or to making any additional changes to the documentation control.

Efficiency

This task was performed with high efficiency. The overall step deviation ratio was 1.07, and the overall

time deviation was 1.62. After learning where to find the function (or refreshing their memory on where

to find it), all participants found the function quickly and performed the task with high efficiency.

Comments


Table 15. Participant Comments for Task 2 – Configure drug-allergy warning intervention


5 Better descriptions for the levels on the "Rx Defaults- Drug Interactions tab" (eg- 3-Highest- Only Major)

Satisfaction

The satisfaction ratings were high, 4.89 across the whole task.

Major Findings

This task was performed without problems. However, the observations from Task1 apply here as well.

What this second task showed was that users don’t have a problem changing the severity settings,

however they still express that it would be good to have slightly more direct labels for the controls.


Test contexts

Creation of clinical decision support interventions was tested in 2 tasks. Task 3 tests the usability of

creating an alert based on a patient's demographic data and asks participants to configure a specific

set of overdue alerts. Task 4 tests the configuration of an alert that combines demographic and

medication based alerts. This task asked users not only to manipulate forms in the Practice Partner

applications, but also to create and load external rules files.

of 48


Task 3 – Create and activate demographics-based decision support (Health

Maintenance alert)

Analysis Notes



Task Description

This task asked participants to create an alert based on a complex demographic (female and between

11 and 18 years of age). It then asked participants to create alerts at ages 12, 14, 16, 18 years.

Risk Analysis

High. Creating alerts can help clinicians to recognize potential issues for patients and allow them to

address those issues during an encounter. However, alerts must be used with care as too many can

lead to alert fatigue, resulting in clinicians missing important alerts due to the possibility of false alarms.

Scoring Notes


Exclusions

None

Quantitative Summary Table 16. Usability Data for Task 3 – Create and activate alert

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)

5 Create and activate

demographics-based decision support (Health

Maintenance alert)

9 100 1.36 113.43

(43.66) 75.12 1.5 .55

4.44

(.73)


20.21

(13.23) 15.38 1.31 .11

4.67

(0.5)

Activation of rule

9 100 1.67 28.24

(7.34) 25.57 1.11 0

4.33

(0.71)

Create specific

overdue dates

9 100 1.24 64.99

(37.3) 35.17 1.85 .44

4.44

(.73)

of 48


Errors Table 17. Participant Errors for Task 3 – Create and activate alert

Participant Error Severity Error Desctiption

3 Minor Initially wrong use of special format of age/month field. Figures out how to use it.

7 Minor Navigation error. Picked wrong menu and went down wrong path, but quickly corrected and found the desired function.

7 Minor Initially wrong use of special format of age/month field. Figures out how to use it.

9 Minor Wrong use of special format of age/month field.

9 Minor Second wrong used of special format. Used another rule as an example for how to use it and corrects her mistake.

Effectiveness

All participants performed this task successfully. One participant committed a minor navigation error

that was quickly corrected. A specially formatted field that asks participants to enter year/month age

led three participants to make minor mistakes in using this format. The system feedback helped them

to figure out what as needed, and one participant looked at some other rules to see how this field had

been filled out. This situation was handled well by the user interface, because the system did not

accept wrongly formatted dates, and gently guided participants to enter the correct information.

Efficiency

Participants performed this task quite efficiently. The overall step deviation ratio was 1.36., and the

overall time deviation ratio was 1.5. The slight struggle with the special age format led to somewhat

longer time deviation of 1.85, but even this subtask was conducted quite quickly.



Table 18. Participant Comments for Task 3 – Create and activate alert


5 None …

7 Didn’t test it…

of 48


Satisfaction

Although participants performed this task without major errors or inefficiencies, the overall satisfaction

rating for this task was a litter lower than for Task 1 or 2 (4.44). This may have been because

participants perceived this to be a cognitively more challenging task (it had more sub goals). But a few

participants also mentioned that they would usually check to make sure the rule really worked. We did

not ask them to do this in our setting, but not being able may have left them feeling like they were not

sure if they had done this task correctly. This is an issue with providing sufficient feedback after the

task has been completed. We will come back to this issue when discussing the Task 4.

Major Findings

In general, participants had little difficulty with this task; the user interface provided good guidance

when it came to entering the correct format for the age. The only possible issue could be providing

better feedbacks about whether a new rule had the desired effect.

Task 4 – Create and activate complex decision support

Analysis Notes



Task Description

This task asked participants to create a complex rule using demographic and medication information.

The task was complicated by a number of things. 1. The system comes with a number of pre-

configured complex rules, and creating new ones does not happen very often. Our participants told us

that they may have had to modify an existing rule, but never had to create one from scratch. 2. To

create a new rule, a conditional rule file must be written using a system specific syntax and elements.

The participants were not familiar with writing these statements. 3. The rule file then has to be stored

in a specific location. 4. The rule file has to be added to a health maintenance trigger that is defined by

system controls. In our case the demographic rule was specified in the rule file, but the medication

rule was specified through the user interface. 5. The rule file then has to be added to the medication

trigger to result in the combination of the two criteria. (6. After the creation, the alert should be tested to

make sure that it works as desired).

We provided training on an example problem that was analogous to the problem in the usability test.

The only difference between the two problems was the name of the medication for which to use the rule

and the specifics of the demographic information to use as a trigger. Participants were allowed to use

the logic statement of the example during the usability test to help them figure out the correct syntax,

because that is how non-programmers often approach a programming problem (use working examples

to modify).

of 48


Despite these supports only 2 participants (out of 9) successfully created a rule. Since most of the time

was spent writing the logic rule and trying different ways to load it into the existing rule we did not

attempt a keystroke-level analysis of this task. We will discuss the findings in narrative and suggest

solutions to some of the discovered problems from that narrative.

Risk Analysis

High. Correctly targeting alerts helps those clinicians who need the information to receive that

information while treating patients, while also keeping the information from clinicians who do not need

it, thereby reducing the possibility of alert fatigue.

Scoring Notes


Exclusions

None


Table 19. Usability Data for Task 4 – Create complex alert rule

Tasks and subtasks

N Task Success

Path Deviation



# Percentage


Mean Observed

(SD)


Mean (SD)

Mean (SD)


9 22

NA, see discussion

below in the result section

12 minutes,

38 seconds

NA, see

discussion

below in the result

section

NA, see discussion below in the result section

NA, see

discussion

below in the result

section

3.00

(0.78)

of 48


Error and Failure Analysis

Table 20. Participant Errors for Task 4 – Causes for overall task failures

Rule file logic Rule file syntax Rule file location Choosing the correct template to attach rule to

Attaching the rule in the correct manner

3 participants failed

Confused larger than with smaller than, or male with female


Did not obey spacing rules, misspelled rule elements


7 participants stsruggled:

sarted with the wrong template (demographic), but eventually figured out they had to start with a medication template

2 failed, started with the wrong type of template


entered the rule in the wrong field, applied the wrong path name, or made additional changes to the template logic

High Severity High Severity High Severity High Severity High Severity



Table 21. Participant Comments for Task 4 – Create complex alert rule


1 Conditional Logic Threads are difficult for me.

5

It would be great to have a "Browse" next to the Rule file line on the HM procedure setup screen, so one does not have to type in the name of the file and can browse and attach it. This will minimize errors and save time.

7 need ability to browse to pick the rule file

6 Did not know I needed to create a rule file. The main screen was not clear on whether or not I could do what I wanted given the out of the box controls.

Effectiveness

Only 2 of our 9 participants succeeded in this task, and even these two participants struggled for a long

time with wrong starts, but eventually ended up with the correct logic, syntax, selecting the correct

template to attach the rule to, and managing to do this in the correct manner.

Efficiency

Even the two participants, who succeeded, struggled tremendously with this task. They needed an

average of 12.5 minutes to finish, and exhibited long sequences of navigating back and forth between

the different task components, example files, the file containing the rule file (to help remember its

of 48


name), and exploring the template name, templates and procedure forms to help them figure out this

task. There were effectively too many task steps to count in any meaningful way.

Satisfaction

The high degree of difficulty is expressed in low satisfaction scores (average of 3.0).

Major Findings

The experimenters were surprised by the magnitude by which the participants struggled with this task,

given that a problem with a very similar structure had been reviewed during the training session, and

participants had even been asked to save that information and used it as a ‘crutch’ during this task.

1. Administrators had an extremely difficult time writing conditional logic statements.

a. They failed to express the correct logic

b. They failed to use the correct syntax

2. It was difficult to find the correct ways to attach the rule file into the templates:

a. It was difficult to understand what part of the complex rule (demographics vs. medication) to

include in the rule, and what ‘trigger’ template to connect this rule to

b. Even when the correct template and procedure had been selected, attaching the rule was

difficult

i. because the correct field was ‘greyed out’ making it look like it could not be edited (2

participants selected another field that was white and appeared much more ‘editable’

ii. Participants expected to enter the whole path name, when only the file name was

needed. The error message did not explain how to fix the problem.

3. Experimental constraints. Several participants mentioned that they would ideally ‘test’ the rule by

creating a test patient and making sure that it appeared in certain test situation that they would

mock up. This was outside the realm of the usability study but may have led to a higher success

rate eventually.

of 48


List of Appendices






Appendix E: Experimenter's Short Biography




of 48



of 48
















familiarize yourself with these new functions, please walk through this short power point.











- Please click anywhere on your screen to take control of the system.


to.

of 48





- When you have started, I won’t be able to any more questions. Do you have any


of 48



of 48








• Your feedback will help us improve the usability of Horizon Ambulatory Care and other EMR functions.

• During the study, you will be performing a few routine tasks using HAC, most of which will be very

familiar to you.

• This presentation provides a review of the routine tasks.

• The following screens present an overview of features in self-guided presentation mode.




on your keyboard.


Study Instructions


• Our session today will last about 30 minutes. During that time you will use HAC and Clinicals

Configuration Tool to perform tasks.









Starting the Study



of 48








Performing the Task













right location.




of 48



of the questions.





of 48


Appendix E: Experimenter's Short Biography












of 48



Introduction

You are the administrator for the McKesson Test Clinic.

In this session you will be configuring the drug-drug interaction settings and a few Health

Maintenance alerts for this clinic.

If you are ready to proceed, please click 'Start'.

Task 1

Task Instructions:

Adjust the severity for drug-drug interaction checks.

Select the lowest setting to display all interventions (both high significance and

low significance interventions).

Task 2

Task Instructions:

Your practice has changed the policy.

Adjust the drug severity to allow only the highest significance interactions to display.

Task 3

Now activate a new Health Maintenance Procedure:

Add “Depression Screen” to the Health Maintenance procedures applicable to 11-

18 year old girls, then …

activate overdue checking for that procedure at ages 12, 14, 16, 18.

of 48


Task 4

Task Instructions:

Add a Health Maintenance procedure “Beta HCG Test” (B-HCG) for:

for female patients who are 55 years old and younger, and

who are taking ACCUTANE .

Beta HCG testing should be done every month.

of 48



1 - Configure Drug Interaction

1. Finding the Prescription Default screen was ...


2. Allowing all interactions to display was ...





Comments:

2 - Adjust Drug Interaction

1. Finding the Prescription Defaults was ...


2. Adjusting the severity setting to high was ...





Comments:

of 48


3 – Create/Activate HM Alert



2. Activating this Health Maintenance alert was ...


3. Creating the overdue alert was ...





Comments:

4 - Create/Activate HM Alert (Complex Alert including Creating of Conditional Rule)

1. Creating the text file with the rule was ...


2. Saving the rule file in the correct location was ...


3. Finding the screen to activate the Health Maintenance report was ...


4. Activating the Health Maintenance Alert was ...


of 48





Comments:

of 48




questionnaires.






















onc hit certification program test results summary for ...€¦ · email:...

Documents