onc hit certification program test results summary for...

Template Version 1

Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016

of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision

ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Part 1: Product and Developer Information

1.1 Certified Product Information

Product Name: ezEMRxPrivate

Product Version: 9.00

Domain: Ambulatory

Test Type: Complete EHR

1.2 Developer/Vendor Information

Developer/Vendor Name: ezEMRx Inc

Address: 25W217 Lake Street

Roselle IL 60172

Website: www.ezemrx.com

Email: [email protected]

Phone: 630-307-7600

Developer/Vendor Contact: Siri Kumar

Part 2: ONC-Authorized Certification Body Information

2.1 ONC-Authorized Certification Body Information

ONC-ACB Name: InfoGard Laboratories, Inc.

Address: 709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

Website: www.infogard.com


Phone: (805) 783-0810

ONC-ACB Contact: Adam Hardcastle

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Adam Hardcastle

EHR Certification Body Manager ONC-ACB Authorized Representative Function/Title

2/28/16 Signature and Date

http://www.infogard.com/

mailto:[email protected]

Template Version 1



2.2 Gap Certification The following identifies criterion or criteria certified via gap certification

§170.314

(a)(1) (a)(17) (d)(5) (d)(9)

(a)(6) (b)(5)* (d)(6) (f)(1)

(a)(7) (d)(1) (d)(8)

*Gap certification allowed for Inpatient setting only

No gap certification

2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification

§170.314 (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4)

(f)(5) Optional & Amb. only (a)(6) (b)(2) (d)(5)

(a)(7) (b)(3) (d)(6) (f)(6) Optional &

Amb. only (a)(8) (b)(4) (d)(7) (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4)

(a)(13) (c)(2) (e)(3) Amb. only

No inherited certification

Template Version 1



Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: 14-2469-R-0004 V1.3 Test Dates: December 26, 27, 30, 31, 2013; January 2, 10, 15, 20, 22, 2014; March 24, 2014; May 2, 2014

3.1 NVLAP-Accredited Testing Laboratory Information

ATL Name: InfoGard Laboratories, Inc.

Accreditation Number: NVLAP Lab Code 100432-0

Address: 709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

Website: www.infogard.com


Phone: (805) 783-0810

ATL Contact: Milton Padilla

For more information on scope of accreditation, please reference http://ts.nist.gov/Standards/scopes/1004320.htm

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:

Milton Padilla

EHR Test Body Manager ATL Authorized Representative Function/Title

2/28/16 Signature and Date

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

Additional Software Applicable Criteria Functionality provided by Additional Software

No additional software required

3.2.2 Test Tools

Test Tool Version

http://www.infogard.com/

mailto:[email protected]

Template Version 1



Test Tool Version Cypress 2.4.0 ePrescribing Validation Tool 10.6 HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.7.0

HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.8.0

HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7.0 HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7.0 Transport Testing Tool 174 Direct Certificate Discovery Tool 2.1

No test tools required

3.2.3 Test Data

Alteration (customization) to the test data was necessary and is described in Appendix A

No alteration (customization) to the test data was necessary

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(13)

§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree

Template Version 1



Criterion # Standard Successfully Tested

(a)(15)(i)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(16)(ii) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(b)(7)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(e)(1)(i)

Annex A of the FIPS Publication 140-2 • Signature algorithm: sha1RSA • Signature hash algorithm: sha1 • Public key: RSA (2048 Bits)

(e)(1)(ii)(A)(2) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(e)(3)(ii)

Annex A of the FIPS Publication 140-2 • Signature algorithm: sha1RSA • Signature hash algorithm: sha1 • Public key: RSA (2048 Bits)

Common MU Data Set (15)


§170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested

Template Version 1



Newer Version Applicable Criteria

No newer version of a minimum standard was tested

3.2.5 Optional Functionality

Criterion # Optional Functionality Successfully Tested

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)


Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)


Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)

No optional functionality tested

Template Version 1



3.2.6 2014 Edition Certification Criteria* Successfully Tested

Criteria # Version

Criteria # Version

TP** TD*** TP TD (a)(1) 1.2 1.5 (c)(3) 1.7.1 (a)(2) 1.2 (d)(1)

(a)(3) 1.2 1.4 (d)(2) 1.4 (a)(4) 1.4 1.3 (d)(3) 1.3 (a)(5) 1.4 1.3 (d)(4) 1.2 (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) 1.2 (d)(7) 1.2 (a)(9) 1.3 1.3 (d)(8) (a)(10) 1.2 1.4 (d)(9) Optional (a)(11) 1.3

(e)(1) 1.7 1.4 (a)(12) 1.3 (e)(2) Amb. only 1.2 1.5 (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) 1.3 1.3 (a)(16) Inpt. only (f)(3) 1.3 1.3 (a)(17) Inpt. only (f)(4) Inpt. only

(b)(1) 1.6 1.3 (f)(5) Optional &

Amb. only (b)(2) 1.4 1.5

(b)(3) 1.4 1.3 (f)(6) Optional &

Amb. only (b)(4) 1.3 1.4

(b)(5) 1.4 1.3.0 (g)(1) 1.6 1.8 (b)(6) Inpt. only (g)(2) 1.6 1.8 (b)(7) 1.4 1.5 (g)(3) 1.3

(c)(1) 1.7.1 (g)(4) 1.2

(c)(2) 1.7.1

*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)

http://www.healthit.gov/certification

Template Version 1



3.2.7 2014 Clinical Quality Measures*

Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested

*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

2 90 136 155 V2

22 117 V2 137 156 V2

50 V2 122 V2 138 V2 157

52 123 139 158

56 124 140 159

61 125 141 160

62 126 V2 142 161

64 127 143 163

65 128 144 164

66 129 145 165 V2

68 V3 130 146 166

69 V2 131 147 167

74 132 148 V2 169

75 133 149 177

77 134 153 179

82 135 154 V2 182

Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113

55 104 114

60 105 171

http://www.cms.gov/

Template Version 1



3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(7) (a)(15) (b)(5)

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(7) (a)(15) (b)(5)

Automated Measure Calculation was not tested

3.2.9 Attestation

Attestation Forms (as applicable) Appendix

Safety-Enhanced Design* B

Quality Management System** C

Privacy and Security D

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product

Template Version 1



Appendix A: Alteration of Test Data

Criteria Explanation 170.314(b)(2) Determined that modified Test Data had equivalent level of robustness to

NIST Test Data

170.314(b)(7) Determined that modified Test Data had equivalent level of robustness to NIST Test Data

170.314(e)(1) Determined that modified Test Data had equivalent level of robustness to NIST Test Data

170.314(e)(2) Determined that modified Test Data had equivalent level of robustness to NIST Test Data

Template Version 1



Appendix B: Safety-Enhanced Design

EHR Usability Test Report of ezEMRxPrivate Version 9.00

Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

Report Prepared By: ezEMRx Customer Teams

Balaji Venkatesh, Nick Daftari

Confidential Page 2 of 28

Table of Contents

Table of Contents ...............................................................................................................................2

EXECUTIVE SUMMARY .......................................................................................................................4

INTRODUCTION .................................................................................................................................5

UCD PROCESS OUTLINE ......................................................................................................................6

DESIGN ACTIVITIES .................................................................................................................................... 6

UCD PROCESS CHART ................................................................................................................................ 7

STUDY METHOD .................................................................................................................................8

PARTICIPANTS ........................................................................................................................................... 8

STUDY DESIGN .......................................................................................................................................... 9

DATA SCORING ........................................................................................................................................ 10

TASKS ...................................................................................................................................................... 11

TEST ENVIRONMENT ............................................................................................................................... 11

TEST FORMS AND TOOLS ........................................................................................................................ 12

USABILITY METRICS ................................................................................................................................. 12

SUMMARY OF TEST RESULTS ............................................................................................................ 13

PARTICIPANTS ......................................................................................................................................... 13

DATA ANALYSIS AND REPORTING ........................................................................................................... 14

DISCUSSION OF THE FINDINGS ............................................................................................................... 15

EFFECTIVENESS ....................................................................................................................................... 15

EFFICIENCY .............................................................................................................................................. 15

SATISFACTION ......................................................................................................................................... 15

MAJOR FINDINGS .................................................................................................................................... 16

AREAS FOR IMPROVEMENT .................................................................................................................... 16


APPENDIX ........................................................................................................................................ 17

Sample - Initial Questionnaire ................................................................................................................ 17

Sample – Informed Consent ................................................................................................................... 19

Sample – Incentive receipt and acknowledgement ................................................................................ 20

Sample – SUS Questionnaire................................................................................................................... 21

Sample – Tasks ........................................................................................................................................ 22

Task 1: Medication list ........................................................................................................................ 22

Task 2: Medication allergy list ............................................................................................................. 23

Task 3: Interaction Check – [D-to-D and D-to-A] ................................................................................ 24

Task 4: Prescribe Medication .............................................................................................................. 25

Task 5: CPOE ........................................................................................................................................ 26

Task 6 : Clinical Information Reconciliation ........................................................................................ 27

Task 7: Clinical Decision support ......................................................................................................... 28


EXECUTIVE SUMMARY

A usability test of ezEMRxPrivate 9.00, an Ambulatory EHR was conducted onsite at participant locations

by ezEMRx. The purpose of this test was to test and validate the usability of the current user interface,

and provide evidence of usability in the EHR Under Test (EHRUT).

During the usability test, six healthcare providers served as participants and used the EHRUT in

simulated, but representative tasks.

This study collected performance data on seven tasks conducted on an EHR:

Computerized provider order entry

Drug-drug, drug-allergy interaction checks

Medication list

Medication allergy list

Clinical decision support

Electronic prescribing

Clinical information reconciliation

During the one-on-one usability test, each participant was asked to review and sign an informed

consent/release form (template included in Appendix); they were instructed that they could withdraw at

any time. Participants all had prior experience with the usage of a previous version of the EHRUT.

The participants were instructed to complete a series of tasks using the EHRUT. During the testing, the

test time was timed and recorded as user performance data on paper. There was no assistance provided

on how to complete the task.

All participant data were de-identified – no correspondence could be made from the identity of the

participant to the data collected. Following the conclusion of the testing, participants were

compensated with $150 (physicians) and $100 (clinical staff) for their time.

Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the

Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the

usability of the EHRUT.

The UCD process applied for EHR Under Test (EHRUT) is not a industry standard process, but rather a

custom process which is broadly based on the ISO 134071 Human centred design processes for

interactive systems.

1 ISO 13407 - Human-centred design processes for interactive systems

ISO 13407 provides guidance on achieving quality in use by incorporating user centred design activities throughout

the life cycle of interactive computer-based systems. It describes user centred design as a multi-disciplinary

activity, which incorporates human factors and ergonomics knowledge and techniques with the objective of


INTRODUCTION

The EHRUT(s) tested for this study was ezEMRxPrivate 9.00, an Ambulatory EHR. Designed to present

medical information to healthcare providers in a clinic and specialty setting, the EHRUT consists of a

workflow based approach of data capture and retrieval. The usability testing attempted to represent

realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the current user interface, and provide

evidence of usability in the EHR Under Test (EHRUT).

To this end, measures of effectiveness, efficiency and user satisfaction, such as time taken to write a

prescription, enter medications, allergies, execute orders with decision support validation, were

captured during the usability testing.

The UCD process applied for EHR Under Test (EHRUT) is not an industry standard process, but rather a

custom process which is broadly based on the ISO 13407 Human centred design processes for

interactive systems.

The UCD study methods (adaptation based on ISO/IEC 25062:20062 and NIST IR 77423) and the UCD

Process (non standard custom process) described in this document were applied for the following

criteria:

· §170.314(a)(1) Computerized provider order entry

· §170.314(a)(2) Drug-drug, drug-allergy interaction checks

· §170.314(a)(6) Medication list

· §170.314(a)(7) Medication allergy list

enhancing effectiveness and productivity, improving human working conditions, and counteracting the possible

adverse effects of use on human health, safety and performance.

http://www.iso.org/iso/catalogue_detail.htm?csnumber=21197

2 ISO/IEC 25062:2006 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) --

Common Industry Format (CIF) for usability test reports.ISO/IEC 25062:2006 provides a standard method for

reporting usability test findings. The format is designed for reporting results of formal usability tests in which

quantitative measurements were collected, and is particularly appropriate for summative/comparative testing.

http://www.iso.org/iso/catalogue_detail.htm?csnumber=43046

3 NISTIR 7742 - Customized Common Industry Format Template for Electronic Health Record Usability

Testing.NISTIR 7742 provides a template for the modified version of Software engineering Software product

Quality Requirements and Evaluation(SQuaRE) Common Industry Format (CIF) for usability test reports (ISO/IEC

25062:2006(E)), the Common Industry Format (CIF) usability test report.

http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907312


· §170.314(a)(8) Clinical decision support

· §170.314(b)(3) Electronic prescribing

· §170.314(b)(4) Clinical information reconciliation

UCD PROCESS OUTLINE

The UCD process applied for EHR Under Test (EHRUT) is not an industry standard process, but rather a

custom process which is broadly based on the “ISO 13407 Human centred design processes for

interactive systems”.

ISO 13407 provides guidance on achieving quality in use by incorporating user centred design activities

throughout the life cycle of interactive computer-based systems. It describes user centred design as a

multi-disciplinary activity, which incorporates human factors and ergonomics knowledge and techniques

with the objective of enhancing effectiveness and productivity, improving human working conditions,

and counteracting the possible adverse effects of use on human health, safety and performance.

DESIGN ACTIVITIES

The User Centred Design process for EHR Under Test (EHRUT) are based on four design activities as

described in “ISO 13407 Human centred design processes for interactive systems”

The four activities initiated early into the product design, and user interface designs are:

- Understand and specify the context of use

- Specify the user and organizational requirements

- Produce design solutions

- Evaluate designs against requirements

The above activities were applied on an iterative nature for every interactive user interface element.


UCD PROCESS CHART

The UCD process and the interdependence of the UCD activities applied on an iterative model on the

EHR Under Test (EHRUT).


STUDY METHOD

PARTICIPANTS

Participants had a mix of backgrounds and demographic characteristics to facilitate the usability study.

The following is a table of participants by characteristics, including demographics, professional

experience, computing experience and user needs for assistive technology. Participant names were

replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Sl

No. Participant ID Gender Age Education

Occupation

/ Role

Professional

Experience

Computer

Experience

Product

Experience

Assistive

Technology

Needs

1 EZUSG20131501 F 23-39 College Medical

Assistant 2 Years Excellent Average None

2 EZUSG20131502 F 40-59 College Nurse 2 Years Excellent Average None

3 EZUSG20131503 M 40-59 MD Physician 10 Years Good Good None

4 EZUSG20131504 F 40-59 College LPN 15 Years Excellent Good None

5 EZUSG20131505 M 23-39 MD Physician 12 Years Good Average None

6 EZUSG20131506 M 40-59 MD/Phd Physician 27 Years Excellent Good None

A total number of Six participants were recruited (total number of participants recruited), and

participated in the usability tests on 12-16-2013 and 12-17-2013. Participants were scheduled for one on

one session(s) with a time of 60 minutes.

Usability tests were conducted on site at participant offices situated in Chicago, IL and Calumet City, IL.


STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that is,

effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs

of the participants. The data from this test may serve as a baseline for future tests with an updated

version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short,

this testing serves as both a means to record or benchmark current usability, but also to identify areas

where improvements must be made.

The study is an adoption of the process as defined in NIST IR 7742. The methodology has been adjusted

to accommodate usability requirements of the ezEMRx EHR and a mechanism to drive user feedback

into product development lifecycles. The participants are engaged within the realm of the clinic to

simulate realistic environments; however the same is controlled with the use of a provided laptop while

under one-on-one study. The task elements and results are documented on paper while the study is in

progress. This methodology injects directed user feedback on usability into product development life

cycle.

During the usability test, the system was evaluated for effectiveness, efficiency and satisfaction as

defined by measures collected and analyzed for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations (comments)

Participant’s satisfaction ratings of the system

Additional information about the various measures can be found in Section on Usability Metrics.


DATA SCORING

The following table describes details how tasks were scored, errors evaluated, and the time data

analyzed.

Measures Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance,

within the time allotted on a per task basis.

The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by

the optimal time for the task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when

constructing tasks.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the

end of the allotted time before successful completion, the task was counted as “Failures.” No task times were taken

for errors.

Not all deviations would be counted as errors. This is expressed as the mean number of failed tasks per participant.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was reviewed. Deviations occur if the participant, for

example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted

incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the

observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency:

Task Time

Each task was timed. Only task times for tasks that were successfully completed were included in the average task

time analysis. Average time per task was calculated for each task.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a

simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate

“Overall, this task was:” on a scale of 1 (Very Easy) to 5 (Very Difficult). These data are averaged across participants.

To measure participants’ confidence in and likeability of the EHRUT overall, the testing team administered the

System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system

frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this

system very quickly.” See full System Usability Score questionnaire in Appendix.


TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities

in a clinic, user might do with this EHR, including:

Computerized provider order entry

Drug-drug, drug-allergy interaction checks

Medication list

Medication allergy list

Clinical decision support

Electronic prescribing

Clinical information reconciliation

Tasks were selected based on 170.314(g)(3) Safety-enhanced design criteria as part of the ONC HIT

Certification Program.

TEST ENVIRONMENT

The EHRUT would be typically be used in a healthcare office or clinic.

In this instance, the testing was conducted at 3 different clinic locations situated in Chicago-Illinois,

Welch-Minnesota and Highland-Indiana. Apart from standard office facilities, each of the clinic locations

(test locations) comprised of comfortable and ergonomic work spaces, and a business class internet

connection with a minimum bandwidth of 25 MBPS for downloads and 10 MBPs for uploads.

For testing, the participant used a laptop computer comprising of a 15.6 inch display, Intel core i3

processor with 4GB memory, and configured with a 32 bit Microsoft Windows 7 as the operating

system. During the test, the screen resolution was set to 1152x768 with color set to “True Color (32

bit)”. The participants used a mouse and keyboard when interacting with the EHRUT. The participants

were instructed not change any of the default system settings.

The application (EHRUT) itself was setup by the vendor using a training database. The application was

setup to be accessible by desktop/laptop computers over a WAN connection using a browser and an

URL. Technically, the system performance (i.e., response time) was representative to what users would

experience with a production implementation.

During the test, the participants used the laptop configured with the Microsoft Windows 7 platform (as

stated above), and Mozilla Firefox ESR Version 17 to access the EHRUT that was setup previously with a

training database on a WAN connection.


TEST FORMS AND TOOLS

During the usability test, the documents used were:

Informed Consent

Initial Questionnaire

Participant Instructions

SUS Questionnaire

Incentive Receipt & Acknowledgement

Examples of these documents can be found in Appendices respectively.

The participant’s interaction with the EHRUT was captured and recorded within the EHR.

USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records, EHRs should support a process that provides a high level of usability for all users. The goal is for

users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To

this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability

testing.

The goals of the test were to assess:

Effectiveness of EHRUT by measuring participant success rates and errors

Efficiency of EHRUT by measuring the average task time and path deviations

Satisfaction with EHRUT by measuring ease of use ratings


SUMMARY OF TEST RESULTS

PARTICIPANTS

A total of six participants were tested on the EHRUT(s). Participants in the test were physicians, nurses

and medical assistants. Participants were compensated $150 (physicians) and $100 (nurse, medical

assistant) for their time. In addition, participants had no direct connection to the development of the

EHRUT(s). Participants are actual end users and were familiar with the use of the EHRUT. Participants

were given the same orientation and level of training.

For the test purposes, end-user characteristics have been identified and translated into this document.

Participants had a mix of backgrounds and demographic characteristics to facilitate the usability study.

The following is a table of participants by characteristics, including demographics, professional

experience, computing experience and user needs for assistive technology. Participant names were

replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Sl

No. Participant ID Gender Age Education

Occupation

/ Role

Professional

Experience

Computer

Experience

Product

Experience

Assistive

Technology

Needs

1 EZUSG20131501 F 23-39 College Medical

Assistant 2 Years Excellent Average None

2 EZUSG20131502 F 40-59 College Nurse 2 Years Excellent Average None

3 EZUSG20131503 M 40-59 MD Physician 10 Years Good Good None

4 EZUSG20131504 F 40-59 College LPN 15 Years Excellent Good None

5 EZUSG20131505 M 23-39 MD Physician 12 Years Good Average None

6 EZUSG20131506 M 40-59 MD/Phd Physician 27 Years Excellent Good None

A total number of Six participants were recruited (total number of participants recruited), and

participated in the usability tests on 12-16-2013 and 12-17-2013. Participants were scheduled for one on

one session(s) with a time of 60 minutes.

Usability tests were conducted on site at participant offices situated in Chicago, IL and Calumet City, IL.


DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability

Metrics section above.

The usability testing results for the EHRUT are detailed below. The results should be seen in light of the

objectives and goals outlined in Section Study Design.

Following is a summary of the performance and rating data collected on the EHRUT.

Measure

Task

N Task Success Path Deviation Task Time Errors Task Ratings

1=Very Easy

# Mean (SD)

Deviations

(Observed /

Optimal)

Mean (SD)

Secs Mean (SD) Mean (SD)

Computerized provider

order entry 6 50% 1 70 50% 2

Drug-drug, drug-allergy

interaction checks 6 67% 1 140 33% 2

Medication list 6 100% 1 63 0 1

Medication allergy list 6 100% 1 46 0 1

Clinical decision support 6 67% 1.3 105 33% 3

Electronic prescribing 6 100% 1 108 0 2

Clinical information

reconciliation 6 67% 1 90 33% 3

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system

based on performance with these tasks to be: 83%. Broadly interpreted, scores under 60 represent

systems with poor usability; scores over 80 would be considered above average.


DISCUSSION OF THE FINDINGS

Discussions with the participants ranged in findings from slowness and usability in areas of CPOE, CDS

and medical reconciliation. Deviations were noted as some participants could not follow directions as

outlined.

It was noted that the slowness related to Drug-to-Drug interactions was due to access to staging systems

as compared to production systems.

Functional concepts of medical reconciliation and CPOE seemed to be challenging and discussions

indicated mandatory requirements of user training and education of requirements.

The presentation of CDS alerts invoked positive and negative responses from the participants. There

were indications of user looking to shut of such alerts as seemed intrusive to regular workings. On an

alternate note, some participants found the CDS alerts to be apt towards clinical decision assistance.

The overall discussions did present a positive mindset from the participants.

EFFECTIVENESS

The analysis indicated the most participants were successful in completing the tasks as presented.

Deviations were noticed on participants wherein the roles restricted functional areas (For, example,

CPOE, CDS functions were more relevant to Physicians than Nurses and Medical Assistants)

EFFICIENCY

The analysis indicated the physicians were able perform all functions successfully. This portrayed

accurate workings in realistic production environments. There were no deviations noted during the

physician based studies. Observations of participants with roles such as Nurses and Medical Assistants

did show deviations, however this is considered as exclusions since tasks such as CPOE and CDS were

not realistic events in daily working and hence the participants could not relate to the functional aspects

of such tasks.

SATISFACTION

To measure participants’ confidence in and likeability of the EHRUT overall, the testing team

administered the System Usability Scale (SUS) post-test questionnaire. The satisfaction index presented

a very positive outlook on usability. The responses have been aggregated and presented in the table

below.

Questions included, “I think I would like to use this system frequently,” “I thought the system was easy

to use,” and “I would imagine that most people would learn to use this system very quickly.” The

responses indicated very positive user usability responses.


Aggregate SUS index -- 1 (Strongly Disagree) to 5 (Strongly Agree)

Index Parameter Index

I think that I would like to use this system frequently 4

I found the system unnecessarily complex 3

I thought the system was easy to use 4

I think that I would need the support of a technical person to be able to use this system 2

I thought there was too much inconsistency in this system 2

I would imagine that most people would learn to use this system very quickly 4

I found the system very cumbersome to use 3

I felt very confident using the system 4

I needed to learn a lot of things before I could get going with this system 3

MAJOR FINDINGS

The major finding based on the study indicated slowness of interaction checks by all participants. This

has however been discounted as this is related to slow response times from third party staging systems.

It was further determined that, slow response times are not present in production environments.

AREAS FOR IMPROVEMENT

Observations and discussions did present several areas of improvement. The highlights are:

Better alert responses to actions on the user interface

Training for users based on roles

More intuitive error descriptors

Improved directions for study


APPENDIX

Sample - Initial Questionnaire

ezEMRx are recruiting individuals to participate in a usability study for an electronic health

record. We would like to ask you a few questions and if would like to participate. This should

only take a few minutes of your time. This is strictly for research purposes. If you are interested

and qualify for the study, you will be paid to participate.

1. Have you participated in a focus group or usability test in the past 6 months?

2. Do you, or does anyone in your home, work in marketing research, usability research, web design […etc.]?

3. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [

4. Which of the following best describes your age? [23 to 39; 40 to 59; 60 - to 74; 75 and older]

5. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]

6. Do you require any assistive technologies to use a computer? [if so, please describe]

Professional Demographics

What is your current position and title? (Must be healthcare provider)

RN: Specialty ________________

Physician: Specialty ________________

Resident: Specialty ________________

Administrative Staff

How long have you held this position?

Describe your work location (or affiliation) and environment?

Which of the following describes your highest level of education? [e.g., high school graduate/GED, some

college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]


Computer Expertise

About how many hours per week do you spend on the computer? [ e.g., 0 to 10, 11 to 25, 26+ hours per

week]

What computer platform do you usually use? [e.g., Mac, Windows, etc.]

What Internet browser(s) do you usually use? [e.g., Firefox, IE, Chrome, etc.]

In the last month, how often have you used an electronic health record?

How many years have you used an electronic health record?

How many EHRs do you use or are you familiar with?

How does your work environment patient records?

On paper

Some paper, some electronic

All electronic

May I get your contact information?

Name of participant:___________________________________________

Address: ____________________________________________________

____________________________________________________

City, State, Zip: _______________________________________________

Daytime phone number: ________________________________________

Evening phone number: ________________________________________

Alternate [cell] phone number: __________________________________

Email address: _______________________________________________


Sample – Informed Consent

Informed Consent

ezEMRx Inc would like to thank you for participating in this study. The purpose of this study is to

evaluate an electronic health records system. If you decide to participate, you will be asked to perform

several tasks using the prototype and give your feedback. The study will last about 60 minutes. At the

conclusion of the test, you will be compensated for your time.

Agreement

I understand and agree that as a voluntary participant in the present study conducted by ezEMRx Inc. I

am free to withdraw consent or discontinue participation at any time.

I understand and agree that the purpose of this study is to make software applications more useful and

usable in the future.

I understand and agree that the data collected from this study may be shared with outside of ezEMRx

Inc and ezEMRx Inc client. I understand and agree that data confidentiality is assured, because only de-

identified data – i.e., identification numbers not names – will be used in analysis and reporting of the

results.

I agree to immediately raise any concerns or areas of discomfort with the study administrator. I

understand that I can leave at any time.

Please check one of the following:

YES, I have read the above statement and agree to be a participant.

NO, I choose not to participate in this study.

Signature: _____________________________________ Date: ____________________


Sample – Incentive receipt and acknowledgement

INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM

Company: ezEMRx Inc,

25W215 Lake Street

Roselle, IL 60125

Printed Name: ___________________________________________________________

Address: _______________________________________________________________

_______________________________________________________________

Signature: _________________________________ Date: _______________

____________________________________________________________________________

Usability Researcher: _____________________________________

Signature of Usability Researcher: ___________________________

Date: _______________

Witness: _______________________________________________

Witness Signature: _______________________________________

Date: _______________


Sample – SUS Questionnaire

SYSTEM USABILITY SCALE QUESTIONNAIRE

Strongly Strongly

Disagree Agree

I think that I would like to use this system frequently

1 2 3 4 5

I found the system unnecessarily

complex

1 2 3 4 5

I thought the system was easy

to use

1 2 3 4 5

I think that I would need the

support of a technical person to

be able to use this system

1 2 3 4 5

I thought there was too much

inconsistency in this system

1 2 3 4 5

I would imagine that most people

would learn to use this system

very quickly

1 2 3 4 5

I found the system very

cumbersome to use

1 2 3 4 5

I felt very confident using the

system

1 2 3 4 5

I needed to learn a lot of things before

I could get going with this system

1 2 3 4 5


Sample – Tasks

Task 1: Medication list

Please Click on the Today's Task Tab as starting point of the task.

On the current visit, patient “Jennifer Doe” indicates that there is change in his medication history. Locate

the medication tab and and complete the following task.

1) View active medication 2) Change the existing medication to “Completed” or “Discontinue” 3) Add new medication 4) Click on “Medication History” to view the medication history.

Success:

Easily completed

Completed with difficulty or help :: Describe below

_______________________________________________________________________

_______________________________________________________________________

Not completed

Task Time: __________ Seconds

Optimal Path : Today's Task -> Click on the patient “ Jennifer Doe” Encounter Task -> Click on

Medications -> Click on medication name and change status to “Discontinue” and Save -> Add new

medication and click on Add -> Click on “History” to view the patient medication history.

Correct

Minor Deviations / Cycles :: Describe below

______________________________________________________________________

______________________________________________________________________

Major Deviations :: Describe below

______________________________________________________________________

______________________________________________________________________

Observed Errors and Verbalizations:

Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:

Overall, this task was: ______

Participant written scale: “Very Easy” (1) to “Very Difficult” (5)

Administrator Comments


Task 2: Medication allergy list


On the current visit, patient “ Jennifer Maker ” indicates that there is change in his allergy list. Locate the

allergy tab and complete the following task

1) View patient active allergy list 2) Change status of one allergy to “Inactive” 3) Add New Allergy 4) Verify patient allergy history

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path : Today's Task -> Click on the patient “ Jennifer Maker” Encounter Task -> Click on

Allergy -> Click on existing Allergy and change status to “Inactive” -> Select new allergy from auto-lookup

field and add “Severity”, “Event”, “Reaction” click on Save -> Click on the “History” to view patient

allergies.

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:





Task 3: Interaction Check – [D-to-D and D-to-A]

Please Click on the Today's Task Tab as starting point of the task. Ensure that the patient has following

drug-drug and a drug-food allergy to the drug chosen. This will put force the participant to find other drugs

and use other elements of the application.

Patient Allergy :

Patient Medication :

After examining the patient “ Jennifer Miller”, you have decided to put this patient on the medication ->

“Lipitor”. Check for any interaction and place an order for this medication

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path : Today's Task -> Click on the patient “ Jennifer Miller” Encounter Task -> Click on

Prescription -> Click on “Drug Search” button -> Search and select the required drug -> Add “Duration”,

“Quantity”, “Sig”, “Refill” Click on Add -> Click on “Prescribe” to view D-to-D and D-to-A interaction.

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:





Task 4: Prescribe Medication


After examining the patient “ Jennifer Major”, you have decided to put this patient on the medication ->

“Lipitor”. Check for any interaction, Choose the pharmacy and ePrescribe for this medication.

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path : Today's Task -> Click on the patient “ Jennifer Major” Encounter Task -> Click on

Prescription -> Click on “Drug Search” button -> Search and select the required drug -> Add “Duration”,

“Quantity”, “Sig”, “Refill” Click on Add -> Click on “Prescribe” to view interaction -> Select Pharmacy ->

Click on “Send” to ePrescribe.

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:





Task 5: CPOE


After examining the patient “ Jennifer Peter”, you have decided to create an orders for lab, radiology and

medication for this this patient. Check for any D-to-D interaction and complete the orders.

1) Select the existing order-set for this patient. 2) Add or remove any one investigation based on the patient requirement 3) Select the required action and execute the order.

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path: Today's Task -> Click on the patient “ Jennifer Peter” Encounter Task -> Click on Order

set -> Select the existing order-set –> Add new lab items, -> Remove lab items -> Select action ->

execute the orders.

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:





Task 6 : Clinical Information Reconciliation


Patient Jennifer Carter was existing patient of yours and had recently underwent surgery and is on

multiple medication and you to reconcile the patient clinical information.

a) Check with patient and add the medication in the list provided

b) Verify the medication against his current medications

c) Add or remove the medication as required.

d) Complete reconcile

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path: Today's Task -> Click on the patient “ Jennifer Carter” Encounter Task -> Click on

medication -> Select medication reconciliation –> Add new medications -> Remove medications ->

complete reconcile.

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:





Task 7: Clinical Decision support


Patient John Smith was existing patient of yours and has come in for the consultation and is currently on

multiple medications / has active problems and has pending results for verification.

1) Click on the encounter task 2) System will trigger an automated clinical decision support based on the patient conditions /

medication / labs 3) Click on the CDS alert and complete the required task.

Success:

Easily completed


_______________________________________________________________________

_______________________________________________________________________

Not completed


Optimal Path: Today's Task -> Click on the patient “Jennifer Smith” Encounter Task -> System will bring

up the CDS alert -> Complete required task (complete the orders / printing out education materials etc) -

>

Correct


______________________________________________________________________

______________________________________________________________________


______________________________________________________________________

______________________________________________________________________


Comments: ______________________________________________________________________________

________________________________________________________________________________________

Rating:




Template Version 1


Page 12 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision

Appendix C: Quality Management System

��

�� !��"��#$� %��&�

��

��

��

��

��

��

�� !��

"��#��"$��%��

�� !��

��

��& ��

�'��"��#�� "(��)��*�+,-�.��"(�

)-./0��$��"��1��

�2��3%��4!��5�6��"��

��"�5��

7��"��

��5��

� �� "�� 8�

�(��"��

�� "��

9��"�� %��1�

� � � � � �

��2�� : ��

��

��%��!��%��%��%��

��"�!��:��

2��%��

9�� ;��,<��,�).�

�

Template Version 1



Appendix D: Privacy and Security

��

�� !��"� ��#��$�

�

�

��

��

��

��

��

��

�� !

��

��

��

��"�#$�

� ��

��

��

��

�

��"�#��$�

� ��

��

��

��

��

��

�

�� %��%��

��$�

� ��

��

��

��

��

��

�

��

��

��

�

��

��

��

��

��

�

��

�� !��"� ��#��

�

��

��

��

��!��

��

��

��

��

��

��

�

��

��

��

��

��

��

��

�

��

��

��

� � � � � �

��

��

��

�

��

��

��

��

��

��&��#��

��

��

��

� � � � � ��

�� '��

��

��

��

�

Template Version 1



Test Results Summary Document History

Version Description of Change Date V1.0 Initial release May 13, 2014

V1.1 Updated Safety-Enhanced Design report February 28, 2016

END OF DOCUMENT

onc hit certification program test results summary for...

Documents