onc hit certification program test results summary for 2014...

Test Results Summary for 2014 Edition EHR Certification 14-2487-R-0029-PRA V1.1 February 28, 2016

©2016 InfoGard. May be reproduced only in its original entirety, without revision of 13

ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Part 1: Product and Developer Information

1.1 Certified Product Information

Product Name: Adaptamed EMR

Product Version: 6.2.0.0

Domain: Ambulatory

Test Type: Complete EHR

1.2 Developer/Vendor Information

Developer/Vendor Name: Adaptamed, LLC

Address: 4276 54th Place

San Diego, CA 92115

Website: www.adaptamed.com

Email: [email protected]

Phone: (619) 330-6087

Developer/Vendor Contact: Mohan Vardhieni

Part 2: ONC-Authorized Certification Body Information

2.1 ONC-Authorized Certification Body Information

ONC-ACB Name: InfoGard Laboratories, Inc.

Address: 709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

Website: www.infogard.com


Phone: (805) 783-0810

ONC-ACB Contact: Adam Hardcastle

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Adam Hardcastle

EHR Certification Body Manager ONC-ACB Authorized Representative Function/Title

2/28/16 Signature and Date

http://www.adaptamed.com/

mailto:[email protected]

http://www.infogard.com/




2.2 Gap Certification The following identifies criterion or criteria certified via gap certification

§170.314

(a)(1) (a)(17) (d)(5) (d)(9)

(a)(6) (b)(5)* (d)(6) (f)(1)

(a)(7) (d)(1) (d)(8)

*Gap certification allowed for Inpatient setting only

No gap certification

2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification

§170.314 (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4)

(f)(5) Optional & Amb. only (a)(6) (b)(2) (d)(5)

(a)(7) (b)(3) (d)(6) (f)(6) Optional &

Amb. only (a)(8) (b)(4) (d)(7) (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4)

(a)(13) (c)(2) (e)(3) Amb. only

No inherited certification



Part 3: NVLAP-Accredited Testing Laboratory Information

Report Number: 14-2487-R-0029 V1.3 Test Date(s): April 2, 4, 8, May 6, June 3, 2014 Location of Testing: InfoGard and Vendor Site

3.1 NVLAP-Accredited Testing Laboratory Information

ATL Name: InfoGard Laboratories, Inc.

Accreditation Number: NVLAP Lab Code 100432-0

Address: 709 Fiero Lane Suite 25

San Luis Obispo, CA 93401

Website: www.infogard.com


Phone: (805) 783-0810

ATL Contact: Milton Padilla

For more information on scope of accreditation, please reference http://ts.nist.gov/Standards/scopes/1004320.htm

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:

Milton Padilla

EHR Test Body Manager ATL Authorized Representative Function/Title

2/28/16 Signature and Date

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

Additional Software Applicable Criteria Functionality provided by Additional Software

SureScripts 170.314(b)(3) ePrescribing

No additional software required

3.2.2 Test Tools

Test Tool Version

http://www.infogard.com/


http://ts.nist.gov/Standards/scopes/1004320.htm



Test Tool Version Cypress 2.4.1 ePrescribing Validation Tool v1.0.4 HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool

HL7 v2 Immunization Information System (IIS) Reporting Validation Tool v1.8.0

HL7 v2 Laboratory Results Interface (LRI) Validation Tool V1.7.0 HL7 v2 Syndromic Surveillance Reporting Validation Tool V1.7.0 Transport Testing Tool V177 Direct Certificate Discovery Tool v2.1

No test tools required

3.2.3 Test Data

Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter]

No alteration (customization) to the test data was necessary

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(13)

§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree



Criterion # Standard Successfully Tested

(a)(15)(i)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(a)(16)(ii) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(b)(7)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(e)(1)(i) Annex A of the FIPS Publication 140-2 • AES 256 • SHA-1

(e)(1)(ii)(A)(2) §170.210(g)

Network Time Protocol Version 3 (RFC 1305)

§170. 210(g) Network Time Protocol Version 4 (RFC 5905)

(e)(3)(ii) Annex A of the FIPS Publication 140-2 • AES 256 • SHA-1

Common MU Data Set (15)


§170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested

Newer Version Applicable Criteria



Newer Version Applicable Criteria

No newer version of a minimum standard was tested

3.2.5 Optional Functionality

Criterion # Optional Functionality Successfully Tested

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)


Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)


Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)

No optional functionality tested



3.2.6 2014 Edition Certification Criteria* Successfully Tested

Criteria # Version

Criteria # Version

TP** TD*** TP TD (a)(1) (c)(3) 1.7.1 2.4.1 (a)(2) 1.2 (d)(1)

(a)(3) 1.2 1.4 (d)(2) 1.4 (a)(4) 1.4 1.3 (d)(3) 1.3 (a)(5) 1.4 1.3 (d)(4) 1.2 (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) 1.2 (d)(7) 1.2 (a)(9) 1.3 1.3 (d)(8) (a)(10) 1.2 1.4 (d)(9) Optional (a)(11) 1.3

(e)(1) 1.7 1.4 (a)(12) 1.3 (e)(2) Amb. only 1.2 1.5 (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) 1.3 1.8.0 (a)(16) Inpt. only (f)(3) 1.3 1.7.0 (a)(17) Inpt. only (f)(4) Inpt. only

(b)(1) 1.6 1.3 (f)(5) Optional &

Amb. only (b)(2) 1.4 1.5

(b)(3) 1.4 1.0.4 (f)(6) Optional &

Amb. only (b)(4) 1.3 1.4

(b)(5) 1.4 (g)(1) (b)(6) Inpt. only (g)(2) 1.6 1.9 (b)(7) 1.4 1.5 (g)(3) 1.3

(c)(1) 1.7.1 2.4.1 (g)(4) 1.2

(c)(2) 1.7.1 2.4.1

*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)

http://www.healthit.gov/certification



3.2.7 2014 Clinical Quality Measures*

Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested

*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

2 V3 90 V3 136 V3 155 V2

22 117 V2 137 156 V2

50 V2 122 138 V2 157

52 123 139 158

56 124 140 159

61 125 141 160

62 126 V2 142 161

64 127 143 163

65 128 144 164

66 129 145 165 V2

68 V3 130 146 V2 166 V3

69 V2 131 147 167

74 132 148 169

75 V2 133 149 177

77 134 153 V2 179

82 135 154 V2 182

Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113

55 104 114

60 105 171

http://www.cms.gov/



3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(7) (a)(15) (b)(5)

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(7) (a)(15) (b)(5)

Automated Measure Calculation was not tested

3.2.9 Attestation

Attestation Forms (as applicable) Appendix

Safety-Enhanced Design* A

Quality Management System** B

Privacy and Security C

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product



Appendix A: Safety-Enhanced Design

Participants used do not meet the requirements for summative usability testing.

No standard or process was identified for user centered design.

EHR Usability Test Report of Adaptamed EMR

Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

Product name and version:

Adaptamed EMR Software Version 6.2.0.0

Date of Usability Test: [04/07/2014] Date of Report: 04/8/2014]

Location of UCD Test: 4276 54th

Place, Suite B, san Diego, CA, 92115

Report Prepared By: Dr. Kumara Prathipati (C.E.O)

Phone: 6197155276

Email address: [email protected]

Mailing Address: 4276, 54th place, suite: B, San Diego, CA, 92115.

Table of Contents

1 PARTICIPANTS

2 STUDY DESIGN

3 TASKS TESTED

4 TEST PROCEDURE PROCEDURES

5 TEST LOCATION-ENVIRONMENT

6 TEST FORMS AND TOOLS

7 PARTICIPANT INSTRUCTIONS

8 USABILITY MEASUREMENT METRICS

9 DATA SCORING

10 RESULTS OF THE TEST AND DATA ANALYSIS

11 DATA LOGGER NOTES

12 POST TEST QUESTIONNAIRE -appendix 5 – 6

13 Appendix 1- SAMPLE RECRUITING SCREENER

14 APPENDIX4-MODERATOR GUIDE

EXECUTIVE SUMMARY

A usability test of Adaptamed EMR Software Version 6.2.0.0 and Ambulatory EHR was conducted on

04/07/2014 in [San Diego CA] by Adaptamed LLC. The purpose of this test was to test and validate the usability

of the current user interface, and provide evidence of usability in the EHR under Test (EHRUT).

During the usability test, 1 healthcare provider matching the target demographic criteria served as participants

and used the EHRUT in simulated, but representative tasks.

This study collected performance data on 7 tasks typically conducted on an EHR:

• § 170.314(a)(1) Computerized provider order entry

• § 170.314(a)(2) Drug‐drug, drug‐allergy interaction checks

• § 170.314(a)(6) Medication list

• § 170.314(a)(7) Medication allergy list

• § 170.314(a)(8) Clinical decision support

• § 170.314(b)(3) Electronic prescribing

• § 170.314(b)(4) Clinical information reconciliation


During the 60 minute one-on-one usability test, each participant was greeted by the administrator and

asked to review and sign an informed consent/release form (included in Appendix 3); they were

instructed that they could withdraw at any time. Participant did prior experience with the EHR.

P ar ti ci pa nts we r e pr o vide d vi s ua l use r guide s p r ior to st ar t o f eac h t e st to s t ud y.

The administrator introduced the test, and instructed participants to complete a series of tasks (given one

at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data

logger(s) recorded user performance data on paper and electronically. The administrator did not give the

participant assistance in how to complete the task.

Participant screens, head shots and audio were recorded for subsequent analysis.

The following types of data were collected for each participant:

1. Number of tasks successfully completed within the allotted time without assistance

2. Time to complete the tasks

3. Number and types of errors

4. Path deviations

5. Participant’s verbalizations

6. Participant’s satisfaction ratings of the system

All participant data was de-identified – no correspondence could be made from the identity of the

participant to the data collected.

Following is a summary of the performance and rating data collected on the EHRUT.

Measure N Task

Success

Path

Deviation

Task Time Errors Task

Ratings

5=Easy

Task

# Mean

(SD)

Deviations

(Observed/

Optimal)

Mean

(SD)

Deviation

s

(Observe

Mean

(SD)

Mean

(SD)

1. Computerized provider

order entry

2. Drug‐drug, drug‐allergy

interaction checks

1 4 1 1080 48/ 0 5

1 5 1 480 230 4

3. Medication list 1 4 1 120 55 5

4. Medication allergy list 1 4 1.5 240 110 4

5. Clinical decision support 1 5 1.8 420 20 1 1

6. Electronic prescribing 1 5 1 180 80 5

7. Clinical information

Reconciliation

1 5 1.6 180 115 1 1

The results from the System Usability Scale scored the subjective satisfaction with the system

based on performance with these tasks to be: 90. In addition to the performance data, the following

qualitative observations were made:

Major Test findings

Participant failed to activate Clinical Decision Support and gave up. Medication reconciliation cannot complete in allocated time and user failed

Identified area(s) of improvement

Drug to Drug interaction screen layout should be made more users friendly. Medication Reconciliation screen needs more easy navigation for users. Clinical Decision support activation window should be more users friendly

INTRODUCTION

The EHRUT(s) tested for this study was Adaptamed EMR Software Version 6.2.0.0 Designed to present medical information to healthcare providers (physicians, physician assistants, nurses) and their administrative staff in an ambulatory setting the EHRUT consists of a web based Electronics Medical Records

(EMR) that is offered to practices as a Software-as-a-Service (Saas)]. The usability testing attempted to

represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR under Test (EHRUT). . To this end, measures of effectiveness, efficiency and user satisfaction, such as task completion rates, task time were captured during the usability testing.

B. PARTICIPANTS

One participant was tested on the EHRUT. Participant in the test is a [MEDICAL CLINIC OFFICE

MANAGER]. Participant was recruited by [Adaptamed LLC] and was compensated [ $ 100 ] for their time. In addition, participant had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of

training as the actual end users would have received. For the test purposes, end-user characteristics were identified and translated into a recruitment screener used to solicit potential participants; an example of a screener is provided in Appendix [1]. Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table of participant by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Part ID

Gender

Age

Education

Occupation/

role

Professional Experience

Computer Experience

Product

Experience

Assistive

Technology Needs

1 1 Female 51 BS Office Manager –

Medical Clinic -

Billing/Clinic

Manager— Helps

provider in

documentation.

6 years 6 Years 4 Years None

3 participants (matching the demographics in the section on Participants) were recruited. 1 selected and participated in the usability test. 0 participants failed to show for the study.

Participants were scheduled for [5 minute] sessions with [5 minute] in between each session for debrief

by the administrator(s) and data logger(s), and to reset systems to proper test conditions. A paper spreadsheet was used to keep track of the participant schedule, and included each participant’s demographic characteristics.

Description of Test Administrator: experience and demographic characteristics:

Test administrator has 15 years of experience in software product design in medical field and financial

applications field. He is a physician by profession and has experience managing 400 employees and held

various managerial positions.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that

is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet the

needs of the participants. The data from this test may serve as a baseline for future tests with an

updated version of the same EHR and/or comparison with other EHRs provided the same tasks are

used. In short, this testing serves as both a means to record or benchmark current usability, but also to

identify areas where improvements must be made.

During the usability test, participants interacted with [1] EHR. Each participant used the system in the

same location, and was provided with the same instructions. The system was evaluated for effectiveness,

efficiency and satisfaction as defined by measures collected and analyzed for each participant:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations (comments)

• Participant’s satisfaction ratings of the system

TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities a

user might do with this EHR:

Task 1: First Impressions (Setup)

What do you notice? What are you able to do here?

Task 2: Computerized provider order entry: Take the participant to the starting point for the task.

Record, change and access Medication order.

Order Allopurinol 100 mg PO daily No 30 tablets

Save this order.

Change Allopurinol to 200 mg PO daily

Save this order

View your order in current medicines window.

Record, change, and access Laboratory order.

Order CBC

Save Order

Cancel CBC

Order CMP

View previously ordered tests

Record, change and access Radiology order.

Order chest x-ray 2 view

Save Order

Cancel chest x-ray 2 views

Order Knee X-ray 2 view

View previously ordered tests

Task 3: Drug-drug, drug-allergy interactions:

Take the participant to the starting point for the task.

Create drug-drug and drug-allergy interventions prior to CPOE completion. Adjust the

severity level of drug-drug interventions.

Metoprolol-Clonidine drug-drug interaction

Sulfa allergy-bactrin Rx

Task 4: Medication List:


a. Enter data: Patient is taking metformin 500 mg PO daily

b. Change Metformin to 1000 mg PO BID

c. Close patient chart. Open patient chart and view current medication list

Record, change and access a Medication List.

Task 5: Medication Allergy List


Record, change and access Medication Allergy List.

a. Enter data: patient is allergic to sulfa, has skin rash, moderate severity

b. Delete Patient is allergic to sulfa

Task 6: Clinical decision support


Activate Clinical Decision Support using problem list, medication list, medication allergy

list, demographics, lab tests and results or vitals as the trigger. Identify User Diagnostic

and Therapeutic Reference information.

Define a intervention (CDS).

Link this to role.

Link the user to role

Open window to view Referance Information

Task 7: Electronic Prescribing

Take the participant to the starting point for the task. Ensure that this patient has a drug-

drug and a drug-food allergy to the drug chosen. This will force the participant to find

other drugs and use other elements of the application.

a. Metoprolol 25 mg po daily

b. Send Erx for the above drug

c. Prescribe clonidine 0.1 mg po qd

d. View drug drug interaction

e. Cancel Clonidine and prescribe Enalapril 5 mg po qd as alternative.

f. Prescribe bactrin DS 1 tab po BID

g. View drug allergy information and cancel the prescription

h. Send enalapril electronically ( do dummy send)

After examining Patient, you have decided to put this patient on – drug name.

Check for any interactions and place an order for this medication.

Task 8: Clinical information reconciliation


Reconcile patient’s active medication list with another source. Reconcile patient’s active

problem list with another source. Reconcile patient’s active medication allergy list with

another source

Tasks were selected based on their frequency of use, criticality of function, and those that may be most

troublesome for users. Tasks will

always be constructed in light of the study objectives.

TEST PROCEDURE:

Upon arrival, participants were greeted; their identity was verified and matched with a name on the

participant schedule. Participants were then assigned a participant ID. 7

Each participant reviewed

and signed an informed consent and release form (See Appendix 3). A representative from the test team witnessed the participant’s signature.

To ensure that the test ran smoothly, two staff members participated in this test, the administrator and the data logger. The Administrator has 3 years of experience is usability testing. The data logger has 3 years of experience in usability testing.

The test administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments.

Participants were instructed to perform the tasks (see specific instructions below): 1. As quickly as possible making as few errors and deviations as possible.

2. W ithout assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use. W ithout using a think aloud technique.

For each task, the participants were given a written copy of the task (standard help manual given to EHR customers) . Task timing began once the administrator finished reading the task and said “GO”. The task time was stopped once the participant indicated they had successfully completed the task or the allocated time is over.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet.

Participants were thanked for their time and compensated. Participants signed a receipt and acknowledgment form (See Appendix 6) indicating that they had received the compensation.

TEST LOCATION:

The test facility included a waiting area and a quiet testing room with a table, desktop computer for the

participant, and recording computer for the administrator. Only the participant and administrator were in

the test room (data logger is not in test room but he sits in adjacent room). All observers and the data

logger worked from a separate room where they could see the participant’s screen and face shot, and listen

to the audio of the session. To ensure that the environment was comfortable for users, noise levels are kept

to a minimum with the ambient temperature within a normal range. All of the safety instruction and

evacuation procedures were valid, in place, and visible to the participants.

TEST ENVIRONMENT

The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted at Adaptamed LLC, 4276 54th Place, Suite B, San Deigo, CA 92115 . For testing, the computer used a Intel i5 desktop computer running Windows 7 Professional. Monitor size is 21 inches. The

participants used a mouse and keyboard when interacting with the ADAPTAMED EHR.

The ADAPTAMED EHR used 1920 x 1080 resolutions (colour sett ings is not recorded) The

application was set up by Usability Test Administrator according to the vendor’s documentation describing

the system set-up and preparation. The application itself was running on a [remote server] using a RDP

connection and test database on a WAN connection. Internet connection speed 1 Mbps was provided for optimal experience. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size or monitor resolution).

INTENDED USERS DESCRITPION:

1. Physicians In Private Practice

2. Front Office Secretaries in medical clinic

3. Back Office Medical Assistant in medical clinic

4. Nurse in medical clinics

5. Office manager in medical Clinics

TEST FORMS AND TOOLS:

During the usability test, various documents and instruments were used, including:

1. Informed Consent to be signed by participants

2. Moderator’s Results Capture Forms

3. Moderator’s Test Instructions Forms/Help Files

4. Post-test Questionnaire

5. Incentive Receipt and Acknowledgment Form

The participant’s interaction with the ADAPTAMED EHR was captured and recorded

Digitally with screen capture software running on the test machine. A web

Camera recorded each participant’s facial expressions synced with the screen capture, and verbal

comments were recorded with a microphone. The test session were electronically transmitted to a nearby

observation room where the data logger observed the test session.

PARTICIPANT INSTRUCTIONS (Speech from test Administrator to participants)

The administrator reads the following instructions aloud to the each participant:

Thank you for participating in this study. Your input is very important. Our session today will last

about [60 minutes]. During that time you will use an instance of an electronic health record. I will

ask you to complete a 8 tasks using this system and answer some questions. You should complete

the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on

your own following the instructions very closely. General introduction/description of the task will

be announced and standard help manual will be provided for each test.

We will provide 5 minutes gap between each test to let you listen to the next test introduction and

read help manual for the next test.

Please note that we are not testing you. We are testing the EHR system, therefore if you have

difficulty all this means is that something needs to be improved in the EHR system. I will be here in

case you need specific help, but I am not able to instruct you or provide help in how to use the

application and complete task.

Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be

useful to you, and how we could improve it. I did not have any involvement in its creation, so

please be honest with your opinions. All of the information captured during the test is confidential

and unidentified. There are a variety of tools that record screens and transmit those recordings

across a local area network for remote observations. Your name will not be associated with your

comments at any time. Should you feel it necessary you are able to withdraw at any time during the

testing you have full choice to withdraw.

Following the procedural instructions, participants were shown the EHR and as their first task, were

given time ([10] minutes) to explore the system and ask questions or make comments. Once this

task was complete, the administrator gave the following instructions: For each task, I will read the description to you, give you time to read standard help manual and say

“Begin.” At that point, please perform the task and raise your hand and say “Done” once you

believe you have successfully completed the task.

I would like to request that you not talk aloud or verbalize while you are doing the tasks.

I will ask you your impressions (subjective opinion) about the task once you are done.

Participants should not use a think-aloud protocol during the testing. Excessive verbalization or

attempts to converse with the moderator during task performance will be strongly discouraged.

Participants will naturally provide commentary, but they should do so, ideally, after the testing.

Some verbal commentary may be acceptable between tasks, but again should be kept to minimum

between the participant and moderator.

USABILITY METRICS

The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of

satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the

usability testing.

The goals of the test were to assess:

1. Effectiveness of ADAPTAMED EHR by measuring participant success rates and failure rates

2. Efficiency of ADAPTAMED EHR by measuring the average task time and path deviations

3. Satisfaction with ADAPTAMED EHR by measuring ease of use ratings

HOW TEST RESULTS ARE SCORED:

DATA SCORING

The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Measures Rationale and Scoring Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to

achieve the correct outcome, without assistance, within the time

allotted on a per task basis.

The total number of successes were calculated for each task and then divided

by the total number of times that task was attempted. The results are

provided as a percentage.

Task times were recorded for successes. Observed task times divided by the

optimal time for each task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by expert performance under

realistic conditions, is recorded when constructing tasks. Target task times

used for task times in the Moderator’s Guide must be operationally defined by

taking multiple measures of optimal performance and multiplying by some

factor [e.g., 1.25] that allows some time buffer because the participants are

presumably not trained to expert performance. Thus, if expert, optimal

performance on a task was [x] seconds then allotted task time performance

was [x * 1.25] seconds. This ratio should be aggregated across tasks and

reported with mean and variance scores.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before

successful completion, the task was counted as an “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would

be counted as errors.11

This should also be expressed as the mean number of failed tasks per participant.

On a qualitative level, an enumeration of errors and error types should be

collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly

with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

It is strongly recommended that task deviations be reported. Optimal paths

(i.e., procedural steps) should be recorded when constructing tasks.

Efficiency:

Task Time

Each task was timed from when the administrator said “Begin” until the Participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for

tasks that were successfully completed were included in the average task time

analysis. Average time per task was calculated for each task. Variance

measures (standard deviation and standard error) were also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post- session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across Participants.

12

Common convention is that average ratings for systems judged easy to use

should be 3.3 or above.

To measure participants’ confidence in and likeability of the [EHRUT]

overall, the testing team administered the System Usability Scale (SUS)

post-test questionnaire. Questions included, “I think I would like to use this

system frequently,” “I thought the system was easy to use,” and “I would

imagine that most people would learn to use this

System very quickly.” See full System Usability Score questionnaire in

Appendix 5. 13

RESULTS OF THE TEST AND DATA ANALYSIS:

The results of the usability test were calculated according to the methods specified in the Usability Metrics

section above. Participants who failed to follow session and task instructions had their data excluded from the

analyses.

Provider ID: #1

List of Tests Excluded:2

Reason Excluded: Unable to complete

Provider ID: #1

List of Tests Excluded due to “unable to interpret results/test irregularity: 0

Reason Unable to interpret: N/A

The usability testing results for the EHRUT are detailed below in a table format for easy view/interpretation.

Task No.

(1) Computerized provider order entry

(2) Drug-drug, drug-allergy interaction checks

(3) Medication list

(4) Medication allergy list

(5) Clinical decision support

(6) Electronic prescribing

(7) Clinical information reconciliation

Task Completion Rates ( 1 = completed; 0 = failed)

Participant ID Task1 Task2 Task3 Task4 Task5 Task6 Task7

#1 1 1 1 1 0 1 0

Success 1 1 1 1 0 1 0

Completion 100% 100% 100% 100% 0% 100% 0%

Rates Participant failed task 7

Task Time (in seconds)

Participant ID Task1 Task2 Task3 Task4 Task5 Task6 Task7

Optimal Time 600 250 65 130 400 100 65

#1 1080 480 120 240 420 180 180

Average 1080 480 120 240 420 180 180

Time

Task Ratings (1 = very difficult; 5 = very easy)

Participant ID Task1 Task2 Task3 Task4 Task5 task6 Task7

#1 5 4 5 4 1 5 1

Average 5 4 5 4 1 5 1

Measure N Task

Success

Path

Deviation

Task Time Errors Task

Ratings

5=Easy

Task

# Mean

(SD)

Deviations

(Observed/

Optimal)

Mean

(SD)

Deviations

(Observed/

Optimal)

Mean

(SD)

Mean

(SD)

1. Computerized provider

order entry

2. Drug‐drug, drug‐allergy

interaction checks

1 4 1 1080 480 5

1 5 1 480 230 4

3. Medication list 1 4 1 120 55 5

4. Medication allergy list 1 4 1.5 240 110 4

5. Clinical decision support 1 5 1.8 420 20 1 1

6. Electronic prescribing 1 5 1 180 80 5

7. Clinical information

reconciliation

1 5 1.6 180 115 1 1

Discussion of the Findings:

Effectiveness: Based on the findings, user was effective using EHR. The amount of time taken by

user to complete the tasks is minimal and felt easy to navigate.

Effectiveness: (Failure): User is unable to navigate to complete the tasks which were failed. May

be we need to improve the work flow design in EHR related to these modules.

Efficiency: The user was able to complete the task in minimum amount of time which is

Reasonable. We found that user was not able to complete a task even though user is in the correct

path and deviated to a different location in the same window. Thus we conclude user is relatively

effective.

Satisfaction: It is stated by the user that system is organized according to the work flow and

logically. There are few suggestions made by user related to Medication Reconciliation module.

But still user is happy the way it is integrated in the system.

DATA LOGGER NOTES:

Task Name: Computerized provider order entry

Optimal time for the task: 600

Allocated time for the task: 1200

Participant ID: 1

Task Start Time: 06.27 PM

Task End Time: 06.45 PM

Task completed: yes

Time taken to complete the task: 1080

No of path deviations (unnecessary clicks): 0

Number of errors committed: 0

Description of Errors:0

Verbalization by Participant:

Data logger comments on Facial Expression: Pleasant

Task Name: Drug-drug, drug-allergy interaction checks

Optimal time for the task: 250 seconds

Allocated time for the task: 500 seconds

Participant ID: 1

Task Start Time: 06. 50 PM


Task completed: yes

Time taken to complete the task: 480 seconds



Description of Errors: 0


Data logger comments on Facial Expression: Pleasant

Task Name:Medication List



Participant ID: 1



Task completed: yes


No of path deviations (unnecessary clicks):0




Data logger comments on Facial Expression: Keenly observing the screen

Task Name: Medication allergy list



Participant ID: 1



Task completed: yes




Description of Errors: unable to find the path to remove allergic information/. Took couple of more

clicks and finished the task.


Data logger comments on Facial Expression: Bit confused, but became relaxed when she find the

path to remove the allergic information.

Task Name: Clinical Decision Support



Participant ID: 1



Task completed: yes

Time taken to complete the task: 420 seconds (Incomplete)

No of path deviations (unnecessary clicks):1


Description of Errors: User was not able to navigate to the accurate location to enable CDS.

Verbalization by Participant: .

Data logger comments on Facial Expression: Participant was not able to enable the CDS and view

it in the problem list. Tried couple of settings but failed to perform.

Task Name: Electronic Prescribing



Participant ID: 1

Task Start Time: 07. 25 PM

Task End Time: 07. 28 PM

Task completed: yes






Data logger comments on Facial Expression:

Task Name: Clinical information reconciliation



Participant ID: 1



Task completed: No

Time taken to complete the task: 180 seconds (Incomplete)



Description of Errors: Unable to choose the Clinical message to view and perform the task.


Data logger comments on Facial Expression: Participant cannot find the patient clinical message

information to perform the task and found a bit disappointed and gave up.

POST TEST QUESTIONNAIRE

Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE

POST TEST QUESTIONNAIRE FOR TEST 1:

Computerized provider order entry : Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)

1. I think that I would like to use this system frequently: 5

2. I found the system unnecessarily complex: 1

3. I thought the system was easy to use: 4

4. I think that I would need the support of a technical person to be able to use this system: 2

5. I found the various functions in this system were well integrated: 3

6. I thought there was too much inconsistency in this system: 2

7. I would imagine that most people would learn to use this system very quickly: 4

8. I found the system very cumbersome to use: 2

9. I felt very confident using the system: 4

10. I needed to learn a lot of things before I could get going with this system: 2


Drug-drug, drug-allergy interaction checks : Score 1 to 5 (1 = strongly disagree; 5 = strongly

agree)












Medication list : Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)












Medication allergy list : Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)












Clinical decision support : Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)












Electronic prescribing : Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)












Clinical information reconcilation Score 1 to 5 (1 = strongly disagree; 5 = strongly agree)











=====================================================================

Appendix 1: SAMPLE RECRUITING SCREENER

The purpose of a screener to ensure that the participants selected represent the target user population as closely as possible.

Recruiting Script for Recruiting Firm

Hello, my name is Kumara Prathipati , calling from Adaptamed LLC. We are recruiting individuals to participate in a usability study for an electronic health record. We would like to ask you a few questions to

see if you qualify and if would like to participate. This should only take a few minutes of your time. This is

strictly for research purposes. If you are interested and qualify for the study, you will be paid to participate.

Can I ask you a few questions?

Customize this by dropping or adding questions so that it reflects your EHR’s primary audience

1. Gender: [ ] Male [X] Female (Recruit a mix of participants)

2. Have you participated in a focus group or usability test in the past 12 months?

[ ] y e s [ X ] n o ( If yes, Terminate)

3. Do you, or does anyone in your home, work in marketing research, usability research, web

design …etc.]? [ ] y e s [ X ] n o ( If yes, Terminate)

4. Do you, or does anyone in your home, have a commercial or research interest in an electronic

health record software or consulting company? [ ] yes [X ] no (If yes, Terminate)

5. Which of the following best describes your age?

[ ] 23 to 39 [X] 40 to 59 [ ] 60 - to 74 [ ]; 75 and older] (need recruit mix)

6. Which of the following best describes your race or ethnic group?

[X ] Caucasian [ ] Asian [ ] Black/African-American [ ]Latino/a or Hispanic

7. Do you require any assistive technologies to use a computer?

[ ] yes [X] no (if so, please describe)

Professional Demographics Customize this to reflect your EHR’s primary audience

8. What is your current position and title? (Must be healthcare provider)

[ ] RN: Specialty:

[ ] Physician: Specialty: _

[ ] Resident: Specialty:

[X] Administrative Staff: Medical Clinic Office Manager

[ ] Other [Terminate]

9. How long have you held this position? 5 YEARS

10. Describe your work location (or affiliation) and environment? (Recruit according to the

intended users of the application) [e.g., private practice, health system, government clinic,

etc.] : Private practice Internal medicine clinic-front office, back office , telpehone

answering, billing

11. Which of the following describes your highest level of education? [e.g., high school

graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other

(explain)] : College Graduate, Post Graduate partially finished

Computer Expertise Customize this to reflect what you know about your EHR’s audience

12. Besides reading email, what professional activities do you do on the computer? [e.g., access

EHR, research; reading news; shopping/banking; digital pictures; programming/word

processing, etc.]: Using microsoft Word, Excle, readign news, banking ( If no computer use

at all, Terminate)

13. About how many hours per week do you spend on the computer?

[ ] 0 to 10 [ ] 11 to 25 [x ] 26+ hours per week

14. What computer platform do you usually use? [ ] Mac [ x]Windows

15. What Internet browser(s) do you usually use? [ x] Firefox [x]IE [x] Chrome

16. In the last month, how often have you used an electronic health record? 10 days

17. How many years have you used an electronic health record? 4 years

18. How many EHRs do you use or are you familiar with? One

19. How does your work environment patient records? [Recruit according to the demographics of

the intended users]

[ ] On paper

[x] Some paper, some electronic

[ ] All electronic

Contact Information If the person matches your qualifications, ask

Those are all the questions I have for you. Your background matches the people we're looking

for. For your participation, you will be paid $100.

Would you be able to participate on 4/07/2014?

May I get your contact information?

Name of participant: Stacy Davidson

Address: 4456 Marlbrough Ave

City: San Diego State: CA, Zip: 92116

Daytime phone number:

Evening phone number:

Alternate [cell] phone number: 619 7232324

Email address: [email protected]


Before your session starts, we will ask you to sign a release form allowing us to videotape your session.

The videotape will only be used internally for further study if needed. W ill you consent to be

videotaped?

This study will take place at [4 2 7 6 5 4 t h P l a c e . S a n d i e g o , C A , 9 2 1 1 5 . ]. I will confirm your

appointment a couple of days before your session and provide you with directions to our office. What time

is the best time to reach you?

Appendix 2: PARTICIPANT DEMOGRAPHICSThe report should contain a breakdown of the key participant

demographics. A representative list is shown below.

Following is a high-level overview of the participants in this study.

Gender

Men [1] Women [2] Total (participants) [3]

Occupation/Role RN/BSN [X] Physician [1] Admin Staff [2] Total (participants) [3]

Years of Experience Years experience [64] Facility Use of EHR All paper [X] Some paper, some electronic

[X]

All electronic [X] Total (participants) [X]

As an appendix to the report, the full participant breakdown (de-identified) should be include

Appendix 4: EXAMPLE MODERATOR’S GUIDE

Only three tasks are presented here for illustration.

EHRUT Usability Test

Moderator’s Guide Administrator: Kumara Prathipati MD

Data Logger: Kiran Ghantasala

Date : 04/07/2014 Time 18:00 Participant # 1

Location 4276 54th place , San Dieog, CA, 92115

Prior to testing

Confirm schedule with Participants

Ensure EHRUT lab environment is running properly Ensure lab and data recording equipment is running properly

Prior to each participant:

Reset application

Start session recordings with tool

Prior to each task:

Reset application to starting point for next task

After each participant:

End session recordings with tool

After all testing

Back up all video and data files

Orientation (7 minutes)

Thank you for participating in this study. Our session today will last 60 to 90 minutes.

During that time you will take a look at an electronic health record system.

I will ask you to complete a few tasks using this system and answer some questions.

We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we

could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with

the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot

answer help you with anything to do with the system itself. Please save your detailed comments until the end of a task or

the end of the session as a whole when we can discuss freely.

I did not have any involvement in its creation, so please be honest with your opinions.

The product you will be using today is describing the state of the application, i.e., production version, early prototype,

etc. Some of the data may not make sense as it is placeholder data.

We are recording the audio and screenshots of our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time.

Do you have any questions or concerns? None

Preliminary Questions (5 minutes) What is your job title / appointment? : Medical Office Secretary/biller

How long have you been working in this role? 6 years

What are some of your main responsibilities? Patient communication/billing/telephone answering/appointments

Tell me about your experience with electronic health records:

Know the billing part very well. Have base knowledge about basic workflow of EHR.

Task 1: First Impressions (30 Seconds) This is the application you will be working with. Have you heard of it? Yes

If so, tell me what you know about it. ?

I used this for billing purpose. I use some times the PMS part of the product to help the doctor office.

Show test participant the EHRUT. Please don’t click on anything just yet.

What do you notice? Looks like an appointment schedule for today.

What are you able to do here? Enter demographics, open chart, and schedule an appointment, Index documents, run

reports. This is the starting point for everything.

Notes / Comments: none

Task 2: CPOE (600 Seconds)


Record, change and access Medication order.

Order Hydralazine 25 mg po bid No 120. Save order. Close chart. Open chart. Change to

Hydraalzine 25 mg po qd no 60

Save order. Close chart. Open chart. View order in current meds.

Record, change, and access Laboratory order.

Order CBC – reason cough

Cancel CBC .Order CMP

Save order. Close chart. Open chart. View previously ordered lab.

Record, change and access Radiology order.

Order chest X ray 2 view – reason = cough

Cancel Chest X ray order.

Order Knee X - Ray Rt knee; reason knee pain.

Save order. Close chart. Open chart. View previously ordered lab.

Success: [x] Easily completed [ ] Completed with difficulty or help: Describe below

[ ] Not completed

Comments:

Task Time: 1080 Seconds

Optimal Path: Patient Search Meds “OK” Button

[x] Correct [ ] Minor Deviations / Cycles: Describe below

[ ] Major Deviations: Describe below

Comments:

Observed Errors and Verbalizations: Comments:

Rating: Overall, this task was: 5

Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)

Administrator / Note taker Comments: None

Task 3: Drug-drug, drug-allergy interactions (250 Seconds)


Create drug-drug and drug-allergy interventions prior to CPOE completion. Adjust the

severity level of drug-drug interventions.

Adjust severity of drug-drug interaction to severe and moderate categories.

Enter current emds : metoprolol succinate 25 mg po qd

Enter new prescription: clonidine 0.1 mg poq d

Now do appropriate action of drug-drug interaction window;

Patient is allergic to sufa.

Order bactrim ds 1 tab po bid x 10 days No 20

And view allergy warning and take appropriate action

Success: [x] Easily completed [ ] Completed with difficulty or help: Describe below

[ ] Not completed

Comments:


Optimal Path: Patient Chart Prescriptions’ folder Give New Rx Search Meds Interactions

[ x] Correct

[ ] Minor Deviations / Cycles :: Describe below

[ ] Major Deviations :: Describe below

Comments:

Observed Errors and Verbalizations: Comments: None


Show participant written scale: “Very Difficult” (1) to “Very easy” (5)


Task 4: Medication List 65 Seconds)


Record, change and access a Medication List.

Add: patient is taking diovan 160 mg po qd , startd ate 2 motnhs agoSave

Change : patient is taking Diovan 40 mg po qod , startd ate 2 months ago Save

View patient current meds

Success: [x ] Easily completed [ ] Completed with difficulty or help :: Describe below [ ] Not completed

Comments:


Optimal Path: Patient Encounter Current Medications “Save” Button

[x] Correct



Comments:

Observed Errors and Verbalizations: Comments: none



Administrator / Note taker Comments:

Task 5: Medication Allergy List (130 Seconds)


Record, change and access Medication Allergy List.

Document allergy to sulfa, skin manifestation rash, moderate severity

Save allergy information

View allergy information

Delete entry because it is data entry error

Success: [ X ] Easily completed [ ] Completed with difficulty or help :: Describe below [ ]Not completed

Comments:


Optimal Path: Patient Past Hx (Folder) Allergies “Save” Button

[ ] Correct

[X] Minor Deviations / Cycles :: Describe below


Comments:

Observed Errors and Verbalizations: Unable to find the path to delete the allergic information. After

couple of attempts participant was able to find it and perform the task. Comments: None




None

Task 6: Clinical decision support (400 Seconds)


Configure Clinical Decision Support using problem list, medication list, medication

allergy list, demographics, lab tests and results or vitals as the trigger. Identify User

Diagnostic and Therapeutic Reference information. Use will configure 1 CDS for

ecah category (problem list, medication list, demographics etc. for CDS)

Define a intervention (CDS).

Link this to role.

Link the user to role

Open window to view Referance Information

Success:

[ ] Easily completed [ ] Completed with difficulty or help :: Describe below

[ X ] Not completed

Comments:Task Time: 420 Seconds

Optimal Path: Admin EMR CDS/Wellness Rules Triggers “Save” Button

[ ] Correct


[ X] Major Deviations :: Describe below

Comments:

Observed Errors and Verbalizations: Comments: Unable to activate the CDS and view the information in problem list . Gave couple of attempts by trying to assign users to roles but failed.

Rating: Overall, this task was:1



Task 7: Electronic Prescribing( 100 Seconds)

Take the participant to the starting point for the task. Ensure that this patient has a drug-drug

And a drug-food allergy to the drug has chosen. This will force the participant to find

other drugs and use other elements of the application.

After examining Patient, you have decided to put this patient on – drug name.

Check for any interactions and place an order for this medication.

Create new prescription metformin 500 mg po bid No 120, zero refills and select

pharmacy, do dummy click on send button but do not clear on real send button

Success:

[ x ] Easily completed [ ]Completed with difficulty or help :: Describe below

[ ] Not completed

Comments: none

Task Time: 180 second

Optimal Path: Patient Search Meds Transmit “OK” Button

[ x] Correct



Comments:

Observed Errors and Verbalizations:

Comments: No errors




Task 8: Med, Allergy, Problem List reconciliation (65 Seconds)

Take the participant to the starting point for the task. Ensure that this patient has drugs, allergies,

problem list imported from out side source. You now decide to merge out side drugs, allergies,

problems with existing list in HER. Pay attention to allergy interactions, drug-drug interactions

during the reconcilation action.

Import new medicnes, allergies, problems into EHR form external list.

Ignore duplicate data (allergies, problems, medications) from external list.

Delete same medicines but with different sig from her list & import form external list

Success:

[ ] Easily completed [ ] • Completed with difficulty or help:: Describe below

[x ] Not completed

Comments:


Optimal Path: reconcile identical drugs reconcile new drugs reconcile same drug but

different dose/freq reconcile drugs exist in EMR but missing in external list “OK” Button

[ ] Correct

[ ] Minor Deviations / Cycles:: None

[x] Major Deviations :: Describe below

Comments: Unable to load external Clinical message to perform the reconciliation.

Observed Errors and Verbalizations:

Comments:

Need screen layout changes

Rating: Overall, this task was: 1:


Administrator / Note taker Comments: none

Final Questions (5 Minutes)

What was your overall impression of this system? Reasonably good

What aspects of the system did you like most? Easy CPOE

What aspects of the system did you like least? Clinical data reconciliation

Were there any features that you were surprised to see? No

What features did you expect to encounter but did not see? No

Is there anything that is missing in this application? No

Compare this system to other systems you have used.

I used other system, since I am not sure if I can mention the name I do not want to say that. The other

system is too difficult to use. This EMR is bit difficult to find the path but when we go to the point its

easy to perform the task. It is much better than other.

Would you recommend this system to your colleagues? Yes

Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE

Strongly Disagree=1 Strongly Agree=5

Rate each question 1 to 5











Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM

Acknowledgement of Receipt

I hereby acknowledge receipt of $ 100 for my participation in a research study run by Test

Company.

Printed Name: Stacy Davidson

Address: 4456 Marlbrough Ave, San Diego, CA: 92116

Signature: Date: 04/07/2014

Usability Researcher: _Dr. Kumara Prathipati

Signature of Usability Researcher:

Date: _04/07/2014

Witness: Nancy Kierstad

Witness Signature: _ Date: 04/07/2014

Description of participants: their experience and demographic characteristics:

1. Test Administrator: See section participants

2. Participant 1: See section participants


©2016 InfoGard. May be reproduced only in its original entirety, without revision Page 11 of 13

Appendix B: Quality Management System

Quality Management System AttestationForm-EHR-37-V02

InfoGard Laboratories, Inc. Page 1

For reporting information related to testing of 170.314(g)(4).

Vendor and Product Information

Vendor Name Adaptamed ,LLC

Product Name Adaptamed EMR

Product Version 6.2.0.0

Quality Management System

Type of Quality ManagementSystem (QMS) used in thedevelopment, testing,implementation, andmaintenance of EHR product.

Based on Industry Standard (for example ISO9001, IEC 62304, ISO13485, etc.). Standard:A modified or “home-grown” QMS.No QMS was used.

Was one QMS used for allcertification criteria or weremultiple QMS applied?

One QMS used.Multiple QMS used.Description or documentation of QMS applied to each criteria:

Not Applicable.

Statement of Compliance

I, the undersigned, attest that the statements in this document are completed and accurate.

Vendor Signature by anAuthorized Representative

Date 4/11/2014



Appendix C: Privacy and Security

Privacy and Security AttestationForm-EHR-36-V02


Vendor and Product Information

Vendor Name Adaptamed, LLC

Product Name Adaptamed EMR

Product Version 6.2.0.0

Privacy and Security

170.314(d)(2) Auditableevents and tamper-resistance

Not Applicable (did nottest to this criteria)

Audit Log:Cannot be disabled by any user.Audit Log can be disabled. The EHR enforces that the audit log is enabled by default

when initially configured

Audit Log Status Indicator:Cannot be disabled by any user.Audit Log Status can be disabled The EHR enforces a default audit log status. Identify the

default setting (enabled or disabled):There is no Audit Log Status Indicator because the Audit Log cannot

be disabled.

Encryption Status Indicator (encryption of health information locallyon end user device):

Cannot be disabled by any user.Encryption Status Indicator can be disabled The EHR enforces a default encryption status. Identify the

default setting (enabled or disabled):There is no Encryption Status Indicator because the EHR does not

allow health information to be stored locally on end user devices.

Identify the submitted documentation that describes the inability of theEHR to allow users to disable the audit logs, the audit log status, and/orthe encryption status:

Identify the submitted documentation that describes the method(s) bywhich the EHR protects 1) recording of actions related to electronichealth information, 2) recording of audit log status, and 3) recording ofencryption status from being changed, overwritten, or deleted by theEHR technology:

Privacy and Security AttestationForm-EHR-36-V02


Identify the submitted documentation that describes the method(s) bywhich the EHR technology detects whether the audit log has beenaltered:

170.314(d)(7) End-userdevice encryptionStoring electronic healthinformation locally on end-user devices (i.e. temp files,cookies, or other types ofcache approaches).

Not Applicable (did nottest to this criteria)

The EHR does not allow health information to be stored locally onend-user devices.

Identify the submitted documentation that describes thefunctionality used to prevent health information from beingstored locally:

The EHR does allow health information to be stored locally on enduser devices.

Identify the FIPS 140-2 approved algorithm used for encryption:

Identify the submitted documentation that describes how healthinformation is encrypted when stored locally on end-userdevices:

The EHR enforces default configuration settings that either enforcesthe encryption of locally stored health information or prevents healthinformation from being stored locally.

Identify the default setting:

170.314(d)(8) IntegrityNot Applicable (did not

test to this criteria)

Identify the hashing algorithm used for integrity (SHA-1 or higher):

Statement of Compliance

I, the undersigned, attest that the statements in this document are accurate.

Vendor Signature by an Authorized Representative

Date 4/11/2014



Test Results Summary Document History

Version Description of Change Date V1.0 Initial release June 6, 2014

V1.1 Updated Safety-Enhanced Design report February 28, 2016

END OF DOCUMENT

onc hit certification program test results summary for 2014...

Documents