director of software development marietta, ga 30067-6407 · 5. formal software functionality and...

Scott Garrison Director of Software Development NCDR, LLC 1090 Northchase Pkwy SE, Ste 150 Marietta, GA 30067-6407

May 30th, 2014

For public release:

NCDR LLC attests to the validity of the information below to satisfy the documentation requirements for testing and

certification of the ONC 2014 Edition criteria: 170.314.g.3 Safety Enhanced Design.

NCDR has adhered closely to the NIST 7741 UCD standard during the development of EHR modules in Boomerang to support safety enhanced design in our software. Our exact process is outlined in the 12 steps below. At a higher level, our process is basically an iterative RAD / Agile software development process. Design, develop, test, and deploy. We maintain an internal Quality Assurance group that tests newly developed software features. Deployment is not done until software has successfully passed through the internal Quality Assurance group. Any defects identified in QA are sent back to development, until software features have passed QA and are marked ready to deploy. Formal description of various steps involved.

1. End User submits new functionality requests to department manager. (Step 1 in Chart A) 2. Department Manager completes project proposal template including workflow design suggestions, and outlines

any expected functionality benefits. (Step 1 in Chart A) 3. After project proposal is reviewed and approved by management, software request is scheduled for next

available AGILE sprint. 4. Software Developer reviews documents and meets with Manager to ask questions, propose alternatives and

finalize intermediate design. (Step 2 in Chart A) 5. Formal software functionality and database design is completed by Software Developer. (Step 2 and 3 in Chart

A) 6. First phase of software development is completed by Software Developer. (Step 4 in Chart A) 7. Software Developer shares beta test version with Managers and select end users in a test environment. 8. The test user group completes first phase UCD testing with assistance / monitoring of QA group. (Step 5 in Chart

A) 9. Feedback and changes are requested if necessary by the Manager or test user group. (Step 5 in Chart A) 10. Next iterative phase of software development is completed to correct any perceived deficiencies. 11. Repeat steps 7-10 as needed until Manager and user sign off on final product. 12. Product is released into production environment.

Chart A: ¹

I hereby attest that all above statements are true, as an authorized signing authority on behalf of my organization.

Scott Garrison

Director of Software Development

May 30, 2014

¹ Robert M. Schumacher. NISTIR 7741 “NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records”. November 201. http://www.nist.gov/itl/hit/upload/Guide_Final_Publication_Version.pdf. Page 31. May 2014

http://www.nist.gov/itl/hit/upload/Guide_Final_Publication_Version.pdf.%20Page%2031

Boomerang EHR System, Version 4.2.00 2014

Page 1 of 61

EHR Usability Test Report of Boomerang, Version 4.2.00 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Full Name of Product and Version Tested: Boomerang Version 4.2.00 Date of Usability Test: June 4th and 5th, 2014 Date of Report: June 6th, 2014 Report Prepared By: NCDR, LLC STL Contact Person, Title and Affiliation: Scott Garrison, Director of Software STL Phone Number: 770-916-7213 STL Email Address: [email protected] STL Mailing Address: 1090 Northchase PKWY SE Suite 150 Marietta, GA 30067

Table of Contents 1 EXECUTIVE SUMMARY 2 2 INTRODUCTION 3 3 METHOD 4 3.1 PARTICIPANTS 4 3.2 STUDY DESIGN 4 3.3 TASKS 5 3.4 PROCEDURE 6 3.5 TEST LOCATION 6 3.6 TEST ENVIRONMENT 7 3.7 TEST FORMS AND TOOLS 7 3.8 PARTICIPANT INSTRUCTIONS 7 3.9 USABILITY METRICS 8 3.10 DATA SCORING 8

4 RESULTS 9 4.1 DATA ANALYSIS AND REPORTING 9 4.2 DISCUSSION OF THE FINDINGS AND RISK ANALYSIS 11

5 APPENDICES 13 5.1 APPENDIX 1: RECRUITING SCREENING DOCUMENT 14 5.2 APPENDIX 2: PARTICIPANT DEMOGRAPHICS 18 5.3 APPENDIX 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM 19 5.4 APPENDIX 4: MODERATOR’S GUIDE 20 5.5 APPENDIX 5: SYSTEM USABILITY SCALE QUESTIONNAIRE 61


Page 2 of 61

1. EXECUTIVE SUMMARY A usability test of NCDR, LLC Boomerang Dental EHR System was conducted on June 4th and 5th, 2014 in Marietta, GA by NCDR, LLC. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). During the usability test, 5 healthcare providers [and/or other intended users] matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 31 tasks typically conducted on an EHR:

(314.a.1) Record Medication Order

(314.a.1) Change Medication Order

(314.a.1) Access Medication Order

(314.a.1) Record Laboratory Order

(314.a.1) Change Laboratory Order

(314.a.1) Access Laboratory Order

(314.a.1) Record Radiology/Imaging Order

(314.a.1) Change Radiology/Imaging Order

(314.a.1) Access Radiology/Imaging Order

(314.a.2) Drug-Drug, Drug-Allergy- Create Interventions Prior to CPOE Completion

(314.a.2) Drug-Drug, Drug-Allergy- Adjustment of Intervention Severity

(314.a.6) Record Medication List

(314.a.6) Change Medication List

(314.a.6) Access Medication List

(314.a.7) Record Medication Allergy List

(314.a.7) Change Medication Allergy List

(314.a.7) Access Medication Allergy List

(314.a.8) Clinical Decision Support- Problem List Interventions

(314.a.8) Clinical Decision Support- Medication List Interventions

(314.a.8) Clinical Decision Support- Medication Allergy List Interventions

(314.a.8) Clinical Decision Support- Demographics Interventions

(314.a.8) Clinical Decision Support- Lab Test and Results Interventions

(314.a.8) Clinical Decision Support- Vital Signs Interventions

(314.a.8) Clinical Decision Support- Identify User Diagnostic and Therapeutic Reference Information

(314.a.8) Clinical Decision Support- Configuration of CDS Interventions by User (Admin Function)

(314.b.3) E- Prescribing- Create Prescriptions

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Medication List with Another Source

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Problem List with Another Source

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Medication Allergy List with Another Source

During the 2 hour one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix 3); they were instructed that they could withdraw at any time. Participants had prior experience with the EHR, but not in any sections relevant to this study. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded user performance data on paper and electronically. The administrator did not give the participant assistance in how to complete the task. Participant screens, head shots and audio were recorded for subsequent analysis.


Page 3 of 61

The following types of data were collected for each participant:

• Number of tasks successfully completed within the allotted time without assistance • Time to complete the tasks • Number and types of errors • Path deviations • Participant’s verbalizations • Participant’s satisfaction ratings of the system

All participant data was de-identified – no correspondence could be made from the identity of the

participant to the data collected. Following the conclusion of the testing, participants were asked to

complete a post-test questionnaire. Participants were not compensated for their time. Various

recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes

Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of

the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.

The results from the System Usability Scale scored the subjective satisfaction with the system based on

performance with these tasks to be: 90.

In addition to the performance data, the following qualitative observations were made: Major findings:

Overall the participants were pleased with the product. The participants expressed ease of use across all tasks. When asked if there were any areas that needed improvement, all participants had no suggested recommendations for improvements or enhancements to the EHR product.

In their overall ratings of the system, all participants stated they felt like Boomerang EHR was very user friendly and would take little effort to learn and implement the system into their daily workflow.

Areas for improvement:

Participants stated the font on each module access button could be changed for easier location. This was not a hindrance for the participants, but did affect the speed of access and entry until they were familiar with their locations for subsequent tasks.

2. INTRODUCTION The EHRUT tested for this study was NCDR, LLC Boomerang Version 4.2.00. Designed to present dental information to healthcare providers in Dental Offices, the EHRUT consists of an all-encompassing EHR system focused on assisting providers with dental care, from the beginning to the end of the patient experience. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide

evidence of usability in the EHRUT. To this end, measures of effectiveness, efficiency and user

satisfaction, such as time it took to complete the tasks and steps taken to complete the tasks, were

captured during the usability testing.


Page 4 of 61

3. METHOD 3.1 PARTICIPANTS

A total of 5 participants were tested on the EHRUT(s). Participants were recruited by Hire Dynamics and were not compensated for their time. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. For the test purposes, end-user characteristics were identified and translated into a recruitment screener

used to solicit potential participants; an example of a screener is provided in Appendix 1.

Recruited participants had a mix of backgrounds and demographic characteristics conforming to the

recruitment screener. The following is a table of participants by characteristics, including demographics,

professional experience, computing experience and user needs for assistive technology. Participant

names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual

identities.

ID Gender Age Education Occupation/Role Professional Experience

Computer Experience

EHR Product

Experience

Assistive Technology

Needs

1 F 25-39 Some College AR Posting Clerk 3 months High High None

2 F 25-39 Some College

Patient Service Representative 3 months Medium None None

3 F 25-39 Some College AR Posting Clerk 6 weeks Medium None None

4 F 25-39

College Graduate

Patient Service Representative 2 months High None None

5 F 25-39

College Graduate

Medicaid Eligibility Verification Specialist 2 months Medium None None

User Demographic Breakdown (Table 1)

Five participants (matching the demographics in the section on Participants) were recruited and 5 participated in the usability test. No participants failed to show for the study. Participants were scheduled for 120 minute sessions with 120 minutes in between each session for

debrief by the administrator(s) and data logger(s), and to reset systems to proper test conditions. A

spreadsheet was used to keep track of the participant schedule, and included each participant’s

demographic characteristics as provided by the recruiting firm.

3.2 STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version


Page 5 of 61

of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with the EHRUT. Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations (comments)

• Participant’s satisfaction ratings of the system Additional information about the various measures can be found in Section 3.9 on Usability Metrics.

3.3 TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including:

(314.a.1) Record Medication Order

(314.a.1) Change Medication Order

(314.a.1) Access Medication Order

(314.a.1) Record Laboratory Order

(314.a.1) Change Laboratory Order

(314.a.1) Access Laboratory Order

(314.a.1) Record Radiology/Imaging Order

(314.a.1) Change Radiology/Imaging Order

(314.a.1) Access Radiology/Imaging Order

(314.a.2) Drug-Drug, Drug-Allergy- Create Interventions Prior to CPOE Completion

(314.a.2) Drug-Drug, Drug-Allergy- Adjustment of Intervention Severity

(314.a.6) Record Medication List

(314.a.6) Change Medication List

(314.a.6) Access Medication List

(314.a.7) Record Medication Allergy List

(314.a.7) Change Medication Allergy List

(314.a.7) Access Medication Allergy List

(314.a.8) Clinical Decision Support- Problem List Interventions

(314.a.8) Clinical Decision Support- Medication List Interventions

(314.a.8) Clinical Decision Support- Medication Allergy List Interventions

(314.a.8) Clinical Decision Support- Demographics Interventions

(314.a.8) Clinical Decision Support- Lab Test and Results Interventions

(314.a.8) Clinical Decision Support- Vital Signs Interventions

(314.a.8) Clinical Decision Support- Identify User Diagnostic and Therapeutic Reference Information

(314.a.8) Clinical Decision Support- Configuration of CDS Interventions by User (Admin Function)

(314.b.3) E- Prescribing- Create Prescriptions

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Medication List with Another Source


Page 6 of 61

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Problem List with Another Source

(314.b.4) Clinical Information Reconciliation- Reconcile Patient’s Active Medication Allergy List with Another Source

Tasks were selected based on 2014 Meaningful Use requirements, frequency of use, criticality of function, and those that may be most troublesome for users.

3.4 PROCEDURE

Upon arrival to the NCDR, LLC Testing Lab, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and signed an informed consent and release form (See Appendix 3). A representative from the test team witnessed the participant’s signature. To ensure that the test ran smoothly, two staff members participated in this test, the usability

administrator and the data logger. The usability testing staff conducting the test were part of the

Boomerang Software Development Quality Assurance Team, with a combined total of 8 years of

experience using the EHRUT.

The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, path deviations, number and type of errors, and comments. Participants were instructed to perform the tasks (see specific instructions below):

As quickly as possible making as few errors and deviations as possible.

Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but not instructions on use.

Without using a think aloud technique. For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Scoring is discussed below in Section 3.9. Following the session, the administrator gave the participant the post-test questionnaire (e.g., the System Usability Scale, see Appendix 5), and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet.

3.5 TEST LOCATION The test facility included a waiting area and a quiet testing room with a table, computer for the participant,

and recording computer for the administrator. Only the participant and administrator were in the test

room. All observers and the data logger worked from a separate room where they could see the

participant’s screen and face shot, and listen to the audio of the session. To ensure that the environment

was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a

normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to

the participants. Testing was performed on June 4th and 5th, 2014.


Page 7 of 61

3.6 TEST ENVIRONMENT The EHRUT would typically be used in a healthcare office or facility. In this instance, the testing was conducted at NCDR, LLC’s EHR Test Laboratory. For testing, the computer used a Wyse Terminal running embedded Windows XP. The participants used a traditional mouse and keyboard when interacting with the EHRUT. The EHRUT used a 1280X1024 LCD monitor with 16M colors. The application was set up by the vendor

according to the vendor’s documentation describing the system set-up and preparation. The application

itself was running on a Citrix environment using a test database on a LAN connection. Technically, the

system performance was representative to what actual users would experience in a field implementation.

Additionally, participants were unable to change any of the default system settings (such as font size or

screen resolution).

3.7 TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

1. Informed Consent

2. Moderator’s Guide 3. Post-test Questionnaire

Examples of these documents can be found in Appendices 3-5 respectively. The Moderator’s Guide was

devised so as to be able to capture required data.

The participant’s interaction with the EHRUT was captured and recorded digitally with screen capture software running on the test machine. A web camera recorded each participant’s facial expressions synced with the screen capture, and verbal comments were recorded with a microphone. The test session was electronically transmitted to a nearby observation room where the data logger observed the test session.

3.8 PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to the each participant (also see the full moderator’s guide in Appendix 4): Thank you for participating in this study. Your input is very important. Our session today will last about 120 minutes. During that time you will use an instance of an electronic health record. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. I will be here in case you need specific help, but I am not able to instruct you or provide help in how to use the application. Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to

you, and how we could improve it. I did not have any involvement in its creation, so please be honest with

your opinions. All of the information that you provide will be kept confidential and your name will not be


Page 8 of 61

associated with your comments at any time. Should you feel it necessary you are able to withdraw at any

time during the testing.

Following the procedural instructions, participants were shown the EHR and as their first task, were given time (120 minutes) to explore the system and make comments. Once this task was complete, the administrator gave the following instructions: For each task, I will read the description to you and say “Begin.” At that point, please perform the task and say “Done” once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once you are done. Participants were then given 31 tasks to complete. Tasks are listed in the moderator’s guide in Appendix

4.

3.9 USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess:

1. Effectiveness of the EHRUT by measuring participant success rates and errors 2. Efficiency of the EHRUT by measuring the average task time and path deviations, including the potential risks due to path deviations 3. Satisfaction with the EHRUT by measuring ease of use ratings

3.10 DATA SCORING The following table (Table 2) details how tasks were scored, errors evaluated, and the time data

analyzed.

Measures Rationale and Scoring

Effectiveness: Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator’s Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor [e.g., 2.5] that allows some time buffer because the participants are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was 60 seconds then allotted task time performance was 150 seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.

Effectiveness: Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failure.” No task times were taken for errors.


Page 9 of 61

The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. On a qualitative level, an enumeration of errors and error types should be collected. This should also be expressed as the mean number of failed tasks per participant.

Efficiency: Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation. It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when constructing tasks.

Efficiency: Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.

Satisfaction: Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants’ confidence in and likeability of the EHRUT overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix 5.

Details of how observed data were scored (Table 2)

4. RESULTS 4.1 DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses. The usability testing results for the EHRUT are detailed below (see Table 3). The results should be seen

in light of the objectives and goals outlined in Section 3.2 Study Design. The key terms and scales used

in the following table are:

Task Success Scale: 1 = Correct, 2 = Minor deviations, 3 = Major deviations Errors Scale: 1 = No errors, 2 = Incorrect or unable to finish Task Rating Scale: 1 = Easy, 2 = Reasonable, 3 = Difficult Task Completion Time: Seconds to complete the given task SD: Standard Deviation


Page 10 of 61

Measure

Task

Effectiveness Efficiency Satisfaction

Task Success

Task Errors

Task Steps

Task Time

(seconds)

Task Ratings

Mean / SD

Value Mean /

SD Mean / SD Mean / SD

1 - CPOE (314.a.1) Record Medication Order 1.4 / 0.55 1 5.8 / 1.3 108 / 30.96 1 / 0

2 - CPOE (314.a.1) Change Medication Order 1 / 0 1 3 / 0 24.6 / 20.26 1 / 0

3 - CPOE (314.a.1) Access Medication Order 1 / 0 1 3 / 0 8.4 / 3.78 1 / 0

4 - CPOE (314.a.1) Record Laboratory Order 1.6 / 0.55 1 6 / 0.96 46.4 / 15.26 1 / 0

5 - CPOE (314.a.1) Change Laboratory Order 1 / 0 1 3 / 0 22.6 / 6.47 1 / 0

6 - CPOE (314.a.1) Access Laboratory Order 1.2 / 0.45 1 3 / 0.45 6 / 1.22 1 / 0

7 - CPOE (314.a.1) Record Radiology/Imaging Order 1 / 0 1 5 / 0 42.2 / 11.9 1 / 0

8 - CPOE (314.a.1) Change Radiology/Imaging Order 1 / 0 1 3 / 0 15 / 6 1 / 0

9 - CPOE (314.a.1) Access Radiology/Imaging Order 1 / 0 1 3 / 0 6.4 / 1.34 1 / 0

10 - CPOE (314.a.2) Create Interventions Prior to CPOE Completion 1.6 / 0.55 1 10 / 0.55 39.6 / 8.38 1 / 0

11 -CPOE (314.a.2) Adjustment of Intervention Severity 1.4 / 0.55 1 11 / 1.41 57.6 / 22.8 1 / 0

12 -CPOE (314.a.2) Adjustment of Intervention Severity (confirmation) 1.6 / 0.55 1 5 / 0.55 18.4 / 3.36 1 / 0

13 - CPOE (314.a.6) Record Medication List 1.2 / 0.45 1 12 / 0.45 77 / 22.84 1.2 / 0.45

14 - CPOE (314.a.6) Change Medication List 1 / 0 1 3 / 0 23.8 / 9.01 1 / 0

15 - CPOE (314.a.6) Access Medication List 1 / 0 1 3 / 0 6.2 / 0.84 1 / 0

16 - CPOE (314.a.7) Record Medication Allergy List 1.4 / 0.55 1 10 / 1.41 63.4 / 38.12 1.2 / 0.45

17 - CPOE (314.a.7) Change Medication Allergy List 1.2 / 0.45 1 5 / 1.34 25 / 19.58 1 / 0

18 - CPOE (314.a.7) Access Medication Allergy List 1 / 0 1 3 / 0 6.4 / 1.95 1 / 0

19 - CPOE (314.a.8) Problem List Interventions 2 / 0.55 1 6 / 1 15.4 / 7.16 1.2 / 0.45

20 - CPOE (314.a.8) Medication List Interventions 1 / 0 1 5 / 0.45 10 / 4.24 1.2 / 0.45

21 - CPOE (314.a.8) Medication Allergy List Interventions 1 / 0 1 5 / 0.45 5.2 / 0.84 1 / 0

22 - CPOE (314.a.8) Demographics Interventions 1 / 0 1 5 / 0.45 9.8 / 3.11 1 / 0

23 - CPOE (314.a.8) Lab Test and Results Interventions 1 / 0 1 5 / 0.45 7.2 / 1.92 1 / 0

24 - CPOE (314.a.8) Vital Signs Interventions 1 / 0 1 5 / 0.45 7.6 / 3.51 1 / 0

25 - CPOE (314.a.8) Identify User Diagnostic and Therapeutic Reference Information 1.6 / 0.55 1 5 / 1.82 17.2 / 7.66 1.4 / 0.55

25 - CPOE (314.a.8) Identify User Diagnostic and Therapeutic Reference Information (confirmation) 1.6 / 0.55 1 8 / 0.84 11.2 / 3.7 1.2 / 0.45

26 - CPOE (314.a.8) Configuration of CDS Interventions by User (Admin Function) 1.4 / 0.55 1 4 / 0.89 12.2 / 3.42 1 / 0

27 - CPOE (314.b.3) E- Prescribing- Create Prescriptions 1.8 / 0.45 1 14 / 1.82 46 / 29.57 1.2 / 0.45

28 - CPOE (314.b.4) Reconcile Patient’s Active Medication List with Another Source 1.6 / 0.55 1 17 / 2.51

56.8 / 34.52 1.8 / 0.45


Page 11 of 61

29 - CPOE (314.b.4) Reconcile Patient’s Active Problem List with Another Source 1.4 / 0.55 1 11 / 0.55 19.2 / 5.72 1.2 / 0.45

30 - CPOE (314.b.4) Reconcile Patient’s Active Medication Allergy List with Another Source 1.2 / 0.45 1 12 / 1.79 21.4 / 7.5 1.2 / 0.45

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system

based on performance with these tasks to be: 91.

4.2 DISCUSSION OF THE FINDINGS AND RISK ANALYSIS

All participants rated 21 of the 31 tasks as ‘Very Easy’. Below are details of all tasks which contained path deviations or technical issues. Task 4 was completed successfully by all participants; however, two participants accidently selected an existing entry prior to data entry. Both participants self-corrected the issue before any modifications were made and added a new order. This was found to be a low risk deviation. Task 19 was completed successfully by all participants; however, one participant selected the “master” intervention button instead of the individual module button. Participant self-corrected and successfully completed the task. This was not deemed to be a high-risk issue. Task 28 was completed successfully by all participants; however, one participant accidently selected the Medication list instead of the DoseSpot (CPOE) application. Participant self-corrected and successfully completed the task. EFFECTIVENESS The EHRUT proved to be functional and successful at handling the required tasks. With proper training, practice and daily use, it is expected that the average user would have no problem using this system. No programming adjustments are needed at this time. EFFICIENCY All tasks were prioritized by user risk, and completed within the estimated optimal task performance time based upon measures of task times and deviations. Times were benchmarked by expert performance in the same environment as participants and multiplied by a 2.5 second time buffer to allow for the fact that the participants in this study are new users to this application. The completion times were less than optimal estimated performance time on 22 of 31 tasks. SATISFACTION The participants were asked to rate each task upon completion on a scale of 1=Very Easy to 5=Very Difficult. 21 out of 31 tasks were rated “Very Easy” and no tasks were rated 3 – 5. Results of the test indicated the application is user friendly and intuitive in most areas of testing. All participants agreed the design and functionality was good, well integrated as and easy to use.


Page 12 of 61

MAJOR FINDINGS Overall the participants were pleased with the product. The participants expressed ease of use across all

tasks. When asked if there were any areas that needed improvement, all participants had no suggested

recommendations for improvements or enhancements to the EHR product.

In their overall ratings of the system, all participants stated they felt like Boomerang was very user friendly

and would take little effort to learn and implement the system into their daily workflow.

AREAS FOR IMPROVEMENT Participants stated the font on each module access button could be changed for easier location. This

was not a hindrance for the participants, but did affect the speed of access and entry until they were

familiar with their locations for subsequent tasks.


Page 13 of 61

5. APPENDICES The following appendices include supplemental data for this usability test report. Following is a list of the appendices provided: 5.1: Recruiting Screening Document 5.2: Participant Demographics 5.3: Non-Disclosure Agreement (NDA) and Informed Consent Form 5.4: End User Testing Tasks 5.5: System Usability Scale Questionnaire 5.6: Incentive receipt and acknowledgment form


Page 14 of 61

5.1 Recruiting Screening Document

NCDR, LLC EHR End User Test Recruiting Screen We are seeking testers to assist us in a usability research project for a dental practice management system. We would like to ask you a few questions to gather participant details. This should only take a few minutes of your time. This is strictly for research purposes. There is no compensation for participating in Training and Testing.

Please answer the following questions: 1. Are you male or female? ______ 2. Have you participated in a focus group or usability test in the past 6 months? Yes/No 3. Do you, or does anyone in your home, work in marketing research, usability research, web design, or other computer work? Yes/No 4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? Yes/No 5. Which of the following best describes your age?

o 23 to 39 o 40 to 59 o 60 to 74 o 75 or older

6. Which of the following best describes your race or ethnic group?

o Caucasian o Asian o Black/African American o Latino/a or Hispanic o Other

7. Do you require any assistive technologies to use a computer? Yes/No If Yes, Please Describe:

____________________________________________________________________________

____________________________________________________________________________

(Continued on next page)


Page 15 of 61

Professional Demographics: 8. What is your current position and title? (Must be dental related) _____Dentist Title ________________ _____Dental Hygienist Title ________________ _____Administrator Title ________________ _____Front Desk Title ________________ _____ Billing Staff Title ________________ _____Other ___________ Title ________________ 9. How long have you held this position? _______ years ________ months _______weeks 10. Describe your work location (or affiliation) and environment: ____________________________________________________________________________ ____________________________________________________________________________ 11. Which of the following describes your highest level of education?

o High School/GED o Some College o College Graduate (RN/BSN) o Postgraduate (DDS/DMD/PhD) o Other (please describe): _________________________________________________

Computer Proficiency 12. What professional activities do you do on the computer?

o Access EHR o Research News o Shopping/Banking o Digital Pictures o Programming o Microsoft Office Products o Gambling

13. About how many hours per week do you spend on the computer?

o 0 to 10 o 11 to 25 o 26 or More



Page 16 of 61

14. What computer platform do you typically use? o Microsoft Windows o Mac o Linux o Other (please explain) __________

15. What Internet browser(s) do you usually use?

o Internet Explorer o Firefox o Chrome o Safari o Other (please explain) __________

16. In the last month, how often have you used an electronic health record? ___________ 17. How many years have you used an electronic health record? _________ 18. How many EHRs do you use or are you familiar with? ___________ 19. How does your work environment record/retrieve patient records?

o On Paper

o Some Paper/Some Electronic

o All Electronic

Contact Information:

Name of participant: ______________________________________

Address/City/State/Zip: ____________________________________________________

Daytime phone: ________________________

Evening Phone: ________________________

Cell phone: ____________________________

Email address: _____________________________



Page 17 of 61

Before your session starts, we may ask you to sign a release form allowing us to videotape your

session. The video recording will only be used internally for further study if needed. Will you

consent to be videotaped? Yes/No

This study will take place at NCDR, LLC headquarters in Marietta, GA. You will participate in the

study at that location.


Page 18 of 61

5.2 Participant Demographics

Gender

Male 0

Female 5

Occupation/Role

Front Desk 0

Billing 0

Administration 3

Executive 0

Hygienist 0

Patient Service 2

Experience

Total Experience 11.5 months

EHR Usage 11 years

All Paper 0

Some Paper/Some

Electronic 0

All Electronic 5

High –Level Overview

ID Gender Age Education Occupation/Role Professional Experience

Computer Experience

EHR Product

Experience

Assistive Technology

Needs

1 F 25-39 Some College AR Posting Clerk 3 months High High None

2 F 25-39 Some College

Patient Service Representative 3 months Medium None None

3 F 25-39 Some College AR Posting Clerk 6 weeks Medium None None

4 F 25-39

College Graduate

Patient Service Representative 2 months High None None

5 F 25-39

College Graduate

Medicaid Eligibility Verification Specialist 2 months Medium None None

Patient Demographics


Page 19 of 61

5.3 Non-Disclosure Agreement (NDA) and Informed Consent Form

NCDR, LLC End User Testing Non-Disclosure Agreement

THIS AGREEMENT is entered into as of __________, 2014, between ____________________

(“the Participant”) and the testing organization NCDR, LLC at 1090 Northchase PKWY SE

Marietta, GA 30067.

The Participant acknowledges his or her voluntary participation in today’s usability study may

bring the Participant into possession of Confidential Information. This means all technical and

commercial information of a proprietary or confidential nature which is disclosed by NCDR,

LLC or otherwise acquired by the Participant, in the course of this study.

By way of illustration, but not limitation, Confidential Information includes trade secrets,

processes, formulae, data, know-how, products, designs, drawings, computer aided design files

and other computer files, computer software, ideas, improvements, inventions, training methods

and materials, marketing techniques, plans, strategies, budgets, financial information, and/or

forecasts.

Any information the Participant acquires relating to this product during this study is confidential

and proprietary to NCDR, LLC, and is being disclosed solely for the purposes of the

Participant’s participation in today’s study.

By signing this form the Participant acknowledges that she or he will receive no monetary

compensation for feedback and will not disclose this confidential information obtained today to

anyone else or any other organizations.

Participant’s printed name: ________________________________________

Signature: _____________________________________ Date: ____________________


Page 20 of 61

5.4 Moderator’s Guide

Test Guide for Safety Enhanced Design

Test Number: 1

Patient Account Number:

Patient Name:

Test Steps:

1) Log into EHR system using the following credentials:

Username: stest1 Password: <provided at time of testing>

2) Locate the test patient by clicking on “Schedule” on the left side of the screen. The Schedule

displays in the right side of the window.

3) Change the Date at the top of the window from today’s date to 05/15/2014 and press Enter.

4) Select the Patient Chart by right clicking on the Patient’s appointment on the Schedule and clicking

on “Chart”.

5) Select Dr. James Getwell from the Provider dropdown list.

6) Open the EHR Modules window, using the “EHR” button.

7) Perform the tasks on the following pages, and rate for ease of use.


Page 21 of 61

(314.a.1) Record Medication Order (Task 1)

Optimal path:

a) Choose the “Order Entry (CPOE)” button to open the CPOE window.

b) Select the “Add” button.

c) Select “Medication” from the dropdown list.

d) Add the following medication entry to the Patient’s CPOE items:

1. Drug: Lorazepam 0.5 mg tablet [RxNorm: 197900]; Dispensing Instructions: 20 count;

Patient Instructions: Take 1 tablet by mouth two times daily; Refills: 1

e) Click “OK” to close the CPOE window, leaving the EHR Modules window open.

User Instructions: Order the medication Lorazepam:

Drug: Lorazepam 0.5 mg tablet [RxNorm: 197900]; Dispensing Instructions: 20 count; Patient

Instructions: Take 1 tablet by mouth two times daily; Refills: 1

Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________

Observed Errors and Verbalizations: ______________________________________________________________________________ ______________________________________________________________________________

Overall Task Rating: _______ Scale: (1) “Very Easy“ to (5) “Very Difficult”

Time to complete: _______________ Administrator Comments: ______________________________________________________________________________ ______________________________________________________________________________


Page 22 of 61

(314.a.1) Change Medication Order (Task 2)

Optimal path:


b) Change the following underlined Medication entry on the Patient’s CPOE items:

1. Drug: Lorazepam 0.5 mg tablet [RxNorm: 197900]; Dispensing Instructions: 20 count;

Patient Instructions: Take 1 tablet by mouth twothree times daily; Refills: 1

c) Click “OK” to close the CPOE window, leaving the EHR Modules window open.

User Instructions: Change the medication order for Lorazepam:


Instructions: Take 1 tablet by mouth twothree times daily; Refills: 1






Page 23 of 61

(314.a.1) Access Medication Order (Task 3)

Optimal path:


b) Find and inspect the following previously-added Medication entry on the Patient’s CPOE

items, to ensure it matches the following entry:

1. Drug: Lorazepam 0.5 mg tablet [RxNorm: 197900]; Dispensing Instructions: 20

count; Patient Instructions: Take 1 tablet by mouth three times daily; Refills: 1


Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed User Instructions: Access and view the medication order for Lorazepam. Locate this medication order and review the

details to ensure they match the following:


Instructions: Take 1 tablet by mouth three times daily; Refills: 1

Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 24 of 61

(314.a.1) Record Laboratory Order (Task 4)

Optimal path:



c) Select “Laboratory” from the dropdown list.

d) Add the following Laboratory entry to the Patient’s CPOE items:

1. Creatinine 24 H Urine Panel [RxNorm: 34555-3]


User Instructions: Order the Laboratory order for Creatinine:

Creatinine 24 H Urine Panel [RxNorm: 34555-3]






Page 25 of 61

(314.a.1) Change Laboratory Order (Task 5)

Optimal path:


b) Change the following underlined Laboratory entry on the Patient’s CPOE item:

1. Creatinine 24 H Urine Panel [RxNormLOINC: 34555-3]


User Instructions: Change the order for Creatinine:

Creatinine 24 H Urine Panel [RxNormLOINC: 34555-3]






Page 26 of 61

(314.a.1) Access Laboratory Order (Task 6)

Optimal path:


b) Find and inspect the following previously-added Laboratory entry on the Patient’s CPOE

items, to ensure it matches the following entry:

1. Creatinine 24 H Urine Panel [LOINC: 34555-3]


User Instructions: Access and view the Laboratory order for Creatinine. Locate this Laboratory order and review the

details to ensure they match the following:

Creatinine 24 H Urine Panel [LOINC: 34555-3]






Page 27 of 61

(314.a.1) Record Radiology/Imaging Order (Task 7)

Optimal path:



c) Select “Radiology / Imaging” from the dropdown list.

d) Add the following Radiology/Imaging entry to the Patient’s CPOE items:

1. Radiologic examination knee 3 views [CPT Code: 70460]


User Instructions: Order the Radiology/Imaging order for Radiologic examination knee 3 views:

Radiologic examination knee 3 views [CPT Code: 70460]

Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________ Observed Errors and Verbalizations: ______________________________________________________________________________ ______________________________________________________________________________




Page 28 of 61

(314.a.1) Change Radiology/Imaging Order (Task 8)

Optimal path:


b) Change the following underlined Radiology/Imaging entry on the Patient’s CPOE item:



User Instructions: Change the Radiology/Imaging order for Radiologic examination knee 3 views:







Page 29 of 61

(314.a.1) Access Radiology/Imaging Order (Task 9)

Optimal path:


b) Find and inspect the following previously-added Radiology/Imaging entry on the Patient’s

CPOE items, to ensure it matches the following entry:



User Instructions: Access and view the Radiology/Imaging order for Radiologic examination knee 3 views. Locate this

Radiology/Imaging order and review the details to ensure the details match the following:







Page 30 of 61

(314.a.2) Drug-Drug, Drug-Allergy- Create Interventions Prior to CPOE Completion (Task 10)

Pre-Loaded Medications:

Warfarin 5 mg tablet once daily, dispense 30, refills 0, 30 day supply.

Valprolic Acid 250 MG capsule once daily, dispense 30, refills 0, 30 day supply.

Pre-loaded Allergies:

Penicillin, status active, reaction hives, onset 5/15/2014

Optimal path:

a) Close the EHR window by selecting the “X” button and then select the “Rx Staff” button at

the bottom of the Chart window.

b) Click on the “Add New Prescription” button.

c) Type in “Amox” and when list populates, select “Amoxicillin (Oral Tablet)”.

d) Select “500 mg”.

e) Enter the following information:

a. Patient Directions: Take once daily.

b. Dispense: 15

c. Refills: 0

d. Days Supply: 3

a) Click on the “Save Prescription” button.

b) Click on the “X” on the top right of the window to exit.

User Instructions: Order medication with drug-drug and drug-allergy interactions. Place the following order:

Amoxicillin (Oral Tablet), 500 mg; Patient Directions: Take once daily.; Dispense: 15;

Refills: 0; Days Supply: 3




Page 31 of 61




Page 32 of 61

(314.a.2) Drug-Drug, Drug-Allergy- Adjustment of Intervention Severity (Task 11)

Optimal path:

a) Select the “X” button on the right hand corner of the Chart window to close.

b) Minimize Boomerang by clicking the “-“ button on the top right hand corner of the Schedule

window.

c) Double click on the “DoseSpot” Icon on the desktop.

d) Enter User Name: <provided during test> and Password: <provided during test>

e) Click the “login” button.

f) Click on the “Show only major drug-drug interactions (hide minor and moderate drug-drug

interactions)” radio button.

g) Click on the “Save Clinic Configurations” button.

h) Click the “X” button on the top right corner of window to close window.

User Instructions: Change the drug-drug interactions.

User Name: <provided at time of test> Password: <provided at time of test>






Page 33 of 61

(314.a.2) Drug-Drug, Drug-Allergy- Adjustment of Intervention Severity (confirmation) (Task 12)

Optimal path:

a) Maximize Boomerang by clicking on the Boomerang icon on the task bar.

b) Right click on the patient’s appointment and choose “Chart”.

c) Select James Getwell as provider from the dropdown list.

d) Click on the “Rx Staff” button in patient’s chart.

e) Verify alerts display according to configuration.

f) Click the “X” button on the top right corner of window to exit.

User Instructions: Verify the alerts display for Dr. James Getwell. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 34 of 61

(314.a.6) Record Medication List (Task 13)

Optimal path:

a) Click on the “EHR” button.

b) Choose the “Medication List” button.

c) Click the “Add” button.

d) Add the following information:

RX Norm Code: 312961

Medication Name: Simvastatin

Strength: 20 mg

Product Form: TABLET, COATED

Route: ORAL

Frequency: Once daily

Start Date: May 15th, 2014

Fill Instructions: Generic substitutions allowed.

e) Click “OK” to close the EHR Medication List window, leaving the EHR Modules window open.

User Instructions: Add a medication the patient takes at home. Enter the following medication:

RX Norm Code: 312961; Medication Name: Simvastatin; Strength: 20 mg; Product Form: CAPSULE; Route: ORAL; Frequency: Once daily; Start Date: May 15th, 2014; Fill Instructions: Generic substitutions allowed.




Time to complete: _______________


Page 35 of 61

Administrator Comments: ______________________________________________________________________________ ______________________________________________________________________________


Page 36 of 61

(314.a.6) Change Medication List (Task 14)

Optimal path:

a) Choose the “Medication List” button to open window.

b) Change the following underlined information in the selected row:



Strength: 20 mg


Route: ORAL



Fill Instructions: Generic substitutions allowed. 1 refill allowed.

c) Click “OK” to close the EHR Medication List window, leaving the EHR Modules window open.

User Instructions: Change the instructions on the patient’s medication list. Make the following change to the patient’s

medication list:

RX Norm Code: 312961; Medication Name: Simvastatin; Strength: 20 mg; Product

Form: CAPSULE; Route: ORAL; Frequency: Once daily; Start Date: May 15th, 2014; Fill

Instructions: Generic substitutions allowed. 1 refill allowed.






Page 37 of 61



Page 38 of 61

(314.a.6) Access Medication List (Task 15)

Optimal path:

a) Click on the “Medication List” button.

b) Find and inspect the following previously-added Medication entry on the Patient’s

Medication List items, to ensure it matches the following entry:



Strength: 20 mg


Route: ORAL



Fill Instructions: Generic substitutions allowed. 1 refill allowed.

c) Click “OK” to close the EHR Medication List window, leaving the EHR Modules window open.

User Instructions: Access and view the patient’s medication list. Locate this medication item and review the details to

ensure they match the following:

RX Norm Code: 312961; Medication Name: Simvastatin; Strength: 20 mg; Product

Form: CAPSULE; Route: ORAL; Frequency: Once daily; Start Date: May 15th, 2014; Fill

Instructions: Generic substitutions allowed. 1 refill allowed.





Page 39 of 61



Page 40 of 61

(314.a.7) Record Medication Allergy List (Task 16)

Optimal path:

a) Click on the “Medication Allergies” button.

b) Click the “Add” button.

c) Add the following information into the new row:

Encounter Date: May 15, 2014

Status: Active

RX Norm Code: 7982

Medication Name: Penicillin G benzathine

Reaction: Hives

Severity: MILD

Event Date: 01/15/2011

Event End Date: <leave blank (00/00/0000)>

d) Click “OK” to close the Medication Allergy List window, leaving the EHR Modules window

open.

User Instructions: Add a medication allergy the patient has. Enter the following medication allergy:

Encounter Date: May 15, 2014; Status: Active; RX Norm Code: 7982; Medication Name:

Penicillin G benzathine; Reaction: Hives; Severity: MILD; Event Date: 01/15/2011; Event

End Date: <leave blank (00/00/0000)>





Page 41 of 61



Page 42 of 61

(314.a.7) Change Medication Allergy List (Task 17)

Optimal path:

a) Choose the “Medication Allergies” button.

b) Change the following underlined information:


Status: Active Inactive

RX Norm Code: 7982


Reaction: Hives

Severity: MILD


Event End Date: <leave blank (00/00/0000)>06/07/2011

c) Click “OK” to close the Patient Medication Allergy List window, leaving the EHR Modules

window open.

User Instructions: Change a medication allergy the patient has. Enter the following changes to the medication allergy

list:

Encounter Date: May 15, 2014; Status: Inactive; RX Norm Code: 7982; Medication

Name: Penicillin G benzathine; Reaction: Hives; Severity: MILD; Event Date:

01/15/2011; Event End Date: 06/07/2011





Page 43 of 61



Page 44 of 61

(314.a.7) Access Medication Allergy List (Task 18)

Optimal path:

a) Choose the “Medication Allergies” button.

b) Find and inspect the following previously-added Medication Allergies entry to ensure it

matches the following entry:


Status: Active

RX Norm Code: 7982


Reaction: Hives

Severity: MILD


Event End Date: 06/07/2011

c) Click “OK” to close the Medication Allergy List window, leaving the EHR Modules window

open.

User Instructions: Access and view the patient’s medication allergy list. Locate this medication allergy item and review

the details to ensure they match the following:

Encounter Date: May 15, 2014; Status: Inactive; RX Norm Code: 7982; Medication

Name: Penicillin G benzathine; Reaction: Hives; Severity: MILD; Event Date:

01/15/2011; Event End Date: 06/07/2011





Page 45 of 61



Page 46 of 61

(314.a.8) Problem List Interventions (Task 19)

Optimal path:

a) Choose the “Patient Problem List” button.

b) Locate and select the Intervention button (orange and yellow lightning bolt button on the

bottom left hand side of the window).

c) View the Intervention.

d) Click the “X” button to close the Problem List Intervention window.

e) Click the “Ok” button on the Patient Problem List window, leaving the EHR Modules window

open.

User Instructions: Access and view the problem intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 47 of 61

CDS (314.a.8) Medication List Interventions (Task 20)

Optimal path:

a) Choose the “Medication List” button.




d) Click the “X” button to close the Medication List Intervention window.

e) Click the “Ok” button on the Medication List window, leaving the EHR Modules window

open.

User Instructions: Access and view the medication Intervention. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 48 of 61

CDS (314.a.8) Medication Allergy List Interventions (Task 21)

Optimal path:

a) Choose the “Medication Allergy List” button.




d) Click the “X” button to close the Medication Allergy List Intervention window.

e) Click the “Ok” button on the Medication List window, leaving the EHR Modules window

open.

User Instructions: Access and view the patient’s medication allergy intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 49 of 61

CDS (314.a.8) Demographics Interventions (Task 22)

Optimal path:

a) Choose the “Patient Demographics” button.




d) Click the “X” button to close the Patient Demographics Interventions window.

e) Click the “Ok” button on the Patient Demographics window, leaving the EHR Modules

window open.

User Instructions: Access and view the patient’s demographics intervention list.






Page 50 of 61

CDS (314.a.8) Lab Tests and Results Interventions (Task 23)

Optimal path:

a) Choose the “Lab Tests/Results” button.




d) Click the “X” button to close the Lab Tests/Results Interventions window.

e) Click the “Ok” button on the Lab Tests/Results window, leaving the EHR Modules window

open.

User Instructions:

Access and view the patient’s laboratory intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 51 of 61

(314.a.8) Vital Signs Interventions (Task 24)

Optimal path:

a) Choose the “Patient Vital Signs” button.




d) Click the “X” button to close the Patient Vital Signs Interventions window.

e) Click the “Ok” button on the Patient Vital Signs window, leaving the EHR Modules window

open.

User Instructions:

Access and view the patient’s vital signs intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 52 of 61

(314.a.8) Identify User Diagnostic and Therapeutic Reference Information (Task 25)

Optimal path:

a) Click the “Ok” button to exit the EHR window.

b) Select the “Close” button to exit the Chart window.

c) Click the “EHR” folder in the tree view to open the Intervention Config window.

d) Select the ”Run” button next to “Intervention Config”

e) Locate the “Reference Resources” for each intervention.

User Instructions:

Locate the Reference Resources.






Page 53 of 61

(314.a.8) Confirm Identify User Diagnostic and Therapeutic Reference Information (Task 26)

Optimal path:

a) Click the “EHR” folder in the tree view to open the Intervention Config window.

b) Select the ”Run” button next to “Intervention Config”.

c) Locate the “Reference Resources” for each intervention.

d) Click on the button (with the picture of eyeglasses and a book) to the right of the Reference

Resource address.

e) Confirm the Reference Resource is enabled.

f) Confirm the Modified and Modified By fields are present and populated.

g) Click the “Ok” to exit the Resource button.

User Instructions: Access and view the patient’s diagnostic and therapeutic reference intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 54 of 61

(314.a.8) Configure Identify User Diagnostic and Therapeutic Reference Information (Task 27)

Optimal path:

a) Select the ”Run” button next to “Intervention Config”

b) Deselect / Select the “Enabled” check box to turn off / on the intervention.

c) Click the “Update” button to save changes.

User Instructions: Confirm the patient’s diagnostic and therapeutic reference intervention list. Success: ( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 55 of 61

(314.b.3) E-Prescribing-Create Prescriptions (Task 28)

Optimal path:

a) Click on “Schedule” on treeview.

b) Right click on patient’s appointment and select “Chart”.

c) Select James Getwell from Provider dropdown list.

d) Click on the “Rx Staff” button.

e) Click on the “Add New Prescription” button.

f) Type in “Levox” and then select “Levoxyl”.

g) Select “100 mcg (0.1 mg)”

h) Enter the following information:

a. Patient Directions: Take once daily.

b. Dispense: 30

c. Refills: 3

d. Days Supply: 30

i) Click on the “Save Prescription” button.

j) Click on the “X” button to exit window.

User Instructions: Order an electronic prescription for the patient. Order the following medication:

Medication Name: Levoxyl ,100 mcg (0.1 mg); Patient Directions: Take once daily.;

Dispense: 30; Refills: 3; Days Supply: 30






Page 56 of 61



Page 57 of 61

(314.b.4) Clinical Reconciliation Active Medication List (Task 29)

Optimal path:

a) Click on the “EHR” button to open window.

b) Choose the “Import CCDA Summary” button to open the CCDA Import window.

c) Click on the “Select a file” button.

d) Using the dropdown, locate file in K: > AATestfiles > Import >, select the file named “CCDA

Import” and click on the “Open” button.

e) Once the Human Readable summary window opens, click the “X” in the upper right of the

window to close the window.

f) Deselect the checkmarks from Problems and Medical Allergies, and then click the “Next”

button.

g) Click the “Next” button.

h) Select the check located next to “I have reviewed/verified that this list has been reconciled

and is current/accurate”.

i) Click on the “Finish” button.

User Instructions: Reconcile the medication list with an outside source.

Success:

( ) Easily completed ( ) Completed with difficulty or help: Describe in Comments ( ) Not completed Comments: ______________________________________________________________________________ ______________________________________________________________________________





Page 58 of 61

(314.b.4) Clinical Reconciliation Active Problem List (Task 30)

Optimal path:

a) Choose the “Import CCDA Summary” button to open the CCDA Import window.

b) Click on the “Select a file” button.

c) Using the dropdown, locate file in K: (Kool data on Ncdrfile) > AATestfiles > Import >, select

the file named “CCDA Import” and click on the “Open” button.

d) Once the Human Readable summary window opens, click the “X” in the upper right of the


e) Deselect the checkmarks from Medications and Medical Allergies, and then click the “Next”

button.

f) Click the “Next” button.

g) Select the check located next to “I have reviewed/verified that this list has been reconciled


h) Click on the “Finish” button.

User Instructions: Reconcile the problem list with an outside source.

Success:






Page 59 of 61

(314.b.4) Clinical Reconciliation Active Medication Allergy (Task 31)

Optimal path:

a) Choose the “Import CCDA Summary” button to open the CCDA Import window.

b) Click on the “Select a file” button.

c) Using the dropdown, locate file in K: (Kool data on Ncdrfile) > AATestfiles > Import >, select

the file named “CCDA Import” and click on the “Open” button.

d) Once the Human Readable summary window opens, click the “X” in the upper right of the


e) Deselect the checkmarks from Problems and Medications, and then click the “Next” button.

f) Click the “Next” button.

g) Select the check located next to “I have reviewed/verified that this list has been reconciled


h) Click on the “Finish” button.

User Instructions: Reconcile the medication allergy list with an outside source.

Success:






Page 60 of 61

Final Questions (20 Minutes):

What was your overall impression of this system?

______________________________________________________________________________ ______________________________________________________________________________

What aspects of the system did you like most?

______________________________________________________________________________ ______________________________________________________________________________

What aspects of the system did you like least?

______________________________________________________________________________ ______________________________________________________________________________

Were there any features that you were surprised to see?

______________________________________________________________________________ ______________________________________________________________________________

What features did you expect to encounter but did not see? That is, is there anything that is missing in this application?

______________________________________________________________________________ ______________________________________________________________________________

Compare this system to other systems you have used.

______________________________________________________________________________ ______________________________________________________________________________

Would you recommend this system to your colleagues?

______________________________________________________________________________ ______________________________________________________________________________


Page 61 of 61

5.5 System Usability Scale Questionnaire

Question

Strongly Disagree

(1) Disagree

(2) Neutral

(3) Agree

(4)

Strongly Agree

(5)

I think that I would like to use this system frequently

I found the system unnecessarily complex

I thought this system was easy to use

I think that I would need the support of a technical person to be able to use this system

I found the various functions in this system were well integrated

I thought there was too much inconsistency in this system

I would imagine that most people would learn to use this system very quickly

I found the system very cumbersome to use

I felt very confident using the system

I needed to learn a lot of things before I could get going with this system


May 30th, 2014

For public release:


certification of the ONC 2014 Edition criteria: 170.314.g.4 Quality Management System.

Test Requirement: DTR 170.314.g.4-1: Identify the use of a QMS for each capability for which certification is sought

This describes the home-grown quality management system (QMS) used by the NCDR software development group to develop Boomerang, our EHR system software. Our process is basically an iterative RAD / Agile software development process. Design, develop, test, and deploy. We maintain an internal Quality Assurance group that tests newly developed software features. Deployment is not

done until software has successfully passed through the internal Quality Assurance group. Any defects identified in QA

are sent back to development, until software feature has passed QA and are marked ready to deploy.

This homegrown QMS system has been used for all criteria submitted for testing and certification under the ONC 2014

Edition criteria.


Scott Garrison


May 30, 2014


May 30th, 2014

For public release:


certification of the ONC 2014 Edition criteria: 170.314(d)7 Encryption of Data at Rest.

[Include the following information based on the EHR System under Test’s functionality]

Test Requirement: DTR 170.314.d.7-1: Determine Encryption Algorithm Used

All EHR data is stored on a centralized database server and there is no data at rest on end-user devices. Because

there is no data at rest on end-user devices , we are not using an encryption algorithm.

Test Requirement: DTR 170.314.d.7-2: Determine Default Setting and Configuration Capability

Boomerang EHR system runs on a centralized Citrix farm, using a Microsoft SQL Server database for centralized

data storage. Terminal devices are used in our offices to access Citrix farm to run Boomerang. User ability to

disable encryption status is not applicable as data is stored in central database and not stored locally on end-

user devices.

Test Requirement: DTR 170.314.d.7-3: Encrypt Data on End-User Device After Use is Stopped


data storage. Terminal devices are used in our offices to access Citrix farm to run Boomerang. Because there is

no data at rest on end-user devices , we are not using an encryption algorithm.

Test Requirement: DTR 170.314.d.7-4: Prevent Local Storage on End-User Device After Use is Stopped


data storage. Terminal devices are used in our offices to access Citrix farm to run Boomerang. There is no data

at rest stored or saved to end-user devices.


Scott Garrison


May 30, 2014


March 8th, 2014

For public release:


certification of the ONC 2014 Edition criteria: 170.314(d)2.

[Include the following information based on the EHR System under Test’s functionality]

1. Does the EHR SUT allow the following?

Disabling the audit log

monitoring and recording of audit log status changes (if disabling is possible)

monitoring and recording of status changes to encryption, if encryption is used to satisfy the end

user device encryption (d)7 criteria

[IN170.314(d)(2)-2.02 / IN170.314(d)(2)-2.09]

The EHR system (Boomerang) does not allow users to disable the audit logs or audit log status. Auditing is

embedded in database stored procedures and audit logs cannot be configured or modified by the users. All

EHR data is stored on a centralized database server and there is no data at rest on end-user devices. User ability

to disable encryption status is not applicable.

2. If the audit log can be disabled, is the default state for audit log and audit log status recording

enabled by default?

No. Auditing is embedded in database stored procedures and audit logs cannot be configured or modified by the users.

3. If applicable, and if the EHR also allows it to be disabled, is the encryption of electronic health

information on end-user devices enabled by default?

No.

All EHR data is stored on a centralized database server and there is no data at rest on end-user devices. User

ability to disable encryption status is not applicable.

4. Does the EHR SUT permit any users to delete electronic health information?

[IN170.314(d)(2)-3.03]

Yes.

Yes, users can delete a very limited amount of EHR data, primarily related to recent functionality add for EHR

certification. Medication history, Medication allergies are primary example where entered data can be deleted.

Any deletions are audited.

5. Does the EHR SUT audit logging capability monitor each of the required actions for all instances

of electronic health information utilized by the EHR SUT in accordance with

the specified standard ASTM E2147-01?

[IN170.314(d)(2)-3.04] Yes. Boomerang EHR SUT monitors and audits all instances of EHR data for the following actions. Addition: Deletion: Changes: Queries: Print: Copy:

6. Describe the method(s) through which the audit logs are protected from being changed,

overwritten, or deleted by the EHR technology itself.

[IN170.314(d)(2)-4.01] There is one centralized method (stored procedure) that records all primary audit records via record insert only. Changes to actual EHR data are tracked via versioning and database triggers.

7. Describe the method(s) through which the EHR SUT is capable of detecting whether the audit logs

have been altered.

NOTE – This type of alteration would be from outside the EHR (e.g. hacking, manual tampering,

other software besides the EHR).

[IN170.314(d)(2)-5.01]

There is one centralized method (stored procedure) that records all primary audit records via record

insert only. Any data deletions would show missing primary key records, any data updates to audit

tracking would update modified date and modified by data on the audit records.


Scott Garrison


March 8th, 2014

director of software development marietta, ga 30067-6407 · 5. formal software functionality and...

Documents