bestnotes, llc bestnotes v1 - drummond group llc€¦ · the cloud-based application itself was...

BestNotes, LLC – 208.543.6646 – 1341 Fiillmore St., Suite 200, Twin Falls, ID 83301 – www.bestnotes.com

BestNotes, LLCBestNotes V1

Jayte BoehlerCTO1341 Fiillmore St., Suite 200Twin Falls, ID83301208 543-6646

For public release:BestNotes, LLC attests that the usability standard/process and usability report submitted for thecertification of [HIT System Product Name] is accurate and complete per the requirements of the ONCcriterion 170.314(g)(3).

Jayte Boehler, CTO

June 24, 2016

BestNotes, LLC – 208.543.6646 – 1341 Fiillmore St., Suite 200, Twin Falls, ID 83301 – www.bestnotes.com

BestNotes, LLCBestNotes V1

Jayte BoehlerCTO1341 Fiillmore St., Suite 200Twin Falls, ID83301208 543-6646

For public release:BestNotes, LLC used the following usability design industry standard in developing and designing theirHIT system, BestNotes V1: NISTIR 7741.

The NISTIR 7741 standard is a user centered design process which aims to ensure that EHRs areefficient, effective, and satisfying to the user. The NISTIR 7741 can be found athttp://www.nist.gov/itl/hit/upload/Guide_Final_Publication_Version.pdf

Jayte Boehler, CTO

June 24, 2016

BestNotes, LLC – 208.543.6646 – 1341 Fillmore St., Suite 200, Twin Falls, ID 83301 – www.bestnotes.com P a g e 1 | 16

EHR Usability Test Report of Bestnotes Version 1

Test Methodology based on NISTIR 7741 (publication November, 2010) Process Approach for Improving the Usability of EHRs

Test Report based on NISTIR 7742 Customized Common Industry Format Template for EHR Usability Testing

BestNotes Version 1

Date of Usability Test: 06/21/2016

Location of Usability Test: Business Facility of volunteer clinicians

BestNotes office listed below for non-clinicians

Date of Report: 06/22/2016

Report Prepared By: BestNotes, LLC Matthew Thompson, Product Owner 208.316.3213 [email protected] 1341 Fillmore Street Suite 200 Twin Falls, ID 83301

http://www.bestnotes.com/

mailto:[email protected]

EHR Usability Test BestNotes V 1


1. EXECUTIVE SUMMARY

A usability test of BestNotes Version 1 was performed on 06/21/2016 by BestNotes, LLC. The areas

tested were in accordance with Meaningful Use Stage 2 §170.314(g)(3) Safety-enhanced design for the

ambulatory setting.

BestNotes integrates with a third-party service, Dr. First, to accomplish medication management and e-

prescribing. When managing medication and e-prescribing, BestNotes users are taken via single sign on

to the Dr. First CPOE. All medications and allergies are brought back into BestNotes for review with the

rest of the patient record. Reconciled medications are pushed from BestNotes to Dr. First before being

incorporated into the BestNotes view of medications. Usability testing of functionality related to

medications except for reconciliation of external medications was performed and reported on by Dr.

First. The usability report generated by Dr. First will be provided as a separate document.

All testing was done in-person after a brief group training. None of the test participants had previous

experience with the portions of BestNotes being tested. The purpose was to determine and provide

evidence of the usability of the user interface of the EHR Under Test (EHRUT). Three clinical therapists

and four non-clinical users were tested.

Prior to testing, an optimal path was determined and a baseline time recorded for each test. The

participants were given a series of pre-defined tasks to complete with general instructions and were not

assisted during the test process. Participants used a test account connected to a test database via local

browser sessions connected to the cloud-based EHR. Test sessions were monitored, recorded and

scored by the testers using a separate system connected via Zoom’s virtual meeting service (zoom.us).

For each test, the following was collected:

Number of deviations from the optimal path

Number of errors encountered

Time of test completion

User feedback

Findings showed that participants rarely deviated from the optimal path and very few errors were

recorded. However, time of completion was higher than expected in some areas. Subjective findings

showed overall satisfaction, but combined objective and subjective results revealed several areas where

improvements could be made.




2. INTRODUCTION

The EHRUT being tested is BestNotes Version 1. BestNotes is a cloud-based EMR/CRM designed

specifically with mental and behavioral health in mind. BestNotes provides a wide gamut of

functionality including patient intake and assessment, scheduling and patient management, clinical

treatment planning and documentation, medication management, labs and reporting.

The purpose of the test is to assess the usability of specific portions of the program related to patient

safety and in accordance with Meaningful Use Stage 2 §170.314(g)(3) Safety-enhanced design for the

ambulatory setting. Data collected from the testing process was both subjective and objective in nature.

3. METHOD

3.1 PARTICIPANTS BestNotes tested six participants in two categories -- three clinical therapists and four non-clinical. The

clinical therapist participants matched the target demographic and were current users of BestNotes but

had no prior exposure to the areas of BestNotes being tested. Similarly, the non-clinical participants

had no prior exposure to the areas of BestNotes being tested.

Participants received a test overview, consent/release form and a non-disclosure form prior to testing.

Samples of these are included as appendices to this report. Participants then received a group training

session on the functionality to be tested similar to testing received by actual system users.

3.2 STUDY DESIGN The User Centered Design process is based on the guidelines detailed in the NISTIR 7741 published by

NIST and dated November 2010. The study design and testing process was based on the template

detailed in the NISTIR 7742 published by NIST.

The objective was to both confirm the proper design and ease-of-use of the areas being tested and to

reveal any errors in functionality and inefficiencies in workflow and design.

For each test, the following was collected:

Number of deviations from the optimal path

Number of errors encountered

Time of test completion

Subjective feedback both verbally and via ease-of-use survey post-test

Feedback will be used to improve areas that were found to be difficult or produced errors, and re-

testing will occur to measure improvements against the current result set.




3.3 TASKS The tasks that were performed by the participants include the following sub-set of features required by

Meaningful Use Stage 2:

§170.314(a)(1) Computerized provider order entry – Laboratory

a) Record Laboratory Order

b) Access Laboratory Order

c) Change Laboratory Order

§170.314(a)(1) Computerized provider order entry – Radiology/Imaging

a) Record Radiology/Imaging Order

b) Access Radiology/Imaging Order

c) Change Radiology/Imaging Order

§170.314(b)(4)/(9) Clinical information reconciliation

a) Reconcile patient’s active medication list with another source

b) Reconcile patient’s active medication list with another source

c) Reconcile patient’s active medication allergy list with another source

§170.314(a)(8) Clinical decision support

a) Problem List Interventions

b) Medication List Interventions

c) Medication Allergy List Interventions

d) Demographics Interventions

e) Lab Tests and Results Interventions

f) Vital Signs Interventions

g) Identify User Diagnostic and Therapeutic Reference Information

h) Configuration of CDS interventions by user (may be an admin type function)

The following tasks were tested and reported on by BestNotes software partner Dr. First:

§170.314(a)(1) Computerized provider order entry

a) Record Medication Order

b) Access Medication Order

c) Change Medication Order

§170.314(a)(2) Drug-drug, drug-allergy interaction checks

a) Create drug-drug and drug-allergy interventions prior to CPOE completion

b) Adjustment of severity level of drug-drug interventions (may be an admin type function)

§170.314(a)(6) Medication list

a) Record Medication List

b) Change Medication List

c) Access Medication List

§170.314(a)(7) Medication allergy list

a) Record Medication Allergy List

b) Change Medication Allergy List

c) Access Medication Allergy List

§170.314(b)(3) Electronic prescribing




A report of the Dr. First test results will be provided as a separate document.

The participant test instructions related to the above tasks are included as APPENDIX 5 in this report.

3.4 PROCEDURE Participants were given a test overview in a group setting where they were asked to fill out the

participant paperwork which included several demographic questions on the overview, a non-disclosure

agreement, and a consent form. The forms are included as APPENDICES 2-5 in this report.

Participants then were separated and paired with a test administrator and received participant test

instructions (APPENDIX 5) with the tasks to be completed. They were notified that they were expected

to complete the tests without help and as efficiently as possible.

The test administrators were connected via remote viewing software to their participant’s computer

and all test sessions were recorded for future review if necessary. Administrators recorded the time of

completion for each task as well as the number of errors and deviations from the optimal path.

Subjective feedback was also solicited verbally after each task, and a general ease-of-use survey was

completed post-test. The ease-of-use survey asked participants to rank the difficulty level of each task

on a scale of 1 to 5, 1 being ‘Easy to Perform’ and 5 being ‘Difficult’.

3.5 TEST LOCATION Testing was performed in the participant’s individual offices during scheduled times to avoid

interruption. Participants used their personal computers while the paired administrator used a separate

laptop to view, record and score the testing. Only the participant and administrator were in the office

during testing.

3.6 TEST ENVIRONMENT The tests were performed at the participant’s facility using their personal computers running Microsoft

Windows with monitor resolutions greater than 1024x768. The participant browser sessions were

initiated by the administrator and participants were logged into test accounts connected to test

databases. All test databases were exact duplicates. Internet connectivity was determined to be

satisfactory and representative of normal user sessions.

The cloud-based application itself was running on a Linux server which employs Apache and MySQL

services among others.

3.7 TEST FORMS AND TOOLS The following forms and documents were used as part of the testing process (samples will be included

as appendices of this report):

1. Orientation Form (APPENDIX 2)

2. Informed Consent Form (APPENDIX 3)

3. Non-disclosure Agreement (APPENDIX 4)

4. Sample Participant Test Instructions (APPENDIX 5)

5. Post-test Feedback




Administrator systems were connected to participants using Zoom’s (zoom.us) remote connectivity

solution. Sessions were observed and recorded using the administrator’s connected system.

3.8 PARTICIPANT INSTRUCTIONS Participants instructions were given as part of an initial orientation. The orientation was given verbally

as well as in written form and included general information and instruction as well as a few preliminary

questions. A copy of the orientation form is included as APPENDIX 2 in this report.

3.9 USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records, EHRs should support a process that provides a high level of usability for all users. The goal is for

users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To

this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability

testing. The goals of the test were to assess:

• Effectiveness of EHRUT by measuring participant success rates and errors

• Efficiency of EHRUT by measuring the average task time and path deviations

• Satisfaction with EHRUT by measuring ease of use ratings

3.10 DATA SCORING

Measures Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome without assistance.

The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as an “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors.




4. RESULTS

4.1 DATA REPORTING Below is a table of all results for each task performed. Column definitions are as follows:

Task: Task being performed Risk: Impact on quality of care with possible values of Low, Medium, and High Success: Average: Number of successes divided by the number of test attempts Success: SD: Standard deviation from the average number of successes Errors: Average: Average number of errors made per test Errors: SD: Standard deviation from the average errors made per test Time: Optimal: Pre-determined optimal time for test completion Time: Average: Average time of successful test completion Time: SD: Standard deviation from the average time of successful test completion Steps: Optimal: Pre-determined optimal number of steps to complete the test Steps: Average: Average number of steps to complete a successful test Steps: SD: Standard deviation from the average number of steps to complete a successful test Difficulty: Subjective test measure on a scale from 1 to 5, 1 being ‘Easy to Perform’ and 5 being ‘Difficult’ Difficulty: Average: Average of the ease-of-use rating provided by responding participants Difficulty: SD: Standard deviation from the average ease-of-use rating

Efficiency:

Task Deviations

The participant’s path (i.e., number of steps taken) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect navigation link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path.

Efficiency:

Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation) were also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering a simple post-test questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Easy) to 5 (Very Difficult). These data are averaged across participants. Common convention is that average ratings for systems judged easy to use should be 3.3 or below.




Success Errors Time Steps Difficulty

Task Risk Average SD Average SD Optimal Average SD Optimal Average SD Average SD

Record Laboratory Order

Low 0.93 0.19 0.14 0.38 73.049 s 75.2 31.18 9 9.7 1.5 1.6 0.9

Access Laboratory Order

Medium 1.00 0.00 0.29 0.49 13.339 s 53.0 63.94 4 4.6 1.1 1.2 0.4

Change Laboratory Order

Medium 1.00 0.00 0.14 0.38 103.074 s 105.3 54.42 15 15.9 1.9 2.0 1.4

Record Radiology/imaging Order

Low 1.00 0.00 0.29 0.49 36.53 s 100.7 32.28 13 13.3 0.5 2.0 0.7

Access Radiology/imaging Order

Medium 1.00 0.00 0.00 0.00 9.114 s 10.6 3.04 3 3.0 0.0 1.0 0.0

Change Radiology/imaging Order

Medium 1.00 0.00 0.00 0.00 22.149 s 18.1 6.13 4 4.0 0.0 1.0 0.0

Configuration of CDS interventions by user

Low 0.86 0.38 0.14 0.38 100.702 s 155.8 69.80 16 16.1 0.4 2.2 1.1

Identify User Diagnostic and Therapeutic Reference Information

High 1.00 0.00 0.00 0.00 12.125 s 15.1 4.56 3 3.0 0.0 1.2 0.4

Problem List Interventions

High 1.00 0.00 0.00 0.00 15.306 s 18.0 12.73 4 4.0 0.0 1.0 0.0

Medication List Interventions

High 1.00 0.00 0.00 0.00 15.306 s 12.2 5.54 4 4.0 0.0 1.0 0.0

Medication Allergy List Interventions

High 1.00 0.00 0.00 0.00 15.306 s 12.7 5.70 4 4.0 0.0 1.0 0.0

Demographics Interventions

High 1.00 0.00 0.00 0.00 15.306 s 13.2 4.36 4 4.0 0.0 1.0 0.0

Lab Tests and Results Interventions

High 1.00 0.00 0.00 0.00 15.306 s 14.0 6.16 4 4.0 0.0 1.0 0.0

Vital Signs Interventions

High 1.00 0.00 0.00 0.00 15.306 s 12.1 3.97 4 4.0 0.0 1.0 0.0

Reconcile patient’s active medication list with another source

Medium 1.00 0.00 0.00 0.00 27.297 s 37.3 45.53 6 6.0 0.0 1.4 0.5

Reconcile patient’s active problem list with another source

Medium 1.00 0.00 0.00 0.00 27.297 s 24.0 14.42 6 6.0 0.0 1.4 0.5

Reconcile patient’s active medication allergy list with another source

Medium 1.00 0.00 0.00 0.00 29.327 s 24.4 6.20 7 7.1 0.4 1.4 0.5




4.2 DISCUSSION OF THE FINDINGS EFFECTIVENESS AND EFFICIENCY

Finding showed that participants rarely deviated from the optimal path or produced errors.

Creation/configuration of a clinical decision support rule had the highest number of errors due to

scrolling issues. Creation and changing of a lab order had the highest number of deviations partially due

to the disconnect between the generation of a lab order and managing existing orders. All other tasks

had no significant errors or deviations. High average time of completion on several task will merit

further review of those areas.

SATISFACTION

All tests averaged 2 or less on a scale of 1 to 5, 1 being ‘Easy to Perform’ and 5 being ‘Difficult’. Overall

satisfaction was therefore relatively high, but user comments and high average time of completion

related to labs and clinical decision configuration reveal that optimizations will need to be made in these

areas. Subjective comments for each test are included in APPENDIX 6.

AREAS OF IMPROVEMENT

Based on the above scoring and analysis, configuration of clinical decision support needs to be

optimized to remove the need to scroll, and the highlighting of required fields would be useful in this

area. Additionally, the generation of lab orders needs to be placed within the context of lab order

management to that the two mechanism aren’t disconnected. Other minor improvements and

suggestions will be considered and possibly implemented prior to re-testing to demonstrate

improvement.




5. APPENDICES

5.1 APPENDIX 1: PARTICIPANT DEMOGRAPHICS

Position

Clinical 3

Non-clinical 4

If clinical, how long have you been working in this role (years)?

4

12

1

Occupation / Role

Administrative 2

Technical 2

PsyD 1

LMSW 2

Tell me about your experience with electronic health records.

All paper 0

Some paper some electronic 1

All electronic 3

no experience 3

Gender

Male 7

Female 0




5.2 APPENDIX 2: ORIENTATION FORM

Orientation - BestNotes MU UI Testing

Thank you for participating in this study. Our testing session today will last approximately 60 minutes.

During that time, you will take a look at an electronic health record system.

I will ask you to complete a few tasks using this system and answer some questions. We are interested

in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could

improve it. You will be asked to complete these tasks on your own trying to do them as quickly as

possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get

lost or have difficulty I cannot answer or help you with anything to do with the system itself. Please save

your detailed comments until the end of a test procedure or the end of the session as a whole when we

can discuss freely.

The functionality you will be testing today is currently in beta. Some of the data may not make sense as

it is placeholder data.

Preliminary Questions

Position

[ ] clinical therapist

[ ] non-clinical

If clinical, how long have you been working in this role (years)? ______

Tell me about your experience with electronic health records.

[ ] all paper

[ ] some paper some electronic

[ ] all electronic

[ ] no experience




5.3 APPENDIX 3: INFORMED CONSENT FORM

Informed Consent - BestNotes MU UI Testing

BestNotes, LLC would like to thank you for participating in this study. The purpose of this study is to

evaluate an electronic health records system. If you decide to participate, you will be asked to perform

several tasks using the prototype and give your feedback. The study will last about approximately 60

minutes. At the conclusion of the test, you will be compensated for your time.

Agreement

I understand and agree that as a voluntary participant in the present study conducted by BestNotes, LLC

I am free to withdraw consent or discontinue participation at any time.

I understand and agree that the purpose of this study is to make software applications more useful and

usable in the future.

I understand and agree that the data collected from this study may be shared outside of BestNotes, LLC

and BestNotes, LLC’s clients. I understand and agree that data confidentiality is assured, because only

de- identified data – i.e., identification numbers not names – will be used in analysis and reporting of the

results.

I agree to immediately raise any concerns or areas of discomfort with the study administrator. I

understand that I can leave at any time.

Please check one of the following:

[ ] YES, I have read the above statement and agree to be a participant.

[ ] NO, I choose not to participate in this study.

Participant’s printed name: ______________________________________________

Signature: ________________________________________________ Date: _____________

Email Address: _____________________________________________




5.4 APPENDIX 4: NON-DISCLOSURE AGREEMENT

Non-Disclosure Agreement - BestNotes MU UI Testing

THIS AGREEMENT is entered into as of June 20, 2016 between

_________________________________ (“the Participant”) and the testing organization BestNotes, LLC.

The Participant acknowledges his or her voluntary participation in today’s usability study may bring the

Participant into possession of Confidential Information. The term "Confidential Information" means all

technical and commercial information of a proprietary or confidential nature which is disclosed by

BestNotes, LLC, or otherwise acquired by the Participant, in the course of today’s study.

By way of illustration, but not limitation, Confidential Information includes trade secrets, processes,

formulae, data, know-how, products, designs, drawings, computer aided design files and other

computer files, computer software, ideas, improvements, inventions, training methods and materials,

marketing techniques, plans, strategies, budgets, financial information, or forecasts.

Any information the Participant acquires relating to this product during this study is confidential and

proprietary to BestNotes, LLC and is being disclosed solely for the purposes of the Participant’s

participation in today’s usability study. By signing this form the Participant acknowledges that s/he will

receive monetary compensation for feedback and will not disclose this confidential information

obtained today to anyone else or any other organizations.

Participant’s printed name: _____________________________________

Signature: ________________________________________________ Date: ___________




5.5 APPENDIX 5: SAMPLE PARTICIPANT TEST INSTRUCTIONS

Sample 1: Test Procedure for Measure a1: Record Lab Order

1. Access patient ‘Pamela Dawson’

2. Prepare a lab order with Nathan Olsen as the ordering provider

From labs CPOE:

3. Add diagnosis ‘w55.11’ (bitten by a horse)

4. Add test ‘lipid 1996 panel’ (Code: ‘24331-1’)

5. Submit the order

Sample 2: Test Procedure for Measure a1: Access a Laboratory Order

1. Access patient ‘Isabella Jones’

2. Navigate to ‘Labs’ from the patient’s clinical area

3. View an order from the list or orders

Sample 3: Test Procedure for Measure a8: Identify User Diagnostic and Therapeutic Reference

Information

1. Access patient ‘Martha Mary’

2. Navigate to the patient’s clinical area

*A CDS alert should appear

3. Open the associated reference material

Sample 4: Test Procedure for Measure b4: Reconcile/Incorporate External Allergies

1. Access patient ‘Mary Williams’

2. Navigate to patient’s clinical area

3. Remove ‘Penicillin V’ from the patient’s external allergies and then reconcile/incorporate the

patient’s remaining external allergies




5.6 APPENDIX 6: POST-TEST USER FEEDBACK

Record Laboratory Order

• Don't want to have to specify diagnosis on the lab CPOE, it should be pulled from the patient's chart.

Access Laboratory Order

• Add a more clear indicator of open and resulted orders. Change Laboratory Order

• Be able to also generate a new lab order for a patient within BestNotes Med/Clinical section in addition to the main patient page.

Record Radiology/imaging Order

• Auto-populate provider in the radiology CPOE form. • The top level clinical menu is not available in all of the main navigation points of

BestNotes. • Be able to generate CPOE order from main patient landing page.

Access Radiology/imaging Order

• None Change Radiology/imaging Order

• To be able to perform this task from patient's med/clinical area.

Configuration of CDS interventions by user • Highlight fields in the form which are required. • Don't want to have to scroll through the configuration form.

Identify User Diagnostic and Therapeutic Reference Information

• None Problem List Interventions

• From within med/clinical be able to quickly redisplay active alerts instead of having to look at the main CDS dashboard.

Medication List Interventions • None

Medication Allergy List Interventions

• None

Demographics Interventions • None




Lab Tests and Results Interventions

• None Vital Signs Interventions

• Make the radio labels selectable. Reconcile patient’s active medication list with another source

• None Reconcile patient’s active problem list with another source

• None Reconcile patient’s active medication allergy list with another source

• None


DrFirst, Inc. - [email protected] - 866-263-6511 (Office) - 301-610-7543 (fax) 9420 Key West Avenue, Suite 230 - Rockville, MD 20850

Follow DrFirst on our Blog: http://blog.drfirst.com/ | Check out our videos at http://www.youtube.com/drfirst

EHR Usability Test Report

Product: Rcopia

Version: 3

Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

Date of Usability Test: June 2013

Date of Report: June 2013 (Updated January 2016)

Report Prepared By: DrFirst, Inc Jason Aquilante, Product Manager 301-231-9510 [email protected] 9420 Key West Ave Suite 230 Rockville, MD 20850

Updated By: DrFirst, Inc. Pravin Conda, Director of Strategic Programs 301-231-9510 [email protected] 9420 Key West Ave Suite 230 Rockville, MD 20850

Note: The following study was developed using the NISTIR 7742 template as a guide for reporting our findings: Customized Common Industry Format Template for Electronic Health Record Usability Testing.


http://blog.drfirst.com/





Executive Summary A usability test of Rcopia, version 3, a modular EHR was conducted throughout the month of June 2013 by DrFirst, Inc. The purpose of this test was to test and validate the usability of the current user interface, and to provide evidence of usability.

During the usability test, 10 healthcare IT users served as participants and used the EHR in simulated, but representative tasks.

This study collected performance data on 11 tasks typically conducted on an EHR. The tasks conducted were related to the following:

x Computerized Provider Order Entry x Drug-Drug/Drug-Allergy Alerts x Updating a Patient’s Medication List x Updating a Patient’s Allergy List x E-Prescribing x Clinical Information Reconciliation

During the 45 minute one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix). Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHR. During the testing, the administrator timed the test and, along with the data logger, recorded user performance data on paper. The administrator did not give the participant assistance in how to complete the task.

The following types of data were collected for each participant:

x Time to complete each task x Number and types of errors x Path deviations





x Participants verbalizations x Tasks completed in the allotted time.

All participant data was de-identified – no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. The following is a summary of the performance and rating data collected on the EHR.

Errors Deviations

Task Participants Task Success

Time to Complete (Avg)

Total Average (Total Observed/Completed Task)

Total (Observed/Optimal)

Average (Observed/Optimal)

1. Create Rx from Fav. List 10 10 4.1 0 0 30/30 1

2. Create Rx from Drug Search 10 7 28.7 3 30% 50/50 1

3. Renew 2 Active Meds 10 10 3.8 0 0 11/10 1.1

4. Set preference to show all warnings

10 3 25.2 7 70% 33/30 1.1

5. Prescribe medication that displays Alert

10 8 16.6 2 20% 60/60 1

6. Add Medication through drug search.

10 7 34.9 3 30% 40/40 1

7. Stop 1 medication 10 10 11.7 0 0 21/20 1.05

8. Add common allergy 10 9 21.7 1 10% 30/30 1

9. Prescribe controlled Rx to EPCS pharmacy

10 5 38.8 5 50% 106/90 1.17

10. Prescribe medication through Rx Report

10 7 12.3 3 30% 20/20 1

11. Merge two patient records 10 3 108 7 70% 94/80 1.175


The




In addition to the performance data, the following qualitative observations were made:

1. Major Findings a. Confusion on how to find the change settings within DrFirst setting feature.

2. Areas for Improvement a. Improvement of user workflows through a standard electronic presribing process.

Introduction The EHR tested for this study was Rcopia, version 3, a modular EHR. Designed to present medical information to healthcare providers in ambulatory healthcare settings, the EHR allows providers to electronically prescribe medications to the pharmacy. The system has evolved to allow practice users to maintain a list of a patient’s medications, allergies and problems. The usability testing attempted to represent realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR. To this end, measure of effectiveness and efficiency (time to perform tasks; total number of deviations; total number of errors; etc) were captured during the usability testing.

Method

Participants A total of 10 participants were tested on the EHR. Participants in the test were individuals that work within an ambulatory healthcare environment. Participants were contacted by DrFirst, Inc staff to participate in the study. In addition, participants had no direct connection to the development of the EHR. Participants were not from DrFirst, Inc. All participants had the same level of training as all other actual end users.

The following is a table of participants by characteristics, including demographics, user role, and product experience. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities. A summary of the participant demographics can be found in the Appendix.

Participant ID Gender User Role Product Experience (Years) 1 20131 M Provider Agent .5 2 20132 F Provider Agent 3 3 20133 F Provider Agent 3 4 20134 F Provider Agent 3 5 20135 F Provider Agent 1 6 20136 M Provider Agent 2





7 20137 F Provider Agent 2 8 20138 F Provider Agent 1.5 9 20139 F Clinical Staff 1.5 10 201210 M Provider Agent 1.5

100% of all participants recruited for the test showed up to participate in the test.

Participants were advised that the test would take 45 minutes; but to allocate 60 minutes for the test. The added 15 minutes was to provide enough time for administrator instructions and time between tasks. A spreadsheet was used to track participant schedules, and included each patient’s demographic characteristics.

Study Design Overall, the objective of this test was to uncover areas where the application performed well – that is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made.

During the usability test, participants interacted with 1 EHR. Each participant was provided the same set of instructions. The system was evaluated for effectiveness and efficiency as defined by measures collected and analyzed for each participant.

x Time to complete each task x Number and types of errors x Path deviations x Participants verbalizations x Tasks completed in the allotted time.

A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including:

x Create Rx from Fav. List x Create Rx from Drug Search x Renew 2 Active Meds x Set preference to show all warnings x Prescribe medication that displays Alert x Add Medication through drug search. x Stop 1 medication





x Add common allergy x Prescribe controlled Rx to EPCS pharmacy x Prescribe medication through Rx Report x Merge two patient records

Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users.

Procedures Upon arrival, participants were greeted and their identity was verified and matched with the participant’s name on the schedule. Participants were then assigned a participant ID. All participants signed an informed consent form prior to the testing. The test administrator witnessed each participant’s signing of the consent form.

To ensure that the test ran smoothly, two DrFirst staff members participated in the administration of the test. The two staff members had at least 2 years’ experience with product and have done testing with prior clients before. The test administrator provided the instructions for each test, and noted all comments from the participants; while the data logger noted all times, deviations and errors.

Participants were instructed to perform the tasks:

x After listening to the instructions from the testing administrator x As quickly as possible x Without assistance

Task timing began after the completion of the verbal instructions from the administrator; and after an acknowledgement from the participant that they were ready to begin. The task time was stopped once the participant indicated they had successfully completed the task.

Following the test, the administrator gave the participant the post-test questionnaire; and then thanked them for their time.

The DrFirst staff member responsible for logging data recorded all participants’ demographic information, task success rate, time on task, errors, and deviations into a spreadsheet.

Test Location The test was administered in a setting where participants were isolated from other participants in the study. Only the test administrator and logger were with the participants while the study was being administered. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range.





Test Environment The computers used for the testing were PCs running on Windows XP or Windows 7. The users were able to connect using a provided Go To Meeting account. Users were tested using multiple resolution on their side but were proctored by a 1280 X 720 resolution screen size and a 32-bit color monitor. Users also used a mouse and keyboard while interacting with the EHR. The Rcopia application is a web-based solution; so all computers were running on high speed internet connections. All participants indicated that system performance during the test was what they were used to seeing during their typical work day.

Test Forms and Tools During the usability test, various instruments and documents were used, including:

x Informed consent x Moderator guide x Post-test questionnaire

Examples of these documents are to be found in the Appendix section.

Participant Instructions The Administrator read the following instructions aloud to each participant:

“Thank you for participating in today’s usability study of Rcopia. In a few minutes, you will be asked to perform a series of tasks and complete a user survey. Please attempt to complete each task as quickly as possible. The idea behind this study is for DrFirst to obtain information on where enhancements are needed in the application based on how quickly, and easily, tasks are being performed in Rcopia.

When it is time to perform each task, I will state the instructions and then tell you to begin. Once you have completed the task, please say ‘Done’. After you have completed the task, I will ask for feedback on the actions you had taken during the task. You will be given a specified amount of time to complete each task. This time will not be communicated to you as we are interested in seeing how long each task does take for you to perform.”

Usability Metrics The goals of this test were to assess:

1. The efficiency of Rcopia by measuring the length of time it takes for a user to complete specific tasks; and the total number of tasks successfully completed during the study.

2. The efficiency of Rcopia by measuring the path deviations taken by the user during the tasks.





3. The effectiveness of Rcopia by measuring the number and types of errors experienced by the user during the tasks.

4. The satisfaction of the user with Rcopia by logging their comments on the tasks.

Data Scoring The table below details how each task was scored.

Measure Rationale and Scoring

Task Time Timing started when the administrator said ‘Begin’. The time ended when the participant said ‘Done’. In the event that the participant finished, and did not say ‘Done’, the administrator stopped the clock when it was clear the participant had completed the task. Task times were only counted if the participant completed the task in the allotted time. The average time per task was calculated for each task.

Errors The task resulted in an error if the participant: failed to finish the task in the allotted time; or, if they became ‘stuck’ and could not proceed without asking for assistance. Task time was not counted when the task resulted in an error.

We calculated the error % for each task by taking the total number of errors for each task and divided that number by the total attempts at the task.

Path Deviations Path deviations were recorded as actions taken during the task that were not part of the necessary actions needed to complete the task.

We calculated path deviations by taking the total number of observed deviations and dividing that number by the total number of steps taken using an optimal path.

Task Success A task was considered a success if the participant completed the task in the allotted time. To calculate the task success rate, we simply divided the total number of successful tasks by the total number of tasks completed.

The time designated for each task was determined by taking the optimal time to complete the task and multiplying it by a factor of 1.25 to allow for those users that may not have been fully trained on the application.





Results

Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the Usability Metrics section above.

The testing results for Rcopia are detailed below. The table below easily identifies the tasks performed and the performance level for each task.

Errors Deviations

Task Participants Task Success

Time to Complete (Avg)

Total Average (Total Observed/Completed Task)

Total (Observed/Optimal)

Average (Observed/Optimal)

1. Create Rx from Fav. List 10 10 4.1 0 0 30/30 1

2. Create Rx from Drug Search 10 7 28.7 3 30% 50/50 1

3. Renew 2 Active Meds 10 10 3.8 0 0 11/10 1.1

4. Set preference to show all warnings

10 3 25.2 7 70% 33/30 1.1

5. Prescribe medication that displays Alert

10 8 16.6 2 20% 60/60 1

6. Add Medication through drug

10 7 34.9 3 30% 40/40 1





search. 7. Stop 1

medication 10 10 11.7 0 0 21/20 1.05

8. Add common allergy 10 9 21.7 1 10% 30/30 1

9. Prescribe controlled Rx to EPCS pharmacy

10 5 38.8 5 50% 106/90 1.17

10. Prescribe medication through Rx Report

10 7 12.3 3 30% 20/20 1

11. Merge two patient records 10 3 108 7 70% 94/80 1.175

Effectiveness Participants in the study experienced errors in all but 3 tasks. However, the bulk of the errors occurred in three areas: Practice Preference Page; Pharmacy search page; and the Merge Application. Errors found in these areas were of no surprise to DrFirst. The Merge application is a newer component to DrFirst users. The practice preference page and the pharmacy search page are two areas that are focal points for future enhancements. Each of these pages present usability issues for end users; and requirements for enhancements are being drafted.

Efficiency Participants in the study, for the most part, followed the optimal paths to complete the assigned tasks. However, there were two exceptions: Pharmacy Search and the Merge Application. As mentioned above; the Merge Application is a newer component to Rcopia and it was expected that users may not be fully aware of the most optimal paths to take when performing the tasks. An abnormal number of deviations were also observed when participants were asked to prescribe a medication to an EPCS-enabled pharmacy. Participants either elected to take extra steps to change the pharmacy; while others forgot to select a pharmacy labeled with a ‘C’ to indicate it is a pharmacy that accepts Controlled Substances.

Satisfaction 9 out of 10 users expressed they were “Very Satisfied” with Rcopia.

Major Findings The study showed no major findings. Overall, the participants verbalized their happiness with the system, but many did stress they would like to see some minor enhancements. Their use of the system has not decreased over time; and most feel the functionality allows them to execute on their job functions without causing additional work.





Areas for Improvement The study did verify two assumptions: enhancements are needed to optimize usability; and there needs to be better means of communication on newer features that may help improve patient safety and improve clinical workflows.

As mentioned previously, we are planning to modify our practice preference (and user preference) page; as well as our pharmacy search area. We want to be sure our users can quickly (but efficiently) navigate through these areas so not to impede on their prescription writing process.

Additionally, we need to find a better means of communication with our users to notify them of new features. Our post-study questionnaire revealed that 8 out of 10 participants were not aware of a particular feature within Rcopia. Improving communication with users will be discussed internally at DrFirst so we can make sure our users are aware of features that will benefit them clinically and operationally.

Appendices The following appendices include supporting data for this usability study.

1. Sample Consent Form

Rcopia Usability Study

Practice Consent DrFirst would like to thank you for your participation in this study. The purpose of the study is to evaluate the usability of the Rcopia electronic prescribing system. Your participation in this study will include performing specific tasks within Rcopia; and completing a short survey following the study. The study should take approximately 30 minutes. The information collected by DrFirst during the study is for research purposes only. Your participation in this study is voluntary, so you are free to withdraw at any point during the study.

By signing below, I agree to participate in the study.

Name of Practice

Name of Participant

Rcopia User Role





Experience w/Rcopia (Yrs)

Date of Study

Location

Signature Date Printed Name Date

2. Sample Post-Test Questionnaire

Rcopia User Survey

1. How would you prefer we communicate new enhancements to your practice? � Email � Members Area Page � Newsletter � We do not care to be notified

2. What is your preferred method of working with DrFirst Support department? � Phone � Email � Online Chat (if this were available)

3. What is your preferred method for creating a prescription? � Drug Search � Favorites

4. How long does it take to enter a new patient in Rcopia (Demographic data only)? � 0-3 minutes � 3-5 minutes � >5 Minutes

5. Which of the following Rcopia tasks would you use on a mobile device? � Viewing a patient’s record � eRx � Updating a Patient’s Med List, Allergy List, Problem List � I would not use Rcopia on a Mobile device





6. If you could change one part of Rcopia, what would you change?

7. If you could add one piece of functionality to Rcopia, what would you add?

8. On a scale of 1-5, how would rate your overall satisfaction with Rcopia? � 1 (Very Dissatisfied) � 2 (Somewhat Dissatisfied) � 3 � 4 (Somewhat Satisfied) � 5 (Very Satisfied)

9. Which of the following functions are available through DrFirst/Rcopia? (You may check more than 1 option) � Patient Education Materials � Electronically Prescribe Controlled Substances � Prescription Coupons for distributing to Patients � Secure email messaging

3. Moderator Guide

Rcopia Usability Study

Computerized Provider Order Entry Tasks to perform during study:

Preparation Task Goal Optimal Paths 1. No Pending Rx’s 2. No Active Medications 3. No Allergies

Create Rx from Favorite List 6 seconds 3

1. No Pending Rx’s 2. No Active Medications 3. No Allergies

Create Rx from Drug Search 34 seconds 5

1. Create 2 Active Medications Renew 2 active medications 8 seconds 1

Clinical Alerts Tasks to perform during study

Preparation Task Goal Optimal Paths

NONE Set preference to have all warnings be shown for Drug-Allergy interactions

27 seconds 3





1. Create Allergy for patient

Prescribe medication that is known to be an allergy for patient.

20 seconds 6

Updating Medication List Tasks to perform during study


Add Medication through drug search 39 seconds 4


Stop 1 medication 16 seconds 2

Updating Allergy List Tasks to perform during study


Add common allergy 31 seconds 3

eRx Tasks to perform during study

Preparation Task Goal Optimal Paths


Use print w/o sending to prescribe controlled substance to EPCS enabled pharmacy.

53 seconds 9

1. Make sure at least one pending Rx for current patient

Prescribe medication through Rx Report 22 seconds 2

Clinical Information Reconciliation Tasks to perform during study

Preparation Task Goal Optimal Paths 1. Make sure a patient Merge patient record from 120 seconds 8





record has been sent through Akario to provider

two different sources.

4. Designated Task Times

Task Time Designated (seconds) 1. Create Rx from Fav. List 6 2. Create Rx from Drug Search 34 3. Renew 2 Active Meds 8 4. Set preference to show all

warnings 27

5. Prescribe medication that displays Alert 20

6. Add Medication through drug search. 39

7. Stop 1 medication 16 8. Add common allergy 31 9. Prescribe controlled Rx to

EPCS pharmacy 53

10. Prescribe medication through Rx Report 22

11. Merge two patient records 120





5. Participants Demographics

Gender Male 3

Female 7

Rcopia User Role Provider 0

Provider Agent 9

Clinical Staff 1

Non-Clinical Staff 0

Front Desk 0

Years of Experience w/Rcopia <1 1

<2 4





>2 5



bestnotes, llc bestnotes v1 - drummond group llc€¦ · the cloud-based application itself was...

Documents