Selecting a Mobile App: Evaluating the Usability of Medical Applications

mHIMSS App Usability Work Group

July 2012

v1.0

Table of Contents

Purpose of this Document ...................................................................................................... 1

What is an “App”? ................................................................................................................. 1

Mobile App Design ................................................................................................................ 2

What is Usability? ................................................................................................................. 2

Practice Guide to Evaluating App Usability ............................................................................ 7

1. Consider Practice Goals and Measurable Objectives ................................................................7

2. Check Other Resources to Supplement App Market Reviews ...................................................8

3. Assess the App’s Usability with Typical Clinical Scenarios ........................................................9

Learn More about Usability ................................................................................................. 12

Appendix A. Mobile Design Tenets ....................................................................................... 14

Appendix B. Sample Usability Task Scenarios ....................................................................... 16

Scenario 1: Pediatric Medication Dose Calculation ........................................................................ 16

Scenario 2: Review of Patient Test Results .................................................................................... 19

Scenario 3: Diabetic Patient Education.......................................................................................... 21

Appendix C. Sample Post-Test Questionnaires ...................................................................... 25

1. System Usability Scale (SUS) ..................................................................................................... 26

2. Additional Sample Usability Post-Test Questions....................................................................... 28

Purpose of this Document

This document has been prepared as an aid to the healthcare provider or IT staff member who is engaged in the selection of mobile apps for his/her practice or hospital organization. While formal, professionally conducted, usability evaluations can provide in-depth product comparisons they may not be a practical consideration for small- to medium-sized practices. This guide outlines some basic steps, to include in a mobile app selection process, to assist practices evaluate mobile app usability, based on current usability recommendations and best practices. While the target audience of this guide is for the provider and smaller practices, the steps outlined in this document can also be used by large practices and healthcare institutions.

What is an “App”?

There are two general types of mobile apps, native and non-native. Native apps reside on a mobile device. Non-native apps utilize browser interfaces or other gateways to provide the end user with information through a mobile device. A native mobile app requires download through a

market place associated with the operating system (OS), such as iTunes for Apple OS devices, Android’s marketplace for Android OS-powered phones, and Research in Motion’s App Store for Blackberry devices. Non-native apps simply require that the mobile device have the appropriate web browser or other interface enabled.

Medical apps fall under many different categories, including but not limited to reference apps, like the Physician’s Desk Reference or WebMD, medical calculators, and apps designed to access electronic health records (EHR) or personal health information (PHI). Medical reference apps constitute 50 percent of the apps currently present in various platform marketplaces1. However, significant growth is anticipated in mobile apps which seek to enhance or augment EHR interaction and provide additional operational functionality surrounding clinical work flow and decision support. With this expansion of mobile apps and the blurring of lines between some apps and medical devices, the Food and Drug Administration (FDA) issued draft guidance in 2011 to manufacturers, distributors and other entities outlining the types of mobile apps the FDA intends to apply its regulatory authorities to in the future2.

1 FirstWordSM, Dossier. (May, 2010).Trends in Mobile Medicine: Smartphone Apps for Physicians

[Market Intelligence Report].

2 U.S. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff - Mobile

Medical Applications. July 21, 2011 [Online]. (URL:

http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm263280.htm )...

This guide focuses on the usability of native apps, presenting varied considerations and resources related to selection. Selection may ultimately be based on a number of factors/resources to include price, user reviews, usability and security. App ratings are currently available through different sources. Usability can be an additional discriminator to supplement user reviews available within the app marketplace.

Mobile App Design

A well-designed medical mobile app will result in a mobile user experience for the provider that is both useful and usable. The design should address the following key areas: data, layout, feedback, and interaction. In a nutshell:

The data is the focus. The interface should be simple and intuitive, with design elements not interfering with the data on a potentially small screen.

The layout incorporates typographic elements that communicate meaning and are consistent across screens.

Feedback can be provided directly to the design team.

Interactions with the interface engage directly with the content and task at hand, keeping cognitive burden to a minimum.

For more details, a set of mobile design tenets developed by mHIMSS is described in Appendix A. Mobile Design Tenets.

What is Usability?

Usability is the effectiveness, efficiency and satisfaction with which specific users can achieve a specific set of tasks in a particular environment3. These usability components can be evaluated and measured (either formally or informally) by means of indirect measures.

Efficiency is generally the speed with which users can complete their tasks. Which tasks and clinic processes must be most efficient for success? Can you establish targets for acceptable times for completion of these tasks?

Effectiveness is the accuracy and completeness with which users can complete tasks. This includes how easy it is for users to make errors using an app. User errors can lead to inaccurate or incomplete patient records can alter clinical decision-making and compromise patient safety.

3 Schoeffel R. The concept of product usability. ISO Bulletin. 2003;34:6‐7.

Ease of learning is also an attribute of usability, although it is not easy to measure in a quick, informal assessment.

User satisfaction is usually the first concept people think of in relation to “usability.” Satisfaction in the context of usability refers to the subjective satisfaction a user may have with a process or outcome. Satisfaction is highly subjective, but routine questionnaires can provide good insight into what and why problems or issues users have with the system.

Platform Optimization is the app’s effective and efficient utilization of a device’s capabilities. As tablet use increases, a number of developers are optimizing apps for higher resolution, which should increase the efficiency and effectives of a tablet-optimized app. While apps developed for smartphones are often available for download and use on tablets, individuals or organizations seeking apps that are optimized for tablets should ensure the app is designed for such applications. In addition, the purchaser should consider device and platform constraints as the mobile industry regularly releases new operating systems and devices. Make sure you have the right app for the right device in the right setting.

Efficiency, effectiveness and satisfaction cannot be taken in isolation—all three components must be evaluated and balanced based on your practice's goals and priorities. Table 1 illustrates example questions of efficiency, effectiveness and user satisfaction.

Table 1: Example of Efficiency, Effectiveness and User Satisfaction Questions

Usability Component

Example Questions

Efficiency Is it reasonable to take two minutes to use the app to compute drug dosing for a medication order?

How many steps should it take to view vital sign measurements of patients at home through apps that remotely connect to bedside monitors?

How long should it take for you to view medical images on a mobile platform?

Effectiveness Is data entry too complex or confusing to perform completely and as intended?

Does the small screen size on your mobile device affect your ability to

accurately interpret radiological images?

Is information poorly presented or subject to misinterpretation?

User Satisfaction

Does the app perform the way you expected?

Usability Component

Example Questions

Would you use the information/data obtained from the app in your clinical decision-making process?

Usability can be compromised by the visual design of the user interface. In fact, applications that use lots of colors or visual elements probably would have a lower usability “score” than well thought out designs that are simpler in appearance and do not increase cognitive burden for the user. Thoughtful use of visual elements can be a supporting factor that enhances product usability.

Usability and functionality are often hard to tease apart. While it is not always necessary to fully distinguish the two, it is important to understand the difference.

The provider evaluation criteria for medical apps are based on both usability principles and mobile design tenets. Current evaluation methods developed to measure efficiency, effectiveness and user satisfaction of information technology are based on usability principles for all software. A subset of these principles can be applied directly to the evaluation of “EMR extension apps”4 or native apps. This set has been assessed in earlier literature for electronic medical record (EMR)5 and put forth as being those of key concern for medical apps. In addition, mobile design tenets—guidelines for developing health information technology on mobile devices—have been translated to provider mobile app evaluation criteria.

Table 2 outlines examples of how a medical app supports the usability principles and mobile design tenets.

4 The term EMR extension apps is used because certain functionality may differ between native apps or web-

enabled apps. Purchasers should fully vet the true functionality of apps that provides EMR/EHR access.

5 HIMSS Usability Task Force. Selecting an EHR for Your Practice: Evaluating Usability v1.2. [Online]. (URL:

http://www.himss.org/content/files/HIMSS%20Guide%20to%20Usability_Selecting%20an%20EMR.pdf )

Table 2 – Usability Principle Attribute Checklist

Usability Principle

Mobile Design Tenet

Example attributes of a “usable” app

Let data scream

Only information, visual elements and functionality necessary to core tasks and decisions are included. Important information stands out, and function options are easy to understand. The focus is the data. The app has a clear, clean, uncluttered screen design.

Speak my sign

What interface?

Screen metaphors are familiar to everyday life, or commonly expected computer experiences for the clinician. Workflows match the clinical practice needs. The app is intuitive and easy to learn; minimal, if any, training is required. Iconography and symbols speak “naturally.”

Grid it Graphic design and layout have the same look and feel, consistent placement of screen elements (e.g., gutters, columns, margins and captions). Terminology and data entry fields are used consistently. Understanding how one screen works helps you understand how other screens work.

Date your users It’s hard to lose data or destroy time-consuming effort with a wrong tap or wrong choice of buttons. If you make a mistake, the app helps you avoid it or provides a method to recover from errors gracefully (the system is “forgiving”). The app provides informative feedback about actions you are about to take or have taken. The app displays explanatory messages when processing information and describes how long it might take.

Speak my sign

Date your users

The app uses the same words that you use (while providing mapping to standardized codes and terms used for data retrieval). List or entry-form choices are clear and unambiguous. Sentences read like your native language (e.g. English).

What interface?

Get physical

The app minimizes the number of steps/gestures it takes to complete tasks; appropriate defaults are always provided. The app provides navigation options such as shortcuts for frequent and/or experienced users. Navigation methods minimize movements, such as

Simplicity

Naturalness

Consistency

Forgiveness

and Feedback

Effective Use

of Language

Usability Principle

Mobile Design Tenet

Example attributes of a “usable” app

scrolling and switching between typing and tapping. Although input methods vary from OS to OS and even within devices, the design ensures that a usable method is present whether the desired input is through typing or the use of gestures. Gestures include single tap, multi-tap, swipe (where the method of input never leaves the screen tracing the letters to input) and auto complete.

Type less + less type

Color carefully

Information on screens includes sufficient white-space and large enough fonts to be read easily with high comprehension. No information is in all upper case. Colors are used to convey meaning (e.g., red to indicate medical urgency), not just for visual appeal.

Efficient

Interactions

Usability Principle

Mobile Design Tenet

Example attributes of a “usable” app

What interface? The app keeps screen changes and visual interruptions to a minimum during completion of a particular task. Visual interruptions include anything (e.g., dialog boxes) that force you to shift visual focus away from the area on the screen where you are currently reading and/or working to address something else, and then re‐establish focus afterward. For example, dialog boxes should be kept to a minimum and should appear in context (adjacent to or just below the control that triggered it).

What interface? Information needed for a particular task or decision is grouped together on a single screen, rather than requiring you to mentally synthesize information from multiple screens. Alerts presented are clear, concise and informative. The app performs calculations automatically.

Practice Guide to Evaluating App Usability

Whether you have downloaded a free medical app, are comparing similar apps or are thinking about purchasing an upgraded version of an app, the following evaluation process steps will assist you in selecting and adopting the most “usable” app:

1. Consider Practice Goals and Measurable Objectives

Consider what goals are most important to your practice when adopting an app, even if just for your own use, and how these goals relate to usability. For example, are you especially concerned about:

● Improving the efficiency of prescription refill requests?

● Effectively using patient encounter data for patient education?

● User satisfaction with a clinical decision support tool?

● How much training will be required for clinicians to become adept with the app? (Ease

of learning.)

Think about simple baseline measures related to your goals. For example, efficiency is typically measured by how long it takes to perform sample tasks during a usability test. Satisfaction and

Preservation

of Context

Minimize

Cognitive

Overload

concerns can be assessed with questions included in a post-usability test questionnaire. Ease of learning can be assessed by having users attempt the same scenario/task with a number of “test tasks” and objectively measure learning based on the number of attempts and measures such as time to completion and success rate. Use your practice goals as a baseline measure for reviewing and/or comparing the apps of interest.

Depending on the specific nature of effectiveness concerns, targeted questions may be appended to the usability test scenarios (answered after the scenario has been completed) or included in the overall questionnaire to be filled out after the full test has been completed. Usability testing is discussed in a subsequent section of this document, with sample scenarios and usability questions provided in the appendices.

2. Check Other Resources to Supplement App Market Reviews

Keep in mind that many of the publicly available medical app reviews are gathered through informal, non-scientific means. Most reviews and ratings are based on user satisfaction data. Organizations are only just beginning to provide information regarding perceived efficiency and effectiveness of medical apps. Use these reviews to help formulate your questions, but do not allow it to override your own assessment based on your practice goals. Hopefully in the future, a framework for assessing the usability of medical apps will be available, similar to that developed by the National Institute of Standards and Technology (NIST) for assessing the usability of EHRs6.

A few resources for medical app reviews:

Resource Description iMedicalApps.com Physician curated and generated reviews with indexed search functions

by app type, physician specialty, and device platform.

iPhone Medical Apps: new, reviews, trends

Reviews focused on iPhone medical applications

Happtique A mobile medical app store developed by healthcare professionals. Happtique is currently developing a medical mobile app certification program to verify clinical appropriateness and technical functionality based on quality and performance standards.

KLAS Reviews on “mobile data systems,” which KLAS describes as “products

6 The EHR Usability Evaluation Protocol (EUP), Technical Evaluation, Testing and Validation of the Usability of

Electronic Health Records (NIST Interagency Report 7804), is available at

www.nist.gov/healthcare/usability/index.cfm. A draft version of the document was released for public comment in

September 2011

Resource Description that focus on providing physicians with access to census, results, and other patient information on a handheld device.”

mobihealthnews News and reviews on the latest medical apps.

Another potential resource to consider is your peers. Check with professional organizations for your specialty to see what they may have to offer. Observe one or two colleagues using the app you are considering. Observe them working with the app. Have a checklist of questions or issues that have arisen so far in your usability evaluation process that relate to their own work.

Find out what challenges these users have had learning and customizing the app with the following questions: How long did it take them to be fully “up to speed”? What was easy or difficult to learn? What workarounds have they had to develop and why? What tasks do they find frustrating due to inefficiencies in the app or complicated design? What kinds of errors do they find are too easy to make? How does the app increase efficiencies?

3. Assess the App’s Usability with Typical Clinical Scenarios

Perform a usability test of the app(s) under consideration. This can be done using fairly simple non-scientific methods, but is a structured means of collecting valuable usability feedback.

a. Create a representative set of clinical scenarios that include the essential and frequent tasks that the app will support. Time permitting, include complex tasks. A complex example: Three different apps are used during a patient assessment—a decision support app to identify differential diagnoses, an app to review the patient’s recent ECG waveform captures and a medication dosing app to help guide the care plan.

If you are testing the app with multiple providers in your practice suggest, including task-specific questions for the participant to answer after completing each test scenario. The questions should focus on efficiency, effectiveness, and satisfaction as well as usability concerns related to each task. See Appendix B for some sample primary care practice scenarios.

A more in-depth review of usability testing is provided in the following NIST publications: Customized Common Industry Format Template for Electronic Health Record Usability Testing (NISTIR 7742) and Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records (NISTIR 7804)

If you are testing the app for your individual practice needs and not gathering feedback from a group of providers, suggest completing the Usability Principle Attribute Checklist outlined in Table 2 of this guide.

b. Prepare a post-usability test questionnaire that captures the overall experience with the app. Include a simple rating scale for each question. See Appendix C for examples.

c. If gathering feedback from a representative set of clinicians in your practice, schedule each of them to participate in a one-on-one hands-on test of the app after a brief demo.

d. Conduct the usability test. The basic elements of a simple usability test are as follows: o Load the app on one or more mobile devices that can be used for testing. Ideally,

the participants should already be familiar with the mobile device type(s) (e.g., iPhone or Android phone) and its basic features.

o Instruct the participants to attempt to perform the scenarios without assistance, “thinking out loud” as they go.

o Each participant should answer scenario specific questions as each scenario is completed, and fill out the general questionnaire at the very end.

e. Try to provide a test environment that closely simulates (or is) the actual usage environment and typical usage conditions.

o Consider screen orientation. Does a certain usage scenario presume use in portrait or landscape mode? Test tasks in both modes if possible.

o Consider how the user will hold the device. Is it likely to be used with one hand or two?

o For any text entry, consider whether the users will one finger type or will be likely to double-finger or double thumb-type. Will this affect performance?

o Are there any other environmental requirements? For example, if the device is to be used in surgery, test the use with a case/cover that meets surgical standards and use with surgical gloves.

f. As an “observer” or usability test facilitator:

o Record the time it takes for the participant to complete each scenario. o Record key comments made during the test (without interrupting the user). o Record whether the participant successfully completed each scenario. o Track common errors made by participants. o Provide each participant with the post-test questionnaire at the end of the test. o Aggregate and synthesize the observations, comments, and responses. o If working alone, consider using low cost video recording so you can go back and

evaluate the users after the live study sessions. There are many free and low cost software apps that let you use a webcam and a laptop to capture the sessions. Below are links to a few usability testing tools/resources: Jing – techsmith.com (free) http://www.usefulusability.com/24-usability-testing-tools/

f. Incorporate the Usability Findings into your Overall Assessment of the App

Whatever app you choose, provide feedback to the vendor to help them improve their product. If possible, also participate in future vendor usability studies and user-centered design activities. It will give you direct input to product improvements and improve the likelihood of their success as well as your own.

Learn More about Usability

You can find more information on the web about basic usability concepts and methods from publications specifically related to app usability. Here are a few resources directed at mobile app usability and mobile app design.

Fling B. (2009). Mobile Design & Development. North Sebastopol, CA: O'Reilly Media. Available at: http://mobiledesign.org/toc.

HIMSS. (June 2009). Defining and Testing EMR Usability: Principles and Proposed Methods of EMR Usability Evaluation and Rating. Available at: www.himss.org/content/files/himss_definingandtestingemrusability.pdf. Accessed March 22, 2012.

HIMSS. (August 2010). Selecting an EHR for Your Practice: Evaluating Usability v1.2. Available at: www.himss.org/content/files/HIMSS%20Guide%20to%20Usability_Selecting%20an%20EMR.pdf. Accessed March 22, 2012.

Lowry SZ, Schumacher RM. (November 2010). Customized Common Industry Format Template for Electronic Health Record Usability Testing (Report No. NISTIR 7742). National Institute of Standards and Technology, Health Information Technology (IT), Usability Available at: www.nist.gov/customcf/get_pdf.cfm?pub_id=907312. Accessed June 12, 2012.

Lowry SZ, Quinn MT, Ramaiah M, Schumacher RM, Patterson ES, Abbott P. (February 2012). Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (Report No. NISTIR 7804). National Institute of Standards and Technology, Health Information Technology (IT), Usability. Available at: , www.nist.gov/customcf/get_pdf.cfm?pub_id=909701.

mHIMSS. Usability, Accessibility, Human Factors & Design. Available at:

www.mhimss.org/resource-topics/usability-accessibility-human-factors-design.

Prayaga S. (n.d.). Mobile User Interface and Usability Guide [Web log post]. Available at: www.scribd.com/doc/33158304/Mobile-UI-and-Usability-Guide; http://pshyama.wordpress.com/2010/06/17/mobile-user-interface-and-usability-guide/ Roberts D. (n.d). Usability.gov. Usability and Mobile Devices. U.S. Department of Health and Human Services' Office of the Assistant Secretary for Public Affairs, HHS Web Communications and New Media Division web site. Available at: www.usability.gov/articles/newsletter/pubs/052010news.html.

U.S. Food and Drug Administration, Center for Devices and Radiological Health and Center for Biologics Evaluation and Research. (July 21 2011) Draft Guidance for Industry and Food and

Drug Administration Staff - Mobile Medical Applications. Available at: www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm263280.htm. Accessed March 22, 2011.

Contributors

Patricia Arellano, MS, Planned Systems International, Inc. (PSI) Janet Bochinski, MSN, PNP Beth Elias, PhD, MS, University of Alabama at Birmingham Shannon Houser, PhD, MPH, RHIA, University of Alabama at Birmingham Hank Head, McKesson Staff Liaison - Thomas Martin, MBA, mHIMSS

Appendix A. Mobile Design Tenets

Below is a list of mobile design tenets as of January 1, 2012 as published by the mHIMSS Design

Tenet Workgroup. A well-designed mobile app should reflect the following design best

practices:

Category Mobile Design Tenet Description Data Let data scream a. The focus is the data. The design of the interface does not

interfere with the display of the data. b. Eighty percent of screen real estate should be dedicated to

data; 20 percent to interface. c. Images, text and numbers are included where visually

appropriate. Only handle information once

a. Efficiency is critical; one piece of data, entered once only.

Layout Grid it a. Graphic design and layout (e.g., gutters, columns, margins, and captions) are consistent across screens.

Type less + less type a. The app incorporates typographic elements that communicate meaning and help users understand context and interactivity. Examples include: Size, weight, style, color, spacing.

b. For readability, a single sans-serif typeface and up to six type treatments for the typeface are used.

c. The app “speaks” the user’s language. d. Content is concise, requiring less reading and uses more

visual guides. e. Layout is part of readability. Eye scan patterns matter.

Scanning the interface is intuitive. Color carefully a. Color is used sparingly.

b. Color helps the information, the interaction, the user experience accomplish the apps’ intended purpose.

Feedback Date your users a. The interface is intuitive when performing primary/common user tasks.

b. Feedback, both good and bad, can be provided directly to the design team.

c. Feedback from early adopters is especially encouraged. d. Repeat customers are cultivated as beta testers.

Interaction What interface? a. Cognitive burden is reduced when dealing with the interface. b. Users engage directly with the content and task at hand

without thinking about interface.

Category Mobile Design Tenet Description

c. The interface makes the most of screen real estate.

Every interface item has a function/purpose. Key tasks have priority. No clutter

d. Efficiency is critical. The app directly manipulates data elements rather than

the interface.

The app displays controls in a progressive manner, only the ones needed at specific points along the intended workflow.

Speak my sign a. Symbolic vocabulary is globally understood. b. Iconography and symbols speak “naturally.”

Repeat customers ROCK

c. The app is designed for different types of users (experienced, novice, inexperienced), who will demand other services to use with the app, and purchase upgrades and future products.

Get physical a. The design judiciously leverages keyboard shortcuts, physical engagement with screens (e.g., touch, swipe, pinch and zoom, and lasso), and power user tools.

b. The app works within mobile device limitations such as:

No hover text feature.

Larger target size. Smaller display.

c. The app leverages new capabilities. Examples include:

Touch-based interactions. Location awareness (GPS).

Orientation, position, motion awareness. Proximity sensitive.

Integrated communications. Integrated communication tools (audio, video, photo). Push notifications (reminders).

Lust to dust a. The app reflects system engineering approach to interface design and the app development lifecycle.

b. The design addresses the entire experience, from inception to implementation; covers the lifecycle of the application.

Appendix B. Sample Usability Task Scenarios

The following examples demonstrate how you can evaluate scenarios for your usability test of an application. You should create your own based on the key practice goals you have defined, the most frequent types of tasks performed as well as the most critical tasks. Scenario specific evaluation criteria can be developed if you are gathering feedback from a representative set of clinicians in your practice. Criteria can be based on the “Usability Principle Attribute Checklist” identified in Table 2 of this guide. The checklist can be used in its current form to evaluate the scenarios if you are evaluating the app for your individual use. Evaluation criteria tailored to the specific scenarios are provided below.

Scenario 1: Pediatric Medication Dose Calculation

A pediatrician completes and documents an assessment of a two-year-old male patient with a diagnosis of bacterial pneumonia. The assessment data is entered in an EHR, which does not have a pediatric dose calculator. One of the treatments in the care plan is Amoxicillin Suspension. The provider uses her pediatric dose calculator via her mobile device to calculate the dose based on the patients age and weight. The dose calculator is not interfaced to the EHR.

Task: Calculate pediatric patient’s medication dose

1. Enter the patient weight and age. 2. Select Amoxicillin Suspension from the list of drugs. 3. Identify the correct Amoxicillin Suspension dose for the child’s age and weight.

Evaluate Task: Identify the correct Amoxicillin Suspension dose for the child’s age and weight

Usability Principle

Example characteristics of a “usable” app

Patient weight and age data entry fields are the focus of the patient data screen.

Access to drug list and drug edit functions are clearly identifiable. You are able to quickly determine the correct drug, concentration and dose. Correct dose is clearly displayed next to weight and age

The app provided all the data elements required for determining the correct medication dose.

You are able to complete the necessary steps to calculate the drug dose. Essential drug administration information is listed.

Simplicity

Usability Principle

Example characteristics of a “usable” app

Essential supporting information required to make a decision is available when calculating the medication dosage.

All the important information you want to quickly reference is readily available using Info Buttons on the display or quickly accessible with little disruption.

You were able to move between patient data input fields, drug lists and dosing calculation screens quickly and easily.

The app alerts you when a patient age and weight are not consistent with normal ranges.

The app drug list supports trade and generic names. The patient weight is expressed in either kilograms and/or pounds. The app provided enough medication list categories or groups allowing for

timely drug identification.

You are able to quickly find Amoxicillin in the drug list. Medications can be sorted by favorites, alphabetically, by broad categories,

searched by name, or by category of your choice. The number of steps required to calculate the drug dosage was not excessive. You have no difficulty entering or searching for data on each screen. You do not encounter data entry errors, e.g. patient weight, due to small screen

size on your mobile device. Screen size did not affect your ability to read drug names, supporting drug data or drug dosing information.

Patient weight, age and other data entry fields are clearly identified and drug lists are easy to read and required drug quickly identified.

Colors are used to convey meaning (e.g., red to identify drugs used in medical urgency or Broselow tape color for selection of weight ranges), not just for visual appeal.

You are able to navigate through drug dosing screens minimal steps. Pop up boxes such as do not interfere with displaying the data fields or dose

result.

Naturalness

Consistency

Forgiveness

and Feedback

Effective Use

of Language

Efficient

Interactions

Usability Principle

Example characteristics of a “usable” app

Screen size does not hinder viewing all data fields simultaneously.

The data on each screen was logically grouped and the number of steps required for calculating the Amoxicillin suspension dose allowed for rapid and accurate dose calculation.

You are able to quickly confirm the dose calculation if necessary. Visual and audio alerts, such as low device memory warnings, are clear, concise

and informative.

Preservation

of Context

Minimize

Cognitive

Overload

Scenario 2: Review of Patient Test Results

A provider sees patient during a follow-up visit to review abdominal ultrasound to rule out gallstones. The provider has access to the local radiology department’s picture archiving and communication (PACS) system’s ultrasound images via his mobile device. The provider’s mobile app uses cloud based radiology (DICOM) viewing and sharing web application. The provider pulls up the images and radiology report, via mobile device, to review and discuss results with patient.

Task: Review patient radiology results

1. Identify and select patient. 2. Select ultrasound. 3. Review ultrasound images.

Usability Principle

Example characteristics of a “usable” app

Unique patient identifiers are clearly displayed allowing for selection of the “right” patient.

Patient's demographic information such as gender and age are displayed prominently and in a location that makes it easy to reference.

You are able to quickly and efficiently identify the right ultrasound images for

the right patient. Patient identifiers and image details are displayed in the same place on each

image and do not interfere with the ultrasound study images. Images take up the majority of the screen enhancing your ability to view and

interpret.

The app allows you to view multiple images at one time. Ultrasound image download time is acceptable and supports clinical

workflow. Screen icons and navigation options are intuitive and are consistent with

common user applications such as web-browsers

Different features of the app are used and behave in a consistent manner. The same gestures such as pinch and zoom, tapping and swiping are used

consistently across screen images. Alerts and other informational messages follow commonly used color and

sound schemes.

Simplicity

Naturalness

Consistency

Usability Principle

Example characteristics of a “usable” app

The app notifies you if you select an image that is not uniquely identified to the patient you have selected.

The app displays explanatory message when processing information such as downloading images from the PACS, and provides feedback on the expected it will take to complete the process.

The app uses the same words that you use (while providing mapping to standardized codes and terms used for data retrieval).

List or entry-form choices are clear, concise and unambiguous. Sentences read like natural English (or the selected language). Iconography and symbols speak “naturally.”

Language that is not information is not included in the design.

Single tap, multi-tap, swipe gestures work consistently and produce the desired navigation result across all ultrasound image screens.

The app minimizes the number of steps/gestures it takes to complete patient and ultrasound selection and navigate through ultrasound images.

Images and patient information can be removed from the device easily when the visit is finished.

Patient identifiers and ultrasound image details are displayed are easy to read.

Colors are used to convey meaning (e.g., change contrast of image), not just for visual appeal.

App allows light calibration capability to control for ambient light and indicates when image of diagnostic viewing quality and non-diagnostic viewing quality.

Pop up boxes do not interfere with the image results. Measurement overlays and contrast tools are available to enhance the MRI

image interpretation. Screen size does not affect your ability to view diagnostic quality images. Manipulating and viewing images is fluid and is not hampered by timing

screen delays.

You are able to quickly synthesize and interpret image findings. Visual and audio alerts, such as low device memory warnings, are presented

are clear, concise, and informative.

Forgiveness

and Feedback

Effective Use

of Language

Efficient

Interactions

Preservation

of Context

The app supports measurement overlay tool to automatically calculate distances between specified points or measurement of an area on images so that you do not have to manually perform the calculations.

Scenario 3: Diabetic Patient Education

A provider at a diabetes clinic is meeting with a patient to review their blood glucose measures and to help the patient meet their dietary and exercise goals. The provider has downloaded the patient’s blood glucose measures for the last two weeks from EHR to the mobile device and is using an app that graphs the measures. Each measure that is above or below the target range is highlighted in red. Additional data such as meal times and exercise times can be added to the graph via a tap and drag menu of icons. When the provider taps on the highlighted measure, educational material is displayed that is appropriate to the measure. Additionally, the app can email this material to the patient at the patient’s request. The trend graph can also be emailed. The trend graph can be resized to allow easier viewing at different levels of detail.

Task: Review patient blood glucose trending with patient

1. Identify and select patient. 2. Select date range for display of blood glucose measures. 3. Add meal and exercise times if desired. 4. Display educational material. 5. E-mail educational material and trend graph if desired.

Usability Principle

Example characteristics of a “usable” app

Unique patient identifiers are clearly displayed allowing for selection of the “right” patient.

Patient's demographic information such as gender and age are displayed prominently and in a location that makes it easy for you to reference.

You are able to quickly and efficiently specify the date range for display of the

blood glucose measures. Trend graph design is concise and uncluttered. Menu of icons for meals and exercise allows for tap and drag to the trend

graph and can be hidden. Trend graph can be resized (made larger or smaller) easily to allow viewing at

different levels of detail.

Minimize

Cognitive

Overload

Simplicity

Usability Principle

Example characteristics of a “usable” app

Trend graph is on a grid with time as the X axis and measures as Y, resizing adjusts from hours to days on the X axis.

Menu of icons for meals and exercise uses appropriate symbols for different types of food and exercise.

Screen icons, navigation and email options are intuitive and are consistent with common user applications such as web-browsers.

Different features of the app are used and behave in a consistent manner. The same gestures such as pinch and zoom, tapping and swiping are used to

be consistent with common smart devices and other apps. Alerts and other informational messages follow commonly used color and

sound schemes.

Naturalness

Consistency

Usability Principle

Example characteristics of a “usable” app

If you select an invalid date range or did not select a date range, the app will let you know and provides informative instructional messages.

The app displays explanatory messages when processing information (such as while generating the trend graph) and describes how long it might take.

The app uses the same words that you use (while providing mapping to standardized codes and terms used for data retrieval).

Trend graph markers are clear, concise and unambiguous. Educational material is written to read like plain English (or the selected

language). Iconography and symbols speak “naturally”.

Language that is not information is not included in the design.

Single tap, multi-tap, swipe gestures work consistently and produce the desired navigation result across all components of the screens.

The app minimizes the number of steps/gestures it takes to complete patient and measures selection and navigate through educational material and email.

Measures and patient information can be removed from the device easily when the visit is finished.

Patient identifiers and trend graph details are displayed are easy to read. Colors are used to convey meaning not just for visual appeal. Educational material is displayed in concise summary form, resizing increases

or decreases level of detail. Audio is used to convey meaning, not just for basic device interaction.

Pop up boxes do not interfere with the image results. Educational material is displayed with the measure in question. Screen size does not affect your ability to view graphs. Manipulating and viewing the trend graph and educational material is fluid

and is not hampered by timing screen delays.

Visual design allows you to quickly synthesize and interpret the trend graph. Measures such as the average glucose level by day or by week are

automatically calculated.

Forgiveness

and Feedback

Effective Use

of Language

Efficient

Interactions

Preservation

of Context

Minimize

Cognitive

Overload

Appendix C. Sample Post-Test Questionnaires

Post-usability test questionnaires are administered to a test participant immediately after completing the test scenarios and before any debriefing. The first questionnaire included in this Appendix is the Digital Equipment Corporation System Usability Scale (SUS)7. This is a general questionnaire that has been used in many usability studies; if used as intended, it has been shown to be a reliable evaluation tool. A brief overview of the SUS can be found in Wikipedia8. For instructions on how to score responses on the SUS as well as a sample scored questionnaire see Brooke9.

The second sample questionnaire illustrates how you might include questions more directly related to the usability principles discussed as key for medical app design. Include questions aligned with your practice goals. For instance, if efficiency in the review of clinical documentation is one of your key concerns, then include questions about the efficiency and ease of use of the documentation tools. This set of questions would not constitute a scientifically reliable tool, just a simple means of collecting subjective responses from your participants in a structured way. There would also be no official means of scoring responses to these questions other than face-value impressions.10

7 Brooke J. (1996). "SUS: a "quick and dirty" usability scale". In P. W. Jordan, B. Thomas, B. A.

Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis.

www.usabilitynet.org/trump/documents/Suschapt.doc.

8 Wikipedia. System usability scale (2011). Retrieved from http://en.wikipedia.org/wiki/System_usability_scale.

9 Brooke J. (1996). "SUS: a "quick and dirty" usability scale". In P. W. Jordan, B. Thomas, B. A.

Weerdmeester, & A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis.

www.usabilitynet.org/trump/documents/Suschapt.doc.

1. System Usability Scale (SUS)

Strongly Strongly Disagree Agree

1. I think that I would like to use this system

frequently.

1 2 3 4 5

2. I found the system unnecessarily complex. 1 2 3 4 5

3. I thought the system was easy to use. 1 2 3 4 5

4. I think that I would need the support of a technical

person to be able to use this system.

1 2 3 4 5

5. I found the various functions in this system were

well integrated.

1 2 3 4 5

6. I thought there was too much inconsistency in this

system.

1 2 3 4 5

7. I would imagine that most people would learn to

use this system very quickly.

1 2 3 4 5

8. I found the system very cumbersome to use. 1 2 3 4 5

9. I felt very confident using the system. 1 2 3 4 5

10. I needed to learn a lot of things before I could get

going with this system.

1 2 3 4 5

2. Additional Sample Usability Post-Test Questions

Strongly Strongly Disagree Agree

1. The app had a clear, clean, uncluttered screen

design.

1 2 3 4 5

2. The app kept screen changes to a minimum during

completion of a task.

1 2 3 4 5

3. The app minimized the number of steps it took to

complete tasks.

1 2 3 4 5

4. Information presented on screens was easy to

comprehend quickly.

1 2 3 4 5

5. Information needed for a specific task was

grouped together on a single screen.

1 2 3 4 5

6. Choice lists were clear and unambiguous.

1 2 3 4 5

7. Clinical documentation tools were efficient to use.

1 2 3 4 5

8. Alerts were only presented at appropriate times.

1 2 3 4 5

9. Data could be entered once then used in multiple

places.

1 2 3 4 5

10. I felt confident I could make a mistake without

losing my work.

1 2 3 4 5

General Disclaimer

The inclusion of an organization name, product or service in this document should not be construed as a HIMSS endorsement of such organization, product or service, nor is the failure to include an organization name, product or service to be construed as disapproval.