rssi

2, P

illto

Armstrong Institute for Patient Safety and Quality, Department of Anesthesiology and Critical Care Medicine, School of Medicine, The Johns HopkinsUniversity, 750 E. Pratt Street, 15th Floor, Baltimore, MD 21202, USA

Article history:Received 26 July 2011Accepted 2 September 2013

low (Manne et al., 2002; Weinberg et al., 2004). In 2003, the

ated that patientse care based on aetropolitan areasrecent report, for

the recommendedisease Control andy be due to a lackate reinforcementpliance with the, limited access toa et al., 2006).

Patient-centric health information technologies (HIT) arebecoming increasingly popular (Perlin et al., 2004; Sunyaev et al.,2010). To offer better care at lower costs, HIT needs to supportand facilitate patient-centered care rather than focusing on isolatedphysician and clinician tasks (Walker and Carayon, 2009). Ifdesigned well, HIT can educate patients on health care topics, givereminders to ensure timely screening, and provide support mech-anisms for patients to facilitate their preventive screening decisions(Institute of Medicine Committee on Quality of Health Care in

* Corresponding author. Tel.: 1 410 455 8627; fax: 1 410 455 1073.E-mail addresses: ozok@umbc.edu (A. Ant Ozok), huijuan.wu@umbc.edu

(H. Wu), melissa.garrido@mssm.edu (M. Garrido), ppronovo@jhmi.edu(P.J. Pronovost), agurses1@jhmi.edu (A.P. Gurses).1 Tel.: 1 455 8834; fax: 1 410 455 1073.2 Tel.: 1 732 932 5230; fax: 1 732 932 1945.3 Tel.: 1 410 637 6261.4

Contents lists availab

Applied Erg

journal homepage: www.els

Applied Ergonomics 45 (2014) 613e628Tel.: 1 410 637 4387.1. Introduction

Each year, thousands of people suffer from potentially pre-ventable illnesses such as heart attack, stroke and cancer. In addi-tion to claiming lives and reducing individuals quality of life, theseillnesses also result in millions of dollars in health care costs(Peterson and Dragon, 1998; Avorn and Shrank, 2008). Screeningcan prevent a considerable amount of these illnesses and reducecosts (Salkeld, 2006). The number of recommended preventivescreening tests has increased in the last decade, but patientscompliance with recommended health screening rates still remains

seminal paper of McGlynn and colleagues indicreceive only 55% of the recommended preventivsurvey of a random sample of 6712 adults in 12 min the US (McGlynn et al., 2003). Based on CDCsexample, only 59% of patients in the US receivecolorectal cancer screening tests (Centers for DPrevention, 2012). These low rates may partiallof awareness by clinicians and patients, inadequmechanisms for patients with regards to comrecommended preventive screening guidelineshigh quality of care, and nancial barriers (MoscKeywords:Personal health recordsUsefulnessUsability0003-6870/$ e see front matter 2013 Elsevier Ltdhttp://dx.doi.org/10.1016/j.apergo.2013.09.005Personal Health Records (PHR) are electronic applications for individuals to access, manage and sharetheir health information in a secure environment. The goal of this study was to evaluate the usefulnessand usability of a Web-based PHR technology aimed at improving preventive care, from both the pa-tients and primary care providers perspectives. We conducted a multi-method descriptive study thatincluded direct observations, concurrent think-aloud, surveys, interviews and focus groups in a suburbanprimary care clinic. Patients found the tailored health recommendations useful and the PHR easy tounderstand and use. They also reported asking useful health-related questions to their physiciansbecause of using the system. Generally, care providers were interested in using the system due to itsuseful content and impact on patient activation. Future successful systems should be better integratedwith hospital records; put more emphasis on system security; and offer more tailored health informationbased on comprehensive health databases.

2013 Elsevier Ltd and The Ergonomics Society. All rights reserved.a r t i c l e i n f o a b s t r a c tUsability and perceived usefulness of pepreventive health care: A case study focucare providers perspectives

A. Ant Ozok a, *, Huijuan Wu a, 1, Melissa Garrido b,

Ayse P. Gurses c, 4

a Department of Information Systems, University of Maryland Baltimore County, 1000 Hb GRECC, James J Peters VA Medical Center, 130 W Kingsbridge Road, Bronx, NY, USAcand The Ergonomics Society. All rional health records forng on patients and primary

eter J. Pronovost c, 3,

p Circle, Baltimore, MD 21250, USA

le at ScienceDirect

onomics

evier .com/locate/apergoghts reserved.

o

the

ErgoAmerica, 2001). Personal Health Records (PHR) are an HIT that canplay a signicant role in improving patients knowledge of andattitudes toward their health care (Powsner, 1998; Brennan, 2000).The Markle Foundation (2003) denes PHR as an electronicapplication through which individuals can access, manage andshare their health information in a private, secure, and condentialenvironment. International Standards Organization (ISO) (2009)indicated: The Personal Health Record of an individual is a re-pository of information considered by that individual to be relevantto his or her health, wellness, development and welfare, and forwhich that individual has primary control over the recordscontent.

PHR can improve health care, including preventive care, througheducating and engaging patients in their own health care (Krist andWoolf, 2011; Tang et al., 2006), and as a result transforming theminto more activated patients (Von Korff et al., 1998; Lorig et al.,1996). Patient activation aims to help patients develop the skills,knowledge and motivation to become an effective participant intheir health (Hibbard et al., 2004, 2005). It has been linked to betterhealth outcomes and lower costs (Bodenheimer et al., 2002; Brittoand Wimberg, 2009). PHR can increase patient activation (Eriksenand Ursin, 2004; Goth, 2008; Ueckert et al., 2003) and improvecommunication between patients and caregivers (Ueckert et al.,2003) (Eriksen and Ursin, 2004; Steele and Lo, 2009). Thesestudies make it clear that patients are more likely to make gooddecisions and take actions to promote their health if they becomeactive participants of their own health care (Bourgeois et al., 2008;Ueckert et al., 2003).

In the early stages of health information systems, PHR-likesystems were not always Web-based (Tang et al., 2006;Srinivasan, 2006). While there are still various platforms support-

Perceived usefulness

Perceived usability

AttitudeBehavioral intention t

use (acceptance)

Fig. 1. Conceptual framework guiding the study (adapted from

A. Ant Ozok et al. / Applied614ing PHR systems, including free-standing/PC-based, universal serialbus (USB)/portable storage-based, and mobile/smart phone-basedplatforms, Web-based systems have so far shown their domi-nancy (Steele and Lo, 2009). Unlike Web-based health portals,which in most cases consist of a collection of standard informationon health issues and illnesses and user-created discussion boards(Weingart et al., 2006). PHR systems typically allow users tomanipulate their own health information, and therefore have thepotential to better satisfy their health information needs andpositively impact health behaviors (Malamateniou andVassilacopoulos, 2010). PHR can motivate patients to take the rec-ommended preventive screening tests and to improve compliancewith drug and other therapies by providing them with eithertailored recommendations and education or electronic remindermechanisms (Palen and Aalokke, 2006; Krist et al., 2011). Theserecommendations and reminders can lead to more informed pa-tientecaregiver interactions and cost savings. PHR can help familymembers manage the health information of specic groups such aschildren, elderly, or terminally disabled patients as well(Greenhalgh et al., 2010; Bourgeois et al., 2008).

PHR systems can also improve quality of health care by sup-porting care providers work. For example, they can help in closingthe health information gap between patients and providers bymaking the episodic nature of care more continuous (Ball et al.,2006) as well as facilitate patient education and shared decision-making.

Although PHR have great potential to improve health care (Tangand Lansky, 2005; Scherger, 2005), in particular preventive care(Kahn et al., 2009; Krist et al., 2011), to date there is limited evi-dence supporting a positive impact of PHR on health care frompatients as well as caregivers perspectives (Tang et al., 2006;Zuckerman and Kim, 2009). A small number of studies identiedbarriers to acceptance and widespread usage concerns about pri-vacy, security and condentiality of health-related data, feasibilityof integration of PHR to electronic health records (EHR), accuracyand completeness of data entered, and usability and usefulness ofPHR systems (Denton, 2001; Liu et al., 2011; Tang et al., 2006;Hargreaves, 2010). For PHR to gain widespread use, it needs to beaccepted and adopted by both patients and care providers, assimilar technology acceptance and adoption issues were reportedin other areas (Venkatesh and Davis, 2000). It is therefore impor-tant to study factors that may potentially affect PHR acceptancefrom both the patients and care providers viewpoints. This willguide us on how to design and implement PHR that will meet theneeds of both stakeholders, and will increase the likelihood ofsuccess.

There is a knowledge gap in the literature regarding patient andcare provider perceptions, attitudes and preferences toward anytype of HIT (Angst and Agarwal, 2009), in particular PHR (Baird

Actual use

Compliance with preventive screening recommendations and lifestyle changes

Technology Acceptance Model) (Davis, 1989; Davis et al., 1989).

nomics 45 (2014) 613e628et al., 2011). A number of studies focused on usability and useful-ness issues from both patient and provider perspectives(Montague, 2010; Martin et al., 2011), but they focused on medicaldevice design, rather than HIT. Furthermore, previous studiesevaluated PHR from either patients (direct users of PHR) or pro-viders (indirect users) perspective, which is only part of the pic-ture. More work is needed to understand the use of PHR systems asa tool that can complement the traditional care, as well as theirimpact on patients, providers, organizations and health care sys-tems based on human factors and ergonomics concepts andmethods (Nazi et al., 2009).

Based on the above factors and the well-known TechnologyAcceptance Model (TAM) (Davis, 1989; Davis et al., 1989), we arguethat high-usability and usefulness of patient-centric HIT can bevaluable in improving patients compliance with the recommendedguidelines and medical outcomes. Fig. 1 presents the overarchingconceptual framework guiding our study, which was adapted fromthe TAM. The framework indicates that a patients or care providers

Ergoattitude toward the PHR system can be determined by two con-structs: perceived usefulness and perceived usability. Instead of theconstruct perceived ease of use included in the original TAMmodel, the modied framework uses the construct of perceivedusability. Due to the broader range of the usability term that in-cludes ease of use as well as a number of other concepts (Nielsen,1993), perceived usability was chosen as one of the main con-structs. Nielsen (1993) denes a high-usability interface as aninterface with high learnability, efciency, memorability andsatisfaction, and one with a low number of errors. In the context ofthis study, perceived usability refers to the degree to which usingthis particular PHR system will be free of effort. We deneperceived usefulness as the degree to which a patient or providerbelieves that using the PHR would increase compliance with preven-tive screening and lifestyle changes. The conceptual frameworkpostulates that if patients and providers have a more positiveattitude toward the technologies they use (in terms of usefulnessand usability), their acceptance, intention to use, and eventuallytheir actual use of these technologies will also be higher.

The purpose of this study was to evaluate the value (i.e., use-fulness and usability) of a specic patient-centered informationtechnology in improving awareness of (for patients) and compli-ance with (for both patients and care providers) preventive careguidelines. Using a case study, we specically aimed at determiningpatients and providers perceptions regarding (1) whether a PHRsystem can be useful in improving patientecare provider in-teractions, preventive health screening, and compliance withhealthier lifestyle behaviors; and (2) the key usability factors ofsuch a system. For the specic technology, we used a Web-based,secure system, MySafe-T.Net, developed as a PHR with the pri-mary aims of (1) improving preventive care and preventivescreening rates and (2) changing patients lifestyles to improvetheir health. MySafe-T.Nets secondary aim is to serve as a healthinformation repository.

Making the technology useful and easy to use for both patientsand care providers is a challenging goal, yet not exclusive to PHRsystems. Health care is lled with situations in which there aredifferent types of users for the same technology (Carayon et al.,2010). Care is often provided in complex socio-technical environ-ments (Buckle et al., 2006; Carayon, 2010). Hence, (re)designingtechnologies based on human factors engineering, specically us-ability and usefulness, principles by taking into account the over-lapping, complementary, and sometimes conicting needs ofmultiple players is crucial if we want to improve overall systemsperformance.

2. Methodology

2.1. Description of the MySafe-T.net system

MySafe-T.net was designed with the goal of preventing futurehealth conditions such as high blood pressure, obesity and cancerand leading people to live a healthy life style by increasing patientactivation. MySafe-T.net is aimed at increasing patients compliancewith preventive screening tests (e.g., colorectal cancer screenings,cholesterol screenings) and lifestyle changes (e.g., physical exercise,eating habits) recommended by the US Preventive Services TaskForce based on an individuals prole including demographics,personal and family health histories, lifestyle and health habits(Shea et al., 1996). It can be used by people of various ages andclinical conditions, including healthy users. The system providespatients with an interactive environment to access their own healthrecords via a Web browser. During the initial use, the system askseach patient to provide information on ve main categories

A. Ant Ozok et al. / Appliedincluding demographics, personal health history, family history,lifestyle, and health habits (Appendices A1, A2 and A3). It developsan individual prole based on this information, and then appliesthe U.S. Preventive Services Task Force (2007) preventive healthscreening recommendations to this prole to produce preventionand lifestyle change (e.g., frequency and intensity of physical ex-ercise) recommendations tailored to each person based on theirindividual risk factors (Appendix A4). MySafe-T.net also providesindividualized educational Web site links (Appendix A5) based oneach patients prole. The recommendations are updated when thepatient makes changes on their stored prole.

Using the system, the patient can print a letter for the careprovider and share this letter with their physician and other careproviders. This letter includes information about the patients de-mographics, own and family health histories, and the recom-mended preventive screening tests and lifestyle changes based ontheir prole. This feature was designed with the goal of improvingthe communication between patients and care providers andincreasing shared decision-making.

2.2. Study site and sample

The study was conducted at a Minnesota suburban primary careclinic afliated with a nation-wide institution. We collected datafrom both patients and care providers. The clinic coordinator con-tacted 36 consecutive patients who had made an appointment atthe clinic for an annual physical exam within a two-week windowin February 2008, described the study briey and gave the contactinformation of the research assistant (MG) to those interested.Twenty-nine patients contacted the research assistant, who pro-vided detailed information about the study and checked each pa-tients eligibility for participation. The inclusion criteria includedhaving scheduled an annual physical exam at the time of the study,having a moderate level of experience with computers and theInternet, and being 21 years of age or older. Of the 24 eligible pa-tients, 22 agreed to participate. Two patients declined due toscheduling issues. The assistant obtained consent from these pa-tients on the phone and instructed them to arrive at the clinic 2 hbefore their scheduled appointment. Each participant was paid $70as an incentive at the end of their visit. Sixteen out of the 22recruited patients participated in the follow-up phone interviewwithin two weeks after their clinic appointment. All study pro-cedures were approved by the Institutional Review Boards of theinvolved institutions.

We also recruited eight care providers (four out of six primarycare physicians and four out of ve medical assistants) employed inthe same clinic. Physicians were included in the study due to theirimportant roles in providing preventive care including recom-mending, ordering and interpreting the results of preventive carescreening, providing lifestyle change recommendations, and pre-scribing medications to patients. Medical assistants almost alwaysinteract with the patients before the physicians on the day of theannual physical exam, take medical histories including thoserelated to preventive care, record vital signs, prepare patients forexaminations, and convey information important for the exam tophysicians. The information included in the PHR overlaps signi-cantly with the information physicians and medical assistants needfrom patients during the annual physical exam.

2.3. Study procedures

We used a multi-method approach (Johnson et al., 2007) foranswering the research questions. We collected data via observa-tions, concurrent think-aloud method, two types of patient in-terviews (one immediately after the tasks and one follow-up

nomics 45 (2014) 613e628 615within 2 weeks of the annual exam), a patient survey, and two

2.3.1. Data collection from patientsThe assistant rst introduced herself and escorted the patient to

a room with computer and Internet access where she providedinformation and instructions about the study. The patient wasasked to complete eight tasks detailed in Table 2 (sample screenshots in Appendix A) and was instructed to think-aloud whileshe/he was completing the tasks, known as the concurrent think-aloud method (Van den Haak et al., 2003). This method allowedus to examine in more depth the perceived usefulness and usabilityissues concerning the system. After the patient completed enteringthe required information to develop the personal prole (de-mographics, personal and family health history, medications andrecent screenings), the system provided tailored preventivescreening and lifestyle change recommendations. The patient wasthen asked to review these recommendations and visit at leastthree Web sites recommended by the system. It also automaticallyprinted the letter for the care provider which the assistant askedthem to share with their medical assistant and the primary carephysician at the beginning of their examination. During theexperimentation, the research assistant observed participants in-teractions with the system and took notes on automation surprises,confusion over instructions, overlooked site functions, and otherpotential problems. Next, patients were asked to complete a writ-ten survey and participate in an interview. They were theninformed that they could login and use MySafe-T.net any timewithin the next two weeks to update their information, review therecommendations, or receive new recommendations. The experi-mentation took about 1 h to complete. A follow-up phone interview

Table 1Summary of data collection methods, time of data collection, participant types, andresearch questions addressed.

Data collection methods Time completed Participanttype

Researchquestionsaddressed

Concurrent think-aloud methodwhile completingthe experimentaltasks provided byresearchers inthe PHR

Day of experiment Patients PHR usabilityfrom patientsperspective

PHR survey (PHRS) Day of experiment Patients PHR usability andusefulness frompatientsperspective

Semi-structuredinterviews

Day of experiment Patients PHR usability andusefulness frompatientsperspective

Follow-up phoneinterviews

2 weeks afterexperiment

Patients PHR usability andusefulness frompatientsperspective

First focusgroup

3 days after allpatient datacollection completed

Primarycarephysicians

PHR usefulnessand usabilityfrom careprovidersperspective

Second focusgroup

6 days afterall patient datacollection completion

Medicalassistants

PHR usefulnessand usabilityfrom care

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628616caregiver focus groups. Table 1 summarizes the data collectionmethods along with their respective completion times, participant

providersperspectivetypes (patients or care providers) and research questionsaddressed.

Table 2List of the experimental tasks and the instructions and additional information given to u

Tasks Users were instructed to:

Task 1: Entering Demographic Information - Enter demographic information- Save their information and move to t

Task 2: Entering Personal Health History - Enter medication allergies and intolerthe corresponding Edit buttons and d

- Enter past and current medical inform- Enter their prior surgical history (surg- Enter prior screening tests- Save and move to next task

Task 3: Entering Lifestyle and Health Habits - Answer a number of questions for the- Classify their habits into one or more

health habits categories

Task 4: Entering Family History - Enter information regarding the medfamily members (acute and chronic il

Task 5: Viewing Ones Prole Summary - Read their summary and make any mgoing back to the corresponding secti

Task 6: Viewing Recommendations - Read health recommendations produbased on their previous entries and thof some algorithms developed basedPreventive Services Task Force (USPS

Task 7: Printing the Letter for the CareProvider

- Keep the printed letter and hand it toassistant and the physician at the begtheir examination

Task 8: Clicking on the Resources Link - Click on and review at least three of tWeb page links recommended by theon patients prolewas conducted around the two-week mark.

2.3.2. The Personal Health Records Survey (PHRS)In order to determine the overall user perceptions concerning

the usability and usefulness of the system and to complement thendings from the qualitative studies, we conducted a survey amongthe patients. The Personal Health Records Survey (PHRS) is a paper-

sers.

Users were informed that:

he next task- They can always modify their demographicinformation later on by clicking on the Demographicsbutton from the main menu

ances usingrop-down menusationery type and date)

- They can return to the main menu using the Back button- They can repeat each procedure until they enter allof their information

systemof the

- Health habits categories include smoking, alcohol use,exercise and diet. Patient information on the habits isused by the system to generate lifestylerecommendations

ical history oflnesses, surgeries)

- They can also add information about their distantfamily members (e.g., uncles and aunts)

odications byonced by the systemrough the useon the U.S.TF) guidelines

- Recommendations are tailored based on theinformation patients provided

- For example, if a patient provided high cholesterolvalues, specic courses of action for loweringcholesterol would be recommended in this sectionbased on age, lifestyle and habits, medical history,family history and the USPSTF guidelines

the medicalinning of

- The letter provides a summary of key clinicalinformation for each patient and preventive healthscreenings recommended for each patient

he externalsystem based

- Patients can get additional information about aparticular health screening and/or lifestyle change

recommendation through these links.

receend

nde

to

follo

main

leteakessingsitetratload, loo

l andthe sethgest

ter u

Ergobased survey aimed at determining patient perceptions regardingMySafe-T.net. It measures eight major constructs presented inTable 3 with corresponding dimensions, used measures, number ofitems, and Cronbachs Alpha values. Thirty-eight of the 51 surveyitems had seven-point Likert scales as response categories rangingfrom 1 (strongly disagree) to 7 (strongly agree). Two items (onmental demand and effort) had 10-point scales (Low to High), andthe rest consisted of demographic questions. The items that

Table 3Description of the personal health records survey.

Section name Constructs measured/sectioncontent

Dimensions

A Usefulness - Relevance of information- Usefulness of site recomm

B Trust and security - Level of trust on the site- Hesitation on entering coto the site

C Impact on personal healthmanagement

- Willingness to use the siterelated information

- Helpfulness of the site inhealth screenings

D Benets (Duplicate questionsfor internal reliability)

- The sites contribution inhealthy lifestyle

E Ease of use/usability - Number of steps to comp- Perceived number of mist

F Overall satisfaction - Willingness to continue u- Advising others to use the- Using the site being a frus

G (part 1) Workload - Mental demand/memorycalculating, rememberingrequired to use the site

- Effort: The level of mentato complete the tasks on

G (part 2) Any other comments/suggestions? Open-ended question on whother comments and/or sugthe MySafe-T.net system

H Demographic information Age, gender, weekly compu

Survey Cronbachs Alpha Values: 0.90, 0.93, 0.81.

A. Ant Ozok et al. / Appliedmeasured usability were in part based on the framework developedby Lin et al. (1997), and the perceived usefulness measures werebased on the framework by Davis (1989) and Davis et al. (1989).

2.3.3. Patient interviews2.3.3.1. Face-to-face patient interviews in the clinic. Following thesurvey, patients were asked to participate in a short semi-structured interview conducted by the research assistant. Theinterview guide (Appendix B) was developed based on the con-ceptual framework guiding the study (Fig. 1, Davis, 1989; Daviset al., 1989; Lin et al., 1997). The goal of the interview was to pur-sue a one-on-one interaction with each patient to get more in-depth information (compared to the survey) about the positiveand negative perceptions of usefulness and usability of the MySafe-T.net and the underlying factors for these perceptions. The usabilityquestions were based on Nielsen (1993) and Shneiderman (1992)and Lin et al. (1997), and the usefulness questions, including easeof use and mental demand, were based on Daviss (1989) frame-work on perceived usefulness. All interviews were digitally recor-ded and transcribed.

2.3.3.2. Follow-up patient phone interviews. A follow-up phoneinterview with each patient was conducted about two weeks afterthe experiment. Sixteen of the 22 initial participants agreed toparticipate in the phone interview; 3 could not be reached after 4attempts to contact via phone, and the rest declined to attend dueto their busy schedules. During this interview, the research assis-tant inquired about (1) whether the patient used the system sincetheir physicians visit and why or why not; (2) how the physicianand the medical assistant reacted to the letter for the care pro-vider; and (3) whether they found the system useful and usable/easy to use in the long-term andwhy or why not. The assistant tookdetailed notes during the interview.

2.3.4. Data collection from care providersWe conducted two focus groups with care providers after

completing data collection from the patients, onewith primary care

Measures used/adapted from # of items Cronbachsalpha values

ived from the siteations

Davis (1989) 7 0.92

ntial informationGefen (2000) 3 0.98

collect health-

wing recommended

Denton (2001) and Prattet al. (2006)

9 0.96

taining a Denton (2001) and Prattet al. (2006)

3 0.91

the tasksmade

Davis (1989)Lin et al., 1997

10 0.76

the site

ing experience

Hackman and Oldham(1980)

6 0.83

: Thinking, deciding,king, searching, etc.

physical demandite

(Hart and Staveland,1988; Human PerformanceResearch Group, 1997;Lin et al., 1997

2 0.89

er patients had anyions regarding

1

sage, frequency, etc. 10

nomics 45 (2014) 613e628 617physicians and one with medical assistants. All recruited careproviders provided care to one or more patients who participatedin the study, which allowed them to examine the letter for the careprovider that the patients sharedwith them as part of the study (atleast twice) before participating in the focus groups. The focusgroup technique was used to allow moderated interactions be-tween care providers and to give providers an opportunity toenhance each others ideas and opinions concerning the system.Both focus groups were moderated by a human factors engineeringexpert (APG). The care providers did not use MySafe-T.net inpractice. However, conducting the focus groups after the patientexperiments ensured that providers had seen the letter for thecare provider for at least two patients. They were also providedwith a detailed demonstration of the system at the beginning of thefocus group by the moderator. The moderator asked questions us-ing the focus group guide in Appendix C. The sessions wererecorded using a digital voice recorder and transcribed. Each ses-sion took under 1 h to complete.

2.3.5. Data analysis2.3.5.1. Quantitative analysis. We limited our quantitative analysisto descriptive statistics due to the limited sample size as well as thesupporting nature of the quantitative data. After data from thesurveys were manually entered, standard data checking and veri-cationwere performed, and descriptive statistics including meansand standard deviations were calculated.

2.3.5.2. Qualitative analysis. To identify patients and care pro-viders perceptions about the value of MySafe-T.net, we analyzed

Table 4Patient demographics and background.

Age

Ergotrustworthiness, and perceived technical reliability of my Safe-T.net(Items 8, 9, 10) received high scores.

Section C focused on the perceived impact of the system onpersonal health management. Patients welcomed the potentialhelp offered by the MySafe-T.net in enabling them to be more incharge of their own health (Item 16) and in having more informa-tive conversations with their physicians (Item 12). They felt moreinformed regarding health risks due to family heritage throughMySafe-T.net (Item 15). Their interest in collecting informationfrom the system before their next physicians visit and the MySafe-T.nets helpfulness in better complying with the recommendedpreventive health screenings receivedmoremoderate scores (Items11 and 13). Participants indicated the system helped them have abetter understanding of the consequences of not following ahealthy life style (Item 19). While still positive, scores were lowerregarding the systems helpfulness in starting and maintaining lifestyle changes (Item 14) and whether the system gave them a betterunderstanding of the consequences of not getting timely screenings(Item 18). Results indicated the positive informational and educa-tional characteristics of the systemwhile scores concerning sharing

Table 6Descriptive statistics for the rst forty items of the PHRS.

# Description Mean S.D.

1 Information relevant 5.50 1.162 Recommendations useful 5.18 1.273 Learned on preventive screening 4.91 1.164 Clinicians awareness of my health 5.73 1.055 Overall usefulness 5.50 1.126 Recommendations page useful 5.36 1.197 Resources page useful 5.59 0.948 Willingness to enter condential info 5.77 1.389 Trusting the site 5.95 1.0210 Finding site technically reliable 6.09 0.9511 Use site for preventive scr. info 5.27 1.6012 Informative conversation w/clinician 5.41 1.6413 Following preventive screenings 5.27 1.4814 Start/maintain lifestyle changes 4.77 1.5615 More informed on health risks 5.36 1.4316 More in charge of own health 5.50 1.3717 More questions to physician 3.77 1.6218 Consequences of noncompliance 4.73 1.3519 Consequences of lifestyle 5.05 1.4620 Benets of compliance 5.18 1.19

A. Ant Ozok et al. / Appliedof this information and making it a more integral part in their in-teractions were lower. Patients were less willing to make radicalchanges in their lifestyles based onwhat they learned fromMySafe-T.net (Item 14). Section D included three questions that were du-plicates of previous questions for calculating the Cronbachs Alphacoefcients (Table 3).

As part of Section E, patients found the system easy to use (Item23) and the screens well-designed (Item 26). They had no difcultyunderstanding MySafe-T.nets content (Item 29) and found thedesigns consistent (Item 32). Overall, perceived user performanceand satisfaction levels with the system were very high. The ma-jority of the patients found screen instructions understandable(Item 30), screens well-designed (Item 27), and the PHR interesting(Item 28) and fast to work on (Item 31). Participants mostly did notthink they made a lot of mistakes (Item 24) and there were toomany task steps (Item 25). No signicant usability problems forMySafe-T.net were detected.

Participants overall satisfaction with the system (Item 36) wasfairly high (about 82% of perfect score). Theywould recommend thesystem to others (Item 33), thought the system was a good use oftheir time (Item 34), and wished to continue using it (Item 35).Mental demand (Item 39, 10- point scale) and overall perceivedeffort to complete the tasks (Item 40, 10- point scale) producedmoderate scores, which indicate that the system requires a mod-erate level of effort.

3.3. Qualitative analysis results

3.3.1. Patients views on the MySafe-T.net systemPositive patient perceptions regarding the system were cate-

gorized as information tailored to individual patients, continuity ofcare, patient activation and improved communication with careproviders. Negative perceptions and improvement suggestionsincluded use of medical terminology that cannot be understood bylay people easily, need for more tailored and individualized infor-mation, and difculty in remembering personal and family medicalhistories. Details are presented below.

3.3.1.1. Positive perceptions3.3.1.1.1. Information tailored to individual patients. During the

interviews, 8 patients reported that they found the tailored andinformative nature of the system based on medical and familyhistory helpful and informative-more informative than generic

# Description Mean S.D.

21 Benets of lifestyle 5.18 1.1922 Enhanced communication 5.32 1.4923 Ease of use of the site 6.27 0.9124 A lot of mistakes made 2.77 1.7625 Too many steps for some tasks 2.45 1.5326 Easy login 6.23 0.8527 Well-designed screens 5.86 1.0128 Generally interesting site 5.82 0.9829 Not difcult to understand 6.14 0.9230 Understandable instructions 5.95 1.0731 Can work fast on site 5.73 1.2132 Consistent designs 6.05 0.9333 Advise others to use site 5.59 1.0734 Good use of time 5.50 1.3435 Will continue using the site 5.45 1.4136 Satised overall 5.73 0.8637 Stressful to use 1.77 1.2838 Frustrating to use 2.00 2.3439 Mental demand 4.77 2.7540 Effort 3.77 2.54

nomics 45 (2014) 613e628 619sites-as evidenced by the following quote.

I like this system because when you put in all your information,then it takes you to specic Web sites that pertain to your issuesor disorders. It is more informative than other systems I haveseen because it is more personal. (Patient #4)

3.3.1.1.2. Continuity of care. Five patients indicated that MySafe-T.net has the potential to improve patient safety by giving themmore control over their own health information and by functioningas a reliable repository of preventive health-related information.They can use the information in MySafe-T.net as they move acrossdifferent care systems and care providers, which can result in morereliable and timely information transfer over the continuum of care.Patients found it useful that the system can alert their physicians ofpossible medicine interactions, prevent duplicate entries, andprovide another medium for communication with their careproviders.

3.3.1.1.3. Patient activation and improved communication withcare providers. Based on the information they learned fromMySafe-T.net, ten patients indicated they have additional questionsfor their primary care physicians regarding some tests, and thesystem allowed them to easily remember these questions. Thesystem is a nice place to start the doctors visit on (Participant #9).

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628620It made communicationwith care providers easier, particularly dueto the tailored recommendations and patient education provided,as exemplied by the following quote.

I should have a certain test done in ve years, but this systemtold me, because of my family history, that three years wouldprobably bemore appropriate for me. I want to talk to my doctorabout this. (Patient #17)

Other positive items identied by the patients included MySafe-T.net (1) focusing on key information, nuts and bolts, avoidingunnecessary details; (2) acting as a reminder for things they shoulddo such as eating better and exercising; (3) being useful because itallows them to track their health record and see possible healthdangers; (4) being easy to use and self-explanatory.

3.3.1.2. Negative perceptions and improvement suggestions3.3.1.2.1. More tailored and individualized information.

Three patients indicated the system could provide more specicinformation tailored to patients conditions. For example, two pa-tients hadmultiple results for a specic test and the system allowedthem to enter only one value:

[.] when I was in the preventive health screening test section,I was only able to enter the results of my most recent choles-terol.[.] However, my cholesterol was higher couple years agobut the system did not allow me to enter this information.(Patient #14)

3.3.1.2.2. Use of medical terminology. While generally patientsindicated the system was easy to use, three patients reported notunderstanding one or more of the terms used in the MySafe-T.net.For example, one patient did not understand the term tubal liga-tion when completing her medical history, while another had ahard time with the instructions for entering her medication infor-mation. Also, one patient indicated: I know I am allergic to eyepolytron hydroxy, but I did not know which category classied that.(Patient #8).

3.3.1.2.3. Remembering personal and family medical history.Two patients had some difculty remembering their own andfamily medical histories in enough detail to enter into the system:

The matter is trying to remember what your background is,what your familys background is. [.] It is a good tool as far as[.] things you probably should know about and you can alwaysadd them in later. (Patient #20)

Overall, interviews indicated that MySafe-T.net was perceivedby patients as a useful and easy to use system. To improve theacceptance of such a system, emphasis should be given to providingtailored information and recommendations to patients, and using alanguage that can easily be understood by laypeople.

3.3.2. Prospects for long-term use by patients based on follow-upinterviews at the two-week mark

Of the 16 participants who agreed to participate in the follow-upphone interview, nine indicated that they used the system in thetwo weeks since their physical exam. All nine used it to access in-formation and resources, and two used it to update their prolesand review the recommendations based on their updated proles.In the interview, participants were presented six statements andwere asked to rate their agreement with those statements on a 7-point Likert scale. Participants moderately agreed that using thesystem allowed them to have amore informative conversationwiththeir physicians (Mean 4.1.4, Std. Dev. 1.88).

Three participants indicated that the physician and/or themedical assistant found the printout potentially useful, with onemedical assistant indicating it would make things easier. Fourpatients indicated they found the output useful during theirconversations. One patient and one physician indicated the pre-ventive screening information was useful, and two indicated thedoctors became quite interested in how the printout was pro-duced and asked for more information. One physician readdirectly from the care provider letter and discussed each itemwiththe patient. One patient became interested in the electronic na-ture of the system and started using the clinics own electroniccharting system after their experience with MySafe-T.net. Oneindicated MySafe-T.net should send optional monthly reminder e-mail alerts for relling their medications, updating their infor-mation, and going to their upcoming laboratory tests and doctorsappointments.

There were few negative comments from patients in the follow-up interviews. One patient found the system irrelevant to theirhealth, one indicated the system was not accurate in its recom-mendations, and two indicated they were too young or toohealthy to use it. Another one indicated that it could be useful if itcould be shared between different hospitals. In spite of these fewnegative comments, all but three patients who participated in thefollow-up interviews stated they would be willing to use the sys-tem on a regular basis.

3.3.3. Care providers views on the MySafe-T.net systemOverall, both the participating primary care physicians and the

medical assistants indicated that the system can improve qualityand efciency in primary care clinics. Physicians found severalfunctions of the system useful, including medical histories of pa-tients and their families, tailored preventive screening recom-mendations, and medical education provided to patients, asevidenced by the following statement:

I think that the history part they enter is nice to have. I can seesomeone I didnt know walking in with one of these [letter forthe care provider], that would be really helpful. (Physician #3)

Both the physicians and medical assistants found the letter forthe care provider helpful as it provided preventive care screeningrecommendations based on each patients condition. Two physi-cians noted that patients who used the system had more detailedquestions during the examination. Other positive points includedthe time-saving aspect of the system, the system being a healthmotivator in both motivating patients to learn more about theirhealth and to keep a healthy lifestyle, and being a limited but goodtechnology aid. (Physician #3)

Finally something to save me time. [.] I dont have to explainthem everything, I mean everything, [.] with this. (Physician #2)

Maybe they can go look up [the Internet] to help them getinformed and be healthy, good health motivator. (Physician #3)

Regarding how to further improve MySafe-T.net, they recom-mended including both generic and brand-name medications forpatients to enter, have pop-up boxes providing more informationon conditions, tests and medical terms, and automatically link tosome information on best practices for better patient education.One physician indicated the system should connect to EMR (elec-tronic medical records) automatically and bi-directionally. Onephysician wanted to see the exact screening dates appearing in thiskind of EMR-integrated system.

It would be nice if they could. [be] potentially integratedwithany EMR. So, for both directions, the patient could still maintaintheir own record . when they go from place to place. (Physi-cian #2)

ErgoTable 7Overview of the qualitative and quantitative ndings: characteristics of the MySafe-T.net system perceived positively by its users and the recommendations forimprovement.

Characteristics desired and availablein MySafe-T.net

Patients Clinicians

Qualitative Quantitative Qualitative

Usefulness-related characteristicsSuggestions/reminders for

lifestyle changesU U U

Suggestions/reminders for earlyscreenings/tests

U U U

Informational web links U UInformation on specic health

conditionsU

Tracking of health records U U

Usability-related characteristicsSimple, parsimonious design U U

Characteristics desired and can beimproved/added in MySafe-T.net

Usefulness-related characteristicsMore individualized and tailored

health information for patientswith different illnesses andeducation levels

U U

Data entry being less relianton ones memory

U

Usability-related characteristicsSimpler, easy-to-understand

language, clear explanationof medical terminology

U

More structured and categorized U U

A. Ant Ozok et al. / AppliedIf it is connected, I could go over [the patients] screening datesand tell them what [screening] they soon need. (Physician #3)

The suggestions were enhancement-focused and for how tomake [the system] better. (Physician #2). One negative commentincluded the system being not for everyone and difcult.(Physician #1) Table 7 summarizes our recommendationsregarding both the structural and content-related PHR designcharacteristics based on this study, along with the information onwhich part of the study those recommendations were deducedfrom.

3.3.4. Proposed design characteristicsTable 8 presents a set of proposed design characteristics as a

result of our analyses of the qualitative and quantitative data ob-tained from patients and care providers. It should be noted that thecharacteristics on Table 8 are not the result of rigorous testing, asour study primarily aimed at understanding the usability andusefulness issues of a PHR system based on a specic case study.However, the data we collected allowed us to provide some insighton the design features that are desired by patients and care pro-viders, based on their needs, limitations and motivations. Therecommendations mostly focus on practical issues such aspersonalized structure, exibility in the number of patient entriesand interoperability with other systems, as well as possible futureexpansions of the system, such as creating push notications (in theform of e-mails and other tools), and syncing with the patientselectronic medical records. The proposed design features can beused as a starting point to develop PHR design guidelines producedfrom patient and caregiver perspectives.

patient entries

Perceived securityMore emphasis on the information

entered being secure andcondential

U4. Discussion

We examined a particular Web-based PHR system for criticalfactors concerning perceived usefulness and usability fromboth thepatients and providers perspectives in a case study. User percep-tions regarding the usefulness of this PHR system were high, withusers showing interest in keeping their health records online andusing them to get educated on health care issues. Patients reportedthat they were more aware of the relevant preventive healthscreening tests and procedures after using the MySafe-T.net. Theyfound the tailored recommendations for preventive screening andlifestyle changes helpful. Furthermore, both patients and careproviders generally found MySafe-T.net a good reminder of healthissues and a relevant information resource. Patients valued theinformation that motivates them to get screenings and tests ontime, as well as having their health information recorded elec-tronically and be easily accessible. Patients also pointed outsimplicity as a positive design factor. Furthermore, some patientsindicated that the PHR system contributed to improved commu-nication with their health care providers, allowing them to havemore effective conversations and ask targeted questions. In addi-tion, participating patients and care providers also expressed arelatively high level of trust in the system with keeping their pri-vate information secure and condential. Although patients foundthe system useful and usable, they also stated that they would feeluncomfortable relying solely on the systems recommendationsregarding their preventive health care. Patients unwillingness tomake important health care decisions based on the informationthey gathered from the site can be seen as one of the strengths ofthe system, as the aim of MySafe-T.net is not to replace but com-plement primary care visits. Patients should consult their physi-cians before making critical decisions related to their health care.Additionally, radical health behavior change is hard to achieve andin general requires multi-level interventions on multiple behaviors(Ory et al., 2002). The system is perceived as an informational toolrather than a tool offering guidelines on health behaviors andlifestyle choices. This view can guide researchers in developingbetter ways to present health information.

In general, care providers found the system promising forimproving preventive care. They indicated that having it as a pa-tients (incomplete) medical information resource is useful, and thesystem provided some information they otherwise did not knowabout the patients. They saw the letter for the care provider as atime saver. While patient Internet portals are relatively common(Weingart et al., 2006), the unique characteristics of this systemincluded providing more tailored, evidence-based content and ahigher level of interactivity. Overall, patients and care providersshowed substantial interest in adopting MySafe-T.net on a perma-nent basis.

The study also revealed several areas that can be focused on toimprove the usability and usefulness of such a system. While pa-tients valued the information obtained from the system, they re-ported that the system would be more valuable with even moretailored information. Furthermore, several patients found some ofthe terminology confusing and unfamiliar, and did not like thesystems high reliance on ones memory. These problems aredifcult to overcome. One potential solution is to integrate thesystem with the patients medical records, which is known to bechallenging from nancial, legal and technological perspectives(Angst and Agarwal, 2009; Bourgeois et al., 2008; Baird et al., 2011).

Although care providers found the letter for the care providerhelpful, some of them also indicated that this feature may be notsuitable for every patient. Furthermore, care providers indicatedthat some pieces of important information was lacking in the sys-

nomics 45 (2014) 613e628 621tem such as the information on generic medications. An ideal

pati

he pnd d

idern ch

ing

ot rns)

sly

t ed

am

Ergosystem would therefore need a more comprehensive database ofmedications, illnesses, health statuses and other information types.

Overall, using a multi-method approach and evaluating MySafe-T.net from both the direct and indirect users perspectives providedvaluable and in-depth information regarding the positive andnegative design characteristics of such a system. A PHR system canbe useful for both patients and care providers and used by patientson a regular basis as a reliable information resource and healtheducation tool. This nding is parallel to the ndings by Denton(2001) and Pratt et al. (2006) who indicated that a well-presented electronic environment can increase PHR use andmotivate patients to manage their health information. The ndingssupport and build upon the health care literature by determiningthe role PHR can play inmaking patientsmore active participants inmanaging their own health. A usable and reliable Web-based sys-tem that presents relevant content on maintaining a healthy life-style, improving communication with caregivers, and emphasis onpreventive health screening can be adopted by patients. The key forhigh adoption is to design a user-friendly and relevant system thatcan provide tailored and comprehensive, yet easy to understand

Table 8Proposed design characteristics for a PHR system for preventive health care.

Proposed design characteristics based on patient responses

Provide summary information on the letter to care provider, including highlights ofscreening

Provide tailored and personalized information based on the medical history and tProvide personalized, specic Web site recommendations on the patients issues aSupport interoperability (compatibility) with other systemsEnsure consistency with other systems (not explicitly stated by patients and provDesign as a reliable repository of patient health information where the patient is iDesign as a virtual place to start the doctors visitPresent key, nuts and bolts information in the form of a practical reminder on eat

keep track of vital statistics such as blood pressure and blood glucose levelsUse simple language, avoid medical terminology as much as possibleProvide exible number of entries by the patient, with entries being non-forced (nPush important information created by the system to users (e.g., e-mail noticatio

Proposed design characteristics based on provider responsesPresent warning signs when patient deviates from recommended norms, consciou

habits are bad, they do not do any exercise, or their blood pressure is too high)Provide automatic, bi-directional syncing with patient EMRsDesign the system as a long-term tool, with one of the primary goals being patienDesign as a time saver (for both patients and caregivers) and a health motivatorProvide a comprehensive database of medications (including generic medication n

A. Ant Ozok et al. / Applied622recommendations to patients based on scientic evidence. Wefocused on preventive care due to the literature indicating thatemploying PHR may improve patient awareness of preventive careguidelines and compliance rateswith the recommended preventivescreening tests and other relatedmeasures. Human factors researchshows that for a technology to positively affect compliance andperformance, one of the rst requirements is for the technology tobe accepted as useful and usable (Karsh et al., 2006; Krist andWoolf, 2011). A PHR that is perceived as useful and usable, inturn, may result in improved health outcomes (Kahn et al., 2009). Inthis study, however, we did not evaluate the impact of a PHR onpatients long-term behavior change (e.g., increased compliancewith recommended diets, increased medication adherence) andoutcome measures on improvements in preventive care (e.g.,reduced morbidities). Future research should evaluate such impact.

The methodological contribution of our study mainly lies in themulti-method approach and the patient-centric design of theexperiment. The quantitative methodologies allowed us to deter-mine patients views of a PHR system with a focus on preventivescreening with regards to well-known constructs that affect tech-nology acceptance (for example, Denton, 2001), while the qualita-tive methodologies allowed us to determine patients andcaregivers attitudes on the usefulness of a PHR system focused onimproving preventive care and suggestions on how such systemscan be improved both content- and usability-wise in future designs.Additionally, there are a limited number of studies in the literaturethat evaluated patient-centric HIT so far using a human factorsengineering approach. Our study was aimed at understanding theperceptions of usability and usefulness of PHR systems based on aspecic system from both the patients and caregivers perspec-tives. Findings from this case study can help researchers in oper-ationalizing usefulness and usability in future PHR design andevaluation studies. Human factors research needs to take multipleusers for the same technology/interface into consideration in manyhealth care settings (Carayon et al., 2010), and qualitative (Gurseset al., 2009; Karsh et al., 2006) and quantitative (Miller et al.,2006) methodologies may need to be used together for investi-gating different user groups perspectives on such systems.

Experiences with many of the technologies introduced intohealth care, especially information technologies, have been medi-ocre at best (Karsh et al., 2010). Most of the health informationtechnologies have been developed under the assumption of a linearand predictable nature of work in health care (Gurses et al., 2011).

Qualitative results Quantitative results

ent history related to preventive U

atients own input Uisorders U

U

s) Uarge U U

U

better and exercising, as well as U U

U U

equired to ll out) UU

or not (such as if their eating U

U

ucation UU

es), illnesses, health statuses U

nomics 45 (2014) 613e628However, health care is, by its nature, team-based, non-linear,event-driven, complex, and full of deviations. Given the complexand safety-critical nature of health care, more systematic andscience-based approaches (Leveson, 2001) that build upon strongpartnerships among product developers, human factors and us-ability engineers, clinicians, patients, and regulators are needed.With the move from paper to electronic record environments, thistype of partnership is a must, not an option to improve quality andsafety of health care.

4.1. Limitations

This study had several limitations. First,we used a particular PHRas a case study, and our ndings may not be generalizable to otherPHR systems. A wide variety of Web-based PHR systems arecurrentlyavailable thatmayvary in the features theyoffer. Our studyfocused on the usability and usefulness issues involving a specicPHR. Due to the fragmented nature of currently available PHR de-signs,wedid not fully assess howsimilarMySafe-T.nets features areto those currently available in other PHR systems. Our study also didnot evaluate in-situ or long-term PHR usability or perceptions ofusefulness. For assessing more general perceptions of health con-sumers regarding PHR systems, differentmethods (e.g., a large-scale

survey of health care consumers who have used PHRs) will need tobe used with a larger sample size. Second, due to the exploratorynature of our study, we did not evaluate the PHR system after it hasbeen used on a long-term basis by patients and care providers.Rather, participants indicated their opinions concerning the systemafter a structured examination of it (with a follow-up interview). It ispossible that participants may have additional and/or differentopinions after using the system on a long-term basis.

Third, our sample size may be viewed as relatively small forcollecting data via a questionnaire. However, we considered asample size of 22 as adequate due to the exploratory nature of thestudy that focused on determining current general trends and at-titudes toward a PHR system aimed at improving preventive care.Additionally, we contacted thirty-six consecutive patients whocalled the clinic for an annual physical exam appointment, resultingin a fairly high response rate of 61%. The survey helped determinethe potential usefulness and usability issues concerning the system,and complemented the data obtained through qualitative methods.It should be noted, however, that a higher number of participantsthat have a broader spectrum including different ailment severitiesand types, as well as different age and education levels, could havehelped in identifying new design characteristics or rening theexisting ones for the Web-based PHR systems to make themappealing to a broader population. With this sample size, partici-pant diversity may not have been adequately captured to furthergeneralize our ndings. Also, there is a body of work focusing onPHR adoption differences due to culture (Hartley, 2004; Fuji et al.,2008; Urowitz et al., 2008; Kahn et al., 2009). Our study did notconsider the role of culture in PHR adoption. Additionally, while our

4.2. Future work

While the PHR systemwas found useful, patients indicated theyprefer an even more tailored system. The recommendations andresources are useful, but sometimes rely on external links. Anagent-based approach, where intelligent programs would dosearches for relevant information in similar patterns to humans(Luck and DInverno, 2001), may increase the accuracy of the rec-ommendations and resources, and the information the site pro-vides can be enhanced with more comprehensive databases thatcan be integrated into the system. Moreover, a future system caninterface with the provider-based electronic health records of thepatients and combine the two information sources. PHR systemsmay also expand to mobile environments, and the concept oftechnology acceptance may go through some changes while beingadapted to health care environments (Holden and Karsh, 2010).

Acknowledgments

This study was funded by the Medical Industry LeadershipInstitute of the University of Minnesota. Dr. Ayse P. Gurses wassupported in part by the Agency for Healthcare Research andQuality K01 grant #HS018762. We would like to thank Dr. DavidMoen and the participating primary care clinic care providers fortheir support of the study. Supported by Dept. of Veterans Affairs,Veterans Health Administration, Health Services Research and

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628 623case study was based on a PHR system focused on preventivehealth, we did not measure the impact of PHR on health outcomes.Finally, 16 out of the 22 patients invited participated in the follow-up interviews, and it is possible that those who did not participatein the follow-up may have different views about the system.A1. The past medicaAppendix A. Sample screen shots from the MySafe-T.netsystemDevelopment Service (CDA 11-201/CDP 12-255). The viewsexpressed in this article are those of the authors and do notnecessarily reect the position or policy of the Department ofVeterans Affairs or the United States government.l history screen.

A2. The cholesterol data entry screen.

A3. The family history screen.

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628624

A4. Updated screening recommendations based on risk factors.

A5. Link to the relevant web site is provided.

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628 625

ecom

ErgoA6. Lifestyle r

A. Ant Ozok et al. / Applied626Appendix B. Patient interview guide

(Usability questions are marked with [1] and usefulness ques-tions are marked with [2].)

1 Do you nd the system easy to use? Why or why not? [1]2 Would you use this system to learn more about and getrecommendations on preventive health screening? [2]

3 If you had control over the design of a perfect computer toolto help you with your health care management, specicallyrelated to preventive health screening and lifestyle choices,what would it be like? What would be on it and why? Whatissues are important to you that you would like the de-velopers of these tools to consider? [1, 2]

5 You seemed to like ... [noted during session]... what did youlike about it? [1,2]

6 What were the major positive aspects of the system? [1,2]7 What were the major difculties you had with this system?[1,2]

8 In your opinion, how can this system be improved? [1,2]9 If this systemwere available to you at home, would you use iton a regular basis? Why or why not? [1,2]

10 Do you plan on asking your physician any questions based onwhat you have learned for this web site? Yes No If yes whatare you planning to ask? [2]

11 What other suggestions do you have to improve this system?[1,2]

Appendix C. Caregiver focus group guide

(Usability questions are marked with [1] and usefulness ques-tions are marked with [2].)mendations.

nomics 45 (2014) 613e628- What are your thoughts about this Web site? Would yourecommend it to your patients? To your colleagues?Why or whynot? [2]

- Do you have any concerns about this Web site? If so, what? [1,2]- How do you think this Web site can affect (1) patient activation(2) patient-physician communication, (3) shared decision-making, (4) compliance with preventive screening, (5) changesin lifestyle choices (e.g., regular exercise, healthy diet, smokingcessation)? [2]

- If you had control over the design of a perfect informationtool for preventive health screening, what would it belike? What would be on it and why? What issues areimportant to you that you would like the developer toconsider? [1,2]

- What are your thoughts about the letter to the doctor func-tionality of the system? How do you think this letter can helpyou? How would you like to modify it? [2]

- What is the process for informing patients about preventivehealth screening? [2]

- What do you do to improve the lifestyle choices of the patients?Do you think this Web site can help you? How? [2]

- If you had all the resources you needed, how would youimprove your current methods to discuss preventivescreening? [2]

- What would be most helpful for you at your practice to improveyour ability to introduce and discuss preventive healthscreening with patients? [1,2]

- What would be most helpful for you at your practice to improveyour ability to introduce and discuss lifestyle choices with yourpatients? [1,2]

- Do you feel it is feasible for your patients to use this Web site?[1,2]

A. Ant Ozok et al. / Applied ErgoReferences

Angst, C.M., Agarwal, R., 2009. Adoption of electronic health records in the presenceof privacy concerns: the elaboration likelihood model and individual persua-sion. MIS Quarter. 33 (2), 339e370.

Avorn, J., Shrank, W.H., 2008. A substantial cause of preventable illneess. Biomed. J.335 (7650) (published online April 8, 2008).

Baird, A., North, F., Raghu, T.S., 2011. Personal health records (PHR) and the future ofthe physicianepatient relationship. In: Proceedings of the 2011 ACM iConfer-ence, Seattle, WA, USA, February 8e11, 2011, pp. 281e288.

Ball, J., Smith, C., Bakalr, R., 2006. Personal health records: empowering consumers.J. Healthc. Inform. Manage. 21 (1), 76e86.

Bodenheimer, T., Lorig, K., Holman, H., Grumbach, K., 2002. Patient self-management of chronic disease in primary care. J. Am. Med. Assoc. 288 (19),2469e2475.

Bourgeois, F., Taylor, P., Emans, S., Nigrin, D., Mandl, K., 2008. Whose personalcontrol? Creating private, personally controlled health records for pediatric andadolescent patients. J. Am. Med. Inform. Assoc. 15 (6), 737e743.

Brennan, T.A., 2000. The Institute of Medicine report on medical errors e could it doharm? (Sounding Board). N. Engl. J. Med. 324 (15).

Britto, M., Wimberg, J., 2009. Pediatric personal health records: current trends andkey challenges. Pediatrics 123 (Suppl. 2), S97eS99.

Buckle, P., Clarkson, P.J., Coleman, R., Ward, J., Anderson, J., 2006. Patient safety,systems design and ergonomics. Appl. Ergon. 37 (4), 491e500.

Carayon, P., 2010. Human factors in patient safety as an innovation. Appl. Ergon. 41(5), 657e665.

Carayon, P.,Karsh,B., Cartmill, R., et al., July2010. IncorporatingHealth IT intoWorkowRedesign: Request for Information Summary Report. AHRQ publication no. 10-0074-EF. The Agency for Healthcare Research and Quality, Rockville, MD.

Cronbach, L., 1990. Essentials of Psychological Testing. Harper & Row Publishing,New York, NY.

Davis, F., 1989. Perceived usefulness, perceived ease of use, and user acceptance ofinformation technology. MIS Quarter. 13 (3), 319e340.

Davis, F., Bagozzi, R., Warshaw, P., 1989. User acceptance of computer technology: acomparison of two theoretical models. Manage. Sci. 35, 982e1003.

Denton, I.C., 2001. Will patients use electronic personal health records? Responsesfrom a real-life literature. J. Healthc. Inform. Manage. 15 (3), 251e259. Fall 2001.

Eriksen, H., Ursin, H., 2004. Subjective health complaints, sensitization, and sus-tained cognitive activation (stress). J. Psychom. Res. 56 (4), 445e448.

Fuji, K.T., Galt, K.A., Serocca, A.B., 2008. Personal health record use by patients asperceived by ambulatory care physicians in Nebraska and South Dakota: across-sectional study. Perspect. Health Inform. Manage./AHIMA, Am. HealthInform. Manage. Assoc. 5.

Gefen, D., 2000. E-commerce: the role of familiarity and trust. Omega 28, 725e737.Goth, G., 2008. Power to the patients. IEEE Distrib. Syst. Online 9, 1e4.Greenhalgh, T., Hinder, S., Stramer, K., Bratan, T., Russell, J., 2010. Adoption, non-

adoption, and abandonment of a personal electronic health record: casestudy of HealthSpace. Br. Med. J. 341, c5814.

Gurses, A., Xiao, Y., Hu, P., 2009. User-designed information tools to supportcommunication and care coordination in a trauma hospital. J. Biomed. Inform.42 (4), 667e677.

Gurses, A., Ozok, A., Pronovost, P., 2011. Time to accelerate integration of humanfactors and ergonomics in patient safety. BMJ Qual. Saf. 21 (4), 347e351.

Hackman, J., Oldham, G., 1980. Work Redesign (Organization Development). Pren-tice Hall, Upper Saddle River, NJ.

Hargreaves, J., 2010. Will electronic personal health records benet providers andpatients in rural America? Telemed. e-Health 16 (2), 167e176.

Hartley, D., 2004. Rural health disparities, population health, and rural culture. Am.J. Pub. Health 94 (10), 1675e1678.

Hibbard, J.H., Stockard, J., Mahoney, E.R., Tusler, M., 2004. Development of the pa-tient activation measure (PAM): conceptualizing and measuring activation inpatients and consumers. Health Serv. Res. 39 (4p1), 1005e1026.

Hibbard, J.H., Mahoney, E.R., Stockard, J., Tusler, M., 2005. Development and testingof a short form of the patient activation measure. Health Serv. Res. 40 (6), 1918e1930.

Holden, R.J., Karsh, B.-T., 2010. The technology acceptance model: its past and itsfuture in health care. J. Biomed. Inform. 43, 159e172.

Hsieh, H., Shannon, S., 2005. Three approaches to qualitative content analysis. Qual.Health Res. 15, 1277e1288.

Human Performance Research Group, 1997. NASA Task Load Index. NASA AmesResearch Center, Moffett Field, CA.

International Standards Organization (ISO), 2009. Personal Health Records: De-nition, Scope and Context. Draft Technical Report. Geneva, 2009.

Johnson, R., Onwuegbuzie, A., Turner, L., 2007. Toward a denition of mixedmethods research. J. Mixed Methods Res. 1 (2), 112e133.

Kahn, J.S., Aulakh, V., Bosworth, A., 2009. What it takes: characteristics of the idealpersonal health record. Health Affairs 28 (2), 369e376.

Karsh, B., Hamilton Escoto, K., Beasley, J., Holden, R., 2006. Towards a theoretical tomedical error reporting system research and design. Appl. Ergon. 37 (3), 283e295.

Karsh, B.T., Weinger, M.B., Abbott, P.A., Wears, R.L., 2010. Health information tech-nology: fallacies and sober realities. J. Am. Med. Inform. Assoc. 17 (6), 617e623.

Krist, H., Woolf, S., 2011. A vision for patient-centered health information systems.

J. Am. Med. Assoc. 305 (3), 300e301.Krist, A., Peele, E., Woolf, S., Rothemich, S., Loomis, J., Longo, D., Kuzel, A., 2011.Designing a patient-centered personal health record to promote preventivecare. BMC Med. Inform. Decis. Mak. 11 (73), 1e11.

Leveson, N., 2001. Safeware: System Safety and Computers. Addison-Wesley,Boston.

Lin, H., Choong, Y., Salvendy, G., 1997. A proposed index of usability: a method forcomparing the relative usability of different software systems. Behav. Inform.Technol. 16 (4e5), 267e277.

Liu, L.S., Shih, P.C., Hayes, G.R., 2011. Barriers to the adoption and use of personalhealth record systems. In: Proceedings of the 2011 ACM iConference, Seattle,WA, USA, February 8e11, 2011, pp. 363e370.

Lorig, K.R., Sobel, D.S., Stewart, A.L., et al., 1996. Evidence suggesting that a chronicdisease self-management program can improve health status while reducinghospitalization: a randomized trial. Med. Care 37 (1), 5e14.

Luck, M., DInverno, M., 2001. A conceptual framework for agent denition anddevelopment. Comput. J. 44 (1), 1e20.

Malamateniou, V.K., Vassilacopoulos, G., 2010. An agent-based application of per-sonal health record in homecare. In: ACM Proceedings of the 3rd InternationalConference on Pervasive Technologies Related to Assistive Environments, June23e25, 2010, Samos, Greece.

Manne, S., Markowitz, A., Winawer, S., Meropol, N., Haller, D., Rakowski, W., Babb, J.,Jandorf, L., Jan. 2002. Correlates of colorectal cancer screening compliance andstage of adoption among siblings of individuals with early onset colorectalcancer. Health Psychol. 21 (1), 3e15.

Martin, J.L., Clarke, D.L., Morgan, S.P., Crowe, J.A., Murphy, E., 2011. A user-centeredapproach to requirements elicitation in medical device development: a casestudy from an industry perspective. Appl. Ergon. 43 (1), 184e190.

McGlynn, E., Asch, S., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., Kerr, A., 2003.The quality of health care delivered to adults in the United States. N. Engl. J.Med. 348, 2635e2645.

Miller, A., Engt, C., Tate, R., Yassi, A., 2006. Evaluation and effectiveness of portableceiling lifts in a new long-term care facility. Appl. Ergon. 37 (3), 377e385.

Montague, E., 2010. Validation of a trust in medical technology instrument. Appl.Ergon. 41 (6), 812e821.

Morbidity and Mortality Weekly Report, Cancer Screening United States, 2010,January 27, 2012. Centers for Disease Control and Prevention, pp. 41e45, 61(3).Accessed online at: www.cdc.gov/mmwr/preview/mmwrhtml/mm6103a1.htm?s_cidmm6103a1_w (accessed 21.06.12).

Mosca, L., Mochari, H., Christian, A., Berra, K., Taubert, K., Mills, T., Burdick, K.A.,Simpson, S.L., 2006. National study of womens awareness, preventive action,and barriers to cardiovascular health. Circulation 113, 525e534.

Nazi, K.M., Hogan, T.P., et al., 2009. Embracing a health services research perspectiveon personal health records: lessons learned from the VA M HealtheVet system.J. General Int. Med. 25 (Suppl. 1), 62e67.

Nielsen, J., 1993. Usability Engineering. Morgan-Kauffman, San Francisco, CA.Ory, M.G., Jordan, P.J., Bazzarre, T., 2002. The Behavior Change Consortium: setting

the stage for a new century of health behavior-change research. Health Educ.Res. 17.5, 500e511.

Palen, L., Aalokke, S., 2006. Of pill boxes and piano benches: home-made methodsfor managing medication. In: Proceedings of the 2006 20th Anniversary Con-ference on Computer Supported Cooperative Work, pp. 79e88.

Patton, M.Q., 2002. Qualitative Research and Methods, third ed. Sage Publications,Thousand Oaks.

Perlin, J., et al., 2004. The Veterans health Administration: quality value, account-ability, and information as transforming strategies for patient-centered care.Am. J. Manage. Care 10, 828e836.

Peterson, A.M., Dragon, C.J., 1998. Improving medication adherence in patientsreceiving home health. Home Healthc. Consult. 5 (9), 25e27.

Powsner, S.M., 1998. Opportunities for and challenges of computerization. Lancet352 (9140), 1617e1622.

Pratt, W., Unruh, K., Civan, A., Skeels, M., 2006. Personal health information man-agement. Commun. ACM Assoc. Comput. Machine. 49 (1), 51e55.

Salkeld, G., 2006. What are the benets of preventive health care. Healthc. Anal. 6(2), 106e112.

Scherger, J.E., 2005. Primary care needs a new model of ofce practice. Bio-Med. J.330 (7504), E358eE359.

Shea, S., DuMouchel, W., Bohamonde, L., 1996. A meta-analysis of 16 randomizedcontrolled trials to evaluate computer-based clinical reminder systems forpreventive care in the ambulatory setting. J. Am. Med. Inform. Assoc. 3 (6),399e409.

Shneiderman, B., 1992. Designing the User Interface: Strategies for Effective Hu-maneComputer Interaction. Addison-Wesley, New York, NY.

Srinivasan, A., 2006. Keeping online personal records private: security and privacyconsiderations for Web-based PHR systems. J. Am. Health Inform. Manage.Assoc. 77 (1), 62e68.

Steele, R., Lo, A., 2009. Future personal health records as a Foundation forComputational Health. Lecture Notes Comput. Sci. 5593, 719e733.

Sunyaev, A., Chornyi, D., Mauro, C., 2010. Evaluation framework for personal healthrecords: Microsoft HealthVault vs. Google Health. In: 43rd Hawaii InternationalConference on System Sciences (HICSS), 5e8 Jan. 2010, Honolulu, HI, pp. 1e10.

Tang, P.C., Lansky, D., 2005. The missing link: bridging the patient-provider healthinformation gap. Health Affairs 24 (5), 1290e1295.

Tang, P.C., Ash, J.S., Bates, D.W., Overhage, J.M., Sands, D.Z., 2006. Personal health

nomics 45 (2014) 613e628 627records: denitions, benets, and strategies for overcoming barriers to adop-tion. J. Am. Med. Inform. Assoc. 13 (2), 121e126.

The Markle Foundation, 2003. Connecting for Health. The personal health workinggroup nal report, July 1, 2003.

U. S. Department of Health and Human Services, 2001. Healthy People 2010. Un-derstanding and Improving Health. Government Printing Ofce, Washington,DC: U.S.

Ueckert, F., Goerz, M., Ataian, M., Tessmann, S., Prokosch, H.-U., 2003. Empower-ment of patients and communication with health care professionals through anelectronic health record. Int. J. Am. Med. Inform. Assoc. 70 (2), 99e108.

Urowitz, S., et al., 2008. Is Canada ready for patient accessible electronic healthrecords? A national scan. BMC Med. Inform. Decis. Mak. 8 (1), 33.

U.S. Preventive Services Task Force, 2007. Accessed online at: http://www.ahrq.gov/CLINIC/uspstx.htm (accessed 18.06.08).

Van den Haak, M., De Jong, M., Schellens, P., 2003. Retrospective vs. concurrentthink- aloud protocols: testing the usability of an online library catalogue.Behav. Inform. Technol. 22 (5), 339e351.

Venkatesh, V., Davis, F., 2000. A theoretical extension of the technology acceptancemodel: four longitudinal eld studies. Manage. Sci. 46 (2), 186e204.

Von Korff, M., Moore, J.E., Lorig, K., et al., 1998. A randomized trial of a lay person-led self-management group intervention for back pain patients in primary care.Spine 23 (23), 2608e2615.

Walker, J., Carayon, P., 2009. From tasks to processes: the case for changing healthinformation technology to improve health care. Health Affairs 28 (2), 467e477.

Weinberg, D.S., Turner, B.J., Wang, H., Myers, R.E., Miller, S., June 2004. A survey ofwomen regarding factors affecting colorectal cancer screening compliance.Prevent. Med. 38 (6), 669e675.

Weingart, S., Rind, D., Toas, Z., Sands, D., 2006. Who uses the patient Internetportal? The PatientSite experience. J. Am. Med. Inform. Assoc. 13, 91e95.

Zuckerman, A.E., Kim, G.R., 2009. Personal health records. Health Inform. 4, 293e301.

A. Ant Ozok et al. / Applied Ergonomics 45 (2014) 613e628628

Usability and perceived usefulness of personal health records for preventive health care: A case study focusing on patients ...1 Introduction2 Methodology2.1 Description of the MySafe-T.net system2.2 Study site and sample2.3 Study procedures2.3.1 Data collection from patients2.3.2 The Personal Health Records Survey (PHRS)2.3.3 Patient interviews2.3.3.1 Face-to-face patient interviews in the clinic2.3.3.2 Follow-up patient phone interviews

2.3.4 Data collection from care providers2.3.5 Data analysis2.3.5.1 Quantitative analysis2.3.5.2 Qualitative analysis2.3.5.3 Proposed design characteristics

3 Results3.1 Participant demographics3.2 Descriptive statistics on elements of PHRS3.3 Qualitative analysis results3.3.1 Patients' views on the MySafe-T.net system3.3.1.1 Positive perceptions3.3.1.1.1 Information tailored to individual patients3.3.1.1.2 Continuity of care3.3.1.1.3 Patient activation and improved communication with care providers

3.3.1.2 Negative perceptions and improvement suggestions3.3.1.2.1 More tailored and individualized information3.3.1.2.2 Use of medical terminology3.3.1.2.3 Remembering personal and family medical history

3.3.2 Prospects for long-term use by patients based on follow-up interviews at the two-week mark3.3.3 Care providers' views on the MySafe-T.net system3.3.4 Proposed design characteristics

4 Discussion4.1 Limitations4.2 Future work

AcknowledgmentsAppendix A Sample screen shots from the MySafe-T.net systemAppendix B Patient interview guideAppendix C Caregiver focus group guideReferences