monitoring consumer satisfaction with inpatient service delivery: the inpatient evaluation of...

Monitoring consumer satisfaction with inpatient service delivery: the Inpatient Evaluation of Service Questionnaire

Thomas Meehan, Helen Bergen, Terry Stedman

Objective:

To report on the development, testing and psychometric properties of a briefconsumer satisfaction measure for use with psychiatric inpatients.

Method:

Focus group discussions with inpatients were used to develop a pool of itemsrelated to satisfaction with hospital stay. A second cohort of 72 inpatients was invited to ratethe 51 items that emerged for importance in contributing to satisfaction. Mean importancescores highlighted 20 items that were subsequently framed into neutrally worded statements.A draft questionnaire comprising these statements was introduced, on a trial basis, in a rangeof inpatient facilities.

Results:

Factor analysis of 356 completed questionnaires yielded three factors comprisinga staff-patient alliance; doctor/treatment issues; and an environmental component. Psycho-metric properties include good response variability and high internal consistency.

Conclusions:

The Inpatient Evaluation of Service Questionnaire addresses many of theshortcomings of existing satisfaction measures. It was developed through extensive con-sumer involvement, it is simply worded, easy to score and appears to perform well with acuteand rehabilitation inpatients.

Key words:

Australian and New Zealand Journal of Psychiatry 2002; 36:807–811

psychiatric inpatient, satisfaction, scale development.

Consumer feedback is now widely promoted in Aus-tralia as a means of informing the planning, delivery andevaluation of mental health services [1–4]. The value ofincluding the patient’s perspective on the services pro-vided is becoming increasingly recognized at a nationaland statewide level. The ‘measurement of patient satis-faction and patient experience of health services, partic-ularly with respect to outcomes’ is one of the majoractivities outlined in the Quality Improvement and

Enhancement Plan for Queensland Health [4]. Althoughthere is no universally accepted method of assessingpatient satisfaction, the use of self-completed question-naires by patients is commonly employed.

The increasing use of satisfaction surveys in themental health field has been driven by the move towardsconsumer participation, the need to provide data forquality assurance/accreditation purposes and as ameasure of treatment outcome [2,5]. While the rush tomonitor patient satisfaction has resulted in an exponen-tial increase in the number of survey instruments avail-able, many of these are poorly designed and suffer fromquestionable validity and reliability [6,7]. A recentreview of satisfaction surveys in the US found that only11% tested interim reliability and only 5% used factoranalysis in their development [7].

The majority of instruments used in the mental healthfield have been developed through items generated byservice providers [7]. In the absence of consumer input,

Thomas Meehan, Senior Lecturer (Correspondence); Terry Stedman, Direc-tor of Clinical Services

Department of Psychiatry, University of Queensland & Service Evalu-ation, & Research Unit, Wolston Park Hospital, Wacol, Queensland4076, Australia. Email: [email protected]

Helen Bergen, Research Officer

Service Evaluation & Research Unit, Wolston Park Hospital, Wacol,Australia

Received 9 October 2001; revised 18 April 2002; accepted 25 June 2002.

808 CONSUMER SATISFACTION WITH INPATIENT SERVICE

many of these existing instruments suffer from low ‘con-tent’ and ‘face’ validity by failing to capture aspects ofsatisfaction particularly relevant and important to con-sumers of mental health services [8–10]. Moreover,purely quantitative measures may fail to uncover seriousdissatisfaction in specific areas. As illustrated by studiesof inpatient psychiatric services in the UK [11,12], satis-faction was extremely positive when assessed using aglobal quantitative scale. Yet in semistructured inter-views, these same patients expressed significant dissatis-faction with many aspects of their treatment [12]. Criticsurge for greater consumer input in the development ofinstruments and the use of open-ended questions tocapture the values and experiences of the patients them-selves [5,13].

The high level of patient satisfaction (typically 75–90%satisfied) provided by many of the surveys in current useis of concern [14]. This lack of variation in responses islikely to be an artefact of the scale design rather than trueperceptions of satisfaction [5]. Many of the surveysreviewed use the ‘yes/no’ response format which doesnot allow for the dispersion of responses at the positiveend of the scale. Ware and Hays found that the ‘E5’format (poor, fair, good, very good and excellent) pro-vided greater response variability and superior predictiveproperties for a number of patient behaviours [15].

Finally, difficulties in conceptualizing satisfaction hasgiven rise to longer survey instruments as researchers tryto capture anything that might contribute to satisfaction[6]. Since inpatient care is now restricted to those indi-viduals requiring stabilization during periods of acutecrisis [16], self-report measures need to be brief, simplyworded and easily administered. However, instrumentsdeveloped in Australia tend to be rather long [17,18],focus on community services [2] or specific clinicalconditions/groups [1,3].

The current study was designed to advance our under-standing of the factors underpinning patient satisfactionin the inpatient setting and to address many of the short-comings of previous survey development. We describethe development and testing of a brief satisfactionmeasure for inpatients, the Inpatient Evaluation ofService Questionnaire (IESQ).

Method

Instrument development employed three separate but related phases.Phase I involved focus group discussions with 66 inpatients at threeacute care units. The aim of the discussion groups (n = 8) was to gener-ate a pool of items related to patient satisfaction with hospital stay. Dis-cussion groups were ceased when participants failed to raise new/additional items to those already identified in previous groups. Thegroup discussions were conducted by the first author (TM) and wereguided by open-ended questions such as, ‘What do you like most/least

about your current stay in hospital?’ and ‘If you could change one thingto make your stay more pleasant what would it be?’ While this proce-dure generated an extensive pool of issues, these were summarizedaround core themes as outlined by Sim [19]. Three service aspects con-sidered important in the literature [3,20] but not raised in the group dis-cussions were added. These included ‘the attention the staff gave toyour concerns and worries’, ‘the standard of privacy in your ward’, and‘the groups/activities provided by the staff’. This resulted in a total of51 items.

In Phase II, a second sample of 72 patients from the same threeacute units was asked to rate the 51 items in terms of importance(1 = ‘not at all important’ to 5 = ‘extremely important’) in contributingto their satisfaction with hospital stay. The items rated most importantin determining satisfaction included ‘being respected by staff’(mean = 4.35) and the ‘quality of service provided by the nursing staff’(mean = 4.24). The ‘number of patients in the ward’ was rated leastimportant (mean = 1.95). Twenty items with a mean importance scoreof ‘3’ or greater were retained. These 20 items were used to constructthe current questionnaire. Additional questions were included tocollect relevant demographic information, to evaluate behaviouralintentions and to allow for freehand comments. The final 29 itemquestionnaire is structured as follows:– 20 items concerning treatment and care, and the services offered by

the hospital, rated using the ‘E5’ format (‘poor’, ‘fair’, ‘good’, ‘verygood’, ‘excellent’);

– 1 item rating overall satisfaction with hospital stay (‘E5’ format);– 2 items rating behavioural intentions (advise a friend with similar

problems to come to the hospital, and intent to return to the hospitalif they had similar problems);

– 2 open-ended questions that enabled patients to provide feedback onaspects of the hospital stay that they liked most and/or liked least;and

– 4 demographic items that have been found in the literature to influ-ence satisfaction (age, gender, number of previous admissions, timein hospital since admission).During Phase III the draft questionnaire was administered to

494 consecutive inpatients who were approaching discharge in acute(n = 3) and rehabilitation (n = 2) facilities. The rehabilitation facilitieswere included to assess the application of the instrument in thisalternate inpatient environment. Three hundred and fifty-six (72%)completed surveys were returned.

During all three phases of the study, patients were excluded if theirstay was less than 7 days. It was felt that exposure of less than 7 dayswould be too brief for patients to assess the inpatient environment andto make valid judgements about aspects of satisfaction [12]. Patientswho were readmitted during the study period were not invited toparticipate in the study again.

Results

Initial analysis explored differences in satisfaction scores for patientsacross the service settings (acute and rehabilitation). Scores for each ofthe 20 scaled items were summed to provide an overall satisfactionscore for each patient. While acute care patients (n = 195) were moresatisfied overall (mean total score = 60.8 vs 55.6), differences betweenthe two service settings were not statistically significant (t = 1.02;df = 349; p > 0.05). Indeed, agreement between the mean total scoresof the rehab cohort and the total sample was significant (Pearson’s

T. MEEHAN, H. BERGEN, T. STEDMAN 809

r = 0.51). Consequently, the completed instruments from both settings(n = 356) were combined and analysed as a single data set from thatpoint.

To examine the response variability of the IESQ, the 20 scaled itemswere subjected to a frequency analysis. The total scores for the itemsshowed good dispersion of responses: 16.6% responded with anaverage rating of ‘poor’; 19.7% ‘fair’; 35.7% ‘good’; 14.7% ‘verygood’; and 13.3% ‘excellent’. The single item rating ‘overall stay inthis hospital’ correlated strongly with the total/summed score of theother 20 items (Pearson’s r = 0.7828, p < 0.005).

Principal components factor extraction (set at 0.40) with varimaxrotation resulted in three factors with eigenvalues greater than one(9.30, 1.29 and 1.21). Considered together, the three factors accountedfor 59% of the total variance. Factor I, accounting for 46.5% of thevariance (Cronbach’s alpha = 0.9316), and describes a staff-patientalliance. In particular, the information and explanations given topatients by staff, the respect shown by staff, the availability of staff, thequality of service from nurses and the opportunity to be involved indecisions about treatment were important contributors to this alliance.The second factor (accounting for 6.5% of the variance with analpha = 0.7830) focused on the treatment environment (cleanliness,privacy, food) and the activities provided for patients. The third factor(accounting for 6.0% of the variance with an alpha = 0.8607) describes

a medical component and comprised items relating to the doctor’savailability and quality of service, as well as explanations abouttreatment and the way treatment was perceived by the patient to meettheir needs. A reliability analysis for the total scale suggests goodinternal consistency (Cronbach’s alpha = 0.9511) with no deletion ofitems considered necessary or appropriate. The factor loadings arereported in Table 1.

Discussion

Ethical clearance for the present study was obtainedfrom each of the hospitals involved. While informedconsent was obtained in writing from those patients whoparticipated in the focus group discussions (phase I) andthe importance ratings (phase II), written consent wasnot obtained from the 356 patients who completed thedraft survey (phase III). As completion of the survey wasleft to the discretion of individual patients who returnedit anonymously, written consent from participants wasdeemed unnecessary.

Table 1. Factor loadings and variance explained by each factor

Item Factor I Factor II Factor III % of variance

Factor I: Staff-patient alliance 46.5%

7 The explanations given to you by the nurses about your care 0.735676 Availability of the nursing staff 0.73385

15 The attention the staff gave to your concerns and worries 0.7154213 The respect you received from staff 0.696931 The information you received about the ward and the

services provided when you were admitted0.69429

9 The quality of the service provided by the nursing staff 0.678728 Availability of your primary nurse/care coordinator 0.65242

19 Feeling safe during your stay in hospital 0.596962 The information you received about your rights while in

hospital0.58915

14 The opportunity to be involved in decisions about your treatment

0.56062

Factor II: Satisfaction with environment 6.5%

12 The activities provided to occupy your time (videos, games,outings, etc.)

0.67872

17 The cleanliness of the ward 0.6660111 The groups/activities provided by the Occupational

Therapist (OT)0.66324

16 The quality of the food provided 0.5758910 The quality of the service provided by Allied Health staff

(social worker/psychologist/OT)0.46183

18 The standard of privacy in the ward 0.42570

Factor III: Satisfaction with treatment 6.0%

5 The quality of the service provided by your doctor 0.798954 The explanations given to you by the doctors about your

treatment0.78197

3 The availability of the doctors 0.6894120 The way the treatment met your needs 0.52219

810 CONSUMER SATISFACTION WITH INPATIENT SERVICE

The first two phases of this study were carried out atthe same three acute psychiatric units. Thus, factors suchas nature and duration of illness were likely to haveremained constant during the 6 months of data collec-tion. During phase III, we again invited patients fromthese same three acute units to participate in the comple-tion of the draft questionnaire. In addition, we includedpatients from a rehabilitation service (n = 161) to assessthe possibility of using the questionnaire in this environ-ment. Although developed in the acute setting, theinstrument seemed to perform equally well in the re-habilitation setting. A review of the freehand commentsprovided by the rehabilitation patients did not supportthe inclusion of additional items or other modificationsto the instrument.

The procedure followed in trialing the draft instrument(i.e. distributing surveys to patients nearing discharge) iscommonly used for monitoring satisfaction in inpatientunits [12]. While attempts were made to ensure that allpatients approaching discharge were invited to completethe draft survey, only 72% did so. Although non-partici-pation may suggest a lack of satisfaction or completesatisfaction [3], the decision to participate will alwaysrest with the patient. There will be a group of patientswho choose not to participate and more expensive andinvasive techniques such as focus group discussions orindividual interviews could be used to solicit their views[12].

The domains covered in the IESQ compare favourablywith other patient satisfaction surveys. Key dimensionsreported in the literature include the social domain,which incorporates staff-patient relations [17,21]; andthe technical domain, which includes treatment andoutcome items [22–24]. In keeping with previous Aus-tralian studies [2,3], a single factor concerned with astaff-patient alliance accounted for almost 50% of thevariance. It is clear that patients distinguish between thetreatment provided and the treatment environment itselfin that the second factor (which contained items relatedto treatment environment) explained much less of thevariance (6.5%). Service providers should not underesti-mate the importance of treatment environment as dissat-isfaction with this may contribute to behaviours such asaggression [25] and absconding [26].

In addition to the scaled items, the IESQ comprisestwo open-ended questions, to elicit comments on aspectsof the hospital stay that patients liked ‘most’ or liked‘least’. These questions were included to allow for iden-tification of particular aspects not covered by the scaleditems, and to discover

why

not just whether inpatientswere dissatisfied. While only 52% of those whoresponded choose to supply freehand comments, thisadditional information did provide important insights

into satisfaction with specific aspects of the services onoffer and was it was clearly valued by service providers.

The high levels of satisfaction (70–95%) found inearlier studies raise concerns about the sensitivity ofsatisfaction instruments [5]. We found that the ‘E5’response format (‘poor’, ‘fair’, ‘good’, ‘very good’ and‘excellent’) did produce good response variability acrossall response options particularly at the positive end of thescale. This finding supports the argument that highersatisfaction ratings are likely to arise from the poorinstrument design rather than lack of instrument sensitiv-ity [27]. Moreover, it suggests that inpatients are capableof making judgements about satisfaction and discrimi-nating between levels of satisfaction.

Our findings support previous studies in that youngerpatients were less satisfied and male patients were moresatisfied. It is clear that other factors such as treatingdoctor, primary nurse and symptomatology may haveinfluenced satisfaction ratings [3,12]. However, it wasnot possible to assess the impact of these variables onsatisfaction as we choose not to collect identifyinginformation. Further development of the instrumentcould explore the impact of mood on satisfaction byinviting patients to provide a self-rating of their mood asdescribed by Eyers

et al

. [3].In conclusion, the IESQ was developed to provide a

brief, user-friendly instrument that overcomes some ofthe shortcomings of existing satisfaction measures. Itcovers a broad range of inpatient concerns, it is simplyworded, easy to score, and is designed to be completedindependently by the inpatient. While it assesses anumber of satisfaction constructs, administration time iskept below 5 min. Ongoing assessment of the instrumentwould be warranted should it be used in settings otherthan acute/rehabilitation.

Acknowledgement

Thanks to the Schizophrenia Fellowship of New SouthWales who provided financial support for the earlystages of this project.

References

1. Rey J, Plapp J, Simpson P. Parental satisfaction and outcome: a 4-year study in a child and adolescent mental health service.

Australian and New Zealand Journal of Psychiatry

1999; 33:22–28.

2. Parker G, Wright M, Robertson S, Gladstone G. The development of a patient satisfaction measure for psychiatric outpatients.


1996; 30:343–349.

3. Eyers K, Brodaty H, Roy K

et al.

Patient satisfaction with a mood disorders unit: elements and components.


1994; 28:279–287.

T. MEEHAN, H. BERGEN, T. STEDMAN 811

4. Department of Health.

Quality improvement and enhancement plan

. Brisbane: Queensland Department of Health, 2001; 1999–2004.

5. Avis M, Bond M, Arthur A. Satisfying solutions? A review of some unresolved issues in the measurement of patient satisfaction.

Journal of Advanced Nursing

1995; 22:316–322.6. Barker D, Orrell M. The psychiatric care satisfaction

questionnaire: a reliability and validity study.

Social Psychiatry and Psychiatric Epidemiology

1999; 34:111–116.7. Kaufmann C, Phillips D.

Survey of state consumer surveys

. Rockville: Substance Abuse and Mental Health Services Administration, 2000.

8. Lebow J. Similarities and differences between mental health and health care evaluation studies assessing consumer satisfaction.

Evaluation and Program Planning

1983; 983:265–274.

9. Nguyen TD, Attkison C, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire.

Evaluation and Program Planning

1983; 6:299–314.

10. Epstein M.

The understanding and involvement project – the challenge of consumer research in the acute psychiatric setting.

Melbourne: Victorian Health Promotion Foundation, 1995.

11. Lovell K. User satisfaction with in-patient mental health services.

Journal of Psychiatric and Mental Health Nursing

1995; 2:143–150.

12. Greenwood N, Key A, Burns T, Bristow M, Sedgwick P. Satisfactions with in-patient psychiatric services.

British Journal of Psychiatry

1999; 174:159–163.13. Williams B, Wilkinson G. Patient satisfaction in mental health

care.

British Journal of Psychiatry

1995; 166:559–562.14. Urquhart B, Bulow B, Sweeney J, Shear M, Frances A. Increased

specificity in measuring satisfaction.

Psychiatric Quarterly

1986; 58:128–133.

15. Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters.

Medical Care

1988; 26:393–402.

16. Department of Health.

Ten year mental health strategy for Queensland.

Brisbane: Queensland Department of Health, 1996.

17. Taylor B, Clarke R. Client feedback – development of an instrument for psychiatric inpatients.

Australian Health Review

1993; 16:231–244.

18. Victoria’s Mental Health Service.

Consumer and carer satisfaction project report on stage one.

Melbourne: Psychiatric Services Division, Health and Community Services, 1996.

19. Sim J. Collecting and analysing qualitative data: issues raised by the focus group.

Journal of Advanced Nursing

1998; 28:345–352.

20. Hansson L, Höglund E. Patient satisfaction with psychiatric services.

Nordisk Psychiatrisk Tidsskrift

1995; 49:257–262.21. Elbeck M, Fecteau G. Improving the validity of measures of

patient satisfaction with psychiatric care and treatment.

Hospital and Community Psychiatry

1990; 41:998–1001.22. Holcomb WR, Parker JC, Leong G. Outcomes of Inpatients

Treated on a VA psychiatric unit and a substance abuse treatment unit.

Psychiatric Services

1997; 48:699–704.23. Holcomb WR, Parker JC, Leong G, Thiele J, Highdon J.

Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients.


1998; 49:929–934.

24. Berghofer G, Lang A, Henkel H, Schmidl F, Rudas S, Schmitz M. Satisfaction of inpatients and outpatients with staff, environment and other patients.


2001; 52:104–106.

25. Barlow K, Grenyer B, Ilkiw-Lavalle O. Prevalance and precipitants of aggression in psychiatric inpatient units.


2000; 34:967–974.

26. Meehan T, Morrison P, McDougall S. Absconding behaviour: an exploratory investigation at an acute inpatient unit.


1999; 33:533–537.27. Ruggeri M. Patients’ and relatives’ satisfaction with psychiatric

services: the state of art of its measurement.

Social Psychiatry and Psychiatric Epidemiology

1994; 29:212–227.28. Attkison CC, Greenfield TK. The client satisfaction

questionnaire (CSQ) scales and the service satisfaction scale – 30 (SSS-30). In: Sederer LI, Dickey B, eds.

Outcomes assessment in clinical practice

. Baltimore: Williams and Wilkins, 1996, 120–127.

monitoring consumer satisfaction with inpatient service delivery: the inpatient evaluation of...

Documents