The measurement and comparison of health system responsivenessNigel Rice, Silvana Robone, Peter C. SmithNigel Rice, Silvana Robone, Peter C. Smith

Centre for Health Economics, University of Centre for Health Economics, University of YorkYork

IntroductionIntroductionPatients’ views and opinions are an essential means

for assessing the provision of health services, to stimulate quality improvements and to measure health systems performance. Traditionally, patients’ views were sought on the quality of care provided and satisfaction with health services. Recently the concept of responsiveness has been promoted as a more desirable measure to judge health systems.

Responsiveness can be defined as the way in which individuals are treated and the environment in which they are treated encompassing the notion of patient experience with the health care system (Valentine et al., 2003)

Responsiveness Domains: Autonomy, Choice, Clarity of communication, Confidentiality, Dignity, Prompt attention, Quality of basic amenities, Social support

ISSUE: data on Responsiveness are self-reported. Individuals, when faced with survey questions about the

functioning of health systems, may systematically interpret the meaning of the available response categories differentially across population sub-groups (Sadana et al., 2002).

Responses will be influenced by individuals' preferences and expectations, which vary systematically across countries, or across socio-demographic groups within a country (REPORTING HETEROGENEITY).

Eg.

POTENTIAL SOLUTION: Use of vignettesVignettes = descriptions of fixed levels of a latent

construct, such as responsiveness.

Eg: “When the clinic is not busy, [Mamadou] can choose which doctor he sees. But most often it is busy and then he gets sent to whoever is free”. How would you rate [Mamadou’s] freedom to choose his health care provider? 1. Very good 2. Good 3. Moderate 4. Bad 5. Very bad

Anchoring vignettes are used to address the issue of reporting heterogeneity. Systematic variation across individuals in the rating of the vignettes can be attributed to reporting heterogeneity (measurement error).

ObjectivesObjectivesExplore the utility of using information from

vignettes to adjust self-reports of health system responsiveness.

• Evaluate the presence of reporting heterogenerity across socio-economic groups within countries, and how it is related to the characteristics of individuals.

• To aid cross country comparison, we stratify countries into three groups according to their Human Human Development IndexDevelopment Index (HDIHDI) (United Nations Development Programme, 2006) and evaluate differences in reporting behaviour among countries in the same group.

• Assess the presence of different reporting behaviour among the three HDI groups of countries.

• Evaluate if the issue of reporting heterogeneity affects the ranking across countries of health systems responsiveness.

Data: Data: The World Health Survey The World Health Survey (WHO, 2001)(WHO, 2001)

70 countries. Survey modes: face to face interview, (90 and 30 minute long) and telephone interviews (4 countries). Samples: randomly selected (+ 18 years), sizes 600 -10,000

9 countries: (high HDI) Mexico, Spain, Malaysia, (medium HDI) India, Philippines, SriLanka, (low HDI) Burkina, Malawi and Ethiopia. Selected for sample size and good psychometric properties.

Dependent Variables:• Respect, Confidentiality, Quality of Facilities and Clarity

of Communication, (4 most important domains in the nine countries selected).

• Response categories: “very good”, “good”, “moderate”, “bad” and “very bad”.

Independent Variables:Individual level Education, Gender, Income, Age. Country level (source: UNDP, 2001) Health expenditures

per capita, GDP per capita.Country and HDI group dummies

MethodologyMethodologyThe Hierarchical Ordered Probit Model (HOPIT), Terza

(1985), Tandon et al. (2003) Method to account for heterogeneous reporting behaviour,

through the use of anchoring vignettes

Two parts: 1) Reporting behaviour (bias) equation: use of vignettes

to model systematic reporting behaviour as a function of covariates

2) Responsiveness equation: model of the mean function of responsiveness, conditional on reporting behaviour in 1)

Assumptions: a) Response consistency: Vignettes are rated consistently

with the rating of own experiences of the service provided.

b) Irrelevance of own provider responsiveness or vignette equivalence: “The level of the variable represented by any one vignette is perceived by all respondents in the same way and on the same unidimensional scale” (King et al., 2004, p.194).

ResultsResults1) Descriptive1) Descriptive StatisticsStatisticsa) Summary freq. for the reporting of own experience

and vignettes about the health care system responsiveness. Eg: Mexico, Clarity of Communication

• Individuals are more polarized in the reporting of own experiences than vignettes

• Vignette ratings exhibits heterogeneity across the response categories. This is an indication of reporting heterogeneity within the country.

b) Ratings of vignette 2 for Clarity of Communication, Mexico, by:

Education Income

For both education and income, for each of the available response categories (i.e. very good”) a gradient provides evidence of reporting heterogeneity across socio-economic groups.

Very bad Bad Moderate good Very good Group 1 Individual 1 pulation 1

Very bad Bad Moderate good Very good ggoodgoo

d

Group 2

0.1

.2.3

.4.5

.6%

own vig1 vig2 vig3 vig4 vig5

i_clarity_communic

mean of verygood mean of goodmean of moderate mean of badmean of verybad

0.1

.2.3

.4.5

.6%

own vig1 vig2 vig3 vig4 vig5

i_time_questions

mean of verygood mean of goodmean of moderate mean of badmean of verybad

2) Tests of homogenous reporting 2) Tests of homogenous reporting Results: within country results (Eg: Mexico)• Evidence of reporting heterogeneity for all response domains. The model indicates reporting heterogeneity as a function of income and education, not as pronounced when considering age and gender.

Results: across countries within HDI group• For all the HDI groups and all response domains, evidence of reporting heterogeneity, particularly by education, income and country dummies.• In the high HDI group the wealthier are more likely to rate vignettes more extremely, in the middle HDI group more positively and in the low HDI group more negatively than their poorer counterparts. In both the high and medium HDI group, the more educated tend to rate vignettes more extremely than the less educated.

Results: across all countries• For all response domains, evidence of reporting heterogeneity, particularly by education, income and HDI group dummies.• For most of the domains the wealthier have higher expectations about responsiveness. The better educated tend to rate vignettes more extremely. Individuals belonging to the medium HDI group have lower expectations about responsiveness than those in the high HDI group.

3) Comparison of Predictions by 3) Comparison of Predictions by OPROBIT and HOPIT OPROBIT and HOPIT (high HDI group (high HDI group model) Spain, Mexico Malaysia, model) Spain, Mexico Malaysia, Clarity of Clarity of CommunicationCommunication

Predicted latent responsiveness

Predicted frequencies of “good” and “very good” responsiveness

Evidence that the responsiveness ranking of Spain, Mexico and Malaysia changes if the model is estimated through the OPROBIT or the HOPIT, when we consider both predicted latent responsiveness and predicted frequencies of “good” and “very good” responsiveness.

ConclusionsConclusionsHeterogeneity in reporting behaviour exists, and appears to be a function of individual socioeconomic characteristics (incomeincome and educationeducation), and characteristics at country and macro-regional levels (captured through country and HDI country and HDI group dummiesgroup dummies). Adjusting for reporting bias impacts on the: Estimated coefficients of the responsiveness mean function when results from the HOPIT model are compared to those from an ordered probit regression. Marginal effects of individual socioeconomic variables (such as education) on responsiveness Ex-post frequencies of reporting each of the five response categoriesRanking of high HDI countries according to their responsiveness level.

0.1

.2.3

.4.5

.6%

primary school completed secondary school completed high school com pleted post graduate degree completed

Mexico g_clari ty_communic vignette 2 by education

mean of verygood mean of good

mean of moderate mean of bad

mean of verybad

0.1

.2.3

.4.5

.6%

1 2 3 4 5

Mexico g_clarity_communic vignette 2 by income

mean of verygood mean of good

mean of moderate mean of bad

mean of verybad

2

3

1

2 1 3

-0.30

-0.20

- 0.10

0.00

0.10

0.20

0.30

0.40

0.50

ESP MEX MYS

latent resp OPROBIT

latent resp HOPIT

2

3

1

2 1 3

-0.30

-0.20

- 0.10

0.00

0.10

0.20

0.30

0.40

0.50

ESP MEX MYS

OPROBIT

HOPIT

78.0%

80.0%

82.0%

84.0%

86.0%

88.0%

90.0%

92.0%

EXP MEX MYS

OPROBIT

HOPIT

ESP MEX MYSlatent resp OPROBIT 0.43 0.38 0.46latent resp HOPIT -0.09 -0.07 -0.17