towards greater objectivity in governance measurement ...ivanynam/research/cgi2/ivanyna... ·...
TRANSCRIPT
Towards Greater Objectivity in Governance Measurement: Second
Generation Citizen-Centric Governance Indicators
Draft
By Maksym Ivanyna, Joint Vienna Institute, Michigan State University,
and Anwar Shah, Bookings Institution, World Bank and SWUFE, China∗
July 06, 2015
Abstract Most widely used governance indicators (e.g. the World Bank’s Worldwide
Governance Indicators) lack information on how citizens evaluate their governments and suffer
from time and cross-country inconsistencies in their assessment methodologies. The first
generation citizen-centric governance indicators by Ivanyna and Shah (2011) attempted to
overcome this and other deficiencies of the WGIs by providing a framework for comparative
assessment of governance quality across countries and over time and by using data from World
Value Surveys to capture citizens’ perceptions of governance environment and outcomes in their
own countries. The citizen based evaluations while addressing the voids of experts’ purely
subjective perspectives, however, are not immune from systemic bias such as indoctrination,
intimidation, and critical citizenship. This paper presents methods for testing and correcting these
biases of public opinion surveys and develops a second generation citizen-centric governance
indicators that incorporate these adjustments. In doing so, it develops more reliable indicators of
governance quality across the globe. The correction procedures presented here would be helpful
in correcting for systemic biases in other measures that use public opinion survey data.
∗ Comments are welcome and may please be addressed to: [email protected] and [email protected]
1. Introduction
Governance quality assessments are now being used to judge various political regimes, conduct
development dialogue, allocate external assistance and to influence foreign direct investment.
In view of these uses, a comparative assessment of governance quality across countries and
benchmarking country performance over time is of growing interest to policy makers, citizens
and scholars alike. However, available worldwide governance indicators are in many aspects
inadequate for use in judging development effectiveness or in allocating external assistance
equitably based upon an objective criteria. 1
One of the most important limitations common to all available composite indexes of governance
is that they do not capture how citizens perceive the governance environment and outcomes in
their own countries. Most of the indexes are either solely based on external armchair experts’
evaluations or mix them with the citizens’ evaluations, with usually much smaller weight given
to the latter. For example, the Worldwide Governance Indicators (WGI), which are perhaps the
most widely used governance measures nowadays, in 2013 only 8 out of 150 representative
datapoints used by these indicators were based on public opinion surveys. If we simply average
these 8 datapoints and compare the resulting country ranking with that of the WGI itself (average
of six dimensions), the mean absolute percentile difference would be 24, i.e. with 215 countries
covered, an average country would be expected to change its ranking by about 50 positions up or
down. It suggests that in most countries, citizens’ assessment of governance quality is at various
with so-called international experts.
To overcome these deficiencies, Ivanyna and Shah (2011) developed a uniform and consistent
global framework for measuring governance based on citizens’ evaluations, the so-called citizen-
centric governance indicators (CGIs). They implemented this framework by using the first five
waves of the World Values Surveys, which combine decent geographical coverage with
acceptable number of governance-related questions.
This paper attempts to improve the measurement methodology used by Ivanyna and Shah (2011)
and to present up to date CGIs using the sixth wave of the WVS, conducted in 2010-2014. The
1 See, for example, Arndt 2008, Arndt and Oman 2006, Kurtz and Schrank 2007, Iqbal and Shah 2008, Langbein
and Knack 2008, Schrank and Kurtz 2008
paper streamlines the aggregation procedure and selects questions to ensure maximum
comparability over time. As a result, citizen-centric governance indicators are computed for 100
countries over six waves of the survey from 1980 to 2014, on average 2.5 times per each country
– 259 country-year measurements.
Public opinion surveys contains useful and unique information for measuring governance. But it
can also be subject to systemic biases, stemming for example from indoctrination in mass-media
or government’s oppression under totalitarian regimes, as ordinary people may be afraid to
answer questions about their governments truthfully for fear of reprisals. On the contrary in open
societies, people may have too high expectations of their government failing which they may
pass harsh judgments on their performance – the so-called phenomenon of “critical citizenship”
developed by Norris (1999). The presence of these biases may limit the usefulness of CGIs
comparisons across countries and through time. Good news is that in carefully constructed public
opinion surveys these biases can be corrected.
This paper develops and implements a method to correct public opinions about governance for
the systemic biases. This is done by by isolating these biases by separating personal (objective)
and government-related (subjective) questions in the surveys. Both groups of questions measure
the concept of governance, but only the latter one is likely to be subject to the above-mentioned
biases.
The paper is organized as follows. Section 2 briefly summarizes a citizen-centric conceptual
framework on measuring governance quality, presented first in Ivanyna and Shah (2011). Section
3 presents an empirical framework, data sources and aggregation techniques. Section 4 presents
the resulting CGIs without the adjustment. Finally, Section 5 discusses the correction for
systemic biases in public opinion, and the resulting adjusted CGIs.
2. Citizen-centric governance indicators: Conceptual framework
This section is a short summary of the corresponding part in Ivanyna and Shah (2011). Here, we
define “governance” as the norms, traditions and institutions by which power and authority in a
country is exercised—including the institutions of participation and accountability in governance
and mechanisms of citizens’ voice and exit and norms and networks of civic engagement; the
constitutional-legal framework and the nature of accountability relationships among citizens and
governments; the process by which governments are selected, monitored, held accountable and
renewed or replaced; and the legitimacy, credibility and efficacy of the institutions that govern
political, economic, cultural and social interactions among citizens themselves and their
governments (see also Huther and Shah, 1998). Note that this definition encompasses both the
governance environment (quality of institutions and processes) as well as governance outcomes.
A stylized view of the public interest can be characterized by four dimensions of governance
institutions, processes and outcomes.
• Responsive Governance. The fundamental task of governing is to promote and pursue
collective interest while respecting formal (rule of law) and informal norms. This is done
by government creating an enabling environment to do the right things – that is it
promotes and delivers services consistent with citizen preferences. Further, the
government carries out only the tasks that it is authorized to do, that is it follows the
compact authorized by citizens at large.
• Fair (equitable) Governance. For peace, order and good government, the government
mediates conflicting interests, is focused on consensus building and inclusiveness and
ensures a sense of participation by all and protection of the poor, minorities and
disadvantaged members of the society.
• Responsible Governance. The government does it right i.e. governmental authority is
carried out following due process with integrity (absence of corruption), with fiscal
prudence, with concern for providing the best value for money and with a view to earning
trust of the people.
• Accountable Governance. Citizens can hold the government to account for all its actions.
This requires that the government lets sunshine in on its operations and works to
strengthen voice and exit options for principals. It also means that government truly
respects the role of countervailing formal and informal institutions of accountability in
governance.
Given the focus on governance outcomes, Table 1 presents some preliminary ideas for discussion
on how to operationalize these concepts in individual country assessments.
Table 1 Governance outcomes and relevant considerations
Governance outcome Relevant considerations
Responsive governance - public services consistent with citizen preferences;
- direct possibly interactive democracy;
- safety of life, liberty and property;
- peace, order, rule of law;
- freedom of choice and expression;
- improvements in economic and social outcomes;
- improvements in quantity, quality and access of public services;
- improvements in quality of life;
Fair governance - fulfillment of citizens’ values and expectations in relation to participation, social justice, and due process;
- access of the poor, minorities and disadvantaged groups to basic public services;
- non-discriminatory laws and enforcement;
- egalitarian income distribution;
- equal opportunity for all;
Responsible governance - open, transparent and prudent economic, fiscal and financial management;
- working better and costing less;
- ensuring integrity of its operations;
- earning trust;
- managing risks;
- competitive service delivery;
- focus on results;
Accountable governance - justice-able rights and due process
- access to justice, information;
- judicial integrity and independence;
- effective legislature and civil society oversight;
- recall of officials and rollbacks of program possible;
- effective limits to government intervention;
- effective restraints to special interest capture.
Source: Shah (2008).
The above simple framework captures most aspects of governance outcomes especially those
relevant for development policy dialogue and can serve as a useful starting point for a consensus
framework to be developed. Once a consensus framework is developed then one needs to focus
on only a few key indicators that represent citizens’ evaluations and could be measurable with
some degree of confidence in most countries of the world and could be defended for their
transparency and reasonable degree of comparability and objectivity.2
Implementation of the above framework requires a worldwide survey with uniform questionnaire
honing on the four dimensions of governance identified above across countries. Such a survey is
yet to be developed. In the following section, we develop rough indexes of governance quality
based upon the survey data, which is already available.
3. Citizen-centric Governance: Empirical Framework
We use World Values Surveys (WVS) as our source of the survey data. It provides a reasonable
compromise between consistency with our conceptual framework and coverage of countries and
years. WVS publishes somewhat dated information (with the time lag of 2-3 years after actual
survey was taken), and only few questions from this survey are relevant for governance
assessments (since the survey is mainly about cultural values, not governance). However, WVS
provides quite comprehensive geographical coverage (100 countries with all major economies
included) combined with acceptable time coverage (six waves, from 1981 to 2014) and its
questionnaire is disclosed.
Appendix Table A1 presents the questions from the WVS that we use to measure governance.
Overall, we pick 15 questions. For quite a few sub-criteria from Table 1 no survey questions are
available. In principle, there are more governance-related questions in the surveys that we could
use, but they are not asked in most waves or in most countries. The questions we pick are asked
in at least 75% of all surveys in all waves, and in at least 90% of the surveys in fifth and sixth
wave. At least 10 sub-criteria from Table 1 can be covered by the questions with sufficient
representation throughout years and countries. The average coverage over all questions and
countries is 93% in all six waves, and 98% in the last two waves. In addition, we drop the
countries for which the coverage is less than 50% - seven out of 239 country-year surveys.
2 See (Andrews and Shah 2005) for details and relevant indicators of an approach that emphasizes citizen-
centric governance and (Shah and Shah 2006) for citizen-centered local governance and relevant indicators
The WVS provides raw survey data, so the observations can be sorted by gender, income,
education of a respondent, as well as by sub national administrative unit or other characteristics.
Then corresponding “adjusted” CGIs can be constructed.
3.1 Aggregation: Empirical model of governance
The underlying assumption of our empirical investigation is that the quality of governance
institutions and processes directly affects governance outcomes. Thus, the quality of governance
should correlate positively with the answers of survey respondents. At the same time, the
answers are random variables, which are subject to personal errors, including potential systemic
biases:
(1)
where is the index of a country, is the index of a respondent (total number of
respondents changes from country to country), and is the index of a question in a survey
(thus of a particular governance outcome). is the answer on question of the respondent
in the country . Each response is normalized to be between 0 and 1, with 0 being the worst
answer, and 1 being the best answer. is the quality of governance in the country , which
obviously does not depend neither on concrete respondent, nor on specific question. Finally,
is the random error, which is assumed to be independently normally
distributed with mean µik and the variance , and both may depend on country and specific
question. If µik is zero then provides an unbiased estimate of gi. This is assumption that we
make further in this section and in Section 4. If µik is not zero then the systemic bias in public
opinion is present, and it has to be corrected. We deal with the correction in Section 5.
Given the assumption of µik=0, the most efficient, unbiased, and consistent estimator for the
governance in country is just the sample mean of weighted averages of citizens’ responses.
The estimator for the governance’s variance is adjusted sample variation:
, (2)
,ijkijkiijkiijk sggs εε −=⇒+=
Mi ..1= iNj ..1=
Kk ..1=
ijks k j
i
ig i
),(~ 2..
ikik
di
ijk N σµε
ik2σ
ijks
i
∑ ∑∑∑∑= === =
−−
==i ii N
j
N
jijk
iijk
i
K
kk
N
j
K
kiijkk
ii s
Ns
Nwgsw
Ng
1
2
11
2
1 1)1(
11)r(av,1ˆ
where the weights w for each question are chosen to minimize the variance of governance
indicator. Roughly speaking, questions with smaller variance of the measurement error ε should
get bigger weight. Since σ’s are not observed, the eventual choice of weights is effectively
arbitrary. We take parsimonious and the most comprehensible approach, and assign equal weight
to each “covered” sub-criterion in Table 1, with the exception of few questions, which seem to
be relatively more far-reaching in assessment of governance (i.e. “satisfaction with life in
general” is clearly more comprehensive than “satisfaction with health”). Such questions get
bigger weight. All weight sum up to one.
4. Citizen-centric governance indicators without adjustment
The maps of the citizen-centric governance indicators for wave 5 (2005-2009) and wave 6 (2010-
2014) of the survey are presented in Figure 1. The surveys were taken in 57 and 59 countries
accordingly. The countries are grouped into four quartiles: the darker is the color the higher is
the CGI. Table 2 presents summary statistics of CGIs over all six waves.
Table 2 CGIs unadjusted: Summary statistics
Wave N Mean S.d. 10th perc. 90th perc
1 (1981-1984) 10 0.61 0.05 0.55 0.68
2( 1989-1993) 17 0.55 0.06 0.47 0.62
3 (1994-1998) 51 0.54 0.07 0.44 0.62
4 (1999-2004) 38 0.57 0.08 0.47 0.67
5 (2005-2009) 57 0.57 0.07 0.47 0.67
6 (2010-2014) 59 0.58 0.07 0.48 0.67
Table 2 shows that quality of governance in 80% of the countries ranges roughly between 0.44
and 0.67 in all six waves, on the [0,1] scale. The mean CGI remains practically unchanged
(except between waves 3 and 4), although the sample of countries changes with each wave.3
3 Ivanyna and Shah (2011) show that in overlapping samples of countries CGIs improved statistically significantly
from wave 3 to wave 4, and did not change from wave 4 to wave 5.
Several observations on individual countries emerge from Figure 1. As expected, most developed
countries (especially Scandinavian countries, Switzerland, Canada, New Zealand) demonstrate
stable and good performance. At the same time, a number of developing countries are also
among the top-performers. East Asian governments (especially, Vietnam, China) get particularly
high plaudits from their citizens. Governance quality is also rated well by the citizens of Ghana,
South Africa, and Turkey. This may appear surprising to a reader in OECD countries, but can be
justified as these governments have an excellent track record of improving economic and social
outcomes for their residents and hence captured in their citizen’s evaluations. On the other side,
countries in Central and Eastern Europe and Latin America consistently get the lowest scores
from their citizens.
Figure 1 Citizen-centric governance indicators (data source WVS, waves 5 and 6)
Figure 2 shows the relationship between the citizen-centric governance indicators and the UN’s
Human Development Index. In general the correlation is positive, but the fit, is very bad.4 Many
countries, which are ranked low by the HDI, get high scores from the CGI. Similar picture
emerges if we compare CGI with other proxies of quality of life or goverance: GDP per capita,
perceived levels of corruption, etc. One reason for this is that public opinion contains
information about governance institutions and outcomes, which is not present in other measures.
Another potential reason could be that public opinion is subject to systematic biases. We explore
the latter in the next section.
4 We exclude SSA countries from the figure for the representation purposes. These countries make the fit even
worse, although the point estimate of the slope of the fitted line is still positive.
Figure 2 Citizen-centric governance indicators vs. Human Development Index, wave 6
5. Removing systematic biases from public opinion
While CGIs may contain information about governance, which is not present in other measures,
public opinion may also suffer from various kinds of systematic biases.5 If this is the case then in
(1) the mean of the random error µik is not equal to zero, so one cannot extract gi from the
citizens’ responses by simply taking a weighted average. The systemic biases distort the real
picture of citizens’ assessment of governance, and in particular make cross-country comparison
misleading. In this section we discuss and implement alternate methods to correct for these
biases.
5 See Bertrand et al. 2001, Heath et al. 2005, King et al. 2004, MacKerron 2012, Olken 2009 for the discussion on
subjective measurements
Public opinion, in particular on the government-related issues, may suffer from at least three
systemic biases. First is the “intimidation” effect, when people are afraid to express their true,
usually negative, opinion about their government, because they think they could be punished for
that. The second bias is the “indoctrination” effect, when mass media in a country praise or
criticize the government disproportionately, so that it distorts public opinion. The third factor,
which may bias public opinion, is the degree of citizen activism and perceived role of
government in a country. In particular, (Norris 1999) argues about the emergence in the 70s in
developed countries of a class of so called “critical citizens” – people, who raise the bar for
government performance to unrealistic levels and become vocal critics once those high
expectations are not met in practice. Adjusting for this “excessive” criticism as well as for
intimidation and indoctrination, is vital for valid cross-country comparison. Below we suggest
how to do it within the WVS framework, and then we discuss other potential biases.
Intimidation, indoctrination and “critical citizenship” affect respondents' answers together with
the quality of governance. We can write down the random error εijk as a sum of the biases and the
true random error, and rewrite (1):
(3)
where similarly to the notation in Section 3.1, is a response of individual in country on a
question , is the quality of governance in country , are the degrees of
intimidation, indoctrination and critical citizenship of individual in country (independent of
question k), and is citizen-, country- and question-specific random error. Its mean is zero if
there are no other systemic biases in the public opinion. and are the coefficients of our
interest. We assume that they can vary by country, as well as by question (f.e. respondents may
be more intimidated to answer some of the questions than others).
The estimation of and is not possible from (3), since we do not observe . However,
the problem can be resolved if we note, or rather assume, that for some questions (call them
“objective”) the effect of “intimidation”, “indoctrination” or “critical citizenship” is likely to be
very low to zero, and for some the effect is likely strong. For instance, when an individual is
asked about the satisfaction with her/his health, it is likely that she/he will not be intimidated to
,_ ijkijikijikijikiijk citcrindings ζµηγ ++++=
ijks j i
k ig i i ji ji j c ic ri n di n _,,
j i
ijkζ
ikik ηγ , ikµ
ikik ηγ , ikµ ig
tell the truth. At the same time, questions like “Do you have confidence in your government?”
may invite all above mentioned biases. Therefore, we can isolate the effect of biases on the
individual responses by differentiating (diffij) between “subjective” and “objective” questions.
Suppose of K questions 1st to K1th are subjective, and the rest are objective. Their corresponding
weights in the CGI are and , see (2). Then for the subjective
questions:
(4)
For the objective questions biases are assumed to be absent:
(5)
Taking a weighted difference between (4) and (5) we can get rid of gi on the right hand side and
get an expression of only observables and random error:
(6)
where γ’, η’, and μ’ are the original coefficients from (3), multiplied by wsub. Expression (6) can
be estimated directly using OLS provided we have corresponding data on indoctrination,
intimidation and critical citizenship. The estimated coefficients can then be used to derive the
true gi from the citizens’ responses (adjusted CGIs):
(7)
We assume the following questions (governance outcomes) to be independent from the
biaseffects:6
6 Not all questions used in the estimation of g must be used in the identification of biases. Questions in doubt (e.g. if
it is hard to say whether the question is objective or subjective) can simply be left out. The bias identification
procedure described in (4)-(6) still carries through as long as there are at least some questions in both groups.
∑ == 1
1
K
k ksub ww ∑ ==
K
Kk kobj ww1
.)_(11
11∑∑==
++++=K
kijkkijiijiijiisub
K
kijkk wcitcrindingwsw ζµηγ
.11
∑∑==
+=K
Kkijkkiobj
K
Kkijkk wgwsw ζ
,'_' '''
11 1
1
ijkijiijiijii
K
Kkijk
obj
subK
kijkkij citcrindins
ww
swdiff ζµηγα ++++=−= ∑∑+==
)_ˆˆˆ(1 '''
1 1ijiijiiji
N
i
K
kijkk
ii citcrindinsw
Ng
i
µηγ ++−= ∑∑= =
- O1. How satisfied are you with the financial situation of your household?
- O2. All things considered, how satisfied are you with your life as a whole these days?
- O3. All in all, how would you describe your state of health today?
- O4. Taking all things together would you say you are [happy, unhappy]?
The following questions, however may invite systemic biases:
- S1. How much confidence do you have in the government?
- S2. How much confidence do you have in the parliament?
- S3. How much confidence do you have in the press?
- S4. How much confidence do you have in the television media?
- S5. How much confidence do you have in the courts?
- S6. How much confidence do you have in the police?
- S7. How much confidence do you have in the armed forces?
- S8. How much confidence do you have in the civil services?
- S9. How much respect is there for individual human rights nowadays in the country?
- S10. How proud are you of your nationality?
- S11. Would you fight for your country?
We use two steps to estimate and - effects of intimidation, indoctrination and “critical
citizenship”. First, we test for indoctrination ( ) on an individual response level within each
country. Then we use country-level regressions to identify effects of intimidation and critical
citizenship. We cannot use individual response level, as we do not have proxies for personal
intimidation or “critical citizenship”.
Adjustment of CGI (7) is less straightforward though as one needs to decide whether the questions in doubt contain
bias and to what extent.
ii ηγ , iµ
iη
5.1 Testing for indoctrination
To identify the indoctrination effect we test whether the individual subjective-objective questions
difference (diffij) depends on the exposure to the mass-media – press, TV and internet. If there is
a statistically significant relationship it means the that mass-media systematically skews citizens’
perceptions/confidence about the government relative to citizens’ perceptions about their own
financial situation, health and life overall, which in turn depend on the governance environment.
To measure individual exposure to the mass media we use the corresponding questions in the
WVS. For the wave 6 of the survey the question is: “'People learn what is going on in this
country and the world from various sources. For each of the following sources, please indicate
whether you use it to obtain information daily, weekly, monthly, less than monthly or never”.
The sources are “daily newspaper” for press exposure, “TV news” for TV exposure, “internet”
for internet exposure. All answers are rescaled to vary from 0 (never) to 1 (daily). For the wave 5
of the survey the mass media related questions are more rudimentary, and hence less suited to
measure the exposure: “Did you watch TV/ read newspapers / read news in internet during the
last week?”. Hence, the exposure variables are dummies: 0 means “No”, 1 means “Yes”.
Nevertheless, we test for indoctrination in both waves. The more people watch TV or read
newspaper / internet news the more they are exposed to possible indoctrination (or excessive
criticism of mass-media).
The statistical summary of all mass media exposure variables is presented in Table 3. The
television remains the dominant source of information for an average person in the world.
Internet is still lagging behind the traditional mass media, even in 2010-2014, but its role is
increasing rapidly (an increase from 0.28 in wave 5 to 0.45 in wave 6), while that of press and
TV remains stagnant.
Table 3 Mass media exposure variables: Summary statistics
Variable N Mean S.d. 10th perc. 90th perc
Wave 5:
Press exposure 78256 0.55 0.48 0 1
TV exposure 75854 0.88 0.33 0 1
Internet exposure 74231 0.28 0.45 0 1
Wave 6:
Press exposure 83109 0.57 0.39 0 1
TV exposure 81063 0.88 0.25 0.5 1
Internet exposure 82696 0.45 0.45 0 1
The exact econometric specification that we use is the following:
(7)
where is a set of individual demographic variables: 1 if unemployed, 1 if student, 1 if
retired, 1 if female, education (1 to 7), relative income (1 to 10), age, 1 if married, interest in
politics (0 – not at all interested to 1 – very interested), and political preferences (1 – left to 10 -
right). These variables are likely to be correlated both with diff and with the exposure to mass
media, so they need to be included in the regression.
The linear regression (7) is run separately by each country-wave – 50 regressions for wave 6, and
46 regressions for wave 5. The summary of coefficients for wave 6 regressions is presented in
Table 4. Magnitudes of the coefficients by countries are presented in Table 5.
First point to note from the tables is that the TV seems to be the primary mean for the
indoctrination and government appraisal in the world. The mass media bias on TV seems to be
present (statistically significant) in 34 countries out of 50, which we tested. Only in one country
out of 34, the TV media is actually too critical about the government. The average coefficient on
the TV exposure is 0.03, and it can go as high as 0.11. With on average 88% respondents
watching the TV news daily the indoctrination could increase CGIs by as much as 9.7 percentage
points – a magnitude large enough to move a median country to as low as 10th percentile.
Table 4 Effect of indoctrination: Summary statistics for wave 6
Variable N Mean S.d. (mean) Min Max
,'' 321 ijkijiijiijiijiiij demogrinetpresstvdiff εϑηηηα +++++=
ijdemogr
Press exposure 50 0.004 0.012 -0.053 0.053
TV exposure 50 0.03 0.019 -0.032 0.105
Internet exposure 50 -0.016 0.011 -0.085 0.02
Indoctrination adjustment in
CGI, total
50 0.022 0.027 -0.035 0.092
Table 5 Mass media bias in public opinion: Magnitude of coefficients by country, 2010-2014
Indoctrination, magnitude of coefficients (
)
Press exposure TV exposure Internet exposure
0.04-0.11 EGY'13, EST'11 LBY'14, TWN’12, CHL'11, RUS'11, SWE'11, COL'12, BLR'11, ARM'11, KOR'10, POL'12, DZA'13, KGZ'11, JPN'10, NZL'11, BHR'14,
TUN'13
0.02-0.04 KAZ'11, ROM'12, WBG'13, LBN'13, AUS'12, IRQ'12, ZAF'13, SWE'11, LBY'14, UKR'11, AZE'11, BLR'11
IRQ'12, LBN'13, TUR'11, RWA'12, NLD'12, MYS'12, CYP'11, MEX'12, AUS'12, DEU'13, UZB'11, USA'11, EST'11, TTO'11, KAZ'11,
ZAF'13, IND'14
PAK'12
-0.02-0.02 30 countries 16 countries 29 countries
-0.04 - -0.02 IND'14, PAK'12, TUN'13 PAK'12 TUR'11, CHL'11, BRA'14, UZB'11, ARM'11, PHL'12, CYP'11, EST'11, YEM'14,
ZWE'12, KOR'10, MYS'12, RUS'11, UKR'11, ROM'12
-0.09 - -0.04 TTO'11, RWA'12, THA'13 BHR'14, BLR'11, THA'13, DZA'13, KAZ'11,
Note: The country-year pairs for which the indoctrination effect was estimated: ARM'11 AUS'12 AZE'11 BHR'14 BLR'11 BRA'14 CHL'11 COL'12 CYP'11 DEU'13 DZA'13 ECU'13 EGY'13 EST'11 GHA'12 IND'14 IRQ'12 JPN'10 KAZ'11 KGZ'11 KOR'10 LBN'13 LBY'14 MEX'12 MYS'12 NGA'11 NLD'12 NZL'11 PAK'12 PER'12 PHL'12 POL'12 ROM'12 RUS'11 RWA'12 SVN'11 SWE'11 THA'13 TTO'11 TUN'13 TUR'11 TWN’12 UKR'11 URY'11 USA'11 UZB'11 WBG'13 YEM'14 ZAF'13 ZWE'12. Coefficients in the category “-0.02-0.02” are all statistically insignificant at 10% level and less.
While newspapers seem to be mostly neutral or sympathetic to the government, but to a much
lesser degree than the television, internet news follow a completely opposite pattern. In 40% of
iii 321 ,, ηηη
the countries in our sample internet news seem to bias the public opinion about the government
downwards. In 60% of the countries internet coverage is neutral, which is much better than for
the TV. The average magnitude of the coefficient on the internet exposure is -0.02, and it can go
as low as -0.09. In some countries overly “critical” internet news seem to provide a balance to
overly “favorable” TV news: countries like Bahrain, Belarus, Kazakhstan, Algeria are among the
“leaders” in the indoctrination by the TV, and at the same time their internet news segments are
one of the world’s most critical.
The internet does not balance out the TV’s indoctrination bias completely. In absolute terms the
coefficients on the internet exposure are on average smaller than the coefficients on the TV
exposure. In addition, people still rely more on the TV than on the internet as their source of
news. As a result, the average indoctrination bias in the CGIs (last line in Table 4) is positive –
0.02, which is around a third of CGIs’ standard deviation. The bias can be as high as 0.09
(Tunisia).
The second noteworthy point is that even though a lot of developing countries, especially those
in Middle East, Europe and Central Asia, seem to be indoctrinated, the mass media bias is also
present in many developed countries – e.g. in Japan, Sweden, New Zealand, USA, Germany,
Netherlands, Australia. Given that the mass media in these countries are unlikely to be captured
by the government, it signifies presence of some sort of ideological or media ownership biases in
the news coverage in these countries. For most of the countries a good practice to get a balanced
view on the government is to diversify sources of information.
5.1 Testing for intimidation and critical citizenship
We do not observe intimidation and “critical citizenship” at the individual respondent level, so
we test for the presence of these biases in a cross-country regression. As a proxy for the
intimidation we use the average score of the country in the “Freedom in the World” index – an
annual publication of the Freedom House, where political and civil rights of the citizens are
assessed. The index varies from 1 (most free) to 7 (least free). Countries rated 1 to 2.5 are
considered “free”, and countries rated 5.5 to 7 are considered “Not Free”. The expectation is that
in more oppressed countries respondents’ public assessment of their governments is more
favorable than their true assessment.
As for the “critical citizenship”, we follow (Norris, 1999) in her definition of a “critical citizen”,
and define the country to be in the stage of “critical citizenship” if it had been classified “Free”
by the Freedom House for at least ten years before the survey was conducted (long period of
stable democracy), and the real GDP per capita in this country (in 2005 prices) was more than 10
thousands US dollars (wealthy population). Most OECD countries enter the group.
Our econometric specification is the following:
(8)
where is the “Freedom in the World” index, is the “critical citizenship”
dummy, diffi is the subjective-objective questions difference, averaged over all residents of
country i and after the adjustment for the indoctrination.
The estimation results are presented in Table 6. We find that both freedom of the county and its
status of “critical citizenship” are statistically significant in explaining biases of subjective
questions in the WVS surveys. The directions of the effects are as expected. One score up in the
Freedom House ranking (which means country becomes less free) does make people more
cautious in answering government-related questions in a public opinion survey, and consequently
overpraise their governments by 0.013 points. From the other side, residents of the countries,
which are in a stage of “critical citizenship”, do have significantly less confidence in their
governments then they should have had. If not hyper “critical”, the residents of these countries
would have given their governments a score 0.039 points higher. The total CGI adjustment for
both biases varies from -0.09 in Uzbekistan to 0.026 in most OECD countries.
Table 6 Effect of indoctrination and "critical citizenship"
Dependent variable - diff
Coef.
freedom 0.013***
,_ iiii citcrfreedomdiff εµγα +++=
ifreedom icitcr _
(0.004)
cr_cit -0.039**
(0.017)
R-squared 0.22
Number of obs 142
Note: *** - significant at less than 1% level ** - significant at 5% level. Method of estimation – OLS. Standard errors in parentheses.
5.3 Adjusted CGIs
Figure 5 CGIs, wave 6, adjusted for indoctrination, intimidation and critical citizenship
Figures 5 and 6 report the adjusted CGIs in 2010-2014 and their comparison with the Human
Development Index. First thing to note is that the adjusted CGIs fit HDI much closer than the
non-adjusted CGIs,7 the relationship becomes statistically significant. Same goes with other
measures of quality of life and governance. Western European countries, U.S., New Zealand, and
Australia are now in upper half of the ranking. Latin American countries somewhat improve
their rankings. Yet, even after the adjustment CGIs still seem to contain information, which is
not captured by other governance and life quality indexes. The scores of East Asian countries,
Uzbekistan, Ghana go down, but these countries still remain in the upper half of the country
ranking. Apparently, there are other reasons for some governments to score so high in the public
opinion polls. In case of East Asia the main of them is probably last decade’s stable economic
growth and development in the region (as it is argued for China by Wang 2005). At the same
time, poor economic performance, political conflicts and corruption in the 90s (and for many
countries up until today) in Central and Eastern European countries keep the scores of the
governments in this regions quite low (although Poland shows significant progress in the last
wave).
7 We still exclude SSA countries from the figure. Including them reduces the fit, but the slope estimate is still
positive.
Figure 6 Adjusted CGIs vs. Human Development Index, wave 6
5.4 Robustness checks, limitations and further research
Our results are robust to a number of checks. First, we change sets of objective and subjective
questions.8 Second, we try different sets of auxiliary control variables in specification (7). Third,
we try different definitions of “critical citizenship” in (8).9 None of the checks leads to a
qualitative changes in the results. Set of countries with significant indoctrination effect remains
almost unchanged. Coefficients on intimidation and “critical citizenship” in (8) remain
significant. Magnitudes of coefficients statistically do not differ from our main specification.
8 In particular, we exclude questions S6-S11 from the list of subjective questions, O2 and O4 from the list of
objective questions.
9 E.g. five years of being “free” and GDP per capita = 5000, or 15 years of being “free” and GDP per capita =
15000.
Even though robust to a number of robustness checks, our procedure of public opinion
adjustment is subject to certain limitations.
First, it rests on the assumption that some governance-related questions are objective, i.e. they
are not subject to systematic biases that we consider. If this is not the case than the objective-
subjective questions differentiation would not capture the full extent of the biases, only its lower
boundary. For example, if in country X people are indoctrinated to overpraise their government,
and they are also led to think too favorably about their personal financial situation and health,
our procedure will only correct for the difference between the two biases. It is reassuring though
that our results do not change much after we rebalance the sets of objective and subjective
questions to exclude the most “doubtful” ones (e.g. “Are you proud of your nationality?”).
Additionally, even if we underestimate the magnitudes of the biases, their signs are likely to be
estimated correctly. It is quite plausible to assume that the questions which we deem “objective”
contain less bias than the questions which we deem “subjective”.
The second limitation of our procedure is that it may not correct for all the biases, which affect
public opinion. For historical, cultural, religious or other reasons respondents may have different
perception of the same governance effort. These differences can translate to a systematic bias at a
country level, which invalidates the cross-country comparison. If the bias appears in both
“objective” and “subjective” questions, then it is particularly hard to identify. We believe,
however, that provided we know the nature of the bias, it is almost always possible to construct a
public opinion survey so that this bias can be later adjusted for.10 This can often be done even
with surveys, like WVS, which were not initially designed to measure governance.
One potential alternative way to adjust for biases is to contrast respondents’ perception with
objective data.11 For example, excessive optimism or pessimism could be identified by looking
at, say, respondents’ perception of their health or crime situation in their neighborhood, and
comparing them with the actual situation. In WVS we do not observe the state of health at a
respondent level, which would be desirable, but we can aggregate the responses at a country
level and contrast them with the country-specific objective data. As a demonstration, Figure 7
10 King et al 2004, King et al 2007, MacKerron 2012 have similar conclusions
11 Approach is similar to Jürges 2007, Olken 2009
shows the average response to a question “How would you describe your state of health today?”
vs. life expectancy, both at a country level. 12 We do find positive correlation between the two:
countries where people live longer rate their state of health higher. Yet, we also find significant
heterogeneity between the countries. For example, in Pakistan in 2012 most people rated their
state of health as good or very good, score of 0.77, whereas in Japan in 2010 the score was more
than 10 percentage points lower. The difference in life expectancy between the two countries is
18 years. It implies that Pakistan’s people are 15% too optimistic relative to the objective
measure of health, while Japan’s people are 16% too pessimistic. In general, the most pessimistic
regions are Central and Eastern Europe and developed countries in East Asia. South Asia and
Middle East seem to be the most optimistic. Interestingly, 80% of the countries in top-20 by the
suicide rate,13 for which we have data, are also in our 25th percentile by the pessimism.
Assuming people apply the same degree of optimism to all other governance-related questions,
CGIs can be adjusted accordingly. For example, Pakistan’s score in 2012 goes down from 0.56
to 0.47 (35th percentile to 70th). Japan’s score in 2010 goes up from 0.52 to 0.61 (46th to 1st
percentile). Our main conclusion carries through though: adjusted CGIs are positively correlated
with quality of life indicators , but they still contain a significant amount of unique information.
12 We exclude Sub-Saharan African countries from the figure because of the unusually high scores these countries
have. Apparently, questions about health carry a certain stigma in these countries, which prompts people to severely
overreport it. We do include these countries in the regression analysis, and add the corresponding regional dummy.
13 According to the World Health Organization. Accessed from:
http://en.wikipedia.org/wiki/List_of_countries_by_suicide_rate (May 3, 2015)
Figure 3 Perception of health (WVS) vs. life expectancy, 2004-2014
6. Conclusions
Listening to the opinions of local residents (nationals) through public opinion surveys is critical
to measuring and monitoring governance as demonstrated by Ivanyna and Shah (2011). This
paper attempts to refine the methodology of such assessment by proposing correction methods to
mitigate systemic biases in opinion surveys as well to update results using recently released data
from Sixth Wave (2010-2014) of World Values Surveys.
The paper identifies and corrects for biases in governance assessment arising from
indoctrination, intimidation, and critical citizenship.. The paper concludes that such adjustments
would invite greater confidence in governance indicators and the resulting measures of citizen
centered governance offer better cross-country and time series comparisons.
A major limitation of the approach advocated in this paper arises from the non-availability of
worldwide survey on governance quality using a uniform questionnaire across countries and over
time. While the WVS is an excellent open source of information, with reasonable geographic,
time and conceptual coverage, the proper measurement of governance requires a specifically
designed survey with stratified random sampling employing a uniform questionnaire across
countries and over time. Differentiating “objective” (personal) from “subjective” (trust)
questions about the government in such survey would enable a researcher to test for the presence
of systemic biases and carry out appropriate corrections. Such corrections could be further
facilitated with comparison of responses with objective data on quality of life indicators from
other sources. This paper has taken first step in achieving greater objectivity in governance
assessment. Much further work lies ahead in establishing full confidence in the validity of
governance indicators for comparative and benchmarking purposes.
REFERENCES
Andrews , M. and A. Shah (2005), “Citizen-Centered Governance: A New Approach to Public
Sector Reform”, in A. Shah, eds., Public Expenditure Analysis, Washington, DC: World Bank,
Chapter 6:153-82.
Arndt, Christiane (2008), “The Politics of Governance Ratings”, International Public
Management Journal 11(3): 275-297
Arndt, C. and C. Oman (2006), Uses and Abuses of Governance Indicators, Development Center
Studies, OECD.
Bertrand, M. and S. Mullainathan (2001), “Do people mean what they say? Implications for
subjective survey data”, American Economic Review 91 (2): 67–72.
Heath, Anthony, Stephen Fisher and Shawna Smith, “The globalization of public opinion
research” (2005), Annual Review of Political Science 8: 297-333
Huther, J. and A. Shah (1998), “Applying a Simple Measure of Good Governance to the Debate
on Fiscal Decentralization”, Policy Research Working Paper 1894, World Bank, Washington,
DC.
Iqbal, K. and A. Shah (2008), “Truth in Advertisement: How do governance indicators stack
up?”, unpublished paper, World Bank Institute, Washington, DC , available at
http;//www.worldbank.org/wbi/publicfinance.
Ivanyna, Maksym and Anwar Shah (2011). Citizen-centric Governance Indicators: Measuring
and Monitoring Governance by Listening to the People . CESifo Forum, 1/2011: 59-71
Jürges, Hendrik (2007), “True health vs. response styles: exploring cross-country differences in
self-reported health”, Health Economics 16(2): 163-178
King, Gary, Christopher Murray, Joshua Salomon and Ajay Tandon (2004), “Enhancing the
validity and cross-cultural comparability of measurement in survey research”, American Political
Science Review 97(4): 567-583
King, Gary and Jonathan Wand (2007), “Comparing incomparable survey responses: Evaluating
and selecting anchoring vignettes”, Political Analysis 15(1): 46-66
Kurtz, M. and A. Schrank (2007), “Growth and Governance: Models, Measures and
Mechanism”, The Journal of Politics 69(2), 538-54.
Langbein, L. and S. Knack (2008), “The Worldwide Governance Indicators and Tautology”,
Policy Research Working Paper no. 4669, July 2008, World Bank, Washington, DC.
MacKerron, George (2012), “Happiness economics from 35000 feet”, Journal of Economic
Surveys 26(4): 705-735
Norris, P. (1999), Critical Citizens: Global Support for Democratic Governance, New York:
Oxford University Press.
Olken, Benjamin (2009), “Corruption perceptions vs. corruption reality”, Journal of Public
Economics 93(7-8): 950-964
Shah, A. (2008), “Demanding to be Served: On Holding Government to Account for Service
Delivery”. In J. de Jong and G. Rizvi , eds., The State of the Access, Washington, DC:
Brookings Institution Press.
Shah, A., and S. Shah (2006), “The New Vision of Local Governance and the Evolving Role of
Local Governments”, in A. Shah, eds., Local Governance in Developing Countries, Washington,
DC: World Bank, Chapter 1: 1-46.
Schrank, A. and M. Kurtz (2008), “Conceptualizing and Measuring Institutions: A View from
Political Science”, unpublished paper, Department of Political Science, Ohio State University
Wang, Z. (2005), “Before emergence of Critical Citizens: Economic Development and Political
Trust in China”, International Review of Sociology 15(1), 155-71.
(WBI) World Bank Insitute (2008), “Worldwide Governance Indicators”,
(info.worldbank.org/governance/wgi/index.asp)
(WVS) World Values Survey (2008), “Official Datafile v.20081015”, World Values Survey
Association (www.worldvaluessurvey.org), aggregate file producer: ASEP/JDS, Madrid.
APPENDICES
TableA1 Governance outcomes: weights and questions assigned
Code Governance criteria
Questions assigned Weight Coverage, %, all waves
Coverage, %, 5th and 6th wave
A Responsive governance
1 safety of life, order, rule of law
How much confidence do you have in police? 0.035 95.8 98.3
3 improvements in economic and social outcomes
How satisfied are you with the financial situation of your household?
0.15 98.3 99.2
4 improvements in quality of life: general
All things considered, how satisfied are you with your life as a whole these days?
0.1 99.2 100
5 improvements in quality of life: health
All in all, how would you describe your state of health today?
0.07 97.5 100
6 peace How much confidence do you have in armed forces?
0.035 94.6 95.7
7 inmprovements in quality of life: happiness
Taking all things together would you say you are [happy, unhappy]?
0.1 99.6 100
B Fair governance
1 social justice, respect for human rights
How much respect is there for individual human rights nowadays in the country?
0.08 76.5 94
2 government represents the whole country
How proud are you to be your nationality? 0.035 98.7 99.1
3 government represents the whole country
Would you fight for your country? 0.035 94.4 99.1
C Responsible governance
1A earning trust: executive branch
How much confidence do you have in government?
0.075 87.9 98.3
1B earning trust: legislative branch
How much confidence do you have in parliament?
0.075 95.4 97.4
2 earning trust: general
How much confidence do you have in civil services?
0.07 96.2 98.3
D Accountable governance
1A access to information, independent mass media - press
How much confidence do you have in press? 0.035 97.1 99.1
1B access to information, independent mass media - television
How much confidence do you have in television?
0.035 89.1 100
2 judicial integrity and independence
How much confidence do you have in courts? 0.07 79.9 96.6
Note: The data source for all questions is World Values Surveys (WVS). Questions are common to all six waves of the survey.
Table A2 CGI, raw and adjusted, waves 5 and 6