methodology v6 - v-dem v6.pdf · natalia stepanova, eitan tzelgov, and yi-ting wang. 2016. “v-dem...
TRANSCRIPT
I N S T I T U T E
Methodology
Copyright © University of Gothenburg, V-Dem Institute,
University of Notre Dame, Kellogg Institute. All rights reserved.
Version 6 - Mar 2016
1
Suggestedcitation:Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,KyleL.Marquardt,ValeriyaMechkova,FarhadMiri,DanielPemstein,JosefinePernes,NataliaStepanova,EitanTzelgov,andYi-tingWang.2016.“V-DemMethodologyv6.”VarietiesofDemocracy(V-Dem)Project.
Authors Collaborators
• MichaelCoppedge–U.ofNotreDame • DavidAltman–PontificiaU.CatólicadeChile• JohnGerring–BostonUniversity • MichaelBernhard–UniversityofFlorida• StaffanI.Lindberg–U.ofGothenburg • M.StevenFish–UCBerkeley• Svend-ErikSkaaning–AarhusUniversity • AdamGlynn–EmoryUniversity• JanTeorell–LundUniversity • AllenHicken–UniversityofMichigan• FridaAndersson–U.ofGothenburg• KyleL.Marquardt–U.ofGothenburg• ValeriyaMechkova–U.ofGothenburg• FarhadMiri–Uof.Gothenburg• DanielPemstein–NorthDakotaStateU.• JosefinePernes–U.ofGothenburg• NataliaStepanova–U.ofGothenburg
• CarlHenrikKnutsen–UniversityofOslo• KellyMcMann–CaseWesternReserve
PamelaPaxton–U.ofTexas• JeffreyStaton–EmoryUniversity• BrigitteZimmerman–U.ofNorthCarolina
• EitanTzelgov–U.ofEastAnglia&U.ofGothenburg
• Yi-tingWang–NationalChengKungU& U.ofGothenburg
2
TableofContents
1. CONCEPTUALSCHEME 4
PRINCIPLES–MEASUREDBYV-DEM’SDEMOCRACYINDICES 4COMPONENTS 10INDICATORS 12SUMMARY 12
2. DATACOLLECTION 15
HISTORYOFPOLITIES 15CODINGTYPES 17EXPERTRECRUITMENT 18EXPERTCODINGPROCESS 22BRIDGE-ANDLATERALCODING 26PHASES 27
3. MEASUREMENT 28
THEQUESTIONNAIRE 28IDENTIFYING,CORRECTING,ANDQUANTIFYINGMEASUREMENTERROR 29
MeasurementModels 30CorrectingErrors 36
VERSIONSOFC-VARIABLES 37ADDITIONALPOSSIBILITIESFORIDENTIFYINGSOURCESOFMEASUREMENTERRORINTHEFUTURE 40
REFERENCES 43
APPENDIXA:V-DEMINDICES,COMPONENTS,ANDINDICATORS 47
3
This document outlines the methodological considerations, choices, and procedures
guidingthedevelopmentoftheVarietiesofDemocracy (V-Dem)project.Part Isetsforth
theconceptual scheme.Part IIdiscusses theprocessofdatacollection.Part IIIdescribes
themeasurementmodelalongwitheffortstoidentifyandcorrecterrors.
Wecontinuallyreviewourmethodology—andoccasionallyadjustit—withthegoal
ofimprovingthequalityofV-Demindicatorsandindices.Wethereforeissueanewversion
ofthisdocumentwitheachnewversionofthedataset.
Additional project documents complement this one.V-Dem Codebook includes a
comprehensive list of indicators, response-categories, sources, and brief information
regarding theconstructionof indices.V-DemCountryCodingUnits explainshowcountry
unitsaredefinedand listseachcountry included inthedataset,withnotespertainingto
the years covered and special circumstances that may apply. V-Dem: Comparisons and
ContrastswithOtherMeasurementProjectssurveysthefieldofdemocracyindicatorsand
situates the V-Dem project in relation to these efforts. V-Dem Organization and
Management introduces the project team, the web site, outreach to the international
community,funding,progresstodate,andsustainability.
Versioning of the current document, V-Dem Codebook, V-Dem Country Coding
Units and V-Dem Organization & Management documents are synchronized with the
release of each new dataset. Versioning of other documents is not synchronized.
(Currently,weareatv5.)
Several configurationsof theV-Demdatasetareavailable, including country-date
andcoder-leveldatasets.Foradditionaldocumentationandguidance,users should refer
totheOtherProjectDocumentationfilethatisappendedtoeachdatadownload.
IntheV-DemWorkingPaperSeries,userscanfindamoretechnicaldiscussionof
the measurement model we use to aggregate coder-level data to point estimates for
country-years (Pemstein et al. 2015, WP #21). Working Paper #6 introduces the
democracy indices. Working Paper #25 details the Electoral Democracy index. Working
Paper#22describestheindexofEgalitarianDemocracy.Additionalworkingpapersprovide
in-depthtreatmentsofmorespecialized indicessuchastheFemaleEmpowerment Index
(#19),theCoreCivilSocietyIndex(#13),thePartySystemInstitutionalizationIndex(#26),
theCorruption Index (#23), andordinal versionsof theV-Dem indices (#20). TheV-Dem
4
WorkingPaperSeriesisavailablefordownloadontheV-Demwebsite(v-dem.net).
V-Dem isamassive,global collaborativeeffort.Anup-to-date listingofourmany
collaborators,withoutwhom this projectwouldnotbepossible, is also availableon the
web site. Collaborators include Program Managers, Regional Managers, International
AdvisoryBoardmembers,theV-DemInstitutestaff(Director,Program-,Operations-,Data
Processing and Data Managers, Assistant Researchers, and Post-Doctoral Fellows and
Associate Researchers), Research Assistants, and Country Coordinators. We are also
especially indebted to some 2,500 Country Experts, whose identities must remain
anonymousforethicalreasons.
The website serves as the repository for other information about the project,
including Country and Thematic Reports, Briefing Papers, publications, grant and
fellowshipopportunities,andthedataitself.Dataforall173countriesincludedinthefirst
publicrelease(V-DemDatasetv5)isalsoavailableforexplorationwithonlineanalysistools
(countryandvariablegraphs,motioncharts,and–soon–globalmaps).
1. ConceptualScheme
Anymeasurementschemerestsonconcepts. Inthissection,wesetforththeconceptual
schemethatinformstheV-Demproject–beginningwith“democracy”andproceedingto
thepropertiesandsub-propertiesofthatfar-flungconcept.Bywayofconclusion,weissue
severalclarificationsandcaveatsconcerningtheconceptualscheme.V-Dem:Comparisons
andContrastsprovidesamoredetaileddiscussion,butwerecaptheessentialpointshere.
Principles–MeasuredbyV-Dem’sDemocracyIndices
Thereisnoconsensusonwhatdemocracywrit-largemeansbeyondavaguenotionofrule
bythepeople.Politicaltheoristshaveemphasizedthispointforsometime,andempiricists
woulddowelltotakethelessontoheart(Gallie1956;Held2006;Shapiro2003:10–34).At
thesametime,interpretationsofdemocracydonothaveanunlimitedscope.
A thorough search of the literature on this protean concept reveals seven key
principles that inform much of our thinking about democracy: electoral, liberal,
majoritarian, consensual, participatory, deliberative, and egalitarian. Each of these
5
principlesrepresentsadifferentwayofunderstanding“rulebythepeople.”Theheartof
thedifferences between theseprinciples is in the fact that alternate schools of thought
prioritize different democratic values. Thus, while no single principle embodies all the
meanings of democracy, these seven principles, taken together, offer a fairly
comprehensiveaccountingoftheconceptasemployedtoday.1
The V-Dem project has set out to measure these principles, and the core values
whichunderliethem.Wesummarizetheprinciplesbelow.
• Theelectoralprincipleofdemocracyembodiesthecorevalueofmaking
rulers responsive to citizens through periodic elections, as captured by
Dahl’s (1971, 1989) conceptualization of “polyarchy.” Our measure for
electoral democracy is called the “V-Dem Electoral Democracy Index.”
We consider this measure fundamental to all other measures of
democracy:wewouldnotcallaregimewithoutelections“democratic”in
anysense.
• The liberal principle of democracy embodies the intrinsic value of
protecting individualandminority rightsagainstapotential “tyrannyof
the majority” and state repression. This principle is achieved through
constitutionally-protectedcivil liberties,strongruleof law,andeffective
checksandbalancesthatlimittheuseofexecutivepower.
• Theparticipatoryprincipleembodiesthevaluesofdirectruleandactive
participation by citizens in all political processes.While participation in
elections counts toward this principle, it also emphasizes nonelectoral
forms of political participation, such as civil society organizations and
other forms of both nonelectoral and electoral mechanisms of direct
democracy.
1Thisconsensusonlyholdsinsofarasmostscholarswouldagreethatsomepermutationoraggregationoftheseprinciplesunderlieconceptionsofdemocracy.Forexample,scholarscanreasonablyarguethatthelistcouldconsistofseven,six,orfiveprinciples;our“principles”maybe“properties”or“dimensions;”and“majoritarian”and“consensual”areactuallyoppositepolesofasingledimension.Asaresult,weintendforthisdiscussiontoassureconsumersofthedataofthecomprehensivenatureofourinventoryofcorevaluesofdemocracy:namely,thatitincludesalmostalltheattributesthatanyuserwouldwanttohavemeasured.
6
• Thedeliberativeprincipleenshrinesthecorevaluethatpoliticaldecisions
in pursuit of the public good should be informed by a process
characterizedbyrespectfulandreason-baseddialogueatalllevels,rather
thanbyemotionalappeals,solidaryattachments,parochial interests,or
coercion.
• Theegalitarianprincipleholdsthatmaterialand immaterial inequalities
inhibit the actual use of formal political (electoral) rights and liberties.
Ideally,allgroupsshouldenjoyequaldejureanddefactocapabilitiesto
participate;toserve inpositionsofpoliticalpower;toput issuesonthe
agenda; and to influence policymaking. Following the literature in this
tradition, gross inequalities of health, education, or income are
understood to inhibit the exercise of political power and the de facto
enjoymentofpoliticalrights.
Theconceptualschemepresentedabovedoesnotcaptureallthetheoreticaldistinctions
atplayinthecomplexconceptofdemocracy.Wehavechosentofocusonthecorevalues
and institutions that the other principles emphasize in their critique of the electoral
conceptionas a stand-alone system.Eachof theseprinciples is logicallydistinct and—at
least for some theorists—independently valuable.Moreover,we suspect that there is a
considerable divergence in the realizationof theproperties associatedwith these seven
principles among the world’s polities. Some countries will be particularly strong on
electoraldemocracy;otherswillbestrongontheegalitarianproperty,andsoforth.
AggregationProcedures
At this point, V-Dem offers separate indices of five varieties of democracy: electoral,
liberal,participatory,deliberative,andegalitarian.Weanticipateprovidingindicesforthe
remaining two principles – majoritarian and consensual – in the near future.2 V-Dem
Codebook contains the aggregation rules for each index and several V-Dem Working
2Themajoritarianprincipleofdemocracy(reflectingthebeliefthatamajorityofthepeoplemustbecapacitatedtoruleandimplementtheirwillintermsofpolicy);andtheconsensualprincipleofdemocracy(emphasizingthatamajoritymustnotdisregardpoliticalminoritiesandthatthereisaninherentvalueintherepresentationofgroupswithdivergentinterestsandview).
7
Papers (present and forthcoming) lay out justifications for the choices made in each
aggregation scheme.Thehigh-level indices,measuring coreprinciplesofdemocracy, are
referredtoasdemocracyindices.
Sartori held that every defining attribute is necessary for the concept. This logic
requires multiplying the attributes so that each of them affects the index only to the
degreethattheothersarepresent.Familyresemblancedefinitionsallowsubstitutability:a
high value on one attribute can compensate for a low value on another. This logic
correspondstoanadditiveaggregationformula.Therearesoundjustificationsfortreating
all of these attributes as necessary, or mutually reinforcing. For example, if opposition
candidatesarenotallowedtorunforelectionortheelectionsarefraudulent,thefactthat
alladultshavevotingrightsdoesnotmattermuchforthelevelofelectoraldemocracy.But
therearealsogoodreasonstoregardtheseattributesassubstitutable.Wherethesuffrage
is restricted, the situation is less undemocratic if the disenfranchised are still free to
participate in associations, to strike and protest, and to access independent media
(Switzerland before 1971) than if they lack these opportunities (Italy under Mussolini).
Even where the executive is not elected, citizens can feel that they live in a fairly
democraticenvironmentaslongastheyarefreetoorganizeandexpressthemselves,asin
Liechtensteinbefore2003.
Becausewebelieveboth thenecessary conditions and family resemblance logics
arevalid forconceptsofdemocracy,ouraggregation formulas includeboth;becausewe
have no strong reason to prefer the additive terms to themultiplicative term, we give
themequalweight. The index is formedby taking theaverageof,on theonehand, the
weighted average of the indices measuringfreedom of association (thick)
(v2x_frassoc_thick), clean elections (v2xel_frefair),freedom of expression
(v2x_freexp_thick),elected executive (v2x_accex), andsuffrage (v2x_suffr) and, on the
other, the five-way multiplicative interaction between those indices. This is half way
betweenastraightaverageandstrictmultiplication,meaningtheaverageofthetwo.Itis
thus a compromise between the two most well-known aggregation formulas in the
literature, both allowing (partial) "compensation" in one sub-component for lack of
polyarchy in the others, but also punishing countries not strong in one sub-component
according to the"weakest link"argument.Theaggregation isdoneat the levelofDahl’s
8
sub-components (with the one exception of the non-electoral component). The index is
aggregatedusingthisformula:
v2x_polyarchy=.5MPI+0.5API
=.5(v2x_accex*v2xel_frefair*v2x_frassoc_thick*v2x_suffr*v2x_freexp_thick)
+.5(1/8v2x_accex+1/4v2xel_frefair+1/4v2x_frassoc_thick+1/8v2x_suffr+
1/4v2x_freexp_thick)
Thesumof theweightsof theadditive termsequals theweightof the interaction
term. The additive part of the formula lets the two components that can achieve high
scores based on the fulfillment of formal-institutional criteria (elected officials and
suffrage) together weigh half as much as the other components that enjoy a stronger
independentstandingintermsofrespectfordemocraticrights(cleanelections,freedom
oforganizationandexpression).3Inanyevent,becausemostofthevariablesarestrongly
correlated, different aggregation formulas yield very similar index values. The official
formula presented here correlates at .94 to .99 with a purely multiplicative formula, a
purely additive formula, one that weights the additive terms twice as much as the
multiplicativeterm,onethatweightsthemultiplicativetermtwiceasmuchastheadditive
terms,andonethatweightssuffragesix timesasmuchas theotheradditiveterms.The
maindifferenceacrosstheseformulas is intheirmeanvalues,withsomebeingcloserto
oneandothers(i.e.themoremultiplicativeformulas)beingclosertozero.
The Electoral Democracy Index also serves as the foundation for the other four
indices.Therecanbenodemocracywithoutelectionsbut,followingthecanonineachof
thetraditionsthatarguesthatelectoraldemocracy is insufficientforatruerealizationof
“rule by the people,” there is more to democracy than just elections. We therefore
combinethescoresforourElectoralDemocracyIndex(v2x_polyarchy)withthescoresfor
the components measuring deliberation, equalitarianism, participation, and liberal
constitutionalism, respectively. This is not an easy task. Imagine two components,
3OnecouldarguethatthesuffragedeservesgreaterweightbecauseitliesonadifferentdimensionthantheothersandisthekeycomponentofoneofDahl’stwodimensionsofpolyarchy(Dahl1971;Coppedgeetal.2008).However,ourformulaallowsarestrictedsuffragetolowertheElectoralDemocracyIndexconsiderablybecauseitdiscountsalltheothervariablesinthemultiplicativeterm.
9
P=Polyarchy and HPC=High Principle Component (liberal, egalitarian, participatory, or
deliberative),4thatwewanttoaggregateintomoregeneraldemocracyindices,whichwe
will call DI (DeliberativeDemocracy Index, EgalitarianDemocracy Index, and so on). For
convenience,bothPandHPCarescaledtoacontinuous0-1interval.Basedonextensive
deliberations among the authors and othermembers of the V-Dem research group,we
tentativelyarrivedatthefollowingaggregationformula:
DI=.25*P1.6+.25*HPC+.5*P1.6*HPC
TheunderlyingrationaleforthisformulaforallfourDIsisthesameasthatforthe
ElectoralDemocracy Index: equalweighting of the additive terms and themultiplicative
term in order to respect both the Sartorian necessary conditions logic and a family
resemblance logic. For example, the degree of deliberation stillmatters for deliberative
democracy even when there is no electoral democracy, and electoral democracy still
matters even when there is no deliberation; but the highest level of deliberative
democracycanbeattainedonlywhenthereisahighlevelofbothelectoraldemocracyand
deliberation.
The more a country approximates polyarchy, the more its combined DI score
should reflect the unique component. This perspective is a continuous version of
theoretical arguments presented in the literature saying that polyarchy or electoral
democracy conditions should be satisfied to a reasonable extent before the other
democracycomponentgreatlycontributestothehighlevelindexvalues.Atthesametime,
itreflectstheviewintheliteraturethat,whenacertainlevelofpolyarchyisreached,what
matters in terms of, say, participatory democracy is how much of the participatory
property is realized. This argument also resembles the widespread perspective in the
quality of democracy literature emphasizing that the fulfillment of some baseline
democracycriteriaisnecessarybeforeitmakessensetoassessthequalityofdemocracy.5
Given this body of literature, it becomes necessary to specify the rate at which a
componentshouldinfluenceaDIscore.Wedosobyraisingthevalueofacomponentby
1.6.We identify this numeric value by defining an anchor point:when a country has a
4TheHPCsareindicesbasedontheaggregationofalargenumberofindicators(liberal=23,egalitarian=8,participatory=21,deliberative=5).
5Foranoverview,seeMunck(2016).
10
polyarchy score of .5 (in practice, this is a threshold on the Electoral Democracy Index
beyondwhichcountriestendtobeconsideredelectoraldemocracies inaminimalsense)
anditsHPCisatitsmaximum(1),thehighlevelindexscoreshouldbe.5.6
Takentogether,theseindicesofferafairlycomprehensiveaccountingof“varieties
of democracy.” The five (soon to be seven) democracy indices constitute a first step in
disaggregatingtheconceptofdemocracy.Thenextstepisthecomponents.
Components
Themain democracy components, already included in the discussion above, specify the
distinct properties associatedwith the principles. TheV-DemElectoralDemocracy Index
consists of five components (each of these components being indices themselves built
fromanumberofindicators)thattogethercaptureDahl’sseveninstitutionsofpolyarchy:
freedom of association, suffrage, clean elections, elected executive, and freedom of
expression.Thecomponent indicesmeasuringthe liberal,deliberative,participatory,and
egalitarianpropertiesof democracy (majoritarian and consensualwill be released in the
near future) follow the principles of democracy described in the previous section – but
without the core, unifying element of electoral democracy. They capture only what is
unique for eachof the principles. As such, these components aremutually exclusive, or
orthogonaltoeachother.
These main democracy components typically have several sub-components. For
example, the liberal democracy component consists of three sub-components, each
capturedwithitsownindex:theEqualitybeforethelawandindividuallibertyindex;the
Judicialconstraintsontheexecutiveindex;andtheLegislativeconstraintsontheexecutive
index.
InadditiontothecomponentandsubcomponentindicesthatarepartoftheV-Dem
democracy indicesconceptual scheme,membersof theV-Demteamhaveconstructeda
series of indices of lower-level concepts such as civil society, party system
institutionalization, corruption, andwomen’s political empowerment.We also list these
6Definetheexponentasp.SettingPolyarchy=.5,HPC=1,andHLI=.5,andsolvingforDI=.25*Polyarchy^p+.25*HPC+.5*Polyarchy^p*HPC,p=log(base0.5)of.25/.75≈1.6.
11
indicesintheappendix.Intotal,V-Demoffers39indicesofcomponents,subcomponents,
and related concepts. TheV-Demdataset includes all of these indices. PublishedV-Dem
working papers already detail many of these indices (e.g. papers #6, #13, #17-20).
Additionalworkingpaperswillprovidefurtherdetailsonotherindices.
Weusetwotechniqueswhenaggregatingintodemocracyindices,components,and
subcomponents, as well as related concepts’ indices. For the first step, going from
indicatorsto(sub-)components,weaggregatethelatentfactorscoresfrommeasurement
model (MM) output. More specifically, we use relevant theoretical distinctions in the
literature togroup interval-levelMMoutput into setsof variables that sharea common
underlying concept.We then randomly select 100 draws from each variable’s posterior
distribution(seedetailsunder“MeasurementModels”below),anduseaunidimensional
Bayesian factor analysis (BFA) to measure this latent concept sequentially for each
randomly-selected draw in each grouping of variables. We then combine the posterior
distributions of the latent factor scores in each variable group to yield the latent factor
scores.Inallanalysesthevariablesgenerallyloadhighlyontheunderlyingfactor.
For the next level in the hierarchy –another subcomponent, a component, or a
democracyindexdependingonthecomplexityoftheconceptualstructure(seeAppendix
A)–we take the latent factor scores from the separateBFAsanduse in combination in
constructingthe“HigherLevelIndices”(HLIs).HLIsarethuscompositemeasuresthatallow
thestructureoftheunderlyingdatatopromulgatethroughthehierarchyinthesameway
astheBFAsdo–andcriticallycarryoverthefullinformationaboutuncertaintytothenext
level in order to avoid allowing the aggregation technique artificially increase the
estimated confidence – while being faithful to the theoretically informed aggregation
formula.FollowingtheformulaofeachHLI(seetheV-DemCodebook),wetakeaveragesor
products of each of the relevant BFA factor score posterior distributions, and then
calculatethepointestimates(means)andconfidenceintervalsacrosstheresultingmatrix
to generate the HLI estimates. For example, the liberal component of democracy index
comprises three elements: equality before the law and individual liberties, judicial
constraintsontheexecutive,andlegislativeconstraintsontheexecutive.Webelievethese
threeelementsaresubstitutiveandthereforetaketheaverageofthesethreeelementsto
constructtheliberalcomponentindex.FortheDIs,weusetheequationsdiscussedabove
toassignweightstothecombinations.
12
Indicators
Thefinalstepindisaggregationistheidentificationof indicators. In identifyingindicators
welookforfeaturesthat(a)arerelatedtoat leastonepropertyofdemocracy;(b)bring
thepoliticalprocessintocloseralignmentwiththecoremeaningofdemocracy(rulebythe
people);and(c)aremeasurableacrosspolitiesandtime.
Indicatorstaketheformofnominal(classifications,text,dates),ordinal(e.g.,Likert-
stylescales),orintervalscales.Somerefertodejureaspectsofapolity–rulesthatstatute
or constitutional law (including the unwritten constitution of states like the United
Kingdom) stipulate. Others refer tode facto aspects of a polity – theway things are in
practice.
Therearesome350uniquedemocracyindicatorsintheV-Demdataset.Welisteach
indicator, along with its response-type, in the V-Dem Codebook. We discuss coding
procedures in greater detail in the next section. The V-Dem dataset contains many
indicators that we do not include in the component and democracy indices discussed
above,thoughtheyarerelatedtodemocracy.Theirabsencereflectsthefactthatwehave
soughttomakethecomponent-anddemocracyindicesasorthogonalaspossibletoeach
other,andalsoasparsimoniousaspossible.Furthermore,wheneverwehavemeasuresof
boththedejureandthedefactosituationinastate,ourindicesbuildprimarilyonthede
facto indicators because we want the measures to portray the “real situation on the
ground”asfaraspossible.
Summary
Tosummarize,theV-Demconceptualschemerecognizesseverallevelsofaggregation:
●Coreconcept(1)●DemocracyIndices(5,soontobe7)
●DemocracyComponents(5)●Subcomponents,andrelatedconcepts(34)
●Indicators(≈350)
As anappendix to this document,weattacha tablewith a completehierarchyof
democracyindices,democracycomponentindices,democracysubcomponentindices,and
13
indicators,aswellasthehierarchyofrelatedconceptindices.
Several important clarifications apply to this taxonomy. First, our attempt to
operationalize democracy does not attempt to incorporate the causes of democracy
(exceptinsofarassomeattributesofourfar-flungconceptmightaffectotherattributes).
Regime-typesmay be affected by economic development (Epstein et al. 2006), colonial
experiences (Bernhard et al. 2004), or attitudes and political cultures (Almond& Verba
1963/1989; Hadenius & Teorell 2005; Welzel 2007). However, we do not regard these
attributesasconstitutiveofdemocracy.
Second,ourquesttoconceptualizeandmeasuredemocracyshouldnotbeconfused
with the quest to conceptualize and measure governance.7 Of course, there is overlap
betweenthesetwoconcepts,sincescholarsmayconsidermanyattributesofdemocracyto
beattributesofgoodgovernance.
Third,werecognizethatsome indicatorsandcomponents (listed intheCodebook)
are more important in guaranteeing a polity’s overall level of democracy than others,
thoughthepreciseweightingparametersdependuponone’smodelofdemocracy.
Fourth, aspects of different ideas of democracy sometimes conflict with one
another.At the levelofprinciples, there isanobviousconflictbetweenmajoritarianand
consensualnorms,whichadoptcontraryperspectivesonmost institutional components.
Forexample,protecting individual libertiescan impose limitson thewillof themajority.
Likewise,strongcivilsocietyorganizationscanhavetheeffectofpressuringgovernmentto
restrict thecivil libertiesenjoyedbymarginalgroups (Isaacn.d.).Furthermore, thesame
institutionmaybedifferently viewedaccording todifferentprinciplesofdemocracy. For
example,thecommonpracticeofmandatoryvotingisclearlycontrarytotheliberalmodel
(where individual rights are sacrosanct and include the right not to vote), but the
participatorymodelsupportsthispractice,since ithasademonstratedeffect inboosting
turnoutwhereversanctionsaremorethannominal.
7SeeRose-Ackerman(1999)andThomas(2010).Inglehart&Welzel(2005)arguethateffectivedemocracy–asopposedtopurelyformalorinstitutionaldemocracy–islinkedtoruleoflaw:aformallydemocraticcountrythatisnotcharacterizedbytheruleoflawisnotdemocraticinthefullsenseoftheterm.InordertorepresentthisthickconceptofdemocracytheymultiplytheFreedomHouseindicesbyindicesofcorruption(drawnfromTransparencyInternationalortheWorldBank),producinganindexofeffectivedemocracy.SeeHadenius&Teorell(2005)andKnutsen(2010)forcriticaldiscussions.
14
Such contradictions are implicit in democracy’s multidimensional character. No
wide-ranging empirical investigation can avoid conflicts among democracy’s diverse
attributes. However, with separate indicators representing these different facets of
democracyitshouldbepossibletoexaminepotentialtradeoffsempirically.
Fifth, our proposed set of democracy indices, components, and indicators, while
fairlycomprehensive,isbynomeansexhaustive.Theproteannatureofdemocracyresists
closure; there are always potentially new properties/components/indicators that, from
one perspective or another, may be associated with this essentially contested term.
Moreover, some conceptions of democracy are difficult to capture empirically; this
difficultyincreaseswhenanalyzingtheseconceptionsovertimeandacrosscountriesona
globalscale.Thisfactlimitsthescopeofanyempiricalendeavor.
Sixth,principlesandcomponents,whilemucheasiertodefinethandemocracy (at-
large), are still resistant to authoritative conceptualization. Our objective has been to
identifythemostessentialanddistinctiveattributesassociatedwiththeseconcepts.Even
so,wearekeenlyawarethatothersmightmakedifferentchoices,andthatdifferenttasks
require different choices. The goal of the proposed conceptual framework is to provide
guidance,notto legislate inanauthoritativefashion.Theschemademonstrateshowthe
various elements of V-Dem hang together, according to a particular set of inter-
relationships. We expect other writers will assemble and dis-assemble these parts in
whateverfashionsuitstheirneedsandobjectives.Inthisrespect,V-Demhasthemodular
qualitiesofaLegoset.
Finally,asshouldbeobvious,thissectionapproachesthesubjectfromaconceptual
angle.Elsewhere(e.g.,intheV-DemCodebookandinV-DemComparisonsandContrasts,
aswellasinworkingpapersfoundontheV-Demwebsite),wedescribetechnicalaspects
ofindexconstructioninmoredetail.
15
2. DataCollection
Theviabilityofanydatasethingescriticallyonitsmethodofdatacollection.V-Demaimsto
achievetransparency,precision,andrealisticestimatesofuncertaintywithrespecttoeach
(evaluativeandindex)datapoint.
HistoryofPolities
Ourprincipalconcern iswiththeoperationofpolitical institutionsthatexistwithin large
andfairlywell-definedpoliticalunitsandwhichenjoyamodicumofsovereigntyorserveas
operational units of governance (e.g., colonies of overseas empires).We refer to these
unitsaspolitiesorcountries.8
Wearenotconcernedmerelywiththepresentandrecentpastofthesepolities.In
our view, understanding the present – not tomention the future – requires a rigorous
analysisofhistory.Theregimesthatexisttoday,andthosethatwillemergetomorrow,are
theproductofcomplexprocessesthatunfoldoverdecades,perhapscenturies.Although
regime changes are sometimes sudden, like earthquakes, these dramatic events are
perhaps sometimes to be understood as a combination of pent-up forces that build up
over longspansof time,notsimplytheprecipitating factorsthatreleasethem.Likewise,
recent work has raised the possibility that democracy’s impact on policies and policy
outcomestakeeffectoveraverylongperiodoftime(Gerringetal.,2005)andthatthere
are indeed sequences in terms of necessary conditions in democratization (Wang et al.
2015).Arguably,short-termandlong-termeffectsarequitedifferent,whetherdemocracy
isviewedasthecauseoroutcomeoftheoreticalinterest.Forallthesereasons,webelieve
thatafullunderstandingofdemocratizationdependsuponhistoricaldata.9
8Wearenotmeasuringdemocracywithinverysmallcommunities(e.g.,neighborhoods,schoolboards,municipalities,corporations),incontextswherethepoliticalcommunityisvaguelydefined(e.g.,transnationalmovements),oronagloballevel(e.g.,theUnitedNations).Thisisnottosaythattheconceptofdemocracyshouldberestrictedtoformalandwell-definedpolities.Itissimplytoclarifyourapproach,andtoacknowledgethatdifferentstrategiesofconceptualizationandmeasurementmayberequiredfordifferentsubjectareas.
9ThisechoesapersistentthemepresentedinCapocciaandZiblatt(2010),Knutsen,Møller&Skaaning(forthcoming),Teorell(2011),andinotherhistoricallygroundedwork(Nunn2009;Mahoney&Rueschemeyer2003;Pierson2004;Steinmo,Thelen,&Longstreth1992).
16
The advantage of our topic – in contrastwith other historicalmeasurement tasks
suchasnationalincomeaccounts–isthatmuchoftheevidenceneededtocodefeatures
of democracy is preserved in books, articles, newspapers archives, and living memory.
Democracyis,afterall,ahigh-profilephenomenon.Althoughasecretiveregimemayhide
thetruevalueofgoodsandservicesinthecountry,itcannotdisguisetheexistenceofan
election; those features of an election that might prejudice the outcome toward the
incumbent are difficult to obscure completely. Virtually everyone living in that country,
studyingthatcountry,orcoveringthatcountryforsomeforeignnewsorganizationoraid
organizationhasaninterestintrackingthisresult.
Thus,weregardthegoalofhistoricaldatagatheringasessentialandalsorealistic,
even if it cannot be implemented for every possible indicator of democracy. V-Dem
thereforeaimstogatherdata,wheneverpossible,backto1900forallterritoriesthatcan
claimasovereignorsemi-sovereignexistence(i.e.theyenjoyedadegreeofautonomyat
leastwithrespecttodomesticaffairs)andserveastheoperationalunitofgovernance.The
latter criterionmeans that they are governed differently from other territories and we
might reasonably expect many of our indicators to vary across these units. Thus, in
identifying political units we look for those that have the highest levels of autonomy
and/or are operational units of governance. These sorts of units are referred to as
“countries,” even if they are not fully sovereign. This means, for example, that V-Dem
provides a continuous time-series for Eritrea coded as an Italian colony (1900-41), a
provinceofItalianEastAfrica(1936-41),aBritishholdingadministeredunderthetermsof
aUNmandate(1941-51),afederationwithEthiopia(1952-62),aterritorywithinEthiopia
(1962-93), and an independent state (1993-). For further details, see V-Dem Country
CodingUnits.Inthefuture,weplantoaddinformationinthedatasetanddocumentation
to link predecessor and successor states, facilitating panel analysis with continuous
country-levelunits.
V-Dem provides time-series ratings that reflect historical changes as precisely as
possible. Election-specific indicators are coded as events occurring on the date of the
election.Wecodeotherindicatorscontinuously,withanoption(thatsomecodersutilize)
tospecifyexactdates(day/month/year)correspondingtochangesinaninstitution.
Date-specificdatacanbeaggregatedat12-monthintervals,whichmaybeessential
17
for time-series where country-years form the relevant units of analysis. The V-Dem
“standard”dataset is inthecountry-year format,wheredate-specificchangeshavebeen
aggregatedtogetherattheyearlevel.However,wealsoprovideacountry-datedatasetfor
userswhowant greater precision. In the data archive accessible via the data download
pageonourwebsite,wealsoprovidetherawcoder-leveldata.Doingsoallowsusers to
inspect thedatadirectlyoruse it for alternateanalyses. Finally, in the samearchivewe
also provide the posterior distributions from the Bayesian ordinal IRT model for each
variabletofacilitatetheirdirectuseinanalyses.
Currently,weareworking toextendV-Demcodingback further inhistorical time,
i.e., to1789, for85sovereigncountriesandforaselectionof indicators.Thiscodingwill
enhance our knowledge of democratic development for countries whose process of
democratizationbeganpriortothetwentiethcentury.Itwillalsoenhanceourknowledge
of the pre-democratic history of all countries, a history that may exert an enduring
influenceoversubsequentdevelopmentsinthe20thand21stcenturies.
CodingTypes
The350+V-DemspecificindicatorslistedinV-DemCodebookfallintofourmaintypes:(A)
factual indicatorscodedbymembersof theV-Demteam, (B) factual indicatorscodedby
Country Coordinators, (C) evaluative indicators based on multiple ratings provided by
experts, and (D) composite indices.Part IofV-DemCodebook describes these indicators
Parts II and III provide a fifth type of indicators: (E) extant data (both factual and
subjective).
We gather Type (A) data from extant sources, e.g., other datasets or secondary
sources,as listed intheCodebook.Thesedataare largelyfactual innature,thoughsome
coderjudgmentmayberequiredininterpretinghistoricaldata.PrincipalInvestigatorsand
Project Managers supervise the collection of these data, which Assistant Researchers
connected to the project carry out using multiple sources, with input from V-Dem’s
CountryCoordinators.
CountryCoordinators,underthesupervisionofRegionalManagers,gatherType(B)
datafromcountry-specificsourcesby.AswithType(A)data,thissortofcodingislargely
18
factualinnature.
Type(C)datarequiresagreaterdegreeof judgmentaboutthestateofaffairs ina
particular country at a particular point in time. Country Experts code these data. These
experts are generally academics (about 80%) or professionals working in government,
media, or public affairs (e.g., senior analysts, editors, judges); they are also generally
nationalsofand/orresidents inacountryandhavedocumentedknowledgeofboththat
country and a specific substantive area. Generally, each Country Experts code only a
selection of indicators following their particular background and expertise (e.g. the
legislature).
Type (D) data consists of indices composed from (A), (B), or (C) variables. They
includecumulativeindicatorssuchas“numberofpresidentialelectionssince1900”aswell
as more highly aggregated variables such as the components and democracy indices
describedintheprevioussectionanddetailedinAppendixA.
WedrawType(E)datadirectlyfromothersources.TheyarethereforenotaV-Dem
product.TherearetwogenresofE-data.Thefirstgenreconsistsofalternativeindicesand
indicators of democracy found in Part II of V-Dem Codebook, which may be useful to
compare and contrast with V-Dem indices and indicators. This genre also includes
alternative versions of the V-Dem indices that are ordinal instead of interval (Lindberg
2015).ThesecondtypeofE-indicatorsconsistoffrequentlyusedcorrelatesofdemocracy
suchasGDP.TheyarefoundinPartIII.
ExpertRecruitment
Type (C) coding–byCountry Experts –involves evaluative judgmentson thepart of the
coder.Asaresult,wetakeanumberofprecautionstominimizeerrorinthedataandto
gaugethedegreeofimprecisionthatremains.10
An important aspect of these precautions is the fact thatwe endeavor to find a
minimumoffiveCountryExpertstocodeeachcountry-yearforeveryindicator.Thequality
and impartiality of C-data naturally depends on the quality of the Country Experts that
10ForaperceptivediscussionoftheroleofjudgmentincodingseeSchedler(2012).
19
provide the coding. Consequently, we pay a great deal of care and attention to the
recruitmentofthesescholars,whichfollowsanexactingprotocol.
First,weidentifyalistofpotentialcodersforacountry(typically100-200namesper
country).RegionalManagers,inconsultationwithCountryCoordinators,usetheirintimate
knowledge of a country to compile the bulk of the experts on this list. Assistant
Researchers locatedattheV-DemInstitute(UniversityofGothenburg)alsocontributeto
thislist,usingreadilyavailableinformationdrawnfromtheInternet.11Othermembersof
theproject team(PIs,PMs,andassociates)mayalsosuggestcandidates.Atpresent,our
databaseofpotentialCountryExpertscontainssome18,000names.
RegionalManagers and Country Coordinators thus play a critical role in the data
collectionprocess.V-Dem’sapproachistorecruitRegionalManagerswhoarenationalsor
residentsofoneofthecountriesineachregionwheneverpossible.TheRegionalManagers
aretypicallyprominentscholars inthefieldwhoareactiveasprofessors intheregion in
question.Insomecases,RegionalManagersarelocatedoutsideoftheregion,iftheyare
currentlyactiveinwell-respectedinternationalthinktanksorsimilarinstitutions.Country
Coordinatorsarealmostalwaysnationalsandresidentsofthecountrytobecoded.They
arealsoscholars,althoughtheyaretypicallymorejuniorthanRegionalManagers.
Using short biographical sketches, publications, website information, or similar
material we compile basic information for each Country Expert: their country of origin,
current location, highest educational degree, current position, and area of documented
expertise(relevantfortheselectionofsurveystheexpertmightbecompetenttocode)to
makesureweadheretothefiverecruitmentcriteria.
RegionalManagers,CountryCoordinators,andotherprojectteammembersreferto
five criteriawhen drawing up the list of potential Country Experts. Themost important
selectioncriterionisanindividual’sexpertiseinthecountry(ies)andsurveystheymaybe
assigned to code. This expertise is usually signified by an advanced degree in the social
sciences, law,orhistory;arecordofpublications;orpositions inoutsidepoliticalsociety
that establish their expertise in the chosen area (e.g. a well-known and respected
11ResearchAssistantsattheUniversityofNotreDamealsosuppliedmorethan3,000namesforallregionsin2011-2013,usinginformationfromtheInternet.
20
journalist; a respected former high court judge). Regional Managers and Country
Coordinatorsmayalsoindicatewhichsurveysapotentialcoderhasexpertisein.Naturally,
potentialcodersaredrawntoareasofthesurveythattheyaremostfamiliarwith,andare
unlikelytoagreetocodetopicstheyknowlittleabout.Asaresult,self-selectionalsoworks
toachieveourprimarygoalofmatchingquestionsinthesurveywithcoderexpertise.
Thesecondcriterionisconnectiontothecountrytobecoded.Bydesign,threeout
of five(60%)oftheCountryExpertsrecruitedtocodeaparticularcountry-surveyshould
be nationals or permanent residents of that country. Exceptions are made for a small
numberofcountrieswhere it isdifficulttofind in-countrycoderswhoarebothqualified
andindependentofthegoverningregime,orwherein-countrycodersmightbeplacedat
risk.ThiscriterionhelpsusavoidpotentialWesternorNorthernbiasesincoding.
The third criterion is the prospective coder’s seriousness of purpose, i.e. her
willingness to devote time to the project and to deliberate carefully over the questions
asked in the survey. Sometimes, personal acquaintanceship is enough to convince a
RegionalManager and a Country Coordinator that a person is fit, or unfit, for the job.
Sometimes, this feature becomes apparent in communications with ProgramManagers
thatprecedetheoffertoworkonV-Dem.Thiscommunicationisquiteintensive,withan
averageof13interactionsbeforecodingisconcluded,andinvolvesrequiringthepotential
coder to read and work with several lengthy, detailed documents. This process readily
identifiespotentialcoderswhoarenotseriousenough.
The fourth criterion is impartiality. V-Demaims to recruit coderswhowill answer
surveyquestionsinanimpartialmanner.Wethereforeavoidthoseindividualswhomight
bebeholdentopowerfulactors–byreasonofcoercivethreatsormaterialincentives–or
whoserveasspokespersonsforapoliticalpartyorideologicaltendency.Closeassociation
(current or past) with political parties, senior government officials, politically affiliated
think-tanks or institutes is grounds for disqualification. In cases where finding impartial
coders is difficult, we aim to include a variety of coderswho, collectively, represent an
arrayofviewsandpoliticalperspectivesonthecountryinquestion.
The final criterion is obtaining diversity in professional background among the
coders chosen for a particular country. For certain areas (e.g., themedia, judiciary, and
civil societysurveys)suchdiversityentailsamixtureofacademicsandprofessionalswho
21
study these topics. It also means finding experts who are located at a variety of
institutions,universitiesandresearchinstitutes.
Afterweighingthesefivecriteria,wegivethe100-200potentialexpertsonourlist
ofcandidatesarankfrom“1”to“3,”indicatingtheorderofprioritywegivetorecruiting
anExpert.TheRegionalManagersandCountryCoordinatorsareprimarilyresponsiblefor
the ranking, but Program Managers and one of the Principal Investigators may review
thesechoices.
Using this process,wehave recruitedover 2,500 scholars and experts fromevery
corneroftheworld.About30percentoftheCountryExpertsarewomen,12andover80
percent have PhDs or MAs and are affiliated with research institutions, think tanks, or
similarorganizations.
In order to preserve confidentiality, V-Dem has adopted a policy of neither
confirmingnordenyingtheidentitiesofCountryExperts.OnlythetwoProgramManagers
are actively involved in this final stage of recruitment (and two of the Principal
Investigators,whohavesupervisoryauthorityovertheprocess)areawareoftheidentities
ofthefinalchosenCountryExperts.Theseindividualsalsohandleallcorrespondencewith
Country Experts, so this confidentiality is not inadvertently revealed through
communication..
Thus,whiletheidentityofothermembersoftheV-Dementerpriseispublicizedon
ourwebsite,wepreservetheconfidentialityofCountryExperts.Severalreasonsliebehind
thisdecision.First,thereareanumberofcountriesintheworldwhereauthoritiesmight
sanctionCountryExperts,ortheirfamiliesorfriends,fortheirinvolvementintheproject.
Second,thereisnowaytopredictwhichcountrymayinthefuturebecomerepressiveand
thereforesanctiontheCountryExperts.Third,weanticipatethatV-Demdatamaybecome
used inevaluationsandassessments internationally inwaysthatcouldaffectacountry’s
status. Thus, one may foresee incentives for certain countries’ governments and other
actors to try to affect their ratings. For all these reasons, we consider it essential to
preserveCountryExpertanonymity.
12ThenumberofwomenamongtheranksofourCountryExpertsislowerthanwewouldhaveliked,anditoccurreddespiteourstrenuousefforts.However,itreflectsgenderinequalitieswithregardtoeducationanduniversitycareersintheworld.
22
ExpertCodingProcess
The two Program Managers at the V-Dem Institute (University of Gothenburg) issue
invitationsuntilthequotaoffivecodersisobtained.13Wereplacethosewhofailtobegin
orcompletethesurveyinareasonabletimeinasimilarmanner.Codersreceiveamodest
honorarium for their work that is proportional to the number of surveys they have
completed.
C-indicatorsareorganizedintofourclustersandelevensurveys:
1. Elections Politicalparties/electoralsystems
2. Executive Legislature Deliberation
3. Judiciary Civilliberty Sovereignty
4. CivilsocietyorganizationsMedia
Politicalequality
Wesuggest(butdonotrequire)thateachCountryExpertcodeatleastonecluster.
In consultation with the Country Coordinators and Principal Investigators, Regional
ManagerssuggestwhichCountryExpertmightbemostcompetenttocodewhichsurveys.
We then consult with the Country Expert about which cluster(s) they feel most
comfortablecoding.Mostcodeonlya fewofthesurveys.Thismeansthat, inpractice,a
dozenormoreCountryExpertsprovideratingsforeachcountry(withatargetoffivefor
eachcountry/indicator/year,asstated).14
AllCountryExpertscarryouttheircodingusingaspeciallydesignedonlinesurvey.
Theweb-basedcodinginterfacesaredirectlyconnectedwithapostgresdatasetwherewe
13BeforeJuly2014,therewasathirdProgramManagerattheKelloggInstituteoftheUniversityofNotreDamewhomanagedmostcountryexpertsinLatinAmericaandafewintheMiddleEastandNorthAfrica.
14Insomerarecases---mainlysmallandunder-studiedcountries---weaskindividualexpertstocodethewholesetofsurveys,simplybecauseexpertsonthevariousspecificpartsofthesurveyarenotavailable.Similarly,itisalsonotalwayspossibletoreachthegoalofhavingfivecountryexpertscodeeachindicatorforthesecountries.
23
storetheoriginalcoder-leveldata.Figure4providesanexampleofthecodinginterface.
ThecodinginterfaceisanessentialelementofV-Dem’sinfrastructure.Itconsistsof
aseriesofweb-basedfunctionsthatallowCountryExpertsandCountryCoordinatorsto(1)
logintothesystemusingtheirindividual,randomizedusernameandself-assigned,secret
password;(2)accesstheseriesofsurveysassignedtothemforaparticularcountry(orset
ofcountries);and(3)submitratingsforeachquestionoveraselectedseriesofyears.
Thecoding interfaceallows formanytypesofquestions (binary,ordinal,multiple
selection,etc.),country-specificandquestion-specificyearmasks(e.g.,allowingthecoding
of elections only in years they occurred), and question-specific instructions and
clarifications.
The interface also requires that, for each rating, experts assign a level of
confidence,indicatinghowconfidenttheyarethattheirratingiscorrect(onascaleof0-
100,where each 10-percent interval has a substantive anchor point), providing another
instrumentformeasuringuncertaintyassociatedwiththeV-Demdata.Weincorporatethis
confidence into the measurement model. Country Experts also have an opportunity to
register uncertainty in the “Remarks” field that lies at the end of each section of the
survey.Here, experts can comment (inprose)onany aspectof the indicatorsor ratings
thatshefoundproblematicordifficulttointerpret.
24
Fig.4ExampleofCodingInterface
Finally, in order to ensure wide recruitment of potential experts, and minimize
confusiondue tounfamiliaritywith English,we translate all type-Cquestions, aswell as
coder-instructionsanddocumentationforthem,intofiveotherlanguages:Arabic,French,
Portuguese,Russian,andSpanish.Approximately15percentofthecoderscodeinanon-
Englishversionofthequestionnaire.CountryExpertsgetasmallremunerationasatoken
ofappreciationfortheirtime.15
15Fromwhatwecantell,thisisnotasignificantthreattocodingvalidity.Fewindividualsseemtohavebeenmotivatedtoconductthisarduouscodingassignmentforpurelymonetaryreasons:V-Dempaysverylittlerelativetowhathighlyqualifiedexpertscouldearnforthesameamountofworkfromotherpursuits.Furtherstrengtheningthispoint,thereseemstobenorelationshipbetweenthewealthofthecountryandourabilitytorecruitcoders:wehavefacedchallengesgettingexpertstoagreetoconductcodingforthe
25
Wetakeanumberofstepstoassureinformedconsentandconfidentialityamong
participants.Theon-linesurveyprovidesfullinformationabouttheproject(includingthis
document)andtheuseofthedata,sothatcodersarefullyinformed.Italsorequiresthat
prospectivecoderscertify that theyaccept the termsof theagreement.Theyaccess the
surveysonlywitharandomizedusernamethatweassignandasecretpasswordthatthey
createthemselves.Westorethedatatheysupplyonafirewall-protectedserver.Anydata
we release to thepublic excludes information thatmightbeused to identify coders.All
personal identifying information is kept in a separate database in order to ensure
confidentiality.
Aspeciallydesignedprogramminginterfaceisemployedtomanagethedatabaseof
potential country experts. It includes many tools that enable us to handle over 2,500
Country Experts while guaranteeing their safety and confidentiality. These tools also
ensureconsistencyininstructionsandinformationsenttoCountryExperts,qualitycontrol
andcleaningofdata,followupandevaluationofthecodingprocess.Itisdirectlylinkedto
thepostgresdatabasewhereratingsarestored.
For example, the roster of hundreds of potential experts for Country X and all
associatedinformationisuploadedintothemanagementdatabaseanddirectlyaccessible
from the interface. Then perhaps 20 or so Country Experts are invited to participate as
coders,using speciallydesigned invitation letters in six languages that is associatedwith
standardizedinformationmaterials.
When a potential Country Expert accepts an invitation, the Program Managers
mark their acceptance, the areas of coding, and assign themas coders for oneormore
countries.Themanagementsoftwarethenautomaticallycommunicateswiththepostgres
database and the coding interfaces, creating a coder ID. The software then creates the
sameIDinrelevanttablesandinterfacecommunications,generatesauseridtobeusedin
the coding interfaces, and sends an email to the new coder with their username and
instructionsforhowto log inandcreateauniqueandsecretpassword.Fromthatpoint,
themanagement software communicates automaticallywith the postgres database and
determines each coder ID’s progress on coding for each of the indicators to which the
poorestaswellastherichestcountriesintheworld.
26
coder is assigned. The software also reports to the Program Managers on the coder
management toolpages.Otherpartsof theprocess including thehandlingof signed tax
forms and applications; as well as payments are similarly connected in the coding
managementtool.
Thecodermanagementtoolisjustoneofover20sophisticatedtoolsamongtheV-
Dem management interfaces in the software. There are tools for management of
countries, rounds of surveys, surveys and questions, country coordinators, regional
managers, for logging activities, analyses of progress on recruitment as well as coding,
planning, and general management. A web-interface portal is connected to the
management software, allowing Regional Managers to securely upload Country Expert
rosterstothedatabasewithouthavingtoshareconfidentialinformationviaemail.
Bridge-andlateralcoding
In addition to regular ratings by multiple Country Experts for C-type indicators, we
encourage Country Experts to conduct bridge coding (coding ofmore than one country
throughtime)and lateralcoding(coding limitedtoasingleyear–2012).Thepurposeof
this additional coding is to assure cross-country equivalence by forcing coders tomake
explicit comparisons across countries. This helps themeasurementmodel estimate, and
correctfor,systematicbiasesacrosscodersandacrosscountriesthatmayresultifCountry
Expertsemployvaryingstandardsintheirunderstandingofaquestion,e.g.,aboutwhata
“high”levelofrepressionmightconsistof.
Throughoutimplementationoftheproject,wehaveencouragedCountryExpertsto
codemultiplecountriesover time -bridge coding.Anexpertwhoagrees tocodeoneor
moreadditionalcountriesreceivesthesamesetofsurveysforthesametimeperiodasthe
originalcountrytheycoded;bridgecodingthereforetypicallycovers1900tothepresent.
Bridge coding helps us better model how Country Experts make judgments between
different response categories, and allows us to incorporate this information into the
estimatedscoreforeachcountry-indicator-year/date.
Bridge coding is most useful when the chosen countries have different regime
histories.ThisgeneratesvarianceacrossaCountryExpert’sratings,whichinturnprovides
information about the coder’s judgments that can be used to inform themeasurement
27
model.Inordertomaximizevariance,andthereforegainasmuchinformationaspossible
abouteachexpert’s thresholdsand reliability,weencourageCountryExperts to select–
from among countries they are familiar with – those that have the most distinctive
historicaltrajectories.
As of March 2016, we have over 390 bridge coders – about 15 percent of all
CountryExperts.Onaverage,theseexpertshavecoded6.1surveysfor2.1countries.
Constraints of time or expertise sometimes prevent Country Experts from
conductingbridge coding. In these situations,weencourageCountryExperts toperform
thesimplertypeofcross-countrycomparisoncalled lateralcoding.That is, inadditionto
theiroriginalcodingofonecountryovertime(e.g.,from1900tothepresent),theycodea
numberofcountriesforasinglepointintime–January1,2012–focusingonthesameset
ofquestions.
SomeCountryExpertshavecodedupto14countries.Moretypically,lateralcoding
extends to a few countries. To date, 350 Country Experts (about 15%) have performed
lateral coding, covering on average of 5.5 countries and 6.3 surveys. As a result, lateral
coding by regular Country Experts has provided linkages equivalent to over 1,100 “fully
covered” countries – in other words, countries that have been “cross-coded” by
lateral/bridgecodingacrossallindicatorsinthedataset.
Phases
In the firstphaseofdatacollection (2012 to2014),weaskedCountryExperts tocodea
cluster(s) of surveys for a single country from 1900 (or the relevant first year for a
particularcountry)totheendof2012.
FromNovember2014toMarch2015weconductedthefirstupdate.Itcovered54
countries–bringingtheirdatacurrentuptoend-2014–andalsoaddedsixnewcountries
(with data from 1900 to 2014). Due to coder attrition, coding for the update was
conducted by amix of returning Country Experts and new Country Experts.When they
codedfor2013and2014,returningCountryExpertssawtheirpreviously-submittedratings
for the years from 2010 to 2012, so as to encourage consistency in ratings over time,
thoughwedidnot allow them to alter those ratings.WeaskednewCountry Experts to
28
codetenyears(2005-2014)soastoensurethattheirscoresoverlapbyanumberofyears
withreturningCountryExperts’ratings.
Wehavenowconcludedthesecondroundofannualupdates,covering2015.This
roundofupdateswilltookplacebetweenDecember2015andMarch2016.Itcovered76
countries,22ofwhichwerealsocoveredinthefirstupdate.Hence,attimeofwriting,the
V-Demdatasetincludesdatafor173countries:upto2012for59countries,andupto2014
or2015for114countries.
Toenhanceconsistencyincodingacrossrounds,returningcoderssawtheirprior
ratings, andwere this timeable to revise them, if theywished to.NewCountry Experts
codedtheyears2015-2015.Finally,weimplementedaseriesofvignettesforeachsurvey
to give us additional leverage on measurement error. The third update takes place
December2016toMarch2017,withthereleaseofdataMarch31.
3. Measurement
Havingdiscussedtheprocessofdatacollection,weproceedtothetaskofmeasurement.
Under this rubric, we include (a) the questionnaire, (b) our measurement model, (c)
methodsof identifyingerror inmeasurement, (d) studiesofmeasurementerror,and (e)
methodsofcorrectingerror.Inprinciple,thediscussionsarerelevantfordifferenttypesof
data(A,B,andCintheV-Demscheme)butmostifnotallofthemaremuchmoreacute
when it comes to expert-based coding of evaluative, non-factual yet critical indicators.
Hence,mostofthefollowingfocusesontheC-typeindicators.
TheQuestionnaire
Themost important featureofasurvey is theconstructionofthequestionnaire itself. In
craftingindicatorstomeasuretheC-typedata,wehavesoughttoconstructquestionswith
bothspecificandclearmeanings,andwhichdonotsuffer fromtemporalorspatialnon-
equivalence.Todesignthesequestions,weenlisted leadingscholarsondifferentaspects
ofdemocracyanddemocratizationasProjectManagers.
We enrolled each Project Manager because of her record of scholarly
accomplishment in a particular area related to issues of democracy (e.g. legislatures,
29
executives, elections, and civil society), with the goal of creating a team that also had
substantiveexperiencesandexpertiseonallregionsoftheworld.ProjectManagersbegan
designingsurvey-questionsintheirareaofexpertisein2009,andwecollectivelyreviewed
andrefinedtheirquestionsoverthecourseoftwoyears.
WeimplementedapilotoftheV-Demsurveyin2011,whichservedasaninitialtest
of our questionnaire. It was implemented for 12 countries, two (one “easy” and one
“hard”) fromeachof the sixmajor regionsof theworldenlistingover120pilot-Country
Experts and resulted in some 450,000 ratings on preliminary indicators. The results
promptedrevisionsinthenextroundofsurveys.Anotherroundofcollectivedeliberation
followed, involving consultationswith scholars outside of the project team. The revised
questionsforC-codingthusenduredseveralroundsofreviewwithProjectManagersand
outside experts over the course of two years before emerging in their final form, as
describedintheCodebook.
Identifying,Correcting,andQuantifyingMeasurementError
Evenwithcarefulquestiondesign,aprojectofthisnaturewillencountererror.Sucherror
may be the product of linguistic misunderstandings (most of our coders do not speak
English as their first language, and some take the survey in a translated form),
misunderstandingsaboutthewayaquestionappliestoaparticularcontext,factualerrors,
errors due to the scarcity or ambiguity of the historical record, differing interpretations
abouttherealityofasituation,variationinstandards,coderinattention,errorsintroduced
bythecoderinterfaceorthehandlingofdataonceithasbeenenteredintothedatabase,
orrandommistakes.
Someof theseerrorsare stochastic in the senseofaffecting theprecisionofour
estimatesbutnot their validity.Othererrors are systematic, potentially introducingbias
intotheestimates thatweproduce. In thissection,we firstdescribethemethodological
toolsweusetomodelandcorrectforsystematicbiasincoders’answerstoourquestions,
aswellas toprovideestimatesof thereliabilityof thesecodings. Wethendescribe the
procedures we use to assess the validity of our estimates. Finally, we explain how we
identifythemostserioussourcesofmeasurementerror,inordertocontinuouslyimprove
30
howwegatherandsynthesizedata.
MeasurementModels
The most difficult measurement problems concern the C-type questions, all of which
require substantial case knowledge and generally somedegree of subjective evaluation.
Having five coders for each of these questions is immensely useful, as it allows us to
conductinter-coderreliabilitytests.Thesesortsoftests–standardinmostsocialscience
studies–areonlyrarelyifeveremployedinextantdemocracyindices.
Whileweselectexpertscarefully,theyexhibitvaryinglevelsofreliabilityandbias,
and may not interpret questions consistently. In such circumstances, the literature
recommends that researchers usemeasurementmodels to aggregate diversemeasures
wherepossible,incorporatinginformationcharacterizedbyawidevarietyofperspectives,
biases,and levelsof reliability (Bollen&Paxton2000,Clinton&Lapinski2006,Clinton&
Lewis2008, Jackman2004,Treier& Jackman2008,Pemstein,Meserve&Melton2010).
Therefore,tocombineexpertratingsforaparticularcountry-indicator-yeartogeneratea
single“bestestimate”foreachquestion,weemploymethodsinspiredbythepsychometric
and educational testing literature (see, e.g., Lord&Novick 1968, Jonson&Albert 1999,
Junker1999,Patz& Junker1999).Theunderpinningsof thesemeasurementmodelsare
straightforward:theyusepatternsofcross-rater(dis)agreementtoestimatevariations in
reliabilityandsystematicbias.Inturn,thesetechniquesmakeuseofthebiasandreliability
estimates to adjust estimates of the latent—that is, only indirectly observed—concept
(e.g.,executive respect for theconstitution, judicial independence,orproperty rights) in
question. These statistical tools allow us to leverage our multi-coder approach to both
identifyandcorrectformeasurementerror,andtoquantifyconfidenceinthereliabilityof
our estimates. Variation in these confidence estimates reflect situationswhere experts
disagree, orwhere little information is available because few raters have coded a case.
These confidence estimates are tremendously useful. Indeed, to treat the quality of
measures of complex, unobservable concepts as equal across space and time, ignoring
dramatic differences in ease of access andmeasurement across cases, is fundamentally
misguided,andconstitutesakeythreattoinference.
ThemajorityoftheC-typequestionsareordinal: theyrequireCountryExpertsto
31
rankcasesonadiscretescale. Take,forexample,thefollowingquestionaboutelectoral
violence:
Question:Inthisnationalelection,wasthecampaignperiod,electionday,andpostelectionprocessfreefromothertypes(notbythegovernment,therulingparty,ortheiragents)ofviolencerelatedto the conductof theelection and the campaigns (butnot conductedby the government and itsagents)?Responses:
0. No. There was widespread violence between civilians occurring throughout the electionperiod,orinanintenseperiodofmorethanaweekandinlargeswathsofthecountry.Itresultedinalargenumberofdeathsordisplacedrefugees.
1. Notreally.Thereweresignificantlevelsofviolencebutnotthroughouttheelectionperiodorbeyondlimitedpartsofthecountry.Afewpeoplemayhavediedasaresult,andsomepeoplemayhavebeenforcedtomovetemporarily.
2. Somewhat.Thereweresomeoutburstsoflimitedviolenceforadayortwo,andonlyinasmall part of the country. The number of injured and otherwise affected was relativelysmall.
3. Almost. There were only a few instances of isolated violent acts, involving only a fewpeople;noonediedandveryfewwereinjured.
4. Peaceful.Noelection-relatedviolencebetweenciviliansoccurred.
Note, inparticular, thattheserankingsdonot followan interval-levelscale. Onecannot
subtractalmost frompeaceful andgetnot really. Furthermore, it neednotbe the case
thatthedifferencebetweennotreallyandsomewhatisthesameasthatbetweenalmost
and peaceful. Perhaps most importantly, although we strive to write questions and
responses thatarenotoverlyopen to interpretation,wecannotensure that twocoders
look at descriptions like somewhat in a uniform way—even when somewhat is
accompaniedbyacarefullyformulateddescription—especiallybecausecodershavewidely
varying backgrounds and references. In other words, one coder’s somewhat may be
another coder’s not really; a problem known as scale inconsistency. Therefore, we use
Bayesian item response theory (IRT)modeling techniques (Fox 2010) to estimate latent
politycharacteristicsfromourcollectionofexpertratingsforeachordinal(C)question.
Specifically, we fit ordinal IRTmodels to each of our ordinal (C) questions. (See
Johnson&Albert1999foratechnicaldescriptionofthesemodels.)Thesemodelsachieve
threegoals.First,theyworkbytreatingcoders’ordinalratingsasimperfectreflectionsof
interval-levellatentconcepts.Withrespecttotheexamplequestionabove,ourIRTmodels
assumethatelectionviolencerangesfromnon-existenttoendemicalongasmoothscale,
and coders observe this latent characteristic with error. Therefore, while an IRTmodel
takesordinalvaluesasinput,itsoutputisaninterval-levelestimateofthegivenlatenttrait
32
(e.g.electionviolence). Interval-valuedestimatesarevaluableforavarietyofreasons; in
particular,theyareespeciallyamenabletostatisticalanalysis.Second,IRTmodelsallowfor
the possibility that coders have different thresholds for their ratings (e.g. one coder’s
somewhatmight fall above another coder’salmost on the latent scale), estimate those
thresholds from patterns in the data, and adjust latent trait estimates accordingly.
Therefore,theyallowustocorrectforthispotentiallyserioussourceofbias.16Thisisvery
important in a multi-rater project like V-Dem, where coders from different geographic,
cultural,andotherbackgroundsmayapplydifferingstandardstotheirratings.Finally,IRT
modelsassumethatcoderreliabilityvaries,produceestimatesofraterprecision,anduse
these estimates—in combination with the amount of available data and the extent to
whichcodersagree—toquantifyconfidenceinreportedscores.
Sinceourcodersgenerallyrateonecountrybasedontheirexpertise,itisnecessary
toutilizelateralcoders.Aspreviouslydescribed,thesecodersratemultiplecountriesfora
limitedtimeperiod(mostlyoneyear,butinsomecasesten).Wehaveatpresentsome350
lateralcoders.Inaddition,wehaveover390bridgecoders,asdiscussedabove.Theseare
coderswho code the full time series (generally 1900-2012) formore than one country,
coveringoneormoreareas (“surveys”).17 Essentially, this codingprocedureallowsus to
mitigate the incomparability of coders’ thresholds and the problem of cross-national
estimates’calibration(Pemsteinetal.2015).Whilehelpfulinthisregard,ourtestsindicate
that,giventhesparsityofourdata,eventhisextensivebridge-coding isnotsufficient to
fully solve cross-national comparability issues. We therefore employ a data-collapsing
procedure.Atitscore,thisprocedurereliesontheassumptionthataslongasnoneofthe
experts change their ratings (or their confidence about their ratings) for a given time
period, we can treat the country-years in this period as one year. The results of our
statisticalmodelsindicatethatthistechniqueisextremelyhelpfulinincreasingtheweight
16Givencurrentlyavailabledata,wemustbuildinassumptions—formally,theseareknownashierarchicalpriors—thatrestricttheextenttowhichcoders’thresholdestimatesmayvary.Informally,whileweallowcoderstolookatordinalrankingslikesomewhatandalmostdifferently,weassumethattheirconceptionsarenottoodifferent.Weareworkingtorelaxtheseassumptionsbycollectingmoredata.TechnicaldetailsareavailableinV-DemWorkingpaperno.19whichwillbeavailableinDecember2015,andfullcodeisreleasedwiththedataset.
17Thuswehavelateral/bridgecodingcoveringtheequivalentofover1,100“fullcoverage”ofallcountry-questions.
33
given to bridge coders, and thus further ameliorates cross-national comparability
problems.
As a final note, our model diverges from more standard IRT models in that it
employsempiricalpriors.Specifically,wemodelacountry-year’s latentscore foragiven
variableasbeingdistributedaccordingtoanormaldistributionwithanappropriatelywide
standarddeviationparameterandameanequaltotherawmeanofthecountry’sscores,
weighted by coder confidence and normalized across all country-years. More formally,
!! ~ !(!! , 1), where ! is the latent score for country-year !, and ! is the normalized
confidence-weighted average from the raw data. In contrast, most standard models
employavaguemeanestimate,i.e.!! ~ !(0,1).Ourapproachofusingempiricalpriorsis
similar to the standardapproach:ourwide standarddeviationparameter still allows for
the model output to diverge from prior as the data warrant. However, our approach
incorporatesouractualpriorbeliefsaboutacountry’sscoreandthusyieldsmoreaccurate
measures.Especially inthecaseofcountrieswithextremevalues,atraditionalapproach
risksbiasingoutputtowardthemean.
FutureversionsofourordinalIRTmodelswillimproveoncurrentestimatesintwo
primary ways. First, hierarchical IRTmodeling techniques (Patz et al. 2002,Mariano &
Junker2007)wouldallowustoborrowstrengthfromdifferentvariableestimates,yielding
more precise measures of each variable. Second, all raters complete a post-survey
questionnaire that asks demographic and attitudinal questions. Coders also report
personal assessments of confidence in their responses to each question. At present, of
these datawe only incorporate confidence into themodel, using it toweight our prior
meanestimates;furtheruseoftheseformsofdatainourmodelswillallowustoteaseout
patternsconcerningbiasesandreliabilityacrossdifferent typesofexperts,andgenerally
improvethequalityofourestimates.
For nominal and some dichotomous variables we use IRT techniques when
sufficient variation exists to identify rater thresholds. For the remaining variables we
providetheunweightedmean.
IdentifyingRemainingErrors
Toevaluatepossibleerrorsweemployanumberoftests,someofwhichareincorporated
34
into themeasurementmodels and others of which are applied ex post to examine the
validityofmodeloutput.
First, we have used data from the post-survey questionnaire that every V-Dem
coder completes to identify potential sources of bias. This survey delves into factors of
possible relevance to coder judgments, such as personal characteristics like sex, age,
country-of-origin,educationandemployment. Italso inquires intoopinions thatCountry
Expertsholdaboutthecountrytheyarecoding,askingthemtoassignapointscoreona0-
100scalesummarizingtheoverall levelofdemocracy in thecountryonJanuary1,2012,
using whatever understanding of democracy they choose to apply. We ask the same
questionaboutseveralprominentcountries fromaroundtheworldthatembodyvarying
characteristics of democracy/autocracy. Finally, the questionnaire contains several
questions intendedtoelicit thecoder’sviewsabouttheconceptofdemocracy.Wehave
runextensive testsonhowwell such individual-level factorspredictscountry-ratingsbut
havefoundthattheonlyfactorconsistentlyassociatedwithcountry-ratings iscountryof
origin (with “domestic” coders being harsher in their judgments). This is also the
individual-levelcharacteristicincludedinthemeasurementmodelestimates.
Inthefuture,weneverthelessplantouseeachpieceofinformationfromthispost-
survey questionnaire to help inform themeasurementmodel, i.e., to enhance precision
andlimitpossibleundetectedbiases.Themeasurementmodelwillalsotakeintoaccount
information we can glean from the performance of the coders that might serve as an
indicationoftheirlevelofattentiveness,effort,andknowledge.Thisinformationincludes
inter-coder reliability (assessed at the coder level across all codings), self-reported
confidence(ineachcoding),numberofcountry-yearscoded(alltogether),codingchanges
(the number of times that a coder changes their coding fromT-1 toT relative to other
codersforthatcountry/indicator,aggregatedacrossallcodings),timeontask(thenumber
of hours a coder is logged into the on-line system, discounted by the number of
country/indicator/yearss/hehascoded),accesses(thenumberoftimestheon-linesurvey
isaccessed),contacts(writingcommentsoraskingquestionsoftheV-Demteamthatare
non-logistical in nature), and response rate (assessed at the country level). (With the
exception of inter-coder reliability, these elements have not yet been included in the
model.)
35
Eachof theaforementioned featureswill alsobe tested independently. Thus,we
will be able to report on whether, and to what extent, each of the observed and self-
reportedfeaturesofthecodersaffectstheirratings.Inparticular,byincludinghierarchical
priors that dependon observed rater characteristics and behavior in our latent variable
model specifications—an approach often referred to as “empirical Bayes”—we can
evaluatetheextenttowhichsuchfeatureshelptoexplainraterbiasandreliability,while
simultaneouslyincorporatingthatinformationintoindicatorestimates.
Inaddition,wewillapplyseveralexpost tests toevaluatethequalityof thedata
emanatingfromthemeasurementmodel.Onesortoftestreliesonthedistributionofthe
data.Ifthedistributionofresponsesforaparticularcountry/indicator/yearisbi-modalwe
haveanobviousproblem:codersdisagreewildly.Thisalsomeansthatthepointestimate
fromthemeasurementmodelisunstable:achangeofcodingforanysinglecoder,orthe
additionofanewcoder,islikelytohaveabigimpactonthepointestimate.Disagreement
as registeredbyabi-modaldistributioncould representa situation inwhich the truth is
recalcitrant – presumably because available information about a topic is scarce and/or
contradictory.Oritcouldrepresenterrorsthatarecorrigible.
A secondapproach tovalidationcomparesV-Dem indiceswithother indices that
purport to measure similar concepts, i.e., convergent validity. For example, a set of
regressionsusingallavailabledataoftheV-DemElectoralDemocracyIndex–andsomeof
itsconstituentindicators–againstPolity2indicatesrelativelyhighcorrelations(Pearson’s
r= .85) and (separately) against FH Political rights (Pearson’s r= .90). Unfortunately,
techniquesof convergent validity are limited in their utility. First,wehave somedoubts
aboutthevalidityofstandardindices(seeComparisonsandContrasts).Second,standard
indices tend to hover at a higher level of aggregation, thus impairing comparability
betweenV-Demindicesandalternativeindices.Indeed,onlyafewextantindicesareclose
enoughinconceptionandconstructiontoprovideanopportunityfordirectcorroboration
withV-Demindices.
A third approach to validation focuses on face validity. Once data collection is
completeforagroupofcountries,RegionalManagersandothermembersoftheV-Dem
teamlookcloselyatpointestimatesinanattempttodeterminewhethersystematicbias
may exist. One major such review was conducted in October 2013 when almost all
36
RegionalManagers,allProjectManagers,ResearchFellows,PIsandstaff,spentfourdays
collectively reviewing all data collated at that point to validate the approach and
aggregationmethods.Theprocessoffacevaliditycheckshassincethenbeenrecurrent.
Finally,inthepresentroundofupdate(2015/2016),weareimplementingaseries
ofvignettesforeachsurveythatCountryExpertscode.Thevignettesarecalibratedatthe
thresholdsbetweenanswercategoriesandwillgiveusadditional leverageonsystematic
differencesinCountryExperts’ratingsdependingontheir“harshness”asraters.Thiswill
furtherreducemeasurementerrorinfuturereleasesofthedata.
CorrectingErrors
We correct problems with factual questions (B-type indicators) whenever the Principal
Investigators,inconsultationwiththerelevantProjectManagers,becomeconvincedthata
better (i.e., more correct) answer is available. Based on analysis of submitted data by
CountryCoordinators,certainvariablesweredesignatedasB+A.UsingtheoriginalB-data
as a point of departure and cross-checking with external resources, we designed and
implemented a coding scheme to re-code these indicators, as the Codebook describes.
Indicatorsaffectedincludeallindicatorsfromthedirectdemocracysurvey,fourindicators
ontheexecutive,fouronelectionsandnineonlegislature.Thedecisiontore-assignthese
indicators was also due to the interaction between question formulation and coder
interpretation, e.g. in some instances the meaning of “plebiscite” was interpreted in a
different way than what the Project Manager envisaged, leading to discrepancies in
coding.
We handle problemswithevaluative questions (C-type indicators)with restraint.
Wefullyexpectthatanyquestionrequiring judgmentwillelicitarangeofanswers,even
whenall codersarehighly knowledgeableabouta subject.A keyelementof theV-Dem
project – setting it apart frommost other indices that rely on expert coding – is coder
independence:eachcoderdoesherworkin isolationfromothercodersandmembersof
theV-Dem team (apart from clarifyingquestions about theprocess). Thedistributionof
responses across questions, countries, and years thus provides vital insight into the
relative certainty/uncertainty of each data point. Since a principal goal of the V-Dem
projectistoproduceinformativeestimatesofuncertaintywedonotwishtotamperwith
37
evidence that contributes to those estimates. Arguably, the noise in the data is as
informativeasthesignal.Moreover,waywardcoders(i.e.,coderswhodivergefromother
coders)areunlikelytohaveastronginfluenceonthepointestimatesthatresultfromthe
measurementmodel’saggregationacrossfiveormorecoders.Thisisespeciallythecaseif
thewaywardcodersareconsistentlyoff-center(acrossalltheircodings);inthiscase,their
weightindeterminingmeasurementmodelscoresisreduced.
That said, there have been instances in which we have altered C-data. A few
questions were largely of factual nature (e.g. number of legislative chambers; if a local
governmentexists,whichofficeswereelectedinaparticularelection,etc.).Sincewelater
acquired enough funding to have assistants conduct the factual coding based on
systematicconsultationofcrediblesources,wedischargedthedatasubmittedbyCountry
Experts for these particular questions and any “downstream” data. For example, if a
CountryExpert indicatedthatthereweretwochambersinthelegislatureforaparticular
year,shethencoded“downstream”inthequestionnaireaseriesofquestionsregarding
boththelowerandupperchamber.Ifourresearchestablishedthatanupperchamberdid
not in fact exist in that particular year,we cleaned the records of data providedby the
expertfortheupperchamber.Thiscleaningaffected19%ofallexecutivedatasubmitted
for thosedownstreamvariables, 7.7%of thedata in theelection surveyand11% in the
legislative survey. These numbers reflect places where coders unnecessarily coded due
eithertoa)problemwiththeskippingfunctioninthesurveys,b)coders’abilitytochange
the pre-coded, factual data, or c) an initial decision, subsequently reversed, to have
CountryExpertstoanswersomeoftheA-coded(morefactual)questions.
Inafinalcase,weremovedoriginalcodingbysomeCountryExpertsbecauseofa
factual misunderstanding (or misunderstanding about response-categories) about the
existenceoftheinternetineraspriortoitsinvention.
In all these situations,wemaintain the original coder-level data in archived files
thatmayberetrievedbyspecialrequestofthePIs.
VersionsofC-Variables
TheV-DemdatasetthencontainsA,B,C,andDindicatorsthatareallunique.Inaddition,
tofacilitateeaseofuseforvariouspurposes,theC-variablesaresuppliedinthreedifferent
38
versions(alsonotedintheV-DemCodebook):
1.“Relative Scale” - Measurement Model Output – has no special suffix (e.g.
v2elmulpar).Thisversionofthevariablesprovidescountry-year(country-dateinthe
alternativedataset)pointestimatesfromtheV-Demmeasurementmodeldescribed
above. The point estimates are themedian values of these distributions for each
country-year.Thescaleofameasurementmodelvariableissimilartoanormal(“Z”)
score (i.e. typicallybetween-5and5,with0approximately representingthemean
forall country-years in the sample) though itdoesnotnecessarily followanormal
distribution.Formostpurposes,thesearethepreferredversionsofthevariablesfor
time-seriesregressionandotherestimationstrategies.
“Measure of Uncertainty” –MeasurementModel Highest Posterior Density (HPD)
Intervals–havethesuffixes–"codelow"and"codehigh"(e.g.,v2elmulpar_codelow
and v2elmulpar_codehigh). These two variables demarcate one standard deviation
upperandlowerboundsoftheintervalinwhichthemeasurementmodelplaces68
percent of the probabilitymass for each country-year score. The spread between
"codelow" and "codehigh" is equivalent to a traditional one standard deviation
confidence interval; a larger range indicates greater uncertainty around the point
estimate.
2. “Original Scale” – Linearized Original Scale Posterior Prediction – has the suffix
“_osp,” (e.g. v2elmulpar_osp). In this version of the variables, we have linearly
translatedthemeasurementmodelpointestimatesbacktotheoriginalordinalscale
ofeachvariable(e.g.0-4forv2elmulpar_osp)asanintervalmeasure.18Thedecimals
in the _osp version indicate the distance between the point estimate from the
linearizedmeasurementmodelposteriorpredictionand the threshold for reaching
thenextlevelontheoriginalordinalscale.Thus,a_ospvalueof1.25indicatesthat
themedianmeasurementmodelposteriorpredictedvaluewasclosertotheordinal
18Morespecifically,weusethemeasurementmodeltoestimatetheposteriordistributionaroundthepredictedprobabilitythatatypicalcoderwouldplaceacountry-yearestimateateachleveloftheoriginalcodebookscale.Wethenlinearlymapthesepredictedprobabilitydistributionsontotheoriginalscale,producingadistributionofinterval-valuedscoresontheoriginalcodebookscaleforeachcountry-year.
39
value of 1 than 2 on the original scale. Since there is no conventional theoretical
justification for linearly mapping ordinal posterior predictions onto an interval
scale,19thesescoresshouldprimarilybeusedforheuristicpurposes.However,since
the_ospversionmapsontothecodingcriteriafoundintheV-DemCodebook,andis
stronglycorrelatedwiththeMeasurementModeloutput(typicallyat.98orhigher),
some users may find the _osp version useful in estimating quantities such as
marginal effects with a clear substantive interpretation. Using the “Ordinal Scale”
estimates---or incorporating the properties of ordinal probit models into the
estimation procedure---is generally preferable to using the _osp estimates in
statistical analyses. That said, if a user uses _osp data in statistical analyses it is
imperative that she first confirm that the results are compatible with estimations
usingMeasurementModeloutput.
“Measure of Uncertainty” – Linearized Original Scale HPD Intervals – have the
suffixes – "codelow" and "codehigh" (e.g., v2elmulpar_osp_codelow and
v2elmulpar_osp_codehigh).Weestimatethesequantitiesinasimilarmannerasthe
Measurement Model Highest Posterior Density Intervals. They demarcate one
standard deviation upper and lower bounds of the interval in which the
measurementmodelplaces68percentoftheprobabilitymassforeachcountry-year
score.Thespreadbetween"codelow"and"codehigh" isequivalent toa traditional
one standard deviation confidence interval; a larger range indicates greater
uncertaintyaroundthepointestimate.
19ThemaintheoreticalandpragmaticconcernwiththesedataisthatthetransformationdistortsthedistancebetweenpointestimatesintheMeasurementModeloutput.Forexample,thedistancebetween1.0and1.5inthe_ospdataisnotnecessarilythesameasthedistancebetweena1.5and2.0.
40
3. “Ordinal Scale” -MeasurementModel Estimatesof Original Scale Value – has the
suffix"_ord"(e.g.v2elmulpar_ord).Thismethodtranslatesthemeasurementmodel
estimates back to the original ordinal scale of a variable (as represented in the
Codebook) after taking coder disagreement andmeasurement error into account.
Moreprecisely, itrepresentsthemost likelyordinalvalueontheoriginalcodebook
scale intowhichacountry-yearwould fall, given theaveragecoder’susageof that
scale. Specifically, we assign each country-year a value that corresponds to its
integerizedmedianordinalhighestposteriorprobabilitycategoryoverMeasurement
Modeloutput.
“MeasureofUncertainty”–OriginalScaleValueHPDIntervals–havethesuffixes–
"codelow" and "codehigh" (e.g., v2elmulpar_ord_codelow and
v2elmulpar_ord_codehigh). We estimate these values in a similar manner as the
Measurement Model Highest Posterior Density Intervals. They demarcate one
standard deviation upper and lower bounds of the interval in which the
measurementmodelplaces68percentoftheprobabilitymassforeachcountry-year
score.Thespreadbetween"codelow"and"codehigh" isequivalent toa traditional
one standard deviation confidence interval; a larger range indicates greater
uncertaintyaroundthepointestimate.
AdditionalPossibilitiesforIdentifyingSourcesofMeasurementErrorintheFuture
A final approach to validationanalyzes various featuresof thedatagatheringprocess in
ordertogaugepossiblesourcesoferror.Thisanalysistakestheformofvariousstudiesin
which a particular issue is probed in an intensive fashion. The following studies are
underwayoronthedrawingboard–thoughwecannotsayforsurehowlongitwilltakeus
tocompletethem.
Onesuchstudywillfocusoncodertypes.Akeychallengetothevalidityisthatdata
maybesubjecttothesubjectiveperceptionsandopinionsofthechosencoders. Is itthe
casethatadifferentsetofcodersmightarriveataverydifferentsetofanswers?Features
of the coders captured in our post-survey questionnaire can be tested systematically
across the entire dataset, as noted. However,we cannot test the potential impact of a
41
differentkindofcodernot includedinourusualsample.Thisstudythereforefocuseson
comparisons across different coder types, e.g., partisans, academics, civil society
professionals,businesspeople,cosmopolitans(thosespeakingforeign languagesandwith
travel or educational experience abroad), educated lay citizens, and less educated lay
citizens.Resultsofthisstudyshouldindicate(a)howfartheconsensusoncodingextends
(i.e., to what types of coders), (b) how much difference the background of the coder
makes,(c)forwhattypesofquestions itmatters,and(d)whichsortsofcodershavethe
most positive view of a country. More generally, we hope to learn more about the
sensitivityofV-DemdatatooursamplingofCountryExperts.
Asecondstudywouldbetofocusoncountrysequencing.Doesitmatterifcoders
have considered other countries prior to coding CountryA? Such a studywould involve
randomizing respondents into two groups. Group 1 is asked to code Country A. Several
weekslater,theyareaskedtocodeahandfulofcountriesincludingCountryA,whichthey
mustre-code.Thecomparisoncasesshould includethosethatare inthesameregionas
wellasacountry(preferablyinthesameregion,orwithahistoryofcolonialinvolvement
intheregion)generallyregardedashighlydemocratic.Respondentsarenotremindedof
theiroriginal codings forCountryAandareencouraged toadjust theiroriginal coding if
they feel that amore accurate assessment is possible, in light of their consideration of
other countries. Group 2 repeats this procedure in reverse. That is, they first code a
handfulofrelatedcountriesandthenareaskedtocodeCountryA.
Athirdstudywouldbetofocusonquestionordering.TheV-Demquestionnaireis
not randomized for several reasons. First, somequestionsmust be asked in a particular
order(laterquestionsareactivatedorskippeddependingupontheanswers).Second,we
wish tomaintain a logical flowacrossquestions and tomake the flowaspredictable as
possible, so that inadvertent errors are minimized. Finally, we wish to maintain
equivalenceacrosssurveys.However,onemayalsowishtoknowwhethertheorderingof
questionson thequestionnaireaffects responses, and if sohow.Toprobe thisquestion
onewouldhavetorandomizequestionswithinasurvey(butnotacrosssurveys),without
upsettingquestionsthataredependentuponothers,andwhilemaintainingsomedegree
of logical flow. For example,wewill reverse the order of questions that are asked first
aboutmenandnextaboutwomen.
42
A fourth study could explore the quality of model-based bias adjustment. In
particular, because coders from different countries may understand both question
wordingsandconceptsindifferentways,twocodersoperatingindifferentcontextsmight
ratetwoidenticalcasesdifferentlyfromoneanother.Acommonapproachtoaddressing
thisproblemistoconstructanchoringvignettes—shorthypotheticaldepictionsofcases—
andthenaskcoderstoevaluatevignettesinadditiontorealcases,andtousedifferences
in vignette evaluations to correct for inter-personal differences in coder perceptions or
understandingsofconcepts (Kinget.al.2004;King&Wand2007;Hopkins&King2010).
Because the vignettes are fixed, these techniques assume that differences in rater
evaluationsmustrepresentdifferencesinpersonalinterpretation,andthensubtractthese
differences from responses for real cases, ostensibly correcting for respondent
incomparability. Similarly, given sufficient overlap in observed coding across raters, our
latentvariablemodelingtechniquescanusepatternsofinter-coderagreementtoidentify
and correct for systematic differences in raters' perceptions and conceptual
understandings. In otherwords, differences in howexperts rate identical cases help to
identify inter-expert variation in interpretation in much the same way that variation in
ratingsoffixedvignettesdoes.Wecanvalidatethisfeatureofthemodelbycomparingits
performance to a vignette-based approach for controlling incomparability in survey
responses.Focusingonasubsetofindicators,wewouldrecruitcountry-expertstoratean
anchoringvignette, theirowncountry,andsomecomparisoncountries. Thenwewould
apply both vignette-based and measurement-model based corrections to responses to
determineiftheyproducecomparableresults.Anexperimentalcomponentcanalsoseek
to determine if vignettes themselves alter coder behavior. In particular, we could use
patternsofagreementbetweenraterstodetermineiftreatedexperts(vignettecondition)
produce codings that are systematicallydifferent froma controlpopulation (novignette
condition).
43
References
Almond,GabrielA.,SidneyVerba.1963/1989.TheCivicCulture:PoliticalAttitudesandDemocracyinFiveNations.NewburyPark,CA:Sage.
Bernhard,Michael,EitanTzelgov,Dong-JoonJung,MichaelCoppedge,&StaffanI.Lindberg.2015.TheVarietiesofDemocracyCoreCivilSocietyIndex.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPaperSeries,No.12.
Bernhard,Michael,ChristopherReenock,andTimothyNordstrom.2004.“TheLegacyofWesternOverseasColonialismonDemocraticSurvival.”InternationalStudiesQuarterly48(3),225-250.
Bollen,KennethA.,PamelaPaxton.2000.“SubjectiveMeasuresofLiberalDemocracy.”ComparativePoliticalStudies33(1):58–86.
Capoccia,Giovanni,DanielZiblatt.2010.“TheHistoricalTurninDemocratizationStudies:ANewResearchAgendaforEuropeandBeyond.”ComparativePoliticalStudies43(8-9):931-968.
Clinton,JoshuaD.,DavidLewis.2008.“ExpertOpinion,AgencyCharacteristics,andAgencyPreferences.”PoliticalAnalysis16(1):3–20.
Clinton,JoshuaD.,JohnS.Lapinski.2006.“MeasuringLegislativeAccomplishment,1877-1994.”AmericanJournalofPoliticalScience50(1):232–249.
Collier,DavidandJamesMahon(1993).“Conceptual‘Stretching’Revisited:AdaptingCategoriesinComparativeAnalysis.”AmericanPoliticalScienceReview87(4):845-855.
Coppedge,Michael,StaffanLindberg,Svend-ErikSkaaning,andJanTeorell.2015.MeasuringHighLevelDemocraticPrinciplesusingtheV-DemData.UniversityofGothenburg,TheVarietiesofDemocracyInstitute:V-DemWorkingPaperseriesNo.6
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,DavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KyleMarquardt,KellyMcMann,FarhadMiri,PamelaPaxton,DanielPemstein,JeffreyStaton,EitanTzelgov,Yi-tingWang,andBrigitteZimmerman.2015.V-Dem[Country-Year/Country-Date]Datasetv5.VarietiesofDemocracy(V-Dem)Project.
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,withDavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KellyMcMann,PamelaPaxton,DanielPemstein,JeffreyStaton,BrigitteZimmerman,FridaAndersson,ValeriyaMechkova,andFarhadMiri.2015.V-DemCodebookv5.VarietiesofDemocracy(V-Dem)Project.
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,KyleMarquardt,ValeriyaMechkova,FarhadMiri,DanielPemstein,JosefinePernes,NataliaStepanova,EitanTzelgov,andYi-tingWang.2015.V-DemMethodologyv5.VarietiesofDemocracy(V-Dem)Project.
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,andVladCiobanu.2015.V-DemCountryCodingUnitsv5.VarietiesofDemocracy(V-Dem)Project.
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,ValeriyaMechkova,JosefinePernes,andNataliaStepanova.2015.V-DemOrganizationandManagementv5.VarietiesofDemocracy(V-Dem)Project.
44
Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,andJanTeorell.2015.V-DemComparisonsandContrastswithOtherMeasurementProjects.VarietiesofDemocracy(V-Dem)Project.
Dahl,RobertA.1971.Polyarchy:ParticipationandOpposition.NewHaven:YaleUniversityPress.
Dahl,RobertA.1989.DemocracyanditsCritics.NewHaven:YaleUniversityPress.Epstein,DavidL.;RobertBates;JackGoldstone;IdaKristensen;SharynO’Halloran.2006.
“DemocraticTransitions.”AmericanJournalofPoliticalScience50(3):551-569.Fox,Jean-Paul.2010.BayesianItemResponseModeling:TheoryandApplications.New
York:Springer.Gallie,W.B.1956.“EssentiallyContestedConcepts.”ProceedingsoftheAristotelian
Society56:167–220.Gerring,John,PhilipBond,WilliamBarndt,andCarolaMoreno.2005.“Democracyand
Growth:AHistoricalPerspective.”WorldPolitics57(3):323–364.Goertz,Gary.2006.SocialScienceConcepts:AUser’sGuide.Princeton:Princeton
UniversityPress.Hadenius,AxelandJanTeorell.2005.“CulturalandEconomicPrerequisitesofDemocracy:
ReassessingRecentEvidence.”StudiesinComparativeInternationalDevelopment39(4):87–106.
Held,David.2006.ModelsofDemocracy,3ded.Cambridge:PolityPress.Hopkins,Daniel,andGaryKing.2010.“ImprovingAnchoringVignettes:DesigningSurveys
toCorrectInterpersonalIncomparability.”PublicOpinionQuarterly:1-22.Inglehart,RonaldandWelzel,Christian.2005.Modernization,CulturalChangeand
Democracy:TheHumanDevelopmentSequence.Cambridge:CambridgeUniversityPress.
Isaac,JeffreyC.n.d.“ThinkingAbouttheQualityofDemocracyanditsPromotion.”Unpublishedms.
Jackman,Simon.2004.“WhatDoWeLearnfromGraduateAdmissionsCommittees?AMultipleRater,LatentVariableModel,withIncompleteDiscreteandContinuousIndicators.”PoliticalAnalysis12(4):400–424.
Johnson,ValenE.,JamesH.Albert.1999.OrdinalDataModeling.NewYork:Springer.Junker,Brian1999.SomeStatisticalModelsandComputationalMethodsthatmaybe
UsefulforCognitively-RelevantAssessment.http://www.stat.cmu.edu/~brian/nrc/cfa/documents/final.pdf
King,Gary,ChristopherMurray,JoshuaA.Salomon,andAjayTandon.2004.“EnhancingtheValidityandCross-culturalComparabilityofMeasurementinSurveyResearch.”AmericanPoliticalScienceReview98(1):191–207.
King,Gary,andJonathanWand.2007.ComparingIncomparableSurveyResponses:NewToolsforAnchoringVignettes.PoliticalAnalysis15:46-66.
Knutsen,CarlHenrik.2010.“MeasuringEffectiveDemocracy.”InternationalPoliticalScienceReview31(2):109-128.
45
Knutsen,CarlHenrik,JørgenMøller,andSvend-ErikSkaaning.Forthcoming.GoingHistorical:MeasuringDemocraticnessbeforetheAgeofMassSuffrage.InternationalPoliticalScienceReview.
Lindberg,StaffanI.2015.OrdinalVersionsofV-Dem’sIndices:ForClassification,Description,SequencingAnalysisandOtherPurposes.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.19.
Lord,FredericM.,andMelvinNovick.1968.StatisticalTheoriesofMentalTestScores.Reading,MA:Addison-Wesley.
Mahoney,James,andDietrichRueschemeyer,eds.2003.ComparativeHistoricalAnalysisintheSocialSciences.Cambridge:CambridgeUniversityPress.
Mariano,LouisT.andBrianW.Junker.2007.“CovariatesoftheRatingProcessinHierarchicalModelsforMultipleRatingsofTestItems.”JournaloftheEducationalandBehavioralStatistics32(2):287-314.
Munck,GerardoL.2009.MeasuringDemocracy:ABridgebetweenScholarshipandPolitics.Baltimore:JohnHopkinsUniversityPress.
Munck,GerardoL.2016.“WhatisDemocracy?AReconceptualizationoftheQualityofDemocracy.”Democratization23(1):1-26.
Nunn,Nathan.2009.“TheImportanceofHistoryforEconomicDevelopment.”AnnualReviewofEconomics1(1):1–28.
Patz,RichardJ.,andBrianW.Junker.1999.“AStraightforwardApproachtoMarkovChainMonteCarloMethodsforItemResponseModels.”JournalofEducationalandBehavioralStatistics24:146-178.
Patz,RichardJ.,BrianW.Junker,MatthewS.Johnson,andLouisT.Mariano.2002.“TheHierarchicalRaterModelforRatedTestItemsanditsApplicationtoLarge-ScaleEducationalAssessmentData.”JournalofEducationalandBehavioralStatistics27(4):341-384.
Pemstein,Dan,KyleL.Marquardt,EitanTzelgov,Yi-tingWang,andFarhadMiri.2015.“TheV-DemMeasurementModel:LatentVariableAnalysisforCross-NationalandCross-TemporalExpert-CodedData.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.20
Pemstein,Daniel,StephenMeserve,andJamesMelton.2010.“DemocraticCompromise:ALatentVariableAnalysisofTenMeasuresofRegimeType.”PoliticalAnalysis18(4):426–449.
Pierson,Paul.2004.PoliticsinTime:History,Institutions,andSocialAnalysis.Princeton:PrincetonUniversityPress.
Rose-Ackerman,Susan.1999.CorruptionandGovernment:Causes,Consequences,andReform.Cambridge:CambridgeUniversityPress.
Sartori,Giovanni.1970.“ConceptMisformationinComparativePolitics.”AmericanPoliticalScienceReview64(4):1033-1053.
Schedler,Andreas.2012.“JudgmentandMeasurementinPoliticalScience.”PerspectivesonPolitics10:1,21-36.
Shapiro,Ian.2003.TheStateofDemocraticTheory.Princeton:PrincetonUniversityPress.Sigman,RachelandStaffanI.Lindberg.2015.TheIndexofEgalitarianDemocracyandIts
Components:V-Dem'sConceptualizationandMeasurement.UniversityofGothenburg,
46
VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.21Steinmo,Sven,KathleenThelen,andFrankLongstreth,eds.1992.StructuringPolitics:
HistoricalInstitutionalisminComparativeAnalysis.Cambridge:CambridgeUniversityPress.
Teorell,Jan.2011.“OverTime,AcrossSpace:ReflectionsontheProductionandUsageofDemocracyandGovernanceData.”ComparativeDemocratization9:1(February)1,7.
Teorell,Jan,MichaelCoppedge,JohnGerring&StaffanLindberg.n.d.2016"MeasuringElectoralDemocracywithV-DemData:IntroducingaNewPolyarchyIndex.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.23
Teorell,Jan,RachelSigman,andStaffanI.Lindbergn.d.2016.V-DemIndices:RationaleandAggregations.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.22
Teorell,JanandStaffanI.Lindberg.2015.TheStructureoftheExecutiveinAuthoritarianandDemocraticRegimes:RegimeDimensionsacrosstheGlobe,1900-2014.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.5
Thomas,MelissaA.2010.“WhatDotheWorldwideGovernanceIndicatorsMeasure?”EuropeanJournalofDevelopmentResearch22(1):31–54.
Treier,Shawn,andSimonJackman.2008.“DemocracyasaLatentVariable.”AmericanJournalofPoliticalScience52(1):201–217.
Wang,Yi-ting,PatrikLindenfors,AkselSundström,FredrikJansson,andStaffanI.Lindberg.2015.NoDemocraticTransitionWithoutWomen’sRights:AGlobalSequenceAnalysis1900-2012.VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.12.
Wang,Yi-ting,AkselSundström,PamelaPaxton,andStaffanI.Lindberg.2015.”Women’sPoliticalEmpowermentIndex:ANewMeasure.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.18.
Welzel,Christian.2007.“AreLevelsofDemocracyAffectedbyMassAttitudes?TestingAttainmentandSustainmentEffectsonDemocracy.”InternationalPoliticalScienceReview28(4):397–424.
47
APPENDIXA:V-DemIndices,Components,andIndicators
DemocracyIndices
Names
Mid-LevelDemocracy
andGovernance
IndicesNames
Lower-Level
Democracyand
GovernanceIndices
Names NamesIndicators
v2_tagIndicesand
Indicators
ElectoralDemocracyIndex
v2x_polyarchy
Expandedfreedomofexpressionindex
v2x_freexp_thick
Governmentcensorshipeffort-Media
v2mecenefm
Governmentcensorshipeffort-Internet
v2mecenefi
Harassmentofjournalists v2meharjrn Mediaself-censorship v2meslfcen Mediabias v2mebias Print/broadcastmediacritical v2mecrit Print/broadcastmedia
perspectives v2merange Freedomofdiscussionformen v2cldiscm Freedomofdiscussionfor
womenv2cldiscw
Freedomofacademicandculturalexpression
v2clacfree
Alternativesourceinformationindex
v2xme_altinf
Mediabias v2mebias Print/broadcastmediacritical v2mecrit Print/broadcastmedia
perspectivesv2merange
Freedomofassociationindex(thick)
v2x_frassoc_thick
PartyBan v2psparban Barrierstoparties v2psbars Oppositionpartiesautonomy v2psoppaut Electionsmultiparty v2elmulpar CSOentryandexit v2cseeorgs CSOrepression v2csreprss Shareofpopulation
withsuffrage v2x_suffr
Percentofpopulationwithsuffrage
v2elsuffrage
Cleanelectionsindex v2xel_frefair EMBautonomy v2elembaut EMBcapacity v2elembcap Electionvoterregistry v2elrgstry Electionvotebuying v2elvotbuy
48
Electionothervotingirregularities
v2elirreg
Electiongovernmentintimidation
v2elintim
Electionotherelectoralviolence v2elpeace Electionfreeandfair v2elfrfair Electedexecutive
index(dejure) v2x_accex
Lowerchamberelected v2lgello Upperchamberelected v2lgelecup Legislaturedominantchamber v2lgdomchm HOSselectionbylegislaturein
practicev2exaphos
HOSappointmentinpractice v2expathhs HOGselectionbylegislaturein
practicev2exaphogp
HOGappointmentinpractice v2expathhg HOSappointscabinetinpractice v2exdfcbhs HOGappointscabinetin
practicev2exdjcbhg
HOSdismissesministersinpractice
v2exdfdmhs
HOGdismissesministersinpractice
v2exdfdshg
HOSappointscabinetinpractice v2exdfcbhsLiberalDemocracyIndex
v2x_libdem
ElectoralDemocracyIndex
v2x_polyarchy
LiberalComponentIndex
v2x_liberal
Equalitybeforethelawandindividuallibertyindex
v2xcl_rol
Rigorousandimpartialpublicadministration
v2clrspct
Transparentlawswithpredictableenforcement
v2cltrnslw
Accesstojusticeformen v2clacjstm Accesstojusticeforwomen v2clacjstw Propertyrightsformen v2clprptym Propertyrightsforwomen v2clprptyw Freedomfromtorture v2cltort Freedomfrompoliticalkillings v2clkill Freedomfromforcedlaborfor
menv2clslavem
Freedomfromforcedlaborforwomen
v2clslavef
Freedomofreligion v2clrelig Freedomofforeignmovement v2clfmove
49
Freedomofdomesticmovementformen
v2cldmovem
Freedomofdomesticmovementforwomen
v2cldmovew
Judicialconstraintsontheexecutiveindex
v2x_jucon
Executiverespectsconstitution v2exrescon Compliancewithjudiciary v2jucomp Compliancewithhighcourt v2juhccomp Highcourtindependence v2juhcind Lowercourtindependence v2juncind Legislative
constraintsontheexecutiveindex
v2xlg_legcon
Legislaturequestionsofficialsinpractice
v2lgqstexp
Executiveoversight v2lgotovst Legislatureinvestigatesin
practicev2lginvstp
Legislatureoppositionparties v2lgoppartDeliberativeDemocracyIndex
v2x_delibdem
ElectoralDemocracyIndex
v2x_polyarchy
DeliberativeComponentIndex
v2xdl_delib
Reasonedjustification v2dlreason Commongood v2dlcommon Respectcounterarguments v2dlcountr Rangeofconsultation v2dlconslt Engagedsociety v2dlengageEgalitariandemocracyIndex
v2x_egaldem
ElectoralDemocracyIndex
v2x_polyarchy
EgalitarianComponentIndex
v2x_egal
Equalprotectionindex
v2xeg_eqprotec
Accesstojusticeformen v2clacjstm Accesstojusticeforwomen v2clacjstw Socialclassequalityinrespect
forcivillibertiesv2clacjust
Socialgroupequalityinrespectforcivilliberties v2clsocgrp
Weakercivillibertiespopulationv2clsnlpct
Equaldistributionofresourcesindex
v2xeg_eqdr
50
Powerdistributedbysocioeconomicposition
v2pepwrses
Powerdistributedbysocialgroup
v2pepwrsoc
Educationalequality v2peedueq Healthequality v2pehealth Powerdistributedbygender v2pepwrgen Encompassing-ness v2dlencmps Means-testedvs.universalistic v2dlunivlParticipatoryDemocracyIndex
v2x_partipdem
ElectoralDemocracyIndex
v2x_polyarchy
ParticipatoryComponentIndex
v2x_partip
Civilsocietyparticipationindex
v2x_cspart
Candidateselection--National/local
v2pscnslnl
CSOconsultation v2cscnsult CSOparticipatoryenvironment v2csprtcpt CSOwomensparticipation v2csgender DirectPopularVote
Index v2xdd_dd
Initiativespermitted v2ddlegci
Initiativessignatures% v2ddsigcip
Initiativessignature-gatheringtimelimit
v2ddgrtlci
Initiativessignature-gatheringperiod
v2ddgrgpci
Initiativeslevel v2ddlevci
Initiativesparticipationthreshold
v2ddbindci
Initiativesapprovalthreshold v2ddthreci
Initiativesadministrativethreshold
v2dddistci
Initiativessupermajority v2ddspmjci
Occurrenceofcitizen-initiativethisyear
v2ddciniyr
Localgovernmentindex
v2xel_locelec
Localgovernmentelected v2ellocelc Localofficesrelativepower v2ellocpwr Localgovernmentexists v2ellocgov Regional
governmentindex v2xel_regelec
Regionalgovernmentelected v2elsrgel Regionalofficesrelativepower v2elrgpwr Regionalgovernmentexists v2elreggov
51
CoreCivilSocietyIndex
v2xcs_ccsi
CSOentryandexit v2cseeorgs CSOrepression v2csreprss CSOparticipatoryenvironment v2csprtcpt Party
Institutionalizationindex
v2xps_party
Partyorganizations v2psorgs PartyBranches v2psprbrch Partylinkages v2psprlnks Distinctpartyplatforms v2psplats Legislativepartycohesion v2pscohesv Womenpolitical
empowermentindex v2x_gender
Womencivillibertiesindex
v2x_gencl
Freedomofdomesticmovementforwomen
v2cldmovew
Freedomfromforcedlaborforwomen
v2clslavef
Propertyrightsforwomen v2clprptyw Accesstojusticeforwomen v2clacjstw Womencivilsociety
participationindex
v2x_gencs Freedomofdiscussionfor
womenv2cldiscw
CSOwomensparticipation v2csgender Percent(%)FemaleJournalists v2mefemjrn Womenpolitical
participationindex
v2x_genpp Powerdistributedbygender v2pepwrgen Lowerchamberfemale
legislatorsv2lgfemleg
ElectoralRegimeIndex
v2x_elecreg
Legislativeorconstituentassemblyelection
v2xel_elecparl
v2eltype v2eltype_0 v2eltype v2eltype_1 v2eltype v2eltype_4 v2eltype v2eltype_5 Legislatureclosed
downoraborted v2xlg_leginter
Legislaturebicameral v2lgbicam Presidential
election v2xel_elecpres
v2eltype v2eltype_6 v2eltype v2eltype_7
52
Chiefexecutivenolongerelected
v2x_hosinter
HOS=HOG? v2exhoshog HOGappointmentinpractice v2expathhg HOSappointmentinpractice v2expathhs Corruptionindex v2x_corr Legislaturecorruptactivities v2lgcrrpt Judicialcorruptiondecision v2jucorrdc Publicsector
corruptionindex v2x_pubcorr
Publicsectorcorruptexchanges v2excrptps Publicsectortheft v2exthftps Executive
corruptionindex v2x_execorr
Executivebriberyandcorruptexchanges
v2exbribe
Executiveembezzlementandtheft
v2exembez
ElectoralComponentIndex
v2x_EDcomp_thick
Freedomofassociationindex(thick)
v2x_frassoc_thick
PartyBan v2psparban Barrierstoparties v2psbars Oppositionpartiesautonomy v2psoppaut Electionsmultiparty v2elmulpar CSOentryandexit v2cseeorgs CSOrepression v2csreprss Shareofpopulation
withsuffrage v2x_suffr
Percentofpopulationwithsuffrage
v2elsuffrage
Cleanelectionsindex
v2xel_frefair
EMBautonomy v2elembaut EMBcapacity v2elembcap Electionvoterregistry v2elrgstry Electionvotebuying v2elvotbuy Electionothervoting
irregularitiesv2elirreg
Electiongovernmentintimidation
v2elintim
Electionotherelectoralviolence v2elpeace Electionfreeandfair v2elfrfair Electedexecutive
index(dejure) v2x_accex
Lowerchamberelected v2lgello Upperchamberelected v2lgelecup
53
Legislaturedominantchamber v2lgdomchm HOSselectionbylegislaturein
practicev2exaphos
HOSappointmentinpractice v2expathhs HOGselectionbylegislaturein
practicev2exaphogp
HOGappointmentinpractice v2expathhg HOSappointscabinetinpractice v2exdfcbhs HOGappointscabinetin
practicev2exdjcbhg
HOSdismissesministersinpractice
v2exdfdmhs
HOGdismissesministersinpractice
v2exdfdshg
HOSappointscabinetinpractice v2exdfcbhs Freedomof
expressionindex v2x_freexp
Governmentcensorshipeffort-Media
v2mecenefm
Harassmentofjournalists v2meharjrn Mediaself-censorship v2meslfcen Freedomofdiscussionformen v2cldiscm Freedomofdiscussionfor
womenv2cldiscw
Freedomofacademicandculturalexpression
v2clacfree