the contribution of comparative research to measuring the policy preferences of legislators

501Comparative Legislative Research

LEGISLATIVE STUDIES QUARTERLY, XXXIII, 4, November 2008 501

GERHARD LOEWENBERGThe University of Iowa

The Contribution of ComparativeResearch to Measuring the PolicyPreferences of Legislators

Comparative legislative research has contributed to an examination of thevalidity of roll-call votes as measures of legislators’ policy preferences. It has promptedan awareness of the influence of legislative structure on the composition of the votingrecord. Comparative research on members’ ideal points has confronted the problemsof selection effects, abstentions, the influence of the agenda setter, and the effect ofparty strategy. It has encouraged the search for alternate measures of members’preferences, including members’ speech, cosponsorship, survey responses, and partymanifestos. In the non-American setting, ideal points have been regarded as group-level, as well as individual-level, variables. The game-theoretic approach to the studyof legislatures has led to the formulation of hypotheses relating legislative structureto members’ ideal points.

Reexamining Roll-call Votes asMeasures of Legislators’ Preferences

The assumption of American exceptionalism and theconceptualization of the U.S. Congress as a unique legislature havebeen formidable obstacles to comparative legislative research (Gammand Huber 2002). The resulting separation between research onCongress and research on other legislatures has impeded the develop-ment of general theories of legislatures. But there has been anothercost, recognized only recently. Studying Congress separately from otherlegislatures has isolated congressional research, depriving research byU.S. scholars of conceptualizations of legislative behavior, relevantdata, and research methods that offer alternatives to those employedin the study of Congress.

The commitment of U.S. legislative scholars to the analysis ofroll-call votes as measures of legislators’ policy preferences is a primeexample. The availability of roll-call data for the U.S. Congress overits entire history has made the use of these data irresistible in congres-sional research. The development of spatial models of voting, advances

502 Gerhard Loewenberg

in computer technology, algorithms like NOMINATE, and Bayesianpoint estimation have spurred advances in the ability to discoverroll-call voting patterns. That ability has further encouraged roll-callanalysis. Because so much of this research has taken place in oneinstitutional setting, it has overlooked the impact of institutional struc-ture on the voting record itself. Research on legislative voting behaviorin the United States has not only ignored non-U.S. legislatures, butalso neglected the 99 state legislative bodies in the United States. Whenlooking at Congress, it has focused very heavily on the House ofRepresentatives alone, paying much less attention to the Senate.

Recently, however, there has been growing awareness of theinfluence of the institutional structure on the voting record. Longitu-dinal studies of voting behavior in the U.S. House, like that describedby Joshua D. Clinton and John Lapinski in this issue, have long had totake account of the significant rules change in the 92d Congress, whichhenceforth allowed recorded votes in the Committee of the Whole.Since that committee is where most amendments are decided, the votingrecord included relatively few votes on amendments before that date.Roll-call studies of the U.S. Senate have also long recognized thedifferences between the chambers—for example, with respect to agermaneness requirement for amendments in the Senate—that causesignificant differences between their voting records. Research todistinguish party from constituency influence on voting behavior has turnedattention to the influence of party leaders in setting the voting agenda andthe influence of party tactics on the inferences that can be drawn from thevoting record. So, even with attention devoted only to the U.S. Congress,it has become evident that differences between House and Senate, changesin voting rules over time, and variations in party tactics all influence thedatabase on which roll-call analysis rests. The expansion of researchon U.S. state legislatures has had the same effect of creating an aware-ness of the institutional structure’s effect on roll-call data.

What is true of differences in one national legislature or amonglegislatures in one country is true in spades for differences amonglegislatures cross-nationally. Studies of voting behavior in non-U.S.legislatures have never been able to rely on roll-call analysis to thesame extent as studies of the U.S. Congress, because recorded votesdid not exist in many of those legislatures. Where they did exist, therecord was often sparse and obviously incomplete. The influence ofparty on voting was taken for granted. With less opportunity to use roll-call data, researchers studying legislatures outside the United States requiredalternative measures of legislators’ preferences, as well as an awarenessof the effect of the structure of the institution on the voting record.


The realization that roll-call votes are not unbiased indicators oflegislators’ preferences therefore results, at least in part, from lookingat the world beyond Congress. Experience studying legislative prefer-ences in other legislatures has refracted back to the use of roll calls inthe study of Congress. We might regard this development as an exampleof the globalization of legislative research, which has some other char-acteristics of globalization in general, such as a division of laborbetween American and non-American scholars. As this division of labormakes use of the comparative advantage that scholars outside the UnitedStates have in research on their own legislatures, we have seencollaborative research with some of the marks of outsourcing. TheQuarterly is devoting a special section in this and the next issues to aset of articles, “Measuring the Policy Preferences of Legislators,” manyof which are the products of cross-national collaboration. Most drawtheir data from non-American legislatures. They exhibit the contribu-tion of comparative research to the problem of analyzing legislators’ideal points. The articles address research questions that are method-ological in the broad sense, not of measurement techniques but of thevalidity of inferences drawn from measures to theoretical concepts.

The Validity of Roll Calls as Measures ofLegislators’ Ideal Points

Although roll-call votes were long regarded as the definitivemeasure of legislators’ preferences on legislation in the U.S. Congress,the study of roll calls over long time periods has identified significantproblems of selection bias. An assessment of the relationship betweencongressional roll calls and legislative enactments over a century showsthat roll calls are useful indicators only for landmark legislation andfor legislative activity in the last 30 years. Surprisingly, no roll callsexist at all in the House of Representatives for nearly half of the top3,500 statutes enacted in the past century in Congress (see Clinton andLapinski’s article in this issue). Relying on roll calls even in studies ofCongress therefore requires careful assessment of the effect of thisuneven record. Because of the obvious difference between the incentivefor recording a vote and the incentive for enacting a law, the existenceof a roll-call record may depend on the political control of the branchesof government, external events, and particular political issues.

The voting record of a legislature is never complete. It variesacross time within a legislature, across chambers in a bicamerallegislature, across legislatures within the same political system andacross systems, as a function of rules of procedure that govern voting,


as a function of party systems and therefore of party strategies, and asa function of political issues. This inescapable variation in the compo-sition of the roll-call record raises questions about the validity of roll-call data as unbiased indicators of legislators’ preferences. Even whenthe record is extensive, there are missing data because members areabsent, because members who are present may nevertheless abstain,and because there is a filter imposed by the procedures that govern theuse of roll calls in voting.

In their article in this issue, Clifford Carrubba, Matthew Gabel,and Simon Hug, who have done extensive work on roll calls in theSwiss Parliament and the European Parliament, address the problemof generalizing from the selection of roll-call votes that happen to beavailable to the universe of decisions in a given parliament. They assessthe effect of the process of selecting decisions for roll-call votes, awarethat party leaders use the instrument of roll calls to discipline members.The authors point out that if the research question is the influence ofparty on parliamentary decisions, then the selection of votes for rollcalls is endogenous to the very characteristic being investigated. Sinceit is impossible to compare how a legislator would have voted had aroll call requested by party leaders not taken place, the authors havedeveloped a theoretical model that generates results for differentassumptions about partisan legislative power and provides testablepredictions. The work done by Carrubba and his associates shows howthe process of selecting roll-call votes in a legislature biases the sampleof roll calls available for analysis, and it reveals the influence of cross-national and cross-temporal differences in the institutional setting thatdetermines that process.

The use of roll-call data is further bedeviled by the difficulty ofinterpreting abstentions. Abstentions cannot be ignored: they arefrequent in many parliaments, they may be intentional acts rather thanrandom occurrences, and they may vary across institutions accordingto voting rules. In their article in this issue, Guillermo Rosas and YaelShomer compare two models that make different assumptions aboutthe inferences to be drawn from abstentions. The authors tested themodels with two legislatures, those of Argentina and Israel, in both ofwhich abstentions are common. On the basis of substantive knowl-edge about the legislative process in these two countries, Rosas andShomer conclude that abstentions cannot be regarded as random. Theyshow that alternative assumptions about the process of abstaining yieldradically different conclusions about the ideological divisions in thesetwo parliaments. Their work shows how important it is to makeinformed assumptions about the inferences to be drawn from missing


data, rather than dismissing these data, assuming that they reflectrandom occurrences.

Alternative Measures of Legislators’ Ideal Points

Awareness of the incompleteness of the voting record has ledother scholars to consider alternative indicators of members’ prefer-ences. Special sections in the next issues of the Quarterly will considera range of such indicators. Two groups of scholars have analyzedlegislator speech and undertaken content analysis for estimatinglegislators’ preferences. Another set of scholars has used data oncosponsorship of bills as measures of legislators’ preferences,comparing revealed preferences in the Argentine and U.S. Congress.The authors recognize that legislators may cosponsor legislationbecause of personal rather than issue-based affinities and that a recordof cosponsorship is bound to be incomplete, because no one shouldassume that a member enters into cosponsorship on all bills that he orshe favors. This line of research attempts to detect the non–issue-basedinfluences on cosponsorship and offers a comparison between co-sponsorship and roll-call data as indicators of policy agreement.

Still another group of scholars has attempted to distinguish partyinfluence from other influences on legislative outcomes in the EuropeanParliament. The authors have modeled votes in the European Parliamentin a Bayesian hierarchical framework to separate party influence fromthe influence of members’ individual preferences. For a measure ofmembers’ individual preferences, the authors relied on responses to aEuropean Policy Research Group survey, comparing these responses withother predictors of the voting records of members of the European Parlia-ment. They found that, while members’ voting behavior appears to beparty disciplined, members’ ideal points indicate wide intraparty variation.

Finally, members’ ideal points have long been identified by theuse of elite surveys. A recent example of this approach employed avariety of measures of members’ ideal points in an analysis of legislativebehavior in Latin America.

The Effect of Party-system and Institutional Structureon the Measurement of Legislators’ Ideal Points

As comparative legislative research has contributed to theexamination of the validity of inferences drawn from the range of avail-able indicators of members’ ideal points, it has also directed attentionto the effect of variance in party-system and institutional structure on


such indicators. When studies focus on single examples of the legisla-tive institution, both institutional and party-system variance are ignored.In the American setting, members’ policy preferences have beenconceptualized as individual-level variables. This tradition reflects theindividualistic conception of the legislature that long predominated inAmerican legislative research. In many, if not most, non-Americansettings, the assumption has been that ideal points are the characteristicsof party groups rather than of individual members. In legislatures withstrongly disciplined political parties, the party groups appear to be theprincipal actors. But identifying the ideal points of parties, particu-larly in legislatures taking relatively few recorded votes, raisesdistinctive challenges. If the unit of observation is a group, then howis its ideal point to be measured? One measure has been developed bythe group of scholars engaged in the Comparative Manifesto Project.The group has devised measures of party policy positions in a largenumber of countries over the entire post–World War II period, relyingon the content analysis of party manifestos. The project has covered2,347 programs issued by 632 parties in 52 countries (Budge et al.2001; Klingemann et al. 2007). These data have been important incomparative legislative research, although they also raise questions ofvalidity. They have made it possible to address a set of research ques-tions that arise in the parliamentary system, where the output of thelegislature takes the form of supporting or dismissing cabinet coali-tions, which are the real loci of policy decisions. This use of party as aunit of observation is one example of how structures—in this case,party systems—affect measurement.

Another example comes from research on how legislative insti-tutions set their agenda. Work by Gary W. Cox and Mathew D.McCubbins (2005) has shown that in the U.S. House of Representa-tives, the majority party has considerable control over the agenda.Through its ability to determine the issues on which votes take place—more than by its ability to impose party discipline on substantivevotes—the majority party can achieve party cohesion. Cox andMcCubbins have provided a theoretical basis for regarding roll-callvotes as indicators of collective decisions on the legislative agenda ineven one of the most individualistic legislatures in the world. Theyhave tested hypotheses derived from their procedural cartel theory withobservations of House members’ votes, which they interpret as evidenceof how infrequently the majority party is defeated. Since majoritycoalitions in many legislatures worldwide exercise effective controlof the agenda, Cox and McCubbins’s work on the effect of agendasetting on roll-call votes has very general implications for the identifi-


cation of legislators’ preferences. Paradoxically, Cox and McCubbinsuse roll-call votes as their principal source of data, and thus they recognizethat they, too, cannot escape problems of measurement validity.

Research Questions Employing Members’ Ideal Points

Cox and McCubbins’s research on agenda setting is an exampleof convergence between the research questions guiding U.S. legislativestudies and the legislative studies of other countries. That convergencelargely results from conceptualizing legislatures in game-theoreticalterms that abstract them from their national contexts. Game theory fitslegislative research particularly well, since legislatures can readily beconceptualized as rule-governed institutions whose members arepurposeful actors. Scholars studying the U.S. Congress were amongthe first to apply game theory to real institutions, and they in turnmotivated the further development of game theory. In the early 1990s,Kenneth A. Shepsle and Barry R. Weingast, two of the leading scholarsusing this approach, regretted that thus far it had been applied only tothe U.S. Congress, “surely the most untypical example of the legislativeinstitution” (Shepsle and Weingast 1994). Eight years later, Shepslenoted that the diffusion of the game-theoretical approach to the studyof legislatures outside the United States demonstrated its generalapplicability (Shepsle 2002). As a result, more and more legislativeresearch across countries addresses the effect not only of agenda setting,but also of rules of procedure and of party leadership on determiningwhat members and parties may and may not vote on. These condi-tions, in turn, define the voting record and set the parameters withinwhich members’ ideal points can be measured. The convergence ofresearch methods and research questions has enabled cross-nationalresearch to contribute to the challenge of measuring legislators’ policypreferences.

The game-theoretic approach has also proven useful in explainingthe origins of particular structural arrangements within legislatures,relying on inferences rather than direct measures of members’ idealpoints. Two examples from opposite ends of the time span show howthis approach has been applied. Douglas C. North and Barry R.Weingast (1989) have explained the abdication of committee power inthe seventeenth-century British parliament as an effort to avoid pork-barrel legislation that would have raised taxes for property holders. Inother words, the rules of the game were changed to constrain particularoutcomes by denying members the possibility of expressing sectionalpolicy preferences. Examining institutional choices made three centuries


later in post-Communist Russia, Steven S. Smith and Thomas F. Remington(2001) explained that the electoral system, party system, committee system,and leadership structure established by the new Russian State Duma at itsorigin reflected the partisan, policy, and electoral goals of its members. Asthe relevance of these goals changed over the first decade of the Duma’sexistence, so did the Duma’s initial organizational characteristics. In neitherof these studies were microlevel data on members’ ideal points brought tobear; in the earlier case, they were not even available. Instead, ideal pointswere inferred from macrolevel outcomes.

One of the most interesting research questions in the study oflegislatures in parliamentary systems is legislatures’ influence on theformation and durability of governing coalitions. Research on thissubject provides another example of the relationship between legislativestructures and ideal points, in this case, the ideal points of parties ratherthan of members. The assumption in parliamentary systems is thatparties seek influence over policy not primarily through the legislativeprocess but by occupying government ministries. The relative strengthsand the ideal points of the parties that are potential coalition partnershave been analyzed to predict the equilibrium among them that willsustain a coalition (Laver and Shepsle 1996). External shocks that upsetthis equilibrium impose transaction costs. A comparison of transac-tion costs across parliamentary systems makes it possible to identifythe institutional-structural characteristics that influence parties’ idealpoints and therefore coalition durability (Mershon 2002; Saalfeld 2007).Attempts to explain the durability of governing coalitions may takedifferent forms in the analysis of presidential systems, but conceptual-izing legislative processes in coalition terms and in a game-theoreticframework is another way to employ the concept of ideal points.

Discussion

The convergence of research methods and research questions, aswell as growing collaboration between U.S. and non-U.S. scholars,has facilitated comparative legislative research and enriched the studyof legislators’ policy preferences. It has prompted the reexaminationof roll-call votes as measures of members’ ideal points. It has expandedthe research questions that can be addressed by data on members’ idealpoints. By conceptualizing legislatures as structures abstracted fromtheir national settings, this new mode of comparative legislative researchhas stimulated the formation of hypotheses regarding the relationshipbetween legislative structures and the voting record. These new hypoth-eses have, in turn, seeded bodies of legislative research in two directions.


First, there is a growing direction of research on single, non-American legislative institutions that employs methods similar to thosedominating U.S. legislative research. Much of this research is publishedin the journals and the languages of the countries in which it is doneand is therefore not as widely known as is the American literature.Language barriers continue to be major obstacles to comparativeresearch. Second, there is a slowly evolving direction of genuinelycomparative research, much of it published in English, that is oftenthe product of collaboration between American and non-Americanscholars. The articles on legislative preferences—the ideal points ofmembers and parties—in this issue of the Quarterly and in subsequentissues exemplify this characteristic of contemporary comparativelegislative research.

Thirty years ago, Malcolm E. Jewell wrote in the first issue ofthe Quarterly that “it has become commonplace among critics oflegislative studies to argue that we need to pay more attention to thedevelopment of theory,” and “it is difficult to develop theories oflegislative institutions by examining only one such institution at onepoint in time” (1976, 1). In the mid-1970s, there were few studiescomparing the U.S. Congress to any other national legislatures andfew comparisons among the state legislatures of the United States, letalone comparisons among non-American legislatures. The relativeproportion of articles being published on the U.S. Congress, on U.S.state legislatures, and on non-U.S. legislatures in the Legislative StudiesQuarterly may not have changed much in three decades. What haschanged is the convergence of research methods and the researchquestions that motivate this research. Undoubtedly this confluencereflects significantly improved communication among legislativescholars across countries. It has resulted in a research paradigm thatpromises an increase in collaborative, genuinely cross-national researchin the future. Language barriers and the disproportionate size of thelegislative research community in the United States compared to thenumber of colleagues in this field in other countries are, admittedly,enduring obstacles that are difficult to surmount. But such obstaclesmay not be as important as the epistemological and methodologicaldifferences that existed in the past, differences that have been substan-tially diminished in the last two decades. The special sections on mea-sures of legislators’ policy preferences in this and subsequent issuesof the Quarterly show a degree of common purpose among legislativescholars that is the result of expanding legislative research beyond theU.S. Congress and moving it toward genuine comparison acrosslegislative institutions.


REFERENCES

Budge, Ian, Hans-Dieter Klingemann, Andrea Volkens, and Judith Bara. 2001. MappingPolicy Preferences: Estimates for Parties, Electors, and Governments, 1945–1998. New York: Oxford University Press.

Carrubba, Clifford, Matthew Gabel, and Simon Hug. 2008. “Legislative VotingBehavior, Seen and Unseen: A Theory of Roll-call Vote Selection.” LegislativeStudies Quarterly 33: 543–72.

Clinton, Joshua D., and John Lapinski. 2008. “Laws and Roll Calls in the U.S. Congress,1891–1994.” Legislative Studies Quarterly 33: 511–42.

Cox, Gary W., and Mathew D. McCubbins. 2005. Setting the Agenda: ResponsibleParty Government in the U.S. House of Representatives. New York: CambridgeUniversity Press.

Gamm, Gerald, and John Huber. 2002. “Legislatures as Political Institutions: Beyondthe Contemporary Congress.” In Political Science: State of the Discipline, ed.Ira Katznelson and Helen V. Milner. New York: W.W. Norton.

Jewell, Malcolm E. 1976. “Editor’s Introduction.” Legislative Studies Quarterly 1: 1.Klingemann, Hans-Dieter, Andrea Volkens, Judith Bara, and Ian Budge. 2007. Mapping

Policy Preferences II: Estimates for Parties, Electors, and Governments inCentral Land Eastern Europe, European Union, and OECD, 1990–2003. NewYork: Oxford University Press.

Laver, Michael, and Kenneth A. Shepsle. 1996. Making and Breaking Governments:Cabinets and Legislatures in Parliamentary Democracies. New York: CambridgeUniversity Press.

Mershon, Carol. 2002. The Costs of Coalition. Stanford, CA: Stanford University Press.North, Douglas C., and Barry R. Weingast. 1989. “Constitutions and Commitment:

The Evolution of Institutions Governing Public Choice in Seventeenth-centuryEngland.” Journal of Economic History 49: 803–32.

Roberts, Jason M. 2007. “The Statistical Analysis of Roll-call Data: A CautionaryTale.” Legislative Studies Quarterly 32: 341–60.

Rosas, Guillermo, and Yael Shomer. 2008. “Models of Nonresponse in LegislativePolitics.” Legislative Studies Quarterly 33: 573–602.

Saalfeld, Thomas. 2007. “Koalitionsstabilität in 15 Europäischen Demokratien von1945 bis 1999: Transaktionskosten und Koalitionsmanagement.” Zeitschrift fürParlamentsfragen 38: 180–206.

Shepsle, Kenneth A. 2002. “Assessing Comparative Legislative Research.” InLegislatures: Comparative Perspectives on Representative Assemblies, ed.Gerhard Loewenberg, Peverill Squire, and D. Roderick Kiewiet. Ann Arbor,MI: University of Michigan Press.

Shepsle, Kenneth A., and Barry R. Weingast. 1994. “Editors’ Introduction: FormalModels of Legislatures.” Legislative Studies Quarterly 19: 145–47.

Smith, Steven S., and Thomas F. Remington. 2001. The Politics of Institutional Choice:The Formation of the Russian State Duma. Princeton, NJ: Princeton UniversityPress.

the contribution of comparative research to measuring the policy preferences of legislators

Documents