election polls in the  21 st century: can they be trusted in the era of cell phones?

ELECTION POLLS IN THE  21ST CENTURY:

CAN THEY BE TRUSTED IN THE ERA OF CELL PHONES?

League of Women Voters Lawrence Township

First Statistical Associations

• Statistical Society of London (1834)

• American Statistical Association (1839)

• International Statistical Institute (1885)

Sir Ronald Fisher and Prasanta Mahalanobis

Anders Kiaer and the Norwegian Census of 1891

Source: Kiaer, Anders N. 1897 “The Representative Method Of Statistical Surveys”, p. 44.

Fisher’s Agricultural Research

• Use of repeated measurements to estimate crop yields (1919)

• Development of randomized experiments, hypothesis testing and tests of statistical significance

• Publishes Statistical Methods for Research Workers in 1925

Gini, Galvani and the Italian Census of 1921

• Need to make room for the 1930 census• Chose 29 of 216 census districts• Averages computed across districts

don’t match population numbers• Both purposive and random sampling

considered inferior to full census tabulations

Source: Feinberg, Stephen E. and Judith M. Tanur. 1995. “Reconsidering Neyman On Experimentation And Sampling: Controversies And Fundamental Contributions.” Probability and Mathematical Statistics Vol. 15, pp. 47-60.

The Impact of the Great Depression• Unemployment clearly a massive problem

• Questions on work developed from 1880 through 1930 censuses

• Population measurements needed between decennial censuses

• Trial Census of Unemployment 1933-34• Enumerative Check Census of 1937• Leads to the Current Population Survey

Source: Card, David. 2011. “Origins of the Unemployment Rate: The Lasting Legacy of Measurement Without Theory” http://davidcard.berkeley.edu/papers/origins-of-unemployment.pdf

1936 Presidential Election• Literary Digest Poll

– 2 million responses– forecasts Republican

victory• Gallup Poll

– 300,000 responses– forecasts

Democratic victory• Roosevelt gets 67% of

the popular vote, 98% of electoral votes

1948 Presidential Election• Gallup Poll

– Use of quota sampling

– 3,250 responses– forecasts Dewey

victory• Truman gets 49.5% of

the popular vote, 62% of electoral votes

• End of quota sampling• Campaign dynamics

American Association for Public Opinion Research

• Founded in 1947• More than 2,200 public

opinion scientists, survey researchers and statisticians are members

• Code of Professional Ethics• Transparency Initiative

Average Errors in Presidential Polls 1948-2008Year # of Polls # of

CandidatesAverage Error

(%)

2008 23 2 1.4

2004 19 2 1.7

2000 19 3 1.7

1996 9 3 1.7

1992 6 3 2.2

1988 5 2 1.5

1984 6 2 2.4

1980 4 3 3.0

1976 3 3 1.5

1972 3 2 2.0

1968 2 3 1.3

1964 2 2 2.7

1960 1 2 1.0

1956 1 2 1.8

1956-2008 (Avg) 1.8

1948 3 3 4.9

Source: Traugott, Michael. 2005. “The Accuracy of the National Preelection Polls” Public Opinion Quarterly 69(5): 649

The 2000 Election: What Went Wrong?

• “If we say somebody’s carried a state, you can pretty much take it to the bank.” – Dan Rather

• Competitive pressures on networks to call the outcome in Florida first

• Voter News Service has problems with following procedure for sampling and interviewing exiting voters and with computer models processing incoming results

Accuracy of the 2008 Polls

Source: http://electoralmap.net/pollsters/popular.txt

Identifying the Likely Voter

Likely Voter Questions

How much thought have you given to the coming presidential election?

How closely have you been following news about the candidates?

Do you plan to vote in the presidential election? How certain are you that you will vote?

Rate your chance of voting in November on a scale of 10 to 1, with 10 being "definitely will vote" and 1 "definitely will not vote.“

How often do you follow what's going on in government and public affairs?

Have you ever voted in your precinct or election district?

How often would you say you vote?

Do you happen to know where people in your neighborhood go to vote?

In the last election, did things come up that kept you from voting or did you vote?

Source: Ask the Expert: Determining Who Is a 'Likely Voter' Scott Keeter, Director of Survey Research August 29, 2012

Problems with Polling in 2012

Source: http://electoralmap.net/pollsters/popular.txt

• Declining Response Rates• Growth of the “Cell-only” population• Shifting attention to state polls after 2000• Increasing popularity of survey by “robo-

call”

Declining Response Rates

Source: http://www.people-press.org/files/legacy-pdf/Assessing%20the%20Representativeness%20of%20Public%20Opinion%20Surveys.pdf

The Growing “Cell-Only” Population

Source: http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201206.pdf

Dual Frame Estimation

Source: http://pewresearch.org/pubs/1601/assessing-cell-phone-challenge-in-public-opinion-surveys

Summary of Differences Between Dual Frame and Landline Estimates

Surveys by Robo-call

• Understanding the math behind a surveySample

size CostMargin of

errorFixed cost 5000

200 $10,000 6.9%Unit cost 25

400 $15,000 4.9%

600 $20,000 4.0%

800 $25,000 3.5%

1,000 $30,000 3.1%

1,200 $35,000 2.8%

1,400 $40,000 2.6%

1,600 $45,000 2.5%

1,800 $50,000 2.3%

2,000 $55,000 2.2%

3,000 $80,000 1.8%

5,000 $130,00

0 1.4%

Rasmussen’s Methodology

“While we do not currently call cell phones directly when conducting our surveys, to reach those who have abandoned traditional landline telephones, Rasmussen Reports uses an online survey tool to interview a demographically diverse panel. Just like our telephone surveys, respondents for this approach are selected on a random basis to insure the reliability of the sample. As a result, you cannot sign up to participate. “

Aggregating State Polls

• Nate Silver’s 538• RealClear Politics• Princeton Election Consortium• PollTracker• Dave Leip's Atlas of U.S. Presidential Elections

RealClear Average

Nate Silver/ NY Times

Princeton Election Consortium

Important Trends

• Early voting• Voter ID movement/Voter fraud issue

– Demographic changes in the US• Increasing automation/improving robo-call

technology• More web-based surveying