Election Predictions: An Empirical Assessment

Election Predictions:

An Empirical Assessment

WILLIAM BUCHANAN POLLSTERS have been publishing predictions of national election results for nearly 50 years. There are now enough instances to make possible a comparison of the empirical with the theoretical standards of accuracy .

Accuracy of election predictions may be expressed by several measures, each useful for a different purpose. These were developed by the Social Science Research Council team which appraised the performance of the American pollsters after their 1948 fiasco (Mosteller, 1949: 54-61). Among these measures are the following: 1. The proportion of polls which correctly predicted the winner.

2. "Average error," the mean of the differences between the predicted percentage and the actual vote percentage, ignoring the sign.

Most easily understood by newspaper readers, this average has the defect of concealing biases-all the prediction errors might be in favor of the more conservative party, or of the ultimate winner or of some other category.

3. Systematic error, the mean of the differences between the predictions and the election percentage, taking account of sign. This reveals the biases, but understates the size of the errors, since large overestimates cancel out equally large underestimates.

4. Nonsystematic error, the standard deviation of the individual er-

Abstract Examination of 155 poll forecasts in 68 national elections since 1949 shows that errors average nearly twice what statistical theory would indicate. Polls predict the division of vote between major parties better than individual party percentages, leading to 85 percent success in picking the winner. The worst failures occurred in a few elections where most polls went wrong. Liberal party votes are correctly forecast, conservatives slightly underestimated. Improved polling methods have not led to better forecasts.

William Buchanan is Professor of Politics at Washington and Lee University.

Public Op~nionQuarterly Vol. 50:222-227 O 1986 by the American Association for Public Opinion Research Published by The University of Chicago Press 0033-362x18610050-2221$2.50

ELECTION PREDICTIONS: AN EMPLRICAL ASSESSMENT

223

rors around the mean of the errors in a series of elections. It tells how widely the predictions vary, discounting systematic error. It is not too meaningful to lay readers, but useful to statisticians since it permits one to compare the performance of polls with the deviations that would be expected for given sample sizes. It is these probability calculations that underlie the familiar "margin of error" statements that appear in all responsible reportage of poll forecasts. However, these calculations are based on probability theory, and do not take account of the actual performance of the polls.

These error calculations may be made for each party's vote as a percentage of the total vote, or as a percentage of the "major party" vote. The latter is what most consumers of polls are interested in, since it has political relevance. It predicts who will come in first and how far ahead of the second-place party.

The SSRC's report showed that through 1948 the polls' nonsystematic error was not excessive, but there had been a decided proRepublican bias, as reflected by the systematic error measure.

Robert M. Worcester (1983) has recently edited a history of political polling that brings together actual and predicted results of 96 election polls in Australia, France, Germany, Britain, Japan, and the Netherlands. I have updated and supplemented these with Canadian, New Zealand and Australian, and U.S. predictions, bringing to 155 the total of forecasts made in 68 elections since 1949. These were the final published percentage predictions of the popular vote for a candidate, party, or coalition in national elections. The data set includes a large majority of such predictions made during the period. Sample size is not always given, but where it is, 1500 or 2000 is the norm, with occasionally larger and rarely smaller samples, none under 900. The assumption for purposes of analysis that the average sample size is 1500 is conservative.

Using the categories established in the SSRC report, average, systematic, and nonsystematic errors were calculated for the party that won the election and also for the one that came in second, each as a percentage of the total vote. Then the percentages of winner and runner-up were added and the winner's percentage of the two-party vote compared to the prediction. This leaves out a residue of votes for minor parties, plus, in a few instances where it was so published, an "undecided" percentage, these being of little interest to consumers of predictions. Since there were several competing forecasts in most elections it seemed sensible to subtract the predicted percentage from the actual percentage; thus errors with a positive sign in the calculations that follow are underestimates and those with a minus are overestimates of what that party actually got.

224

WILLIAM BUCHANAN

The published "margin of error" percentages are derived from the statistical formula for the standard error:

where p is the proportion we are trying to estimate, q = 1 - p, and n is the number of cases in the sample.

The statistical analog for this calculation is a vat containing millions of marbles (equivalent to the electorate) of which about half are white (the leading party's vote). The pollster scoops up a bucketful of marbles (roughly 1500) time after time, counts them, throws them back, stirs the vat and scoops up another bucketful (for 155 samples). If done with marbles we could count on close to 95 percent of the sample estimates varying from the proportion of white marbles in the vat by no more than than the margin of error based on the formula above, in this case 2.5 percent: 1.962/(.5 x .5) / 1500 = .025.

Merely describing this ideal sampling arrangement is enough to show the contrast to what election forecasters are really doing. Voters aren't susceptible to being stirred and replaced; similar ones are clumped together and surveys sample these neighborhood or telephone prefix clusters. Some aren't home and some don't have phones. Some haven't made up their minds and some won't vote, though they may think they will (we don't know what color marbles these are, or indeed whether they are marbles at all). Each poll has its own adjustment procedure, often a valued trade secret, for coping with these problems.

The customary "margin-of-error" stipulations in the media are based upon the formula for simple random sampling, which rounds off to plus-or-minus 2.5 percent for 1500 cases, 2 percent for 2000 cases.

With adjustment for cluster sampling these figures would be around 3.5 percent and 3 percent, respectively.

The 95 percent probability criterion calls for an average of one "bad" prediction-i.e., one outside the specified margin-in every 20 attempts. National elections are infrequent and we need the full sample of 155 to make use of the principle, which would call about 8 errors of this size.

The calculations of the relevant averages appear in Table 1. The average absolute error for the 155 predictions is close to 2 percentage points-not a bad record. For winning parties the systematic error is +0.45. Thus the winner's percentages were underestimated by half a ppint on the average. Fortunately, the polls also underestimated the percentage of the second party by 0.29, which gave them a better record for their two-party predictions of - 0.04-a trivial overestimate of the winner's percentage.

225

ELECTION PREDICTIONS: AN EMPIRICAL ASSESSMENT

Table 1. Accuracy in Percentage Points of 155 National Poll Predictions, 1950-1984.a (Negatives are overestimates.)

Winning Party Average error Systematic error Nonsystematic error

2.02

2.14

Two-Party Split 2.16

+ 0.45

+ 0.29

- 0.04

2.62

2.85

2.73

Systematic error

By decade 1950s 1960s 1970s 1980s

Runner-up

Conservative

LiberaliLabor

+0.89

-0.08

Systematic

Nonsystematic

N

0.09 -0.39 0.15 - 0.12

2.89 2.15 2.60 3.14

13 29 67 46

" Australia 20, Canada 16, France 26, West Germany 9, Japan 14, Netherlands 1, New Zealand 4, United Kingdom 36, United States 29.

The nonsystematic error is an indication of how closely these predictions cluster around the actual percentages, after allowing for these biases. This error component may be thought of as the net effect of all those uncorrected-for not-homes, nonvoters, undecideds, and mindchangers, plus random sampling errors, with some of the errors canceling out others. We may compare the standard deviations for these elections with the theoretical ones derived by the statisticians' formula, which represents the error due to sample size alone. Standard error from the formula above is The vote for the average winning party is 47 percent of the total, so we insert .47 for p and 1500 for n, our approximation of the average sample size, to get a theoretical standard error of 1.29 percentage points, which should be comparable to our nonsystematic errors.

But it is not. Standard deviation of the errors for winning parties turns out to be 2.62 percentage points, twice what would be expected.

Multiplying by 1.96 we get a margin-of-error estimate of 5.1 percent, which we may compare to the theoretical margins of 2.5 percent to 3.5 percent as calculated above. Where theory would lead us to expect 7 or 8 errors larger than 3.5 percent, there were in fact 3 1 errors of this magnitude in the two-party predictions. As Paul Perry (1979) of the Gallup organization has observed: "One cannot expect to achieve in surveys an accuracy consistent with the theory of random sampling."

On the basis of the long-term performance of the polling profession as a whole, press accounts of predictions should say "5 percent margin of error" where they now say "3 percent" and make comparable adjustments to all such estimates based on sample size.

226

WILLIAM BUCHANAN

Ideologues of left and right are convinced that pollsters rig their predictions to favor the other party. There is a greater likelihood, however, that the method itself is biased toward one side or the other.

Perhaps the most perplexing problem faced in the adjustment decision is how to screen out those citizens who express an intention but are unlikely to reach the voting booth to implement it. Pollsters do this by eliminating or down-weighting on the basis of other questions about past voting habits, registration, interest in the election, and so on (Lipset, 1980; Traugott and Tucker, 1984; Wiener, 1976). If the screen is too loose, the intentions of nonvoters will be counted; if it is too tight some potential voters will be left out. There are also socioeconomic biases in sampling: e.g., the rich can restrict interviewers' access to their dwellings and can have unlisted phones; the poor are suspicious of interviewers and also may have no phones. The problem of election prediction is whether these screening, sampling, and other biases cancel out or whether they cumulate, as they obviously did in 1948.

We can shed some light on the question by grouping the parties into the more conservative ones (winners 103 times) and liberal or labor parties (winners 52 times) and comparing the systematic errors. It appears that there was an underestimate of conservative votes by nearly a percentge point ( + 0.89) and a trivial overestimate of the liberalflabor vote (-0.08). These led to predicting a liberal victory 17 times when the conservatives won (17 percent wrong) compared with 5 bad forecasts when the liberal or labor party won (10 percent wrong).

We might expect the predictions to have improved over the years since 1949, particularly in the 1980s, when random-digit-dialing and widespread telephone ownership have made possible forecasts based on data collected only a day or two before the election. Apparently not.

The results by decades in Table 1 show no consistent pattern by either error measure.

In general the most serious errors occurred in a few elections in English-speaking democracies where several polls made the same mistakes. In the British election of 1970, five polls overestimated the Labor vote by an average of more than 4 percentage points, leading to a two-party error of some 3.5 points in favor of a party that lost the popular vote by 2.5 points. Thus 4 out of 5 forecasts picked the wrong winner. The year 1983 was in some respects even worse; the average of six polls was an overprediction of the Conservative vote by over 3 points and an underprediction of Labor by about 2 points, resulting in a two-party bias toward the Tories of nearly 4 points. But Mrs. Thatcher had such a commanding lead that no prediction was wrong and the error attracted less comment than the smaller one in 1970. Bad years in Australia were 1980, when a 3-point average underestimate of the Liberals caused five polls to go astray in a very close election, and 1984,

ELECTION PREDICTIONS: AN EMPIRICAL ASSESSMENT

227

when three out of four polls predicted a much wider margin for Labor than eventuated. In 1976 and 1980 in the U.S., both major parties were seriously underestimated, but the underestimates offset each other in the two-party percentages and put most of the polls on the right side. In all, 22 predictions (out of 155) pointed to the wrong winner, half of them occurring in these debacles.

The consistency with which almost all the polls in a few elections made errors in the same direction and of about the same magnitude is revealing. These are situations in which the electorate is volatile and people make up or change their minds in the last few days about whether and/or how they will vote. Survey procedures based on experience in more stable situations may not work as well. The popular stereotype that "the polls are always wrong" is incorrect. They are on the wrong side in only one prediction out of every seven. A better oversimplification would be: "Whenever the polls are wrong, they are all wrong."

To summarize, examining the performance of most of the published national election predictions since 1949 in a variety of political cultures and electoral systems shows: (1) The margin of error estimates based on probability theory seriously understate the empirical range of error.

(2) Winners' percentages are underestimated on the average by about half a point and runners-up by a quarter point. (3) These compensating underestimates lead to better predictions of the two-party distribution and hence to picking the winner about 85 percent of the time. (4) There is no evidence that either biases or random errors have been reduced over the years. (5) There is a tendency to underestimate the conservative vote. (6) The bad predictions are made in a few elections when most of the polls err in the same direction.

References Lipset, Seymour Martin 1980 "Different polls, different results in 1980 politics." Public Opinion, August1 September, p. 19.

Mosteller, Frederick, Herbert Hyman, Phillip J. McCarthy, Eli S. Marks, and David B. Truman 1949 The Pre-election Polls of 1948. New York: Social Science Research Council.

Perry, Paul 1979 "Certain problems in election survey methodology." Public Opinion Quarterly 43:322-25.

Traugott, Michael W., and Clyde Tucker 1984 "Strategies for predicting whether a citizen will vote and estimation of political outcomes." Public Opinion Quarterly 48:330-43.

Weiner, Sanford L. 1976 "The competition for certainty: the polls and press in Britain." Political Science Quarterly 91:673-696.

Worcester, Robert M. 1983 Political Opinion Polling. New York: St. Martin's Press.

http://www.jstor.org

LINKED CITATIONS - Page 1 of 1 -

You have printed the following article: Election Predictions: An Empirical Assessment William Buchanan The Public Opinion Quarterly, Vol. 50, No. 2. (Summer, 1986), pp. 222-227.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198622%2950%3A2%3C222%3AEPAEA%3E2.0.CO%3B2-8

This article references the following linked citations. If you are trying to access articles from an off-campus location, you may be required to first logon via your library web site to access JSTOR. Please visit your library's website or contact a librarian to learn about options for remote access to JSTOR.

References Certain Problems in Election Survey Methodology Paul Perry The Public Opinion Quarterly, Vol. 43, No. 3. (Autumn, 1979), pp. 312-325.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28197923%2943%3A3%3C312%3ACPIESM%3E2.0.CO%3B2-U

Strategies for Predicting Whether a Citizen Will Vote and Estimation of Electoral Outcomes Michael W. Traugott; Clyde Tucker The Public Opinion Quarterly, Vol. 48, No. 1. (Spring, 1984), pp. 330-343.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198421%2948%3A1%3C330%3ASFPWAC%3E2.0.CO%3B2-7

The Competition for Certainty: The Polls and the Press in Britain Sanford L. Weiner Political Science Quarterly, Vol. 91, No. 4. (Winter, 1976-1977), pp. 673-696.

Stable URL: http://links.jstor.org/sici?sici=0032-3195%28197624%2F197724%2991%3A4%3C673%3ATCFCTP%3E2.0.CO%3B2-Y