Procedures for Evaluating by Juan Pardo

Procedures for Evaluating

Trends in Public Opinion

D GARTH TAYLOR

T H I s PAPER shows how to use the percentage difference models described by Davis (1976) and by Grizzle et al. (1969) to analyze trends in public opinion data. The first part of the paper is an attempt to persuade the reader that most of the interesting theories of public opinion change can be thought of as statistical models which make formal predictions about percentages or percentage differences. The next part shows how to choose the appropriate model (or theory) to describe the change by examining a series of goodness-of-fit tests. Finally, a few detailed examples show the application of the method to patterns of change in public opinion data which are difficult to describe, both verbally and statistically. Change and Change Models in Public Opinion Research

Trend analysis in public opinion research begins with a theory of change (or lack of change) to be tested or a data set (usually consisting of a series of percentages or percentage differences) to be described as showing a "significant" or "nonsignificant" pattern of

Abstract This article expresses a variety of theories of public opinion change in terms of a formal model for analyzing public opinion data. Statistical criteria are proposed whereby models of varying complexity can be accepted or rejected. The observation is made that certain substantive theories of change are more parsimonious than others in terms of their statistical predictions. The model for analyzing change presented here is based on the premise that the most parsimonious theories should be accepted or rejected first. D Garth Taylor is an Assistant Professor in the Department of Political Science at the University of Chicago and Senior Study Director at the National Opinion Research Center. Public Opinion Quarterly 1980 by The Trustees of Columbia University Published by Elsevier North Holland, Inc. 0033-362X18010044-086/$1.75

EVALUATING TRENDS I N PUBLIC OPINION

change. Usually t h e researcher has both. I n o u r experience, a n array of possible theories c a n b e used t o describe a set of d a t a , a n d t h e ultimate theoretical conclusion depends o n t h e most parsimonious statistical model which c a n b e used t o describe t h e data. It is important to recognize t h e predictions about t h e statistical patterns in public opinion d a t a which a r e m a d e in various theories o f change. Following a r e some examples of theories o f change that have appeared recently in t h e literature, and their associated statistical predictions. Theory I : Plus qa change, plus c'est la mBme chose. This is a theory that things don't really change. Statistically, the prediction is that a set of proportions only departs randomly from a fixed value or that a series of percentage differences are not statistically different from zero. For instance, Sullivan et al. (1979) claim that within a proper framewgrk for conceptualization and measurement, survey research shows that the level of tolerance in America has not changed in recent decades. Theory 2: Social statics. The key proposition here is that differences endure. Some recent examples in the literature are studies of enduring regional and cultural differences (Reed, 1972; Glenn and Simmons, 1967), enduring class differences in political behavior (Alford, 1963), and the enduring effects of education (Hyman et al., 1975). Each study argues that, controlling for other changes in the society, the correlations between certain sets of variables remain constant. Theory 3: Regular processes described by transitive verbs in gerund form. Studies on assimilating blacks into the electoral process (Converse, 1972), rnassifying or differentiating social differences (Glenn, 1967) and diminishing social class differentials (Mayer, 1959) are in this category. Each study suggests steady, gradual change in intergroup differences-in some theories the differences are growing, while in others the differences are diminishing. The point is the regularity of the process of change and the suggestion that the data might be described by a linear pattern of increasing or decreasing differences in the population. Theory 4: Social catastrophe. Analyses of fertility expectations (Blake, 1974), confidence in national leaders (Smith et al., 1979), and attitudes toward foreign involvements (Gallup, 1972) all argue that trends in these measures change erratically as the result of sudden developments in the world economic or political situation. The public opinion measures show a great deal of statistically significant "bounce," and the hypothesis is that the change cannot be described by any of the preceding, simpler models. I n analyzing a set of public opinion data, w e typically w a n t to know which theory describes t h e change observed in t h e data. M o r e formally, this question becomes, which of t h e statistical models associated with t h e four theories most parsimoniously describes t h e data?

D GARTH TAYLOR

The next section illustrates the procedures used to decide this question. Statistical Procedures

The steps in the statistical analysis are to derive the trend in public opinion which is predicted under each theory of change and then, using a chi-square goodness-of-fit test, decide which theory provides the best and most parsimonious fit to the data. In this exposition the analyses will all be focused on describing trends in intergroup differences in public opinion (i.e., the object of attention is the pattern of change in a series of percentage differences). The procedures described here can be simplified to study trends in percentages. Table 1 shows the trend from 1965 to 1973 in support for abortion among Protestants (excluding Baptists) and Catholics. We note that in 1965 both groups were predominantly opposed to abortion for the Table 1. The Trend in Religious Differences in Response to the Question: "Tell me

whether or not you think it should be possible for a pregnant woman to obtain a legal

abortion if the family is poor and cannot afford another child."

Data

Surveya Date Protestantsb Percent No (N) Catholics

Percent No

AIP0721 12/65

SRS870 11/65

AIP0788 9/69

GSS72 3/72

GSS73 3/73

80.6 (803)

79.4 (807)

75.3 (815)

51.5 (756)

42.8 (755)

83.8

83.2

82.6

62.1

60.3

( N (853) (327) (356) ) (364) , (358) Statistical Analysis Category difference (Base=ProtestantJ Hvoothesis

Catholic

a) no difference b) constant difference C) linear change reduction from linear term

Model

Decision

d=O d = dp d = a + bx

94.4 29.6 .3

5 4 3

<.05 1.05 >.05

reject reject accept

29.3

<.05

significant

Final Model Catholic: d = -1.34 + ,021 (year-1900) a SRS870 is a survey conducted by the Survey Research Service of the National Opinion Research Center. GSS refers to General Social Surveys, conducted by the National Opinion Research Center, funded by the National Science Foundation. AIP0721 and AIP0788 are surveys conducted by the American Insititue of Public Opinion (Gallup). Protestants, not including Baptists.

EVALUATING TRENDS IN PUBLIC OPINION

stated reason (if the family is poor and cannot afford another child), and the religious difference is small (about 3 percent). By 1973 both groups were more in favor of abortion and the religious differences are greater (about 17 percent.) Which theory adequately and parsimoniously describes the pattern of religious differences shown in Table l? To answer this question we need to determine: (a) the pattern in the data predicted under each theory; (b) the appropriate statistical procedure for deriving the predicted percentage differences under each model; and (c) the closeness (goodness-of-fit) between the predicted and observed percentage differences for each model. The Plus ca change theory predicts that the percentage difference is actually zero in each survey year and that the differences observed in Table 1 are nothing more than random sampling fluctuations. In this case, the predicted percentage difference is supplied by the theory--each difference should be zero-and so there is no further statistical estimation required. We will learn that this theory does not fit the data in Table 1 very well. The exact goodness-of-fit test is described below. The social statics theory predicts that the religious difference is constant-that there is some true, nonzero value for the religious difference which characterizes each survey and that the pattern of differences is really random fluctuation around this true difference. The intuitive solution is to take the average of the percentage differences as the pooled estimate of the "true" difference. For full statistical power, however, it is more efficient to take a weighted average of the survey results, where the results for any survey are weighted proportionally to the amount of confidence we place in the accuracy of that survey. There are several reasons for weighting the results for each survey proportionate to the amount of confidence we have in that survey. It can be the case, although it is not in Table 1, that surveys at different times differ greatly in the number of cases in the sample. It is an accepted standard of survey procedure that proportions from a larger sample should be given greater weight in reckoning the significant pattern in the data. We also note that there are two Gallup surveys and three NORC surveys represented in the time series in Table 1. These surveys may differ in the efficiency of the sampling procedures, in the size of the sample clusters, or in other procedures such as interviewer training or interviewer instructions which will have implications for the effective sample size (and hence error variance) of the survey results (Hansen, 1951; Glenn, 1975; Newman, 1976). The most efficient estimate of the constant true score predicted by the

DGARTHTAYLOR

social statics theory for describing the data in Table 1 is a weighted estimate of the pooled percentage difference for the five surveys (Goodman 1963). The procedure for estimating the weighted pooled percentage difference is as follows: 1 . Compute the percentage difference and the variance of the percentage difference for each year. According to textbook formulas the percentage difference is p , - pz and the variance of the percentage difference is the sum of the variances for each percentage. The variance of a percentage is (p* (1-p))lN where N is the number of cases the percentage is based on. If the data are not from a probability sample or if there are other reasons to suspect the quality or effective sample size of the data, the place to quantify this reservation is in the estimate of the variance of the percentages for that survey. The formula for pooling the results of the surveys assigns less weight to the less reliable surveys. 2. The weights for pooling the percentage differences are inversely proportional to the variance of the percentage difference. The weights are arrived at by taking the reciprocal of the variance for each percentage difference and dividing this by the sum of the reciprocals of the variance for each percentage difference. i.e. :

where W indexes the weights, V indexes the variances and k indexes the number of surveys pooled for the weighted average. 3. The pooled weighted percentage difference is the sum of the weight for each conditional D times the difference for conditional D.

The variance of D, can be used to assess the statistical significance of the pooled weighted average. This statistic is:

In sum, the social statics theory predicts that each percentage difference is only randomly different from the weighted pooled percentage difference. The statistical test for this conclusion will be given below (the model does not fit the data in Table 1). The third theory, the transitive gerund theory, predicts that the

EVALUATING TRENDS IN PUBLIC OPINION

percentage differences are regularly becoming larger or smaller. The rhetoric of this theory is easily stated in a linear regression format: the predicted percentage difference at any time is equal to a constant plus some factor adjusted for the amount of time elapsed between observations:

+ b * (Time)

(4)

where D indexes the predicted percentage difference.

In the linear regression analysis of the predicted percentage differences, the same considerations apply as earlier regarding the importance of weighting the results of each survey inversely proportional to the variance of the percentage difference for that survey. Some explanations of how to do this appear in statistics texts under discussions of weighted least squares or heteroscedasticity (Wonnacot and Wonnacot, 1970). In the language of regression analysis, the goal is to minimize the weighted squared deviations from the regression line, where the weights are defined as before in Eq. (1). The formulas for a weighted least squares analysis of the model in Eq. (4) are: b

where W, Dk

D Tk -

(D, - D) * (T, W,, * ( T , - T)2

(5)

the weight for each percentage difference as before = the percentage difference in the k t h survey = the simple mean of the Dk = the time (year and month) of the kth survey = the mean of the Tk

The two important substantive comments to make about this procedure are: (1) the difference between these estimates and the regular regression results is the presence of the weights in the numerator and denominator of Eq. (5); and (2) the same weighted least squares principles apply to more complex models for describing nonlinear patterns of change. The transitive gerund theory predicts that the percentage differences in Table 1 are predicted (within sampling error) by a regression model such as the one shown in Eq. (4). It turns out that this is the theory which is appropriate for describing the data in Table 1-the religious differences are growing gradually larger.

DGARTHTAYLOR

Before describing the goodness-of-fit tests that are used to choose between theories of change, it is useful to consider what the catastrophe theory would have predicted for the pattern of differences in Table 1. The catastrophe theory predicts that there are significant fluctuations in the data which are not captured by a constant difference model or a linear trend model for the percentage differences. Since a linear change model subsumes the constant difference model (the constant difference model is a linear change model with b = O ) , the prediction of the catastrophe theory is essentially that the linear change model does notfit. We will see that the linear change model does fit the data in Table 1 and so the catastrophe theory is not appropriate there. The chi-square test for the goodness of fit for any model compares the observed percentage differences with the percentage differences predicted under the model. The number of degrees of freedom for the chi-square is equal to the number of surveys being compared minus the number of parameters estimated in the model for the predicted percentage differences. Under the plus Ca change theory there are k tables (in our notation) and the theory provides the only parameter required for the analysis-the hypothesis is that all differences are equal to zero. Therefore there are k degrees of freedom for theplus ca change hypothesis. The social statics theory uses the data to estimate 1 parameter, the pooled weighted percentage difference and so there are k-1 degrees of freedom for this test. The transitive gerund theory estimates two parameters-the slope and intercept of the regression equation-and so there are k-2 degrees of freedom here. The social catastrophe predicts that the linear regression model does not fit and so there is no further estimation for this theory: if the linear model fits we accept the transitive gerund theory, if it does not fit we accept the catastrophe theory. Once the predicted values are calculated, the goodness of fit of the theory is assessed using the following chi-square formula:

where

k Dk

= =

p =

the number of tables being compared the observed percentage difference in the k t h table the expected percentage difference in the kth table under some model the number of parameters estimated for the model the variance of the observed percentage difference in the k t h table

EVALUATING TRENDS IN PUBLIC OPINION

The expected percentage difference is subtracted from the observed value, the difference is squared and divided by the variance of the observed percentage difference, and the results are added over the number of surveys available. Table 2 summarizes the models that are being tested, the predicted values under each theory of change, and the chi-square calculation for each goodness-of-fit test. The sequence of parameters estimated and models tested is an extremely important aspect of the general procedure presented here for evaluating trends in public opinion. The four theories of opinion change were not initially presented as a sequence or hierarchy of hypotheses. Indeed, they are substantively different interpretations of the process of change in public opinion. However, it is important to recognize that from a statistical point of view, the theories are ordered in their complexity in at least two ways: as we move down the list the later theories require a greater number of parameters to be estimated in order to "explain" the patterns in the data; and, the theories constitute a nested hierarchy of hypotheses. We have already noted that the later theories "subsume" the earlier theories. For instance, if the plus qa change theory fits the data, then the social statics theory will also fit because the "no difference" theory is the same as a "constant difference" theory with the difference equal to zero. Thus, the principle of parsimony becomes important as the basis for our choice of theories to explain the data. The goal of the sequence of statistical tests is to choose the theoretical model which most parsimoniously describes the data. If the simplest theory does not "fit" (i.e., if the chi-square is too large given the number of degrees of freedom), then we move on to test the next most complex model, and so on. The series of models is organized in such a way Table 2. A Summary of the Predicted Percentage Differences and the Chi-Square Test Under Each Hypothesis for Describing Change in Public Opinion Data

Hypothesis

1. Zero difference 2. Constant

3. Linear change

Predicted Values for Each Tnhle

Zero Weighted average Weighted least squares estimate

(Dk - zero)2 Vk (Dk - ~ 0 0 1 ) ~

(k)

(k-1) Vk 2 (Dk - DkY ik-2) VVk A

DGARTHTAYLOR

that each successive model requires one more parameter to be estimated and hence uses up one more degree of freedom in explaining the data. The difference in chi-squares between successive models is the test of significance (evaluated on one degree of freedom) for the additional parameter that was required for the more complicated model. Thus, a sequence of tests that begins with the "no difference" model and tests the goodness of fit of each successive model and the significance of the parameter which is added for each successive model produces an unambiguous procedure for arriving at the most parsimonious model for describing a series of percentage differences in public opinion data. This sequence of tests is shown in Figure 1. Final Model (check for outliers)

Hypothesis No difference (df = k )

-D=O

N. S.

Model Type

reject

Constant value (df = k-1)

Test reduction X 2 Significant-D from No difference model on one df C ~ . ~ - ~

reject

Linear trend (df = k - 2 )

Occept

-x2

= constant borderline significance

2 2a

N.S

Test reduction x2 from Constant difference model

<..

Significant-D

+ b

* (time)

~ linear ~

3 3a

trend

I '-'-"' \

= constant

Test reduction X 2 from Constant difference model on one d f

Significant -,Substantial linear component

N. S-.

Nonlinear trend

Explore further nonlinear or multiple linear models

Figure 1. Decision Rules for Evaluating Trends in Public Opinion

EVALUATING TRENDS IN PUBLIC OPINION

There are seven possible outcomes for the sequence of tests proposed here. If one accepts the hypothesis that all differences could be zero (theplus Ca change theory) then one arrives at Model Type 1. If one finds that the "no difference" theory does not fit the data but the "constant difference" does, then one accepts the social statics explanation. Because of the character of the chi-square distribution, it is possible that the "constant difference" model is required to fit the data, but the pooled weighted percentage difference is only of marginal statistical significance. This is determined by subtracting the chisquare for the "no difference" model from the chi-square for the "constant difference" model and assessing the difference in chisquare on one degree of freedom. This is the test of significance of the pooled percentage difference. If the D, is significant, we arrive at model type 2, if it is not we choose model type 2a. A similar set of procedures applies for testing the "linear change" model. The model may fit the data or it may not (in which case we accept the catastrophe theory). If the model does fit, then we assess the significance of the linear term (i.e., the slope) by subtracting chi-square and testing for significance on one degree of freedom. If the linear model does not fit, there is still the possibility that there is a significant linear component to the change in the percentage differences. This is examined using the same difference in chi-squares. If the linear model does not fit the data, but the chi-square reduction for the linear component is significant then we choose model 3b. In the next part of the paper we will apply the decision rules to some examples of trends to be analyzed in public opinion. At the end of the section are a few further details and caveats of a methodological nature. Some Examples

For the data in Table 1, the decision rules lead to the adoption of the linear change model (type 3). The sequence of chi-square tests is shown in the section of the table labeled "statistical analysis." The "no difference" and "constant difference" models do not fit, the "linear change" model does fit, and the reduction from the linear term is highly significant. The final model states that the religious difference changed by about 2.1 percentage points a year between 1965 and 1973 (b = .021). Table 3 shows the percentage of men and women approving of abortion if the woman became pregnant as a result of rape. In each survey the sex difference is not very large and the sign of the difference varies depending on the year. Both these factors conspire to

D GARTH TAYLOR

Table 3. The Trend in Sex Differences in Response to the Question: "Tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if she became pregnant as a result of rape."

Data Surveya Date Male Percent No (A') Female Percent No (N)

Category Difference (Base =Male) Female

~ ~ ~ 8 7 0 GSS72 11/65 3/72 40.4 (675) 40.2 (715)

GSS73 3/73

GSS74 3/74

15.6 (673)

14.5 (668)

17.2 (778)

12.6 (752)

20.8 (751) 21 .O (761) Statistical Analvsis

Hypothesis

Model

Decision

a) no difference

d =0

1.8

>.05

Final Model

Female: d = 0

a SRS870 is a survey conducted by the Survey Research Service of the National Opinion Research Center. GSS refers to General Social Surveys, conducted by the National Opinion research Center, funded by the National Science Foundation.

make the "no difference" model the appropriate one for describing the data. Table 4 shows the trend in the percent favoring abortion in case of rape for three educational groups. For polytomous variables, we follow Davis's strategy (1976) and choose one category as the base category, analyzing the trend in the difference between the base category and each of the nonbase categories in the percent favoring abortion. In our example "less than high school education" is the base category. The statistical analysis shows that we accept a different model for each of the nonbase categories. For the difference between high school and less than high school education, the chi-square for the "constant difference" model is 10.7 with 3 degrees of freedom. This chi-square would be insignificant if we had applied correction factors to the variances of the percentages to account for the clustering effects in the construction of the survey samples. The simple route to the chi-square correction is to divide the chi-square by some factor between 1.2 and 2 before assessing the significance of any comparison (Kish, 1965:162; Kish, 1957:159; Taylor, forthcoming). The exact correction factor depends on the particulars of the survey but an extremely conservative procedure is always to divide the chi-squares by 2 before the significance test. Looking at the trend in the college vs. less than high school dif-

EVALUATING TRENDS IN PUBLIC OPINION

Table 4. The Trend in Educational Differences in Response to the Question: "Tell me whether or not you think it should be possible for a pregnant woman to obtain a legal abortion if she became pregnant as a result of rape."

Data Survey

Date

Education category

Less than high school

Fercent No

(N) High school graduate

Percent No

(N) College

Percent No

(N) Statistical Analysis Category Difference fBase=<HS) High school graduate College

Hvoothesis a) b) a) b) c)

Model

no difference constant difference no difference constant difference linear change in

difference reduction from linear term

d=O d = dp d=O d = dp d=a

+ bx

102.8 10.7 273.4 34.1

4 3 4 3

<.05

reject

<.05 <.05

reject

14.4

<.05

reject

19.7

<.05

significant

Decision

Final Model High school graduate College

d = -.I25

d d

= =

-.I89 -1.54

+ .0187

a = .0130 a = .0122

(year-1900)

ference, we see the final model is type 3b. The linear model of change does not fit, but there is a substantial linear component to the pattern of percentages. Therefore neither of the final models reported is adequate for describing the pattern in the data. The constant difference is an inappropriate model because there is change over time in the percentage difference. The linear change model is inappropriate because there are significant deviations from the linear trend. The appropriate strategy for the analyst in this situation would be to explore further models or to construct a verbal explanation of the pattern of the percentage differences which takes account of the results shown in Table 4. OUTLIERS

After applying the decision rules in Figure 1, it is a good idea to examine the data for outliers before reporting a final model. When

DGARTHTAYLOR

comparing surveys over time, outliers are a distinct possibility. "Wild observations" can come from slight differences in question wording or interviewer instructions, differences in sampling procedures from study to study or among survey organizations, or from mechanical errors in processing older studies with incomplete documentation. There is no single, unambiguous procedure for spotting outliers, but the following rule is consistent with the statistical literature (Anscornbe and Tukey, 1963; Tukey, 1962): An individual observation may be considered an outlier if the chi-square for the deviation of that observation from the modeled estimate is too large. Therefore, the chi-square test for an outlier is:

The sampling correction should be applied to this formula, and so we might summarize the decision rule as follows: an observation should be studied more carefully and possibly rejected from the analysis if the corrected chi-square for the deviation of that observation from the modeled estimate is significant at the .O1 level (6.6 or greater). ANOTHER METHODOLOGICAL PROBLEM

When applying the decision rules it is possible to arrive at a negative chi-square for the contribution of an added parameter. In this case the chi-square should be taken as zero for purposes of the analysis. This is not due to rounding error or a fault in the logic of the system; rather, the reason lies in the distribution theory for the statistics and the fact that the methods we propose here are "approximate" methods. Conclusion

The purpose of this paper is not to invent new statistical procedures. Rather, the goal is to take the weighted least squares model for percentage differences (Davis, 1976; Grizzle et al., 1969) and link the steps in this model to a series of theories which might be used to describe change in public opinion data. Like most calls for applied statistics in substantive research, we emphasize the ordering of hypotheses and the use of significance tests to unambiguously choose between alternative explanations.

EVALUATING TRENDS IN PUBLIC OPINION

T h e discussion addresses t h e question o f h o w to find t h e most parsimonious description of a series of percentage differences. But t h e principles o f t h e analysis apply t o a n y statistic that is relevant t o public opinion research (the only limitation being that t h e statistic t o b e analyzed must have a variance that c a n b e calculated.) Thus, t h e procedures can b e used to find t h e most parsimonious model for a series of percentages (although t h e hypothesis that all t h e percentages a r e equal to z e r o is not of much interest.) Likewise, b y analyzing trends in t h e logarithms of odds ratios, w e a r e simply committing ourselves to a particular series of hypotheses in testing log linear models (Davis, 1975). Finally, other studies of trends in gamma statistics (e.g., Verba et al., 1976) might well have used s o m e variant of t h e decision rules suggested here t o analyze t h e data.

References Alford, R. L. 1963 "The role of social class in American voting behavior," Western Political Quarterly 16: 180-94. Anscombe, F. J., and J. Tukey 1963 "The examination and analysis of residuals," Technometrics 5:141-60. Blake, Judith 1966 "Ideal family size among white Americans," Demography 3: 154-73. 1974 "Can we believe recent data on birth expectations in the United States?" Demography 11:25-44.

Converse, P. E.

1972 "Change in the American Electorate," in Angus Campbell and Philip Converse (eds.), The Human Meaning of Social Change. New York: Russell Sage Foundation. Davis, James A. 1975 "The log linear analysis of survey replications," in Kenneth Land and Seymour Spilerman (eds.), Social Indicator Models. New York: Russell Sage Foundation. 1976 "Analyzing contingency tables with linear flow graphs," Sociological Methodology. San Francisco: Jossey Bass. Gallup, George 1972 The Gallup Poll: Public Opinion 1935-1971. New York: Random House. Glenn, Norval 1967 "Massification and differentiation: some trend data from national surveys," Social Forces, 46: 172-80. 1975 "Trend studies with available survey data: opportunities and pitfalls," in J. Southwick (ed.), Survey Data for Trend Analysis. Williams College Center, Roper Public Opinion Research. Glenn, Norval, and J. L. Simmons 1967 "Are regional cultural differences diminishing?" Public Opinion Quarterly 31:176-93.

100

D GARTH TAYLOR

Goodman, Leo 1963 "On methods for comparing contingency tables," Journal of the Royal Statistical Society (A) 126:94- 108. 1976 "The relationship between modified and usual multiple regression approaches to the analysis of dichotomous variables," Sociological Methodology. San Francisco: Jossey Bass. Grizzle, James E., C. Frank Starmer, and G. Koch 1969 "Analysis of categorical data by linear models," Biometries 25:489-504. Hansen, Morris, W. Hurwitz, E . Marks, and W. Mauldin 1951 "Response errors in surveys," Journal of the American Statistical Association 46: 147-90. Hyman, H. H., J. Wright, and J. Reed 1975 The Enduring Effects of Education. Chicago: University of Chicago. Kish, Leslie 1957 "Confidence intervals for clustered samples," American Sociological Review 22: 154-65. 1965 Survey Sampling. New York: Wiley. Mayer, K. 1959 "Diminishing class differentials in the United States," Kyklos XIII:605-28. Newman, Karen 1976 "The use of AIPO surveys: to weight or not to weight," in James A . Davis (ed.), Studies of Social Change Since 1948 Volume 1 . Chicago: National Opinion Research Center. Nie, N., S. Verba, and J. Petrocik 1976 The Changing American Voter. Cambridge: Harvard University Press. Reed, John Shelton 1972 The Enduring South. Lexington, Mass.: D.C. Heath. Smith, Tom W., D Garth Taylor, and Nancy Mathiowetz 1979 "Public opinion and public regard for the federal government," in A. Barton and C. Weiss (eds.), Making Bureaucracies Work. Beverly Hills, California: Sage. Sullivan, John L., James E. Pierson, and George Marcus 1979 "Political intolerance: the illusion of progress," Psychology Today 12:87-91. Taylor, D Garth forth- "Analyzing qualitative data," in P. Rossi, J. Wright, and A. Andercom- son (eds.), The Handbook of Survey Research. New York: Acaing demic Press. Tukey, John 1962 "The future of data analysis," Annals of Mathematical Statistics 33:l-67. Wonnacot, R. J., and T. H. Wonnacot 1970 Econometrics. New York: Wiley .