Effects of Question Wording and Format by Juan Pardo

Effects of

Question Wording and Format

on Political Attitude Consistency

GEORGE F. BISHOP, ROBERT W. OLDENDICK

A N D ALFRED J. TUCHFARBER

N E OF THE more enduring controversies in American public opinion research centers on the question of whether the average voter thinks ideologically (cf. Axelrod, 1967; Bennett, 1973; Bishop, 1976; Converse, 1964, 1975; Field and Anderson, 1969; Litwak et al., 1973; Nie and Andersen, 1974; Nie et al., 1976; Pierce, 1975; Robinson and Holm, 1976; Weissberg, 1976). Although there exist some serious disagreements about how to best conceptualize or operationalize "ideology," or about whether the term should even be used at all, there seems to be little question that the major empirical indicators of mass political sophistication-attitudinal consistency and issue votinghave risen sharply since the middle 1950s (see especially Nie et al., 1976). There is also considerable consensus in the literature about when the critical change occurred: at the time of the 1964 American

Abstract Numerous studies in the past decade have suggested that the American voter has become increasingly sophisticated, ideologically, and that this is due largely to the separate or combined impact of the growing educational attainments of the U.S. public and the greater salience of politics in the 1960s and 1970s. This study, however, hypothesizes that much of the change might be due to a simple methodological artifact--changes in question wording and format-and reports data from a recent national experiment which strongly supports such an interpretation. The generalizations derived from The American Voter may, therefore, not be quite as "time-bound" as the current literature would lead us to believe. George F. Bishop is senior Research Associate, Robert W. Oldendick is Research Associate, and Alfred J . Tuchfarber is Director, Behavioral Sciences Laboratory, University of Cincinnati. The research reported in this paper was supported by a grant from the National Science Foundation (SOC77-10109). The authors wish to thank Richard Dawson, Program Director for Political Science at NSF, for his encouraging support of this project. Public Opinion Quarterly 1978 by The Trustees of Columbia University 0033-362X/78/0042-00811$01.75 Published by Elsevier North-Holland, Inc.

GEORGE F. BISHOP ET AL.

national election. For it is exactly at this point that all the trend lines for attitude consistency, issue voting, and the like take a sudden upward leap not only in the aggregate but also among the well-educated and the not so well-educated, and among those interested in politics and those not at all interested. The explanation of all these trends, however, has become rather elusive. Some reseaarchers have tried to trace the growth of mass electoral sophistication to the salience of political issues in the 1960s (see, e.g., Bennett, 1973; Nie et al., 1976; Niemi and Weisberg, 1976:76-79), whereas others have emphasized the electorate's increasing educational attainments as a significance causal factor (cf. Bishop, 1976; Miller et al., 1976). But neither of these interpretations can adequately account for the sudden, steplike surge in ideological sophistication during the 1964 election period. For one thing, there is no direct evidence that the electorate became any more interested in politics at this time than at any previous period in the past two decades (see, e.g., Converse, 1972:332-33). Furthermore, we know that educational attainment in the mass public climbed only gradually in the decisive phase, 1960-1964 (see Executive Office of the President, 1973:75-110) and could thus explain only a small part of the total increase. What then accounts for the abrupt shift in the structure of American politial behavior? We have reason to believe that a much more fundamental factor is responsible, one which inflated political attitude consistency (and issue voting) in all segments of the population: changes in question wording and format. For we now know that at the pivotal point of the 1964 election the Michigan Survey Research Center made some major modifications in the wording and format of the basic issue questions used by numerous researchers to measure the long-term trends in issue voting and attitudinal structure. Prior to 1964 almost all of the domestic and foreign policy questions in the SRC election studies appeared in a five-point, Likert format with response alternatives ranging from "strongly agree" to "strongly disagree"-including a middle category, "not sure; it depends." But in 1964 SRC changed the issue items to a more balanced, dichotomous-choice format. And in 1968 they introduced still another variation by expanding these dichotomous-choice questions into a seven-point, semantic differential type of format. An illustration of these changes will be helpful. Take, for example, the issue of the federal government's responsibility for employment (and living standards). From 1956 to 1960 SRC interviewers had asked respondents to what extent they agreed or disagreed-on a five-point scale-with the statement: "The government in Washington should see to it that everybody who wants to work can find a job." The interview-

QUESTION WORDING AND ATTITUDE CONSISTENCY

ers had also asked the following filter question immediately before reading each such statement: "Do you have an opinion on this or not?" In 1964, however, respondents received this new variation of the question and filter: In general, some people feel that the government in Washington should see to it that every person has a job and a good standard of living. Others think the government should just let each person get ahead on his own. Have you been interested enough in this to favor one side over the other? (If Yes) What is your feeling? Do you think the government should see to it that every person has a job and a good standard of living or should it let each person get ahead on his own? And beginning in 1972, this item reads: Some people feel that the government in Washington should see to it that every person has a job and a good standard of living. Others think the government should just let each person get ahead on his own. And, of course, other people have opinions somewhere in between. Suppose people who believe that the government in Washington should see to it that every person has ajob and a good standard living are at one end of this scale-at point number 1. And suppose that the people who believe that the government should let each person get ahead on his own are at the other end-at point number 7 . Where would you place yourself on this scale, or haven't you thought much about this? To some researchers these variations may seem trivial; others may argue that while such modifications could affect our estimates of the amount or direction of change in political attitude distributions over time (i.e., "the marginals"), they would not appreciably influence the degree of relationship between items, which represents the indicator most commonly used to operationalize consistency and issue voting. But recent experiments in question wording by Schuman and his associates (Schuman and Presser, 1977; Schuman and Duncan, 1944) demonstrate that this is simply not the case. Not only did variations in wording and format alter the marginals in their studies, but the changes also significantly affected the amount of association between sociodemographic and attitudinal variables. These findings thus suggested that a good part of the increase in mass political sophistication reported in the literature might be due to a methodological artifact. An opportunity to test this hypothesis arose when the National Opinion Research Center recently released data from a set of experiments designed, among other things, to evaluate the effects of the differences in wording and format that we have described for the SRC issue questions.

The authors wish to thank Professor Norman Nie and NOKC librarian Patrick Bova for releasing these data. Needless to say, neither Professor Nie nor the NOKC staff bear any responsibility for the analyses and interpretations presented here.

GEORGE F. BISHOP ET AL.

Method As part of a National Amalgam Survey in December of 1973 NORC interviewers administered the three different forms of the seven SRC issue questions to randomized subsamples of about 500 respondents each: (1) an X-form which presented the items in a pre-1964, agreedisagree format, (2) a Y-form which put them in the 1964, dichotomous-choice format, and (3) a Z-form which arranged them into a seven-point format of the kind used by SRC, beginning with the 1968 election (see the Appendix). For purposes of comparability with previous analysis by Nie and his colleagues, which represents the currently dominant theoretical perspective on American electoral behavior, we computed gamma coefficients between all possible pairs of issues within each of the experimental subsamples, using their exact guidelines for recoding the five-point, Likert items into three "statistically comparable" categories: liberal, centrist, and conservative. These guidelines were: "(1) to make as even as possible the proportions of the population in each of the three categories, while (2) not permitting the first guideline to place respondents on the agree and disagree side of an issue in the same category (Nie and Andersen, 1974546-47)." We also applied their guidelines for recoding the seven-point items into three comparable categories (see Nie et a]., 1976:366-69). Finally, we replicated their procedure of using the volunteered response- "other, dependsw-as a middle category for the 1964, dichotomous-choice format. Each of the gamma coefficients reported in Table 1 below is therefore based on a 3 x 3 crosstabulation. We also want to re-emphasize that the differences described in Tables 1 and 2 below are based on randomized subsamples all interviewed in 1973.

Results and Discussions Looking first at just the gamma coefficients for the 1956 and 1964 formats (Table 1) we find two relatively distinct clusters of differences: (a) issue pairs involving questions about either government power or the cold war (Group B) and (b) those not including either of these items (Group A). In the latter cluster it is clear that the coefficients computed from the 1964 formats are generally higher in magnitude. The most telling example, and the one which produced the most conspicuous differences, is the correlation between attitudes toward black welfare (i.e., fair treatment in jobs and housing) and racial integration of public schools. Using the 1956 format, the coefficient is only -.25 compared to an astounding .77 with the 1964 format. What we find so impressive

QUESTION WORDING AND ATTITUDE CONSISTENCY

Table 1. Gamma Coefficients for Issue Pairs In Three Randomized Subsamples Receiving Different Question Wording Formats

Question Wording Format Issue Pair

Group A

Economic welfare Economic welfare Economic welfare Economic welfare School aid School aid School aid Medical care Medical care Black welfare

Pre-1964

1964

1968-76

.14

.07

-Medical care

-School aid

-Black welfare

-School integration

-Medical care

-Black welfare

-School integration

-Black welfare

-School integration

Average Gammaa Group B

Economic welfare -Government Economic welfare -Cold war

Medical 'care -Government Medical care -Cold war

School aid -Government School aid -Cold war

Black welfare -Government Black welfare -Cold war

School integration -Government School integration -Cold war

Governmeiit power-Cold war

Average Gamma

power

.14

SOURCE:NORC Amalgam Survey 4179 (December, 1973). "The average gamma is based on the absolute values of the coefficients.

about this particular contrast is that the question in both formats concerned a very obvious common symbol: blacks (Negroes). It did not, therefore, require any unusually sophisticated act of ideological abstraction to behave consistently toward this relatively concrete attitude object (cf. Converse, 1964:241-42). Furthermore, the differences in wording between the two versions were not so glaring compared to some others--e.g., cold war and government power questions (see the Appendix). Yet the gap in the magnitude of association generated by the two forms can only be described as massive. The reader may also remember that it was this correlation between black welfare and school integration which increased so dramatically in the 1964 election data reported by Nie and Andersen (1974:548), exceeding all the other issue correlations by substantial amounts. Given what we now know about

GEORGE F. BISHOP ET AL.

the greater magnitudes of association created by the 1964 format, we can see how such artifacts might have inflated the averaged gamma coefficients which figured so promimently in their interpretations of the SRC data. When we study the coefficients in Group B, however, we notice hardly any discrepancies and even a few apparent reversals (see Table 1). One possible explanation for the lack of differentiation on these items may be that they have lost much of their relevance as political issues (see Bennett and Oldendick, 1977; Nie and Andersen, 1976:125-28); this would seem to fit with the overall pattern of low and negligible associations for these questions. In other words, the correlations for the cold war and government power measures are so close to zero in the 1973 population that there is not much opportunity for any dissimilarities to emerge. But there is another reason why we should not expect to find any coherent pattern of variation between the 1956 and 1964 versions of these issues; and that is, they are so discontinuous in wording-despite claims to the contrary in the literature-that they are probably measuring relatively distinct kinds of attitudes (see the Appendix, Forms X and Y , questions 6 and 7). Consider the issue of government power. Both the 1956 and the 1964 questions on this issue involve a common symbol-"the government"-but the later version, which talks about the government "getting too powerful," encompasses a much broader range, cutting across a variety of specific issues that might occasion the use of federal authority (see the 1956 version in the Appendix: Form X, question 6 and the 1964 version: Form Y, question 6). While one would expect responses to these two versions to correlate, if administered concurrently or on separate occasions to the same individuals, the magnitude of association between them would not necessarily be high or even moderately so. For example, some respondents might believe that, in general, the government in Washington is getting too powerful, but that in particularly critical economic areas such as employment, the housing industry, utilites, and energy conservation, the government should play a much stronger role; and in certain other domains--e.g., racial composition of public schools, sexual behavior, and the use of marijuana-the government should stay out entirely. Similar constraints apply to the questions concerning the cold war. Although both the 1956 and 1944 variations appear to tap an underlying dimension of feelings toward "communism" or "csmmunists," they each focus on rather different, though not necessarily contradictory, policy options. The pre-1964 form of the question centers on the desirability of military support for anti-communist countries, whereas the 1964 version revolves around the more general matter of negotia-

QUESTION WORDING AND ATTITUDE CONSISTENCY

tion and conflict resolution. Clearly, many people could "consistently" support both policies: keeping troops overseas to bolster the military defense of anti-communist nations and yet, at the same time, favor a policy of "negotiation" rather than "confrontation," as it is often referred to today. To summarize, we believe the absence of a coherent pattern of differences between issue pairs involving the 1956 and 1964 versions of the cold war and government power questions stems from two factors: their overall decline in discriminative power for the contemporary electorate and their extraordinary noncomparability . We have yet to discuss the correlations computed from the recoded seven-point scales (see Table 1, column 3). To say the least, they present us with a mixed picture. In some instances they either roughly match o r exceed the magnitudes of the corresponding coefficients for the pre-1964 and 1964 formats (e.g., medical care-cold war, economic welfare-school aid); in other instances they tend to be substantially lower (e.g., school aid-medical care, medical care-black welfare); and in still others they fall somewhere between (e.g., school aid-school integration, black welfare~schoolintegration). One source of extraneous variation which might account for some of the bewildering patterns we have uncovered in Table 1 could be differences in the amount of filtering generated by the three question formats, two of which involved a distinct kind of filter question (pre-1964 and 1964 versions) and one of which (1968-1976 form) used no filter at all (see the Appendix). While this variable was not subjected to systematic manipulation and control in the NORC experiment, we can get some clues about how it might have operated by scrutinizing the marginals for the filtered responses along with the traditional indicator of nonattitudes: the "don't know" responses (see Table 2). The discrepancies in the magnitude of filtering among the three formats are so large that we must wonder whether they are not at least as important as, if not more important than, the variations in the wording of the questions themselves. At one extreme we have hardly any screening out of individuals who lack opinions, with the exception of a handful of volunteered "don't knows"; at the other end we find the 1964 filter eliminating large numbers of respondents-in one instance, almost a third of the entire subsample (the government power issue); and we discover yet another distribution for the pre-1964 filter question that is intermediate to the other two. It is tempting to attribute most of the variance in Table 1 to that in Table 2. For we know from recent studies (Francis and Busch, 1975; Sudman and Bradburn, 1974) that individuals giving don't know answers and other forms of nonsubstantive responses to such attitude questions tend to be significantly less

GEORGE F. BISHOP ET AL

Table 2. Percentage Distributions of Don't Know, No Opinion and No Interest Response to 7 Issue Questions in 3 Randomized Subsamples

Receiving Different Question Wording Formats

Question Wording Format (N = 495)

(N = 501) Issue

Economic welfare School aid Medical care Black welfare School integration Government power Cold war

Don't Know

No Opinion

Don't Know

No Interest

Don't Know

2.4 2.0 1.4 2.4 1.6 5.6 4.0

6.8 7.2 4.0 9.4 8.8 16.6 8.2

0.2 0.6 0.4 0.2 0.0 0.2 0.8

27.7 23.4 14.9 24.8 29.7 31.3 18.0

1.6 1.2 1.2 1.2 2.0 3.0 1.6

NZ Amalgam Survey 4179

SOURCE:

( N = 493)

(December, 1973).

educated and less involved in politics than their more opinionated counterparts. And since it is apparent that the 1964 filter screened out many more of these kinds of respondents than the other forms, the higher inter-item correlations observed for this format in Table 1 could be the result of massive self-selection by more politically sophisticated respondents. But while differences in filtering may account for some significant portion of the variation in correlations produced by the three formats, our experiments with this variable indicate that there is still a good deal of residual variance to be accounted for. Some of this may be due to inherent psychometric properties of the various question forms (e.g., greater reliability and validity of dichotomous-choice items), and some may stern from interactions between respondent attributes (e.g., acquiescence response style) and item characteristics (e.g., the agreedisagree format of the pre-1964 issue questions). It also means that further experiments are needed to disentangle the effects of question wording, format, and filtering. But just the evidence we have uncovered here should suffice to convince even the most skeptical reader that there are methodological artifacts of disturbing proportions in the Michigan Electoral Series, artifacts which probably account for much of the change in mass political sophistication that began with the 1964 national election. And if we accept this conclusion, it requires nothing less than a radical revision s f the new theoretical dogma about the emergent ideology of the contemporary American electorate, an electorate which may, in fact, be no more sophisticated than its counterpart

QUESTION WORDING AND ATTITUDE CONSISTENCY

of the 1950s. T h u s the generalizations derived from The Anzerican Voter (Campbell e t al., 1960) may be less "time-bound" than w e thought they were.

References Axelrod, Robert 1967 "The structure of public opinion on policy issues." Public Opinion Quarterly 3 1:49-60. Bennett, Stephen E. 1973 "Consistency among the public's social welfare policy attitudes in the 1960's." American Journal of Political Science 17:544-70. Bennett, Stephen E., and Robert W. Oldendick 1977 "The power of the federal government: the case of the changing issue." A paper presented at the annual conference of the Midwest Political Science Association, Chicago, Illinois, April. Bishop, George F. 1976 "The effect of education on ideological consistency." Public Opinion Quarterly 40:337-48. Campbell, Angus, Philip E. Converse, Warren E. Miller, and Donald E. Stokes 1960 The American Voter. New York: Wiley. Converse, Philip E. 1964 "The nature of belief systems in mass publics." Pp. 206-261 in D. E. Apter (ed.), Ideology and Discontent. New York: Free Press. 1972 "Change in the American electorate." Pp. 263-337 in A. Campbell and P. E. Converse (eds.), The Human Meaning of Social Change. New York: Russell Sage. 1975 "Public Opinion and Voting Behavior." Pp. 75-169 in F. I. Greenstein and N. W. Polsby (eds.), Handbook of Political Science, Vol. 4: Nongovernmental Politics. Reading, Mass.: Addison-Wesley. Executive Office of the President: Office of Management and Budget 1973 Social Indicators 1973. Washington, D. C.: U . S. Government Printing Office. Field, John O., and Ronald E. Anderson 1969 "Ideology in the public's conceptualization of the 1964 election." Public Opinion Quarterly 33:380-98. Francis, Joe D., and Lawrence Busch 1975 "What we now know about 'I don't knows'." Public Opinion Quarterly 39:207-18. Litwak, Eugene, Nancy Hooyman, and Donald Warren 1973 "Ideological complexity and middle-American rationality." Public Opinion Quarterly 37:317-32. Luttbeg, Norman R. 1968 "The structure of beliefs among leaders and the public." Public Opinion Quarterly 32:398-409. Miller, Arthur H., and Warren E. Miller 1976 "Ideology in the 1972 election: myth or reality-a rejoinder." American Political Science Review 70:832-49. Miller, Arthur H., Warren E. Miller, Alden S. Raine, and Thad A. Brown 1976 "A majority party in disarray: policy polarization in the 1972 election." American Political Science Review 70:753-78.

GEORGE F. BISHOP ET AL.

Nie, Norman H., and Kristi Andersen 1974 "Mass belief systems revisited: political change and attitude structure. " Journal of Politics 36540-9 1. Nie, Norman H., Sidney Verba, and John R. Petrocik 1976 The Changing American Voter. Cambridge: Harvard University Press. Niemi, Richard G., and Herbert F. Weisberg 1976 Controversies in American Voting Behavior. San Francisco: W. H. Freeman. Pierce, John C. 1975 "The relationship between linkage salience and linkage organization in mass belief systems." Public Opinion Quarterly 34:102-110. Robinson, John P., and John D. Holm 1976 "The role of ideology and party identification in the 1972 election." A paper presented at the Second Annual Conference of the Midwest Association for Public Opinion Research, Chicago, Illinois, November. Schuman, Howard, and Otis D. Duncan 1974 "Questions about attitude survey questions." Pp. 232-51 in H. L. Costner (ed.), Sociological Methodology 1973- 1974. San Francisco: Jossey-Bass. Schuman, Howard, and Stanley Presser 1977 "Question wording as an independent variable in survey analysis." Sociological Methods and Research 6: 151-70. Sudman, Seymour, and Norman M. Bradburn 1974 Response Effects in Surveys. Chicago: Aldine. Weissberg, Robert 1976 "Consensual attitudes and attitude structure." Public Opinion Quarterly 40:349-59.

Appendix Issue Questions in Three Formats

No opinion

Agree strongly

Agree but not very strongly

Not sure, it depends

Disagree but not very strongly

Disagree Don't strongly know

1. T he government in Washington ought to see to it that everybody who wants to work can find a job. Do you have an opinion on this or not? 2. If cities and towns around the country need help to build more schools, the government in Washington ought to give them the money they need. Do you have an opinion on this or not? 3. The government ought to help people get doctors and hospital care at low cost. Do you have an opinion on this or not? 4. If (Blacks!Negroes) ure not getting fair treatment in jobs and housing, the government ought to see to it that they do. Do you have an opinion on this or not?

QUESTION WORDING AND ATTITUDE CONSISTENCY

5. The government in Washington ought to stay out of the question of whether white and (BlackiNegro) children go to the same school. Do you have an opinion on this or not? 6. The government should leave things like electric power and housing for private businessmen to handle. Do you have an opinion on this or not? 7. The U.S. should keep soldiers overseas where they can help countries that are against Communism. Do you have an opinion on this or not? FORM Y

1. In general, some people feel that the government in Washington should see to it that every person has a job and a good standard of living. Others think the government should just let each person get ahead on his own. Have you been interested enough in this to favor one side over the other? (If Yes) What is your feeling; do you think: the government should see to it that each person has a job and a good standard of living, or do you think the government should let each person get ahead on his own? 2. Some people think that the government in Washington should help towns and cities provide education for grade and high school children; others think that this should be handled by the states and local communities. Have you been interested in this enough to favor one side over the other? (If Yes) Which are you in favor of: getting help from the government in Washington, or handling it at the state and local level? 3 . S ome say the government in Washington ought to help people get doctors and hospital care at low cost, others say that the government should not get into this. Have you been interested enough to favor one side over the other? (If Yes) What is your position; should the government in Washington: help people get doctors and hospital care at low cost, or stay out of this? 4. Some people feel that if (BlacksINegroes) are not getting fair treatment in jobs, the government in Washington ought to see to it that they do. Others feel that this is not the federal government's business. Have you had enough interest in this question to favor one side over the other? (If Yes) What is your feeling; do you think: the government in Washington should see to it that (BlackINegroes) get fair treatment in jobs, or that this is not the federal government's business? 5. Some people say that the government in Washington should see to it that white and (BlacWNegro) children are allowed to go to the same schools. Others claim that this is not the government's business. Have you been concerned enough about this question to favor one side over the other? (If Yes) What is your feeling; do you think: the government in Washington should see to it that white and (BlacWNegro) children go to the same schools, or do you think that the government should stay out of this area as it is not its business? 6. Some people are afraid the government in Washington is getting too powerful for the good of the country and the individual person. Others feel that the government in Washington is not getting too strong for the good of the country. Have you been interested in this enough to favor one side over the other? (If Yes) What is your feeling; do you think: the government is getting too powerful, or do you think that the government has not gotten too strong?

GEORGE F. BISHOP ET AL.

7. Some people think that the United States should keep soldiers overseas so that they can help countries protect themselves against Communism. Others feel that these countries should take care of themselves and that we should not send troops abroad. Have you been interested in this issue enough to favor one side over the other? (If Yes) What is your feeling; d o you think: the U.S. should keep soldiers overseas to help countries protect themselves against Communism, o r d o you think that these countries should take care of themselves and we should not send troops abroad? FORM Z

1. Some people feel that the government in Washington should see it to that every person has a job and a good standard of living. They would be at point 1 on the card. Others think the government should just let each person get ahead on his own. They would be at point 7 on the card. Others have opinions somewhere in between. Where d o you place yourself? 2. Some people think the government in Washington ought to help people get doctors and hospital care at low cost. They would be at point 1 . Others say that the government should not concern itself with this matter. They would be at point 7. Where would you place yourself? 3. Some say the government in Washington ought to help towns and cities provide education for grade and high school children. They would be at point 1. Others think that this is a matter that should be handled by the states and local communities. They would be at point 7. Where would you place yourself? 4. Some people think that the recent attempts to improve conditions for Blacks in America should be speeded up. They would be at point 1. Others think that these efforts should be slowed down; they would be at point 7. Where would you place yourself? 5. Some people say that the government in Washington should see to it that white and (BlacWNegro) go to the same schools. They would be at point 1. Others claim that this is none of the government's business. They would be at point 7. Where would you place yourself? 6. Some people are afraid the government in Washington is getting too powerful for the good of the country and the individual person. They would be at point 1. Others feel that the government in Washington is not getting too strong for the good of the country. They would be at point 7. Where would you place yourself? 7. Some people think that the United States should keep soldiers overseas whenever they can help countries protect themselves against Communism. They would be at point 1. Others feel that these countries should take care of themselves and that we should not send troops abroad. They would be at point 7. Where would you place yourself?