Effects of Question Order on Survey Responses SAM G. McFARLAND
THE impact of the order of questions upon survey responses has had only limited attention. Recent texts on survey methods have either ignored the issue altogether (Smith et al., 1976; Weisburg and Bowen, 1977) or have merely given unsupported, common sense instructions for the ordering of questions (Backstrom and Hursh, 1963; Miller, 1970; Moser and Kelton, 1974; Thomas et al., 1975; Kornhauser and Sheatsley, 1976; Dillman, 1978; Bradburn and Sudman, 1979). Systematic research on question order effects has been infrequent and has produced mixed results. Sayer (1939) showed that attitudes toward radio advertising were more positive when the key question followed either a question on the willingness t o pay for advertisement-free radio or a series of questions about specific consumer items. Cantril (1944) found that question order affected the willingness of the public to allow enlistment in foreign armies. Respondents to the 1976 National Opinion Research Center survey were Abstract Two question orders were used in a Kentucky statewide random survey. In nonoverlapping sections, four questions which asked the respondents to express their general interests in politics and religion and their general evaluations of the gravity of the economic and energy situations either preceded or followed series of specific questions on the same issues. The respondents expressed significantly greater interest in politics and religion when these general questions followed the specific questions, but evaluations of the economic and energy crises were not significantly affected by question placement. Implications for the arrangement of questions within surveys are discussed. Sam G. McFarland is a Professor in the Department of Psychology at Western Kentucky University. This study was conducted as a part of a regular semiannual survey by the Kentucky Center for Survey Research and Services. The author wishes to thank Tom Madren and John Peterson, the codirectors of the center, for facilitating this research, and Howard Schuman and Fred Heilizer, for their comments on an earlier draft of this paper. Public Opin~onQuarterly Vol. 45:208-215 @ 1981 by The Trustees of Columbia University Published by Elsevier North-Holland, Inc. 0033-362Xi6110045-208/$2.50
EFFECTS OF QUESTION ORDER
209
more likely to view the federal tax burden as "about right" rather than "too high" when the question followed the evaluations of federal spending in 11 specific areas (cited in Turner and Krauss, 1978). Other studies sponsored by NORC have failed to show the predicted order effects (Sudman and Bradburn, 1974). Bradburn and Mason (1974) proposed four specific types of possible order effects due to saliency, redundancy, consistency, and fatigue, but their own research found no order effects. Delamater and MacCorquodale (1975) found that responses to a series of questions on the respondents' sexual behaviors were not substantially affected by the placement of the series within a longer questionniare. Turner and Krauss (1978) have argued, however, that many of the significant differences in public confidence in national leaders found by different surveys are best explained by the differing contexts in which the questions appeared within the surveys. One common recommendation for survey construction is that questions which ask for a general evaluation on a particular issue should precede questions about its more specific aspects (cf. Kornhauser and Sheatsley, 1976). This recommendation is apparently based on the untested assumption that the specific questions create a saliency or a specific set which might affect answers to the more general questions, whereas the general questions are less likely to influence responses on more specific subissues. This assumption served as the principal hypothesis for the present study. Because of the increased saliency of the issues, respondents were expected to express greater interest in religion and politics and greater concern about the economic and energy crises when a general question on each of these issues followed rather than preceded a series of special questions on the same topic. The magnitude of these order effects, if they occur, may also vary as a function of the demographics of the respondents. For example, respondents with little education may be more susceptible to order effects than are the more educated. As a consequence, the arrangement of questions in a survey may be more or less critical depending upon the particular population under study. Without prior hypotheses, the variations in the strength of the order effects were examined as a function of the respondents' sex and education. The effects of income were not examined because of its correlation with education, r = .41, and race was omitted because the small sample of nonwhites was not sufficient for partitioning. Duncan and Schuman (1977) have suggested that changes in question order might alter the patterns of relationship between the responses given to questions, as well as altering the frequencies of
SAM G . McFARLAND
210
particular answers. In the present study, for example, the strengths of the correlations between answers to the general and specific questions on particular topics might be either enhanced or diminished by placing the specific questions first. With no prior logical or empirical basis for predicting which, if either, would occur, the variations in the strength of the relationships between the specific and general questions as a function of question order were examined without advance hypotheses.
Method Five hundred and sixteen respondents were interviewed by telephone between January 10 and 21, 1980. The survey used a modified random direct dialing technique to insure a random survey of all Kentucky households with telephones. In four nonoverlapping sections, respondents were asked series of questions on their attitudes toward energy, the economy, politics, and religion. Each section contained a question which assessed the respondent's general attitude toward the area as a whole and a series of specific questions related to particular facets of the topic. For example, the general question on energy asked, "How would you describe the current energy problem in the United States: as extremely serious, somewhat serious, or not serious at all?' Respondents were also asked their specific attitudes toward nuclear energy, the causes of the recent gasoline shortage, strip mining regulations which might increase coal costs, and the windfall profits tax on oil companies. The three other general questions were (1) "During the next year, do you think that the economy will get better, get worse, or stay about the same?" (2) "In general, how interested would you say you are in politics: very interested, somewhat interested, or not very interested?'and (3) "How interested would you say you are in religion: very interested, somewhat interested, or not very interested?' The identical survey was administered in two forms: the general attitude question on each issue either preceded or followed the series of specific questions. The number of specific questions ranged from three on the economy to five on religion and energy. With the exception of the energy and economy sections, which appeared in immediate succession, the relevant sections were separated by at least five questions on other topics. Each form was administered to one-half of the sample, alternating with each phone call. The interviewers were not informed of the purpose of the alternation. The responses to all questions were analyzed by chi squares. The variations in responses due to question order, education, and sex
EFFECTS OF QUESTION ORDER
211
were partitioned by procedures outlined by Winer (1962:629-32). The strengths of the relationships between responses to the general and specific questions were either examined by partitioned chi squares or by tests for significant differences between product-moment correlations. The former method was used when responses to either the general or specific question were clearly nominal or ordinal data (e.g., What is your religious preference: Protestant, Catholic, Jewish, or some other religion?). The latter method was used when responses to both the general and the specific questions appeared to generate approximations to interval data (e.g., "About how often do you pray: several times a day, once a day, a few times a week, once a week or less, or never?').
Results As seen in Table 1, the respondents expressed significantly more interest in politics and religion when these questions followed the Table 1. Impact of Question Order upon Responses to General Questions of Interest and Evaluation Question Order
General Question
I . How would you describe the current energy problem in the United States? a. Extremely serious b. Somewhat serious c. Not serious at all 2. During the next year, do you think the
economy a. Will get better b. Will get worse c. Stay the same 3. In general, how interested would you say
you are in politics: a. Very interested b. Somewhat interested c. Not very interested 4. In general, how interested would you say you are in religion: a. Very interested b. Somewhat interested c. Not very interested
Generul Specij?c Question Questions First First % % Sinnificance
212
SAM G . McFARLAND
series of specific questions on the same issues. Their evaluations of the gravity of the economic and energy crises were not significantly affected by question order. The trend on the energy question was in the expected direction, but did not approach significance, x2(1) = 2.38, p > .25. No evidence was found to indicate that the strength of the order effects varied with sex andlor education. None of the interactions between education, sex, and question order were significant for any of the four general questions. As an example of the consistency of order effects across education levels, Table 2 presents the responses to the general question on political interest as a function of education and question order. The partitioned chi square showed that responses varied as a function of education x2(2) = 8.81, p < .025, as well as by question order, but the interaction between education and question order did not approach significance, x2(2) = 1.17, p > .25. This consistency is similar for all the general questions. The responses to the 17 specific questions also were not significantly affected by the question order. None of the partitioned main effects for question order on the chi squares were significant at probability levels of less than .25. The partitioned interactions between question order, education, and sex also showed no evidence of any order effects for any subgroups of respondents. In general, the question order had little effect upon the strengths of the relationships between the general and the specific questions. Only two of the 17 relationships were significantly different at probability levels of .05 or less: The correlation between the general question on interest in religion and the specific question "About how often do you pray . . . ? w a s r = .51 ( N = 248) when the general question came first and r = .64 ( N = 241) when the specific questions were first. Table 2. Responses to the Question, "In general, how interested would you say you are in politics . . ," as a Function of Education and Question Order
.
Education High School or Less
More Than High School
General Specific Question Questions First First % % Combined
General Specijic Question Questions First First % % Combined
A . Very interested 22 39 30 B . Somewhat interested 55 47 52 C . Not interested 22 14 18 at all ( N = 1 6 1 ) ( N = 146) ( N = 3 0 7 )
37 46
46 48
42 47
17 7 11 ( N = 9 1 ) ( N = 103) ( N = 166)
213
EFFECTS OF QUESTION ORDER
This difference was significant, p < .02. Similarly, the correlations between the general question on interest in religion and the specific question "Did you, yourself, happen to attend church in the last seven days?'were r = .33 (N = 248) and r = .49 (N = 296) with the general question first and last, respectively. This difference was also significant, p < .02.
Discussion While question order effects are not ubiquitous, these findings indicate that the question order should be carefully planned in the construction of every survey. Furthermore, since the effects of question order are consistent for both sexes and across education levels, question order effects are as much a concern when the survey is on a restricted population (e.g., college faculty, male plant managers) as when it covers the general population. How can order effects be controlled? Perrault (1976) has suggested the creation of several random question orders with computer assistance. The present study suggests a different approach. While we cannot know with certainty whether the "true" responses are those which precede or those which follow the specific questions, logic suggests that unprimed responses are probably closer to the respondents' real feelings. In light of the present results, the common recommendation that general questions should precede questions on more specific content appears justified. In addition to the practical difficulties, several random arrangements of questions appear less likely to accurately represent the population since the saliency of an issue might have been enhanced for some respondents but not for others. In retrospect, the general questions on politics and religion were probably more susceptible to order effects than those on energy and the economy. The former asked only for the respondents' interest while the latter asked for discriminating judgments. Interest appears to be more easily influenced by question order than is evaluation. The responses requiring evaluation on the very specific issues were not affected by the order manipulations, although it must be noted that these manipulations involved only the presence or absence of a single general question on each issue. Nevertheless, these results indicate that questions may vary their susceptibility to order effects. They suggest, as general hypotheses, that the more specific the content of a question and the more concrete the required response, the less susceptible the question is to order effects. Conversely, the more diffuse the question's content and the more
SAM G . McFARLAND
214
vague the response required, the greater the possibility of these effects. These results also suggest, in a post hoc fashion, particular circumstances in which question order variations might affect the strength of the relationship between responses to general and specific questions. In both instances where asking the specific question first strengthened the relationship in the present study, the specific questions were actually behavioral indices of the general question: How often one prays and whether or not one has attended church in the past seven days are objective indications of one's religious interest. Asking these questions first apparently gave the respondents concrete, behavioral references for answering the general question. Thus it is not surprising that the relationships were strengthened. In contrast, responses to most other specific questions were not behavioral indices of the general question. The specific question "How about the Republican Party: Do you think it will get stronger over the next ten years, or will it get weaker?'increased the saliency of politics in general and with the other specific questions enhanced the respondents' apparent interest in politics. Since the respondents' answers to this question were not behavioral indices of their own political interest, however, asking this specific question first did not strengthen its relationship to the general question. The difference suggests an additional general hypothesis: When the respondents' answers to one question are a behavioral index of another question, asking the former question first will enhance the strength of the relationship between the responses to the two questions. These new hypotheses await additional testing.
References Backstrom, C. H., and G. D. Hursh 1963 Survey Research. Chicago: Northwestern. Bradburn, N. M., and W. M. Mason 1964 "The effect of question order on responses." Journal of Marketing Research 157-64. Bradburn, N. M., and S. Sudman 1979 Improving Interview Method and Questionnaire Research. San Francisco: Josey Bass. Cantril, H. 1944 Gauging Public Opinion. Princeton: Princeton University Press. Delamater, J., and P. MacCorquodale 1975 "The effects of interview schedule variations on reported sexual behavior." Socological Methods and Research 4:215-36. Dillman, D. A. 1978 Mail and Telephone Surveys: The Total Design Approach. New York: Wiley.
EFFECTS OF QUESTION ORDER
215
Duncan, O., and H. Schuman 1977 "An experiment on order and wording of questions." Unpublished manuscript, Department of Sociology, University of Arizona. Kornhauser, A., and P. Sheatsley 1976 "Questionnaire construction and interview procedure." In C. Sellitz, L. Wrightsman, and S. Cook. Research Methods in Social Relations. New York: Holt, Rinehart & Winston. Miller, D. 1970 Handbook of Research Design and Social Measurement. New York: Basic Books. Moser, C. A., and G. Kelton 1974 Methods in Social Investigation. New York: Basic Books. Perrault, W. D. 1975 "Controlling order-effect bias." Public Opinion Quarterly 35:544-51. Sayer, J. 1939 "A comparison of three indices of attitude toward radio advertising." Journal of Applied Psychology 23:23-33. Smith, D. L., K. F. Johnson, D. W. Paulson, and F. Socket 1976 Political Research Methods: Foundations and Techniques. Boston: Houghton Mifflin. Sudman, S., and N. Bradburn 1974 Response Effects in Surveys. Chicago: Aldine. Thomas, J. C., S. C. Smith, and J. J. Hall 1975 "Polling and survey research." Arlington, Va., National School Public Relations Associations. Turner, C. F., and E. Krauss 1978 "Fallible indicators of the subjective state of the nation." American Psychologist 33:456-70. Weisberg, H. F., and B. D. Bowen 1977 An Introduction to Survey Research and Data Analysis. San Francisco: W. H. Freeman. Winer, B.J. 1962 Statistical Principles in Experimental Design. New York: McGrawHill.
http://www.jstor.org
LINKED CITATIONS - Page 1 of 1 -
You have printed the following article: Effects of Question Order on Survey Responses Sam G. McFarland The Public Opinion Quarterly, Vol. 45, No. 2. (Summer, 1981), pp. 208-215. Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198122%2945%3A2%3C208%3AEOQOOS%3E2.0.CO%3B2-2
This article references the following linked citations. If you are trying to access articles from an off-campus location, you may be required to first logon via your library web site to access JSTOR. Please visit your library's website or contact a librarian to learn about options for remote access to JSTOR.
References The Effect of Question Order on Responses Norman M. Bradburn; William M. Mason Journal of Marketing Research, Vol. 1, No. 4. (Nov., 1964), pp. 57-61. Stable URL: http://links.jstor.org/sici?sici=0022-2437%28196411%291%3A4%3C57%3ATEOQOO%3E2.0.CO%3B2-5