Context Effects on Survey Responses

Context Effects

on Survey Responses

to Questions About Abortion

HOWARD SCHUMAN,

STANLEY PRESSER, A N D JACOB LUDWIG

ALTHOUGHcontext effects in survey data are always a possible problem, they pose a particularly serious threat when surveys are used to study social change. In such research, a response difference due to questionnaire context is quite likely to be interpreted as a reflection of true change. Yet is is difficult to hold context constant from one survey to another: questions are almost always repeated on a selective basis, and it is rare for an earlier question order to be maintained perfectly. Even a vehicle like the National Opinion Research Center's General Social Survey (GSS), designed in part to measure change, rotates items in and out of the survey for practical reasons, which in turn alters context. Thus, it is important that we Abstract The proportion of favorable responses to h general question about abortion was significantly smaller in an NORC survey than were responses to the same item in an SRC survey. We hypothesized that questionnaire context was the main source of the difference-in one survey the general item followed a more specific question about abortion-and carried out two split-ballot question order experiments that provide strong support for the hypothesis. The context effect seems to arise from the generality of the abortion question; the specific question itself showed no change due to context.

Furthermore, there is some evidence that ambivalence toward the issue of abortion is a factor in the sensitivity of the general question to context. The context effect in these experiments is also notable for producing contrast rather than consistency between responses to adjacent questions on the same issue. Implications of the findings for the study of social change are emphasized.

Howard Schuman is a Professor of Sociology and Program Director, Institute for Social Research, University of Michigan. Stanley Presser is a Research Associate in the Institute for Research in Social Science and Visiting Assistant Professor, Department of Sociology, University of North Carolina. Jacob Ludwig is a doctoral student in sociology at the University of Michigan. This article is based on research supported by the National Science Foundation under Grant No. SOC-7804021.

Publlc Opinion Quarterly Vol. 45:216-223 Published by Elsevier North-Holland, Inc.

1981 by The Trustees of Columbia Univers~ty 0033-362X18110045-2161$2.50

CONTEXT EFFECTS ON SURVEY RESPONSES

217

understand the nature of question order effects, identifying so far as possible the kinds of items and contexts that invite such effects, as well as their size, direction, and frequency.

Fortunately there has been renewed interest in this problem, prompted in part by a concern for social indicators (see Turner and Krauss, 1978, and Smith, 1979), and about a quarter of all the available experimental reports on context have appeared in the past five years (e.g., Gibson et al., 1978; Kalton et al., 1978; Duncan and Schuman, 1980; McFarland, 1981; Sigelman, 1981). Not all experiments on context show effects (e.g., Bradburn and Mason, 1964), and no doubt there is a bias toward publishing only positive results, but enough effects have turned up to make it clear that they must be reckoned with. (See Schuman and Presser, 1981, for a review of this literature.)

We discuss here a context effect that, in addition to being rather sizable and to having been replicated twice, is important for several reasons: 1. The effect was discovered accidentally in the course of a typical attempt to "borrow" an item from a major national survey (the GSS), rather than having been contrived for demonstration purposes.

Moreover, the focal question is about an important social issue (abortion) where studies of attitude change are of considerable interest and where it is crucial to separate true change from effects due to context.

2. Whereas the usual assumption about context effects is that they result from attempts by respondents to be consistent from one item to the next, our experiments indicate that contrast as well as consistency can be produced by contextual influence.

3 . The effect to be described is based on the ordering of two items, one general and one specific, but only responses to the general item are altered by context. This accords with other literature (noted below) suggesting that broad summary type questions are especially susceptible to contextual redefinition.

Data and Results In 1979 we needed for purposes not relevant to this report a relatively general question on abortion, and decided to borrow a suitable one from NORCs General Social Survey, thus allowing use of the GSS data for comparative purposes. Results with the abortion item in our March 1979 national telephone survey, however, differed by 18 percent from those reported by NORC in 1978, as shown in Table 1.

SCHUMAN, PRESSER, AND LUDWIG

218

Table 1. Discrepancy Between NORC and SRC Results on General Abortion Question " D o you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children?'

Yes No

a

NORC GSS-78

SRC 79-March

40.3% 58.4% 59.7 41.6 100 100 (1,483) (490) X 2=48.34. d f = 1 . D < .001

Adjusted SRCa 54.9% 45.1 100

SRC results standardized on NORC educational distribution.

This difference was far beyond any that could be accounted for by chance.

Since our telephone survey tended to underrepresent less educated persons relative to the GSS sample, and low education is associated with opposition to abortion, a small part of the difference could be explained in this way. Enough remained after standardization for this difference, however, to suggest a possible context effect, for the general abortion item was the only question we asked on that issue, whereas in the GSS the item was preceded by a much more specific question on abortion in the case of a defect in the unborn child. Of course, there also were other possible explanations, such as true change over time or a difference between telephone and face-to-face modes of interviewing, which needed to be ruled out.

We included both the general abortion item and the more specific defective-child item in a split-ballot question order experiment in June 1979. The questions were adjacent to each other, but with their order reversed on the two questionnaire forms. (The question immediately preceding the abortion items concerned labor unions, and no earlier items dealt in any way with abortion issues.) Table 2 presents the cross-tabulation of the two items separately by questionnaire form, so that both the marginal differences and the intercorrelations can be compared for the two orderings. Examination of the marginals at the bottom of the subtables shows that the general abortion item received substantially more support (by 13 percent) when asked first (which is equivalent to being asked alone) than when asked after the child-defect item. The latter item, on the other hand, was unaffected by its placement. Furthermore, almost all of the effect on the marginals of All our data come from 1979 Survey Research Center (SRC) national monthly telephone surveys using RDD sampling. The final Ars and response rates were, respectively, 534 and 73 percent in March, 651 and 74 percent in June, and 679 and 66 percent in August. The NORC sample is described in NORC (1980). Missing data do not appear to be influenced by question order, and are omitted from tables.

CONTEXT EFFECTS ON SURVEY RESPONSES

219

Table 2. Order Effects on Abortion Itemsa Order: SpecificIGeneral General Abortion Item Yes No

Abortion Specitic Item

Yes No

Order: GenerallSpeciJic General Abortion Item Yes No

: : I "1/ :1 ni f : :

48.1

51.9

1

100% (293)

M:; 60.7

39.4

1

100% (305)

Gamma = .90

Gamma = .8 1 General abortion marginals by order: X 2 = 9.52, df = 1, p < .O1 Specific abortion marginals by order: X 2 = 0.11, df = 1, n.s.

Three-way interaction (response to general item by response to specific item by order): X 2 = 0.99, df = 1, n.s.

" Carried out in SRC-79 June. Percentages are based on grand totals. The general item is the one shown in Table 1. The specific item was worded as follows: "Do you think it should be possible for a pregnant woman to obtain a legal abortion if there is a strong chance of serious defect in the baby?"

the general item is due to persons who said yes to the defective-child item, though a trend in the same direction occurs among those who said no to the defective-child item.

Since the specifictgeneral order increases the difference in margin a l ~between the two items over what it would be if the items were asked separately, the effect is one of contrast rather than of consistency. Note also that the effect produced experimentally accounts for much of the nonexperimental difference shown in Table 1, especially when the original SRC results are adjusted to take account of the NORC-SRC difference in educational distributions.

Because of the importance of the abortion results, we replicated the experiment in SRC-29 August, and included in the same survey an item to obtain religious preference and a question on ambivalence discussed below.2 The basic order effect appeared even more strongly this second time: agreement with the general abortion item is 17.4 percent higher when it comes before rather than after the defectivechild item (X2 = 19.66, df = 1, p < .001), whereas there is no significant order effect (X2 = 0.27) for the defective-child item itself.

(Intercorrelations are also quite similar to those in Table 2.) Thus the context effect on the general abortion item seems to be both large and highly reliable.

The exact interpretation of the effect is less clear than its reliability.

In the August 1979 survey, the question immediately preceding the abortion experiment concerned beliefs about the Soviet Union, and no earlier questions dealt with abortion.

220

SCHUMAN, PRESSER, AND LUDWIG

We call it a part-whole effect because the general abortion item contains the specific one in the sense that agreement to the general item ordinarily implies agreement to the specific item, but the reverse is not true. (Compare the bottom left and top right cells in either half of Table 2.) One plausible explanation for the effect turns on the fact that there are a number of different reasons for supporting legalized abortion. A possible defect in an unborn child is a specific reason that appeals to a large part of the population, as is shown by the marginals for that item. When the more general item is asked first, some respondents may say yes but mainly with such a specific reason in mind. When the item on abortion because of a defective child is asked first, however, this indicates to respondents that the general item which follows does not refer to that specific case. Thus respondents who are reluctant to favor abortion except within narrow limits should find it easier to oppose the general rationale after having favored (and "subtracted") the more specific rationale about the defective child.

A part-whole or subtraction process can be conceptualized as purely cognitive, a redefinition of what the general abortion item is about. It is also possible that an element of ambivalence is involved, so that a respondent who has agreed to the defective-child item feels freer to swing to the other side when faced with the general item.

These two hypotheses, although appealing, are not entirely consistent with the finding in Table 2 that a shift occurs not only among those saying yes to the defective-child item, but among those saying no as well. Although the latter trend is based on too few cases to be at all sure of its reliability, it occurs in the August replication as well.

Perceptual contrast offers still another line of interpretation, with order influencing the perceived gap between the two items (cf. the "Crespi phenomenon" noted in McGuire, 1969:213).

In order to determine whether ambivalence is involved in the order effect, we included at the end of the August 1979 questionnaire a question intended to tap ambivalence about abortion, as shown in Table 3. (In an attempt to avoid a further context effect, the original abortion items and the question on ambivalence were separated by over 50 questions on automobiles and the energy shortage.) For those who show ambivalence by saying that the abortion issue is one they find difficult to decide on, the order effect is quite large (28 percent) and highly reliable; for those who are very certain about the issue, only a nonsignificant trend (10 percent) toward an order effect appears. However, while the ambivalence item does condition the order effect, it may also be conceptualized as a measure of strength of opinion, rather than of ambivalence in the sense of conflicted attitudes, and thus may simply reflect a greater resistance to order effects

221

CONTEXT EFFECTS ON SURVEY RESPONSES

Table 3. Attitude on General Abortion Item by Question Order and Ambivalence Very Certain General Order Question on Abortiona GeneralISpecific SpecificIGeneral

Favor Oppose

68.1% 58.0% 42.0 31.9 100 100 (135) (150) XZ=3.15, df= 1, n.s.

Three-way interaction: X 2 = 4.42, df = 1, p < .O5

A Difficult Issue Order GeneralISpecific SpecificIGeneral

67.8% 39.9% 32.2 60.1 100 100 (174) (143) X 2 = 25.07, df = 1, p < .001

a Carried out in SRC-79 August. The question was worded as follows: "The last question is on a different subject. Some people are very certain about their feelings about when legal abortions should be permitted. Other people see this issue as a difficult one to reach a decision on. Would you say that you are more like those who are very certain, or that you are more like those who see this issue as a difficult one to reach a decision on?"

by those who feel strongly about an issue. The two interpretations are not easily di~tinguished.~ We also examined the abortion order effect by religion (Protestant vs. Catholic), sex, and education. Only religion shows a possible interaction, but it is not even borderline in significance (.lo < p < .20). For now, we conclude that both religious categories, both sexes, and three educational groupings are all susceptible to the order effect shown in Table 2.

Discussion Given a nonexperimental comparison, followed by two highly significant experimental results, the context effect for the abortion item appears to be quite reliable. It is also large, the average of the two experimental differences being 15 percent, certainly sufficient to lead to serious misinterpretation if the context difference occurred in a comparison of survey results from two different time points.

We know of only one other report of a similar effect due to context.

Kalton et al. (1978) asked respondents whether driving standards were getting lower for (a) people in general, and (b) for young drivers On the assumption that those who shift their responses to the general item would be especially ambivalent, we expected ambivalence to be higher for pro-abortion respondents when the general question came first (and included the shifters as pros) than when it came second. This does occur. Similarly, we expected ambivalence to be higher for anti-abortion respondents when the general item came second (and included the shifters as antis). Surprisingly, this does not happen. Complicating the interpretation is the fact that the ambivalence item itself, though placed more than 50 items after the main abortion questions, shows a borderline ( p < .lo) difference in response between forms: there is more ambivalence registered on the form with the general item coming first. Only replication can establish whether this difference points to a remarkable order effect that "jumps" across 50 items, or whether it is a result of chance.

222

SCHUMAN, PRESSER, AND LUDWIG

specifically. When the general question came second, respondents were significantly less apt to say standards had gone down than when it came first; there was no effect of order on the question about young drivers. This also can be described as a contrast effect: after having been asked about young drivers, respondents' views of drivers in general are moved further away (in the positive direction) from their views of young drivers. Furthermore, the Kalton et al. analysis yields a meaningful three-way interaction, since the order effect occurred only for respondents who were themselves classified as older (45 and over), and they are probably the people most likely to perceive young drivers as especially reckless. The authors also suggest a subtraction explanation for the order effect: after answering about young drivers, respondents redefine the general drivers item to exclude young drivers and therefore give a less negative response.

A second part-whole experiment reported by Kalton et al. (1978) manipulated an item about the noise of traffic in general and an item specifically about the noise of truck traffic. Again there is a significant order effect that seems to involve contrast, but this time it is only the specific item that is affected, contrary to the other part-whole experiments we have discussed. No obvious reason for this difference in results is apparent in the nature of the items, nor do Kalton et al. offer an explanation.

Our results and those reported by McFarland (1980) and Smith (1979) also suggest that context effects are especially likely when an investigator attempts to summarize a complex issue with a single item. Sensible as such an abstraction may seem to the investigator, it may not allow for either the ambivalence or the specific qualifications in the minds of respondents. As a result, the summary item takes on different meanings in different contexts as respondents attempt to interpret the general question.

The development and reporting of social indicators assumes that we can chart trends in attitudes, and beliefs in much the same way that trends in more factual data are studied over time. Possible context effects, which have often in the past been ignored in the single self-contained survey, pose a major challenge when analysis of differences between surveys is the goal. Moreover, it remains to be seen whether even the reasonable step of holding context constant will always allow unambiguous conclusions about attitude change, since context can conceivably interact not only with response, but with time as well.4 One such instance appears in Schuman and Presser (1981), where a question replicated over two points in time in one context yields different conclusions about the direction of change from the same question replicated in a different context.

CONTEXT EFFECTS ON SURVEY RESPONSES

223

References Bradburn, Norman M., and William M. Mason

1964 "The effect of question order on responses." Journal of Marketing

Research 157-61.

Duncan, Otis Dudley, and Howard Schuman

1980 "Effects of question wording and context: an experiment with religious indicators." Journal of the American Statistical Association 75:269-75.

Gibson, Christina O., Gary M. Shapiro, Linda R. Murphy, Gary J. Stanko 1978 "Interaction of survey questions as it relates to interviewerrespondent bias." 1978 Proceedings of the Section on Survey Research Methods. Washington, D.C.: American Statistical Association.

Kalton, Graham, Martin Collins, and Lindsay Brook 1978 "Experiments in wording opinion questions." The Journal of the Royal Statistical Society Series C 27:149-61.

McFarland, Sam G. 1981 "Effects of question order on survey responses." Public Opinion Quarterly 45:208-15.

McGuire, William J. 1969 "The nature of attitudes and attitude change." Pp. 136-314 in Gardner Lindzey and Eliot Aronson (eds.), The Handbook of Social Psychology (2nd ed.), Vol. 3. Reading, Mass.: Addison-Wesley.

National Opinion Research Center 1980 General Social Surveys, 1972-1980: Cumulative Codebook. Chicago: NORC, University of Chicago.

Schuman, Howard, and Stanley Presser 1981 Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. New York: Academic Press (forthcoming).

Sigelman, Lee 1981 "Question-order effects on presidential popularity." Public Opinion Quarterly 45: 199-207.

Smith, Tom W. 1979 "Happiness: time trends, seasonal variations, inter-survey differences, and other mysteries." Social Psychology ~ u a r t e r l ~ 42: 18-30.

Turner, Charles F., and Elissa Krauss 1978 "Fallible indicators of the subjective state of the nation." American Psychologist 33:456-70.

http://www.jstor.org

LINKED CITATIONS - Page 1 of 2 -

You have printed the following article: Context Effects on Survey Responses to Questions About Abortion Howard Schuman; Stanley Presser; Jacob Ludwig The Public Opinion Quarterly, Vol. 45, No. 2. (Summer, 1981), pp. 216-223.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198122%2945%3A2%3C216%3ACEOSRT%3E2.0.CO%3B2-H

This article references the following linked citations. If you are trying to access articles from an off-campus location, you may be required to first logon via your library web site to access JSTOR. Please visit your library's website or contact a librarian to learn about options for remote access to JSTOR.

References The Effect of Question Order on Responses Norman M. Bradburn; William M. Mason Journal of Marketing Research, Vol. 1, No. 4. (Nov., 1964), pp. 57-61.

Stable URL: http://links.jstor.org/sici?sici=0022-2437%28196411%291%3A4%3C57%3ATEOQOO%3E2.0.CO%3B2-5

Effects of Question Wording and Context: An Experiment With Religious Indicators Otis Dudley Duncan; Howard Schuman Journal of the American Statistical Association, Vol. 75, No. 370. (Jun., 1980), pp. 269-275.

Stable URL: http://links.jstor.org/sici?sici=0162-1459%28198006%2975%3A370%3C269%3AEOQWAC%3E2.0.CO%3B2-3

Effects of Question Order on Survey Responses Sam G. McFarland The Public Opinion Quarterly, Vol. 45, No. 2. (Summer, 1981), pp. 208-215.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198122%2945%3A2%3C208%3AEOQOOS%3E2.0.CO%3B2-2

Question-Order Effects on Presidential Popularity Lee Sigelman The Public Opinion Quarterly, Vol. 45, No. 2. (Summer, 1981), pp. 199-207.

Stable URL: http://links.jstor.org/sici?sici=0033-362X%28198122%2945%3A2%3C199%3AQEOPP%3E2.0.CO%3B2-%23

http://www.jstor.org

LINKED CITATIONS - Page 2 of 2 -

Happiness: Time Trends, Seasonal Variations, Intersurvey Differences, and Other Mysteries Tom W. Smith Social Psychology Quarterly, Vol. 42, No. 1. (Mar., 1979), pp. 18-30.

Stable URL: http://links.jstor.org/sici?sici=0190-2725%28197903%2942%3A1%3C18%3AHTTSVI%3E2.0.CO%3B2-W