M E T H O D O L O G I C A L BIAS I N

M E T H O D O L O G I C A L BIAS I N

P U B L I C O P I N I O N SURVEYS

BY F R E ' D E R I C K W I S E M A N * Statistically designed sample surveys have enabled pollsters to gauge public opinion on a wide range of issues. In such surveys, selection of a data collection technique is generally based on four criteria: (I) cost; (2) completion time; (3) response rate; and (4) response bias. Typically, more weight is placed on the first three factors and, as a result, adequate attention has not been given to the latter consideration. T h e study described in this paper looks at one type of response bias-that which results from the use of a specific data collection method. More specifically, this research uses a controlled experimental design in order to determine whether responses given in a public opinion polling are influenced by the method used to collect the data. Three methods are investigated: (I) mail questionnaire; (2) telephone interview; and (3) personal interview.

METHODS

Residents of a suburban Boston community were polled on nine current issues, both local and national. In order to determine the influence of the data collection technique, three experimental groups were formed and asked identical questions. Members of the first group received a mail questionnaire, while those in the second and third groups had telephone and personal interviews, respectively. If there were n o technique bias, then one would expect identical results (except for random variation) in each of the three experimental groups.

Critical to the reasoning above is the assumption that the experimental groups are equivalent samples from a common population. A two-stage sampling process was used in an attempt to satisfy this requirement. In stage one, the population was divided into twelve mutually exclusive and collectively exhaustive clusters, and a random selection of a street within each cluster was made. In stage two, those residing on each of the selected streets were sequentially assigned to groups in the following order: mail-telephone-mail-personal.

Approximately the same number of potential respondents was obtained for each of the twelve street listings.

As can be seen, there were twice as many potential mail respondents

* Frederick sity.

Wiseman is Assistant Professor of Marketing a t Northeastern Univer-

FREDERICK WISEMAN

as potential telephone or personal interview respondents. This particular sample allocation was made because of (1) the belief that the response rate from the mail questionnaire would be significantly lower than that from the other two methods; and (2) the desire to obtain a p proximately the same number of completed interviews for each of the three data collection techniques. T o improve upon the mail questionnaire response rate, a follow-up postcard was sent three days after the original mailing. In addition, on an experimental basis, approximately 25 percent of this group received a prior telephone notification. This brief notification informed potential respondents that they would be receiving a questionnaire in the mail within the next few days and that their cooperation in filling it out would be greatly appreciated.1 RESULTS

T h e response rates achieved for each of the three techniques are given in Table 1.2 As can be seen from Table 2, the number of completed interviews using the mail questionnaire was considerably larger than that for either the personal or the telephone interview. This particular result was unexpected and was primarily due to the relatively large number of questionnaires returned from those households which had received prior telephone notification.

Sample Validation. Socioeconomic and demographic data were obtained for each of the respondents. No statistically significant differences were found among the groups on any of the following characteristics: sex, marital status, age, occupation, income, and religion.

TABLE 1 Data Collection Technique Mail questionnaire Prior notificationa No prior notification Telephone interview Personal interview

Attempts 75 245 160b 160

Completions 50 (67%) 107 (47%) 102 (64%) 96 (60%)

The position of the stamp on the self-addressed return envelope made it possible to identify all returns coming from households receiving prior notification.

Not included are 56 households for which there was no telephone number.

1 Both these procedures have been found to have a positive effect on the response rate. See, for example, Morton L. Brown, "Use of a Postcard Query in Mail Surveys," Public Opinion Quarterly, Vol. 29, 1965, pp. 635-637; and James E. Stafford, "Influence of Preliminary Contact on Mail Returns," Journal of Marketing Research, Vol. 3, 1965, pp. 410-411.

2 Telephone and personal interviews were obtained by students in an advanced undergraduate marketing research course.

BIAS IN OPINION SURVEYS

107

TABLE 2 SURVEY RESULTS

BY DATA COLLECTION TECHNIQUE

Percent Yes Answer9 Issue

Mail

Telephone Personal

X2

1. In favor of Congress's decision to eliminate funds for the SST 2. In favor of an all-volunteer army within two years 3. In favor of a reduction in the size of the Massachusetts House of Representatives 4. In favor of legalizing marijuana 5. In favor of making birth control devices readily available to unmarried people 6. In favor of legalizing abortion 7. In favor of the Court's decision in finding Lt. Calley guilty 8. In favor of giving state aid to Catholic schools 9. In favor of lowering the legal drinking age to eighteen a

Excluding "Don't know" responses.

Significant a t the .05 level.

Significant a t the .O1 level.

Thus, the previously described sampling process did, in fact, generate equivalent groupings of respondents. No differences in response patterns were found between those receiving and those not receiving prior notification in the mail questionnaire group.

Suruey Results. T h e results of the public opinion polling are shown in Table 2. Significant technique bias was present on only two of the nine questions: "Do you believe that birth control devices should be readily available to unmarried people?" and "Should abortion be legalized in Massachusetts?" These issues are both personal, and, in each instance, the largest percentage of socially undesirable responses was obtained in the mail questionnaire while the smallest percentage was obtained in the telephone interview.

Further analysis of the two significant questions revealed that the response bias was related to the religious preference of the respondent.

On the birth control issue, 75 percent of the Catholics who received a mail questionnaire were in favor of making birth control devices readily available to unmarried people, compared with only 44 percent of those Catholics receiving either a telephone or personal interview. O n the abortion issue, Jewish members of the sample varied most depending on which method was used to collect the data. Ninetyeight percent of

108

FREDERICK WISEMAN

Jews receiving the mail questionnaire agreed with the proposition, compared with approximately 73 percent of those responding to one of the other data collection methods. Again, the largest percentage of socially undesirable responses was obtained with the mail questionnaire.

T h e problem of respondents not revealing socially undesirable traits or characteristics has been previously discussed by Edwards3 and, more recently, by Dohrenwend.4 T h e results of the present research suggest that research techniques affording greater privacy of response are more likely to reduce this problem than those affording less privacy.

A possible solution to the problem when personal interviews are used is to employ a data collection technique for which the probability of respondents' giving untruthful answers is relatively small. Such a technique is the randomized response model which was first proposed by Warner.6 This promising technique, based on probability theory, has been empirically tested in a recently completed North Carolina abortion study. Results and discussion of this research, together with the methodological framework of the model itself, can be found in Greenberg et. a1.6 CONCLUSION

T h e major finding obtained from this research is that responses given in a public opinion polling are not always independent of the method used to collect the data. Response bias is likely to be a problem in telephone and personal interviews whenever the question being asked is one for which there exists a socially undesirable response. T h e significance of this result is twofold: (I) pollsters must be especially careful in designing sample surveys in which the objective is to measure public sentiment on sensitive issues; and (2) new data collection techniques, such as the randomized response model, must be developed so as to reduce the probability that respondents give socially desirable responses when, in fact, they do not hold such viewpoints.

3 A. L. Edwards, The Social Desirability Variable In Personality Assessment and Research, New York, Dryden Press, 1957.

4 Barbara Snell Dohrenwend, "An Experimental Study of Directive Interviewing," Public Opinion Quarterly, Vol. 34, 1970, pp. 117-125.

5 S. L. Warner, "Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias," Journal of the American Statistical Association, Vol. 60, 1965. pp. 63-69.

6B. G. Greenberg, R. R. Kuebler, Jr., J. R. Abernathy, and D. G. Horvitz, "Application of the Randomized Response Technique in Obtaining Quantitative Data," Journal of the American Statistical Association, Vol. 66, 1971, pp. 249-250.