Effects of Anonymity by Juan Pardo

Effects of Anonymity

and Social Setting

on Survey Responses

RICHARD C. WILDMAN sources of bias in survey research are numerous. Phillips (197 1: 12-49), for example, has listed anywhere from 13 to 18 different sources of potential bias in survey research and suggests that a substantial proportion of the findings resulting from such methods are probably invalid. Of the many sources that can contribute to bias in social surveys, two which lend themselves to examination by mail survey are setting and anonymity (or its lack): these are the focus of the present study. In arguing that setting alone can produce bias, Phillips (1971:33-34) cites the early work of Maslow and Mintz, whose results (Maslow and Mintz, 1956; Mintz, 1956) showed that rating of facial photographs by subjects varied according to the "esthetic" nature of the room in which the rating took place. In more recent work, Alutto (1970) found that businessmen were more likely to complete open-ended items on a questionnaire mailed to their office than one mailed to their home. In most studies, however, setting is seen as part of the more general defining aspects of the overall situation resulting in particular types of role enactment (see Orne, 1962; Sarbin, 1964; Slomcznski, 1969; Hermans, 1970; Philips, 197 1 ). Anonymity, or its lack, is also frequently considered as a potential RrEN-rlAL

Abstract Anonymity was tested by placing an identifying number on half of the questionnaires on a mail survey designed to elicit reactions from teachers concerning teacher unions. Bias due to social setting was tested by mailing half the questionnaires to teachers at their home addresses, the other half to teachers at their school addresses. In a two-by-two factorial design, anonymity and social setting had no appreciable effect on response or speed of return, although social setting did have an effect on return rate. Respondents receiving the questionnaire at their school address had a higher return rate than those receiving it at home. Richard C . Wildman is Assistant Professor of Sociology at Bradley University, Peoria, IIlinois. This report is based o n a study which was begun in collaboration with the late Dr. Wayne E. Gregg, University of Nebraska-Lincoln. The author wishes to thank Marvin Leavy of Bradley University for his comments on an earlier draft. POQ 41 (1977) 74-79

EFFECTS OF ANONYMITY AND SOCIAL SETTING

source of bias in mail surveys. Respondents who can be identified may give more "socially desirable" responses than those who remain anonymous. Fuller (1974), however, concludes from her review of the literature that the risk of significant bias is relatively small. While a few researchers have found some systematic bias, others have not, even when the information is somewhat sensitive (see Fischer, 1946; Ash and Abramson, 1952; Hamel and Reif, 1952; Rosen, 1960; Backer and Bakal, 1970; Kosen et al., 1970; Fuller, 1974). King (1970), for example, found no difference in admission of drug use, in return rate, or in response to attitudinal items in a mail survey of Dartmouth undergraduates, half of whom received questionnaires with identifying numbers, half of whom did not. Fuller (1974) herself, however, reports an 1 1 percent higher return rate and some social desirability response bias for one of the identified groups in the mail questionnaire she examined. Method

Public school teachers, as well as the community in which they teach, are often split over the issue of teacher unions. In a midwestern community where teachers were unrepresented by union, we mailed a single-page questionnaire of four items concerning unions, strikes, and the teaching profession to 320 randomly selected public school teachers.' Each envelope contained a cover letter, a questionnaire, and a self-addressed stamped postcard with the response categories printed on it. The cover letter included the name of the research organization conducting the research (rather neutral in tone, we felt), the usual request for the respondent's help in making the survey a success, and the usual assurances of confidentiality to the respondent. Our analysis was based on the returns from this single mailing. A two-by-two factorial design was utilized to test the effects of social setting and anonymity in completing and returning the responses to the questionnaire. This procedure allowed us to examine anonymity, social setting, and interaction or the additive effects (if any) for the two conditions.

' The introductory remarks on the questionnaire read as follows: "This is a period when many occupational groups have found it necessary to organize into unions to protect and further their interests. In some areas public school teachers have organized into unions, as you are undoubtedly aware. The following four questions ask for your opinion regarding the unionization of teachers." The four items were: I. D o you believe that it is necessary for teachers to unionize to protect their economic and professional interests? 2. Would you join and actively support a teachers union in (name of community) if the opportunity occurred? 3. In order to maintain and insure an income level for teachers relative to the income level of other occupational groups, it is probably necessary for teachers to unionize and make use of the strike or the threat of a strike. In regard to the above statement, would you say that you (strongly agree . . . strongly disagree)? 4. If the teachers of (name of the community) were unionized, would you support a strike if it were called by that union?"

RICHARD C. WII.DMAN

The test of social setting was accomplished by mailing half of the letters to teachers at their home address, the other half to teachers at their school address. T o identify the mailing by postcard, we had printed two sets of postcards on slightly different weights of paper. This procedure allowed us to easily sort the returned postcards, although the cards themselves appeared identical to the naive observer. Random selection assured equivalence in these two mailings. The test of anonymity and its lack was accomplished by placing what appeared to be an identifying number handwritten in ink in the upper righthand corner of the postcard for half of the home and half of the school mailings, again randomly determined. Findings

Of the 320 letters mailed out, 210 postcards were returned, for a completion rate of 66 percent. No difference was found for the two groups in the anonymity condition: 105 (66 percent) out of 160 in both mailings re16 more postcards were received from the turned the p o s t ~ a r dHowever, .~ school mailing than from the home mailing, 113 (71 percent) versus 97 (61 percent). A difference of proportions test for these values yields a z score of 1.89. The probability of this occurring by chance is .06 for a twotailed test. No interaction is apparent for the two conditions, anonymity and setting. The rate at which the postcards were returned under each condition was examined also. Our measure of time was mail delivery days beginning with the day the first postcard was received. The rate of return for all conditions was remarkably similar regardless of the way the time variable was considered. For example, comparing those returns for the first four days with later returns in a two-by-four table including our four conditions yielded a x2 value of only 1.01; the probability of this occurring by chance approximates .80. The third variable examined was the response to the questionnaire items. Judicious dichotomizing of the four response categories for all postcards returned yielded a Guttman scale with a coefficient of reproducibility of .91 and a coefficient of scalability of .80. A breakdown of the From our initial examination of,the returned postcards it was clear that the postcards with the identification number in the upper righthand corner were being perceived as threatening, or at least as undesirable, by some respondents. Thirteen (12.4 percent) out of the 105 respondents returning postcards for the "identifiable" mailing had torn, cut, or inked over the identification number in the upper righthand corner of the postcard in such a way as to make it unreadable, six in the home mailing and seven in the school mailing. While it is possible that some of these returned postcards came from the anonymous mailing, having no other way to distinguish the two conditions than by the number or its absence in the upper righthand corner, we nevertheless assumed that all of these cards were in fact from respondents in the identifiable condition and took the defaced cards as evidence of a perceived threat on the part of respondents in this condition.

EFFECTS O F ANONYMITY AND SOCIAL SETTING

Table 1. Scores on Guttman Scale by Anonymity and Social Setting Scores

0 1-3

Not Anonymous

Anonymous

School

Home

School

Home

22 (39) 21 (38) 13 (23)

17 (35) 19 (39) 13 (27)

20 (35) 20 (35) 17 (30)

16 (33) 15 (30) I7 (35)

scores for all 2 10 returns produced the following results, with the higher scores representing a more favorable response t o teacher unions: Score on Guttrnan Scale

0 1 2 3 4

36 12 9 14 29

When the distribution of scores on this scale was examined by test condition, once again only minor differences were found. Table I, which has combined the scores of 1 to 3 on the Guttman scale, represents the results ~ b t a i n e d Even . ~ consideration of the two extreme sets of scores in Table 1 (the outside columns), the anonymous, home mailing versus the identifiable, school mailing, yields a x2 of 1.87 with an associated probability level of .39 at 2 degrees of freedom, considerably above the generally accepted levels of significance. This finding suggests that even the combined effects of setting and anonymity, o r its lack, are minor, if they exist at all in our sample. Discussion and Conclusion

Alutto (1970) obtained a higher rate of completion for open-ended questions in mailings to offices than to homes. The higher return rate from respondents in the school mailing found in our study may be similar. Alutto suggested that questionnaires received at home may be regarded as an intrusion upon one's private time, while questionnaires received at one's office may be regarded as more acceptable. An office mailing, according to Alutto, may encourage cooperation as a professional gesture on the part of the respondent. Regardless of the reasons, if response is relatively unaffected as in the findings reported here, it would seem to be advantageous to use office or work addresses whenever this alWith scores of I to 3 not collapsed, a yeilding a probability level of .93.

x2 value of 5.81 at 12 degrees of freedom results,

RICHARD C. WILDMAN

ternative is available, though the generalizability of this conclusion may be limited by both occupational type and survey topic. Our findings concerning the rapidity with which responses were returned offer little evidence for favoring any one of the alternatives tested. They all appear equally acceptable. Here, too, Alutto (1970) reported findings similar to ours for social setting-no differences between his home and office mailing in speed of return. The examination of attitudes concerning teacher unions by the test variables, anonymity and social setting, revealed n o apparent risk t o social surveys from these two variables. This was true even though evidence did exist in the form of defaced postcards (destroying identification number) that at least some respondents in the identifiable subsample (with numbers on postcard) found that condition threatening. O u r review of the literature did not lead us t o expect any marked discrepancy in these test conditions considered separately. We did believe, however, that an additive effect of the two variables in combination might reach a level of statistical significance. Our reasoning was straightforward enough. A teacher at school where the teaching role would be most salient, we felt, might respond more as a professional than as a worker. The potentially public nature of the response suggested by the identification number, we believed, might result in a. more cautious response, one less militant regarding unions and strikes. After all, these teachers were not unionized, considerable opposition t o unions did exist among teachers and the school system as a whole, and we had not revealed o u r purpose in conducting the survey beyond brief introductory remarks. Considering all these factors we thought that a statistically significant result might emerge when compared with the anonymous, home mailing.4 This proved not to be the case. Neither setting nor anonymity, alone or in combination, appeared t o affect response beyond chance levels; though, again, the generalizability of these results may be limited by the nature of the setting and survey topic. References

Alutto, J . A. 1970 "Some dynamics of questionnaire completion and return among professional and managerial personnel: The relative impacts of reception a t work site or place of residence." Journal of Applied Psychology 54:43032. Ash, P., and E. Abramson 1952 "The effect of anonymity on attitude questionnaire response." Journal of Abnormal a n d Social Psychology 47:722-23. Phillips (1971:20) has criticized survey research and social science in general for failing to consider various sources of bias in combination, when bias is considered at all: examining them one at a time ignores the additive or accumulative effects of several in conjunction.

EFFECTS OF ANONYMITY AND SOCIAL SETTING

Becker, G., and D. A. Bakal 1970 "Subject anonymity and motivational distortion in self-report data." Journal of Clinical Psychology 26:207-9. Fischer, R. P. 1946 "Signed versus unsigned personal questionnaires." American Psychologist 5:3 15. Fuller, C . 1974 "Effect of anonymity o n return rate and response bias in a mail survey." Journal of Applied Psychology 59:292-96. Hamel, L., and H. G . Reif 1952 "Should attitude questionnaires be signed?" Personnel Psychology 5 3 7 92. Hermans, H. J. M . 1970 "The influence of situation and condition o n the responses t o questionnaires." Sociologia Neerlandica 6: 1 - 13. King. F. W. 1970 "Anonymous versus identifiable questionnaire in drug usage surveys." American Psychologist 25:982-985. Kosen, D., C . Kitchen, M . Kochen, and D. Stodolosky 1970 "Psychological testing by computer: Effect on response bias." Educational and Psychological Measurement 30:803-10. Maslow, A. H., and N . L. Mintz 1956 "Effects of esthetic surroundings: I. Initial short-term effects of three esthetic conditions upon perceiving 'energy' and 'well-being' in faces." Journal of Psychology 4 1 :247-54. Mintz, N . L. 1956 "Effects of esthetic surroundings: 11. Prolonged and repeated experience in a 'beautiful' and 'ugly' room." Journal of Psychology 41:459-66. Orne, M . T . 1962 "On the social psychology of the psychological experiment: with particular reference t o demand characteristics and their implications." American Psychologist 17:776-83. Phillips, D. L . 1971 Knowledge from What. Chicago: Rand McNally. Rosen, N. A. 1960 "Anonymity and attitude measurement." Public Opinion Quarterly 24:675-79. Sarbin, T . R . 1964 "Role theoretical interpretation of psychological change." Pp. 176-219 in P. Worchel and D. Byrne (eds.), Personality Change, New York: Wiley. Slomczynski, K. M . 1969 "Conditions of interveiw-their impact upon statements of respondents." Polish Sociological Bulletin 20:125-35.