R. Keith Kelsall* Anne Poolet Annette Kuhnl by Juan Pardo

R. Keith Kelsall* Anne Poolet Annette Kuhnl

The questionnaire in a sociological reseaich project The well-publicized antagonism to the I 97 I census has posed yet again the problem of obtaining reliable data on which to base important policy decisions while at the same time bearing in mind the desirability of safeguarding individual privacy. The Registrar General is certainly not alone in facing this dilemma-in fact, he has the questionable advantage of compulsion on his side, while other researchers have to use the gentle art of persuasion to elicit compliance. Indeed, for the success of any survey, the importance of motivation to respond cannot be overstated, and such motivation presents particular problems when the selfadministered questionnaire, as opposed to the interview, is used as a research tool. I n this paper, attention is drawn to one recent attempt to resolve some of the research problems to be faced in using the postal questionnaire as a means of obtaining data. A postal questionnaire was employed in a large scale follow-up survey of university graduates of 1960, six years after they had taken their first degrees.1 Several factors indicated the appropriateness for this purpose of a postal survey, not the least of which were the size of the graduate sample-more than ten thousand men and women were involved-and its wide geographical dispersal both within the United Kingdom and overseas. Moreover, the group to be surveyed was very select in the sense of being composed of uniformly highly educated people2 who were expected to have little difficulty in understanding and answering the various questionnaire items. Perhaps most important, the information sought was of a relatively precise factual nature and could be effectively collected without the need of any personal contact with the respondent via an interviewer.

* Roger Keith Kelsall M.A.

Professor and Head of Department

t Anne Poole B.A.(ECON.), Independent Research Worker

$ Annette Kuhn B.A.(ECON.) Independent Research Worker All at the Higher Education Research Unit, Department of Sociological Studies, University of Sheffield

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

CONTACT

Nevertheless, from the beginning the survey presented certain difficulties, in particular regarding the establishment of initial contact with the graduates. The sample was drawn with the help of the Registrars and Appointments Secretaries of the co-operating institutions, who provided the last known addresses of the men and women concerned: but these addresses were inevitably out of date by the time six years had elapsed, and the problem of actually getting into contact with graduates was exacerbated by the fact that, since graduation, many of the women among them had married and acquired new surnames. The significance of these difficulties lay, of course, in the need to achieve a good response rate3 by first of all ensuring that the graduates actually received our correspondence. I n practice, contact is a most important element in response, and this applies in both interview and questionnaire surveys. I n either case, the most obvious drawback arising from poor response TAB LE

Analysis o f response: addresses

Men

Women

(N =

4,702)

(N = 3,582)

% 88

% 83

I00

Response after : I address tried 2 addresses tried More than 2 tried ALL RESPONSE

is that the respondents may be unrepresentative of the sample or population surveyed, rendering the final results of the investigation unreliable and the investigation itself a waste of time and money. However, with careful preparation certain steps can be taken effectively to maximise contact with sample members, and several devices were used to this end in the present survey.4 First of all, envelopes were printed with a request to forward if necessary. The analysis of response in Table I would indicate that, since in the vast majority of cases a completed questionnaire was received after only one address had been tried, this was an especially successful tactic; and also that the original recipients of our correspondence, if they were not the graduates themselves, had often been good enough to forward the envelopes to the appropriate addresses.5 This particular strategy would at the same time have had the effect of warning the recipient that the contents of the buff-coloured envelope were important, that this was not merely a circular, and that it should be read rather than destroyed.

345

R. X. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

Realizing that graduates tend to marry other graduates and that therefore some degree of intermarriage within the sample was likely, each sample member whose wife or husband also graduated in 1960 and had not as yet received a questionnaire was asked to let us know his or her marriage partner's name on graduation, and university. This was particularly useful in the case of graduate wives whose changes of name might well have reduced the chances of successful contact by the usual channels. Of major impact in the minimization of non-contact was the use of the 'snowball technique', which had been used with enormous success in an earlier follow-up survey of women in the teaching profession6 in which problems very similar to those raised in our own survey had to be TABLE 2

Analysis of response: questionnaires Men

Women

(.V = 4,702)

3,582)

ALL RESPONSE

I00

Mean number of questionnaires sent

1'34

1'37

(N =

Response after : I questionnaire sent 2 questionnaires sent 3 questionnaires sent 4 questionnaires sent 5 questionnaires sent 6 questionnaires sent

* Less than

per cent

faced. I n the present instance, a list of fellow graduates of 19607 was sent out with each questionnaire, and respondents were asked to note down the recent addresses of any people on the list with whom they were still in touch, and to return the list with the completed questionnaire. I n the vast majority of cases in which second and subsequent addresses were tried (see Tables I and 3), those addresses were obtained from the snowball lists. Several other attempts were made at a later stage to trace people with whom it appeared contact had as yet not been made: searches in telephone directories for addresses of those with unusual names, in professional directories in the case of graduates whose degrees were particularly vocationally oriented, and the following-up of various clues and pieces of information taken from snowball lists. 346

R. K. Kelsall, A. Poole, A. Kuhn TABLE

The questionnaire in a research project

3 A n a h i s of non-response :addresses

address tried addresses tried 3 addresses tried q addresses tried 5 addresses tried 6 or more addresses tried

Men

Women

(.V = 1,310)

(N =

34 7

781)

7 I

A L L NON-RESPONSE

* Less than

1 Y

I00

100

per cent

CO-OPERATION

Apart from non-contact, the other element in non-response is failure, for whatever reason, of the potential respondent to co-operate in the research once contacted. While it is true that in a university-educated group few if any people will have been unable to reply to most of the items in the questionnaire, the possibility remained that some would not in fact wish to do so. With regard to both these aspects of non-response, however, systematic study has yielded a considerable body of knowledge, so that the achievement of a high degree of co-operation from a sample can be seen to depend to a large extent upon factors within the control of the investigator.8 There are certain fairly standard procedures for encouraging people, once contacted, to participate, and the tactics decided upon in this case owed much to the experience of other research teams.9 Nevertheless, the response rate finally achieved was the result of many months of our own very careful preparation of the questionnaire and accompanying documents with the recipient in mind. Although it was generally agreed that the survey should seek data on the career patterns of graduates during the six years or so between first graduation and the time of the investigation, there were some differences ofview as to the amount and nature of any other information to be sought. From one point of view, of course, it was essential to include certain questions if-apart from the mere collection of statistical information-we were to achieve anything which could be described as sociologically significant. Here was an ideal opportunity to examine the graduates' situation in terms of more or less well-formulated theories regarding the nature of the British educational and occupational systems. But there was some hesitation lest the inclusion of too many

347

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

questions or of questions on particular topics might have a n adverse effect on response rates. This dilemma was in the event largely resolved through the judicious use of pilot studies and pretests, the effectiveness of which rests on the relationship to the eventual respondent group itself of the tested group. In this case, great care was taken to contact a group of people who would in many respects be similar to the 1960 graduate sample, graduates of Sheffield University in 1958, 1959 and 196010 being selected for the purpose. The pilot study involved a group of men who took their first degrees at Sheffield in 1960 and a similar group of women who graduated in 1959, and it indicated that only a minority of graduates would decline TABLE

Analysis of non-response :questionnaires Afen (N =

Women

1,310)

781)

2.65

2.70

(N =

questionnaire sent

questionnaires sent

3 questionnaires sent

4 questionnaires sent

5 questionnaires sent

6 questionnaires sent

7 questionnaires scnt

Mean number of questionnaires sent

* Less than

per cent

to answer questions not immediately related to their education and careers. I n particular, they willingly gave information about both their families of origin (including their parents' education and occupations) and their present family circumstances. I t appeared, too, that they experienced little difficulty in recalling all their changes of job since graduation, and it was decided on the basis of this evidence that the inclusion of questions on social background and on every job since graduation would not substantially jeopardize response rates. At the same time, however, the graduates involved in the pilot study seemed to have some difficulty in expressing, on a self-completed questionnaire, their opinions on certain issues, and found it hard to answer hypothetical questions which sought their attitudes to university education. O n this basis, and after further consultation with colleagues 348

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

working on a similar project, it was decided to restrict the use of such questions to the minimum and, where possible, attempt any investigation of attitudes through using specifically behavioura1 questions. Much care and a great deal of time was spent on constructing the questions themselves, which were eventually pretested on another group of graduates, this time those men and women who had taken degrees at Sheffield respectively in 1959 and 1958. This enabled any ambiguities in instructions and questions to be detected, while the pilot survey suggested that graduates were not on the whoIe averse to completing a questionnaire which at that time covered fourteen sides of mimeographed paper. There has been considerable discussion as to the effect on response rates of length of questionnaire, the general feeling being that potential respondents are deterred if asked to answer too many questions. It seems, however, that length is only one of the variables involved, and that it may be of less significance than certain others. Scott,ll for instance, has suggested that it is essential in attempting to secure a high response rate to arouse the recipient's interest in the enquiry, and Sletto points out that 'when we have constructed questionnaires to be mailed, we have too frequently overlooked the importance of motivating the recipient, keeping his undivided attention, and sustaining his interest until the task is completedY.l2Our own investigation was about graduates, and the people actually taking part in it were graduates who were provided with an opportunity to write at some length about themselves and to give their views on issues which concerned them and people like them, knowing that their answers were of great interest to the research team. Moreover, certain measures were adopted in the construction of the questionnaire which were intended to awaken, promote and sustain sufficient interest to ensure its completion and return. In the first place, having settled the coverage and construction of the individual questions, they were ordered so as to minimize the emotional content of the first few pages, which were in fact confined to questions which would appear directly relevant to the stated purpose of the investigation. Questions about graduates' family building and their social backgrounds, which might have given offence to a minority, were restricted to the later sections of the questionnaire, in the belief that the decision to respond may be made in a proportion of cases upon inspection of the first few pages. The lesser significance of questionnaire length as compared with interest is amply demonstrated in the case of the women graduates, who were asked (in a pullout section which the men did not receive) to answer thirteen additional questions relating to their home and work roles and the nature of any incompatibility between them. This reference to the difficulties women graduates experience in their careers 349

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

because of their family commitments doubtless aroused interest in many of them, with the result that in spite of their rather longer questionnaire, the ultimate response rate for women was 81 per cent as compared with 77 per cent for men (see Table 5 ) . Even with such a highly literate sample as this it was decided at the outset to present questions, where possible, in a highly structured, precoded form, so that respondents could in the main complete the questionnaire simply by putting circles around the appropriate answers. Where this was not possible, however, and open-ended questions had to be used, the difficulties experienced in replying were clearly minimal, and this factor is unlikely to have exerted any adverse effect on response rates. I n any case, the opportunity to elaborate at some length on certain TABLE

5 Response rates

Sample, N Respondents, N Percentage response -

---

Men

Women

Total

6,118 4,702 77

4,430 3,582 8I

10,548 8,284 79

topics may well have afforded compensation for any extra effort involved. Another important aspect of motivation to respond lies in the physical appearance of the questionnaire itself. For this reason, a graphic designer was commissioned who was able to create an aesthetically pleasing and compact questionnaire in the form of an I 14-inch by 54-inch booklet printed in two colours. By this means even two hitherto very unwieldy questions set out in tabular form were made to appear much less daunting, and clearer and easier both to read and to answer than they were in their original mimeographed state. We were unable to pretest this layout and design, but a small experiment was undertaken which indicated that the extra effort and expense involved here had been well worthwhile. The 1960 graduates, men from one university who as alternates were excluded from the main sample, were sent copies of the original duplicated questionnaire by the same post as an equal number of their counterparts were sent theprintedone: the alternates were not of course aware that they were not taking part in the survey in the usual way. A record was kept of the replies received from the two groups for ten weeks after posting, by which time the men in the sample were substantially more likely to have returned completed questionnaires than were members of the control group (Table 6). Although there could obviously have been several factors associated with this differential response, the result nevertheless does suggest that the care taken with the design and 350

R. K. Xelsall, A. Poole, A. Kuhn

The questionnaire in a research project

neither the questionnaire booklet nor the accompanying letter had to be folded for inclusion. At the time of posting, each envelope (franked to save time) also contained the appropriate snowball list, a letter signed by the ViceChancellor of the addressee's university endorsing the project and encouraging support, and a preaddressed stamped reply envelope for the completed questionnaire. Recent research has indicated that a stamped reply envelope is more effective in eliciting response than a franked one. According to Scott, this is because the stamp appears to the recipient as a piece of unspent money. As such, he may feel, it cannot be thrown away; the envelope cannot be used for a private purpose because it carries a printed heading indicating the sponsor; the stamp cannot be steamed off because that would be patently dishonest and in any case not worth the trouble; and it cannot easily be ignored because a fresh . . . stamp constantly catches the eye. Faced with this situation the recipient's only comfortable alternatives are to lose it or to return the questionnaire. With a franked label he has the third choice of ignoring it and the fourth of throwing it away, without any feeling that he is throwing away money.14

By these various means a 'package' was built up in the hope that graduates who received it would, albeit with varying degrees of enthusiasm, want to complete the questionnaire, fill in the snowball list where possible, and return both. MAILING A N D RESPONSE

I n the realization that graduates would be receiving and completing questionnaires over several months, it was decided from the point of view of ensuring meaningful findings that in answering questions relating to their current situations, they should all be referred to one particular date: I October 1966 was chosen for this purpose, the day in fact on which the questionnaires were sent out. As a result of our conscious attempts to combat non-response, we received, in response to this first mailing, completed questionnaires from as many as 57 per cent of all the men and 61 per cent of all the women included in the sample. However satisfactory these may be as initial response rates, they are clearly not high enough to ensure confidence in the reliability of any findings based upon them, and as is usually the case in postal questionnaire surveys, a programme of reminders was required. I n order to operate such a programme, it was essential to identify graduates as they responded, for which purpose each member of the sample had been assigned an individual serial number, which was stamped unobtrusively at the end of his or her questionnaire. 353

R. K. Kelsall, A. Poole, A. Kuhn

The questionilaire in a research project

Since reminder letters have been found to be the most potent means yet discovered of increasing returns,l5 up to four reminders were sent to each graduate in the attempt to reach those who had not returned their questionnaires and for whom no new addresses had come to light. The reminder letter referred to our original communication, p:inted out that we did not seem to have had a reply, and stressed that it was certainly not too late for co-operation to be of value, and that a reply a t any stage would be most appreciated. Graduates were also reminded of who the sponsors were, and of the importance of a good response. Snowball lists were included, and the offers made in the initial covering letter were renewed: to furnish anyone interested with details of the survey's findings, and to supply the addresses of any fellow graduates with whom the recipient had lost touch. With each reminder letter, graduates also received all the original enclosures, including the questionnaire itself. This is recognized as the ideal procedure, but in view of the expense involved it is relatively rarely put into practice.16 As Table 2 shows, 24 per cent of the men and 26 per cent of the women respondents did reply to a second or subsequent questionnaire, the intervals between successive waves of reminders depending, as in most investigations of this type,l7 on the flow of returns. Parallel with the following-up of the original addresses, other addresses-obtained for the most part from returned snowball listswere being tried (see Tables I and 3), and in fact 1 2 per cent of the men and I 7 per cent of the women respondents returned their questionnaires after two or more addresses had been tried. The differences between the sexes as to both the number of addresses tried (Table I ) and the mean number of questionnaires sent before response was secured (Table 2) reflects the fact already mentioned that married women were more difficult to trace: this difficulty was, however, more than offset by their greater propensity to respond once contacted. As regards non-response, contact could not of course have been made either with the members of the sample known to be dead at the time of the survey (28 men and 12 women) or with those for whom no address had ever been available (78 men and 55 women), and after allowing for these people there remained 2,091 non-respondents (1,310 men and 781 women) who had been sent at least one questionnaire. A handful of this number wrote to the Unit declining to take part in the survey, but otherwise it was impossible to tell to what extent non-response was due to genuine non-contact as opposed to failure to complete and return a questionnaire, and it may or may not be true in this instance to say that as far as can be judged, failure to respond was, in the vast majority of the cases where it occurred, due to the questionnaire not having reached the person for whom it was intended, and not to any unwillingness on [his or] her part to co-operate in the survey.18 354

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

I t is hardly surprising to find that more addresses had been tried in the case of non-respondents than in that of respondents (Table 3). I n all, 33 per cent of the men and 42 per cent of the women non-respondents had had questionnaires sent to them at two or more addresses. The largest single group had been sent three questionnaires (the number usually sent to a single address before giving up), and the average number of questionnaires sent was approaching three (Table 4), as opposed to just over one in the case of respondents. With a sample of many thousands, the follow-up programme was time-consuming, and it was some fifteen months after the first questionnaires had been sent out that it was finally decided, in view of the high rates of response by then achieved (see Table 5 ) , to call a halt to data collection. This did not mean, however, that the next stages of the operation were held up for as long as this: on the contrary, the coding frame had been designed and the bulk of the coding completed within twelve months, and the coding of questionnaires which came in later was completed concurrently with the preparation of the computer programme to be used in the analysis. While a n overall response rate of 79 per cent is highly satisfactory, it does not necessarily remove the possibility of some bias in the results,lg but in this case it was possible to establish to a certain extent the direction and degree of any bias by reference to some basic data on the nonrespondents in the sample. At the beginning of the investigation, staffs of each of the co-operating institutions provided last known address and subject of degree for each of the sample members; and at a later stage they were asked to furnish details of the type and class of degree taken by non-respondents and non-contacts. Even a few basic details such as these can help to establish the representativeness of the survey's findings, although they do not tell us the reasons for non-response. It was possible, therefore, to check whether the non-respondents and non-contacts differed from the respondents in terms of subject, tyFe and class of degree, and university group, since any significant differences would have introduced systematic bias into the survey's findings. I n fact, taking men and women together, distributions by university group and faculty were remarkably similar for the two groups, while there was a slight tendency for those with good degrees (first and upper seconds) to be over-represented among the respondents (Table 7 ) . 2 0 CONCLUSION

A recent large scale postal survey of university graduates six years after their graduation achieved a remarkably high response rate and so overcame one of the major drawbacks of the postal questionnaire as a research tool. Although the sample was in many ways ideally suited to 0

355

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

this means of data collection, problems of contact were paramount because of the unreliable nature of many addresses and the additional complications of geographical mobility and, in the case of many women graduates, change of surname. It was essential, therefore, to devise means of maximizing contact and, once contact was made, persuading people to complete and return their questionnaires. I n the hope that our experiences may be helpful to those who follow, we have reported some of the techniques used-some well known, others less so. Clearly, the basis of a good response rate can be built up with a suitable sample and with due care as to the design of the questionnaire itself; and we have demonstrated here how effective in terms of response was the use of a specially designed and printed questionnaire. I n this favourable situation, a potentially good response can be further increased by a careful mailing and reminder programme, in the present case complemented by the extractionof new addresses by the 'snowball list' technique. The response rate eventually achieved-79 per centis substantially higher than is normally obtained in postal questionnaire surveys and, most important in a government-sponsored project, it meant that analysis could proceed in the knowledge that reliance could confidently be placed on any of the findings which eventually emerged.

Notes I. Further information about the survey and details of some of its findings can be found in Six Years After, Sheffield University: Department of Sociological Studies, I 970; and Graduates: the Sociolngy of an Elite, London: Methuen, 1972, both by the present authors. 2. The sample comprised every woman and every other man who took a first degree at a British university in 1960. A list of the institutions taking part can be found in Six Years Afer, p. 91. Graduates in all faculties except those of Medicine, Dentistry and Veterinary Science, were included. 3. I t seems that response to postal questionnaires tends in general to vary between 20 and 70 per cent. See W. J. Goode and P. K. Hatt, Methods in Social Research, New York: McGrawHill, 1952, p. 173. 4. These measures were also aimed at contacting British graduates who had moved abroad. The sample specifically

excluded graduates originating from overseas who had subsequently returned to their home countries. 5. In many cases, the address provided by a university was that of the graduate's parents. 6. R. K. Kelsall, Women and Teaching, London: H.RI.S.O., 1963, p. 7. 7. In general, the lists included people who had graduated from the same university and with the same degree title, although there were exceptions to this: for instance, Oxford, Cambridge and London snowball lists were divided into colleges. 8. R. F. Sletto, 'The Pretesting of Questionnaires', Amer. Sociol. Rev., vol. 5 (1940), 193-200. See page 195. g. One of us had previously directed several successful follow-up surveys, including one of three cohorts of women entrants to the teaching profession, and this experience was invaluable in the present survey. We are also indebted to

R. K. Kelsall, A. Poole, A. Kuhn

The questionnaire in a research project

Mrs Christina Holbraad, who was involved in a similar survey at the L.S.E., for a great deal of practical assistance. 10. The 50 per cent of men excluded as alternates from the sample proper. 11. C. Scott, 'Research on Mail Surveys, J.Roy. Stut. Soc. (A), 124, 1961, 143-205. See page 178. 12. R. F. Sletto, op. cit., p. 194. 13. Christopher Scott, op. cit., examined five Government postal surveys for their bearing on mail survey tech-

niaue. He refers extensivelv to the work of other researchers, and his paper includes a substantial bibliography. 14. Ibid., p. 170. 15. Ibid., p. 164. 16. Ibid., pp. 164 and 166. 17. Ibid., p. 166. 18. R. K. Kelsall, op. cit., pp. 7-8. 19. W. J. Goode and P. K. Hatt, op. cit., p. 180. 20. Full analyses of response and nonresponse appear in Chapter I of Six Years Afkr.