(Division of Research Techniques, London School of

(Division of Research Techniques, London School of Economics and Political Science)

Random Sampling and Quota Sampling In the sampling of human populations the selection of the final sample units is usually carried out by one of two methods: random sampling or quota sampling: Of the former there are many variants, ranging from pure random selection (by random numbers or the lottery method) to more or less systematic selection from some complete record of the population. Area sampling, predominantly used in the United States, is a form of random sampling. Whatever the details of the sampling designs or frames employed, one characteristic is-or should be-common to all random sampling. The selection of the sample units is carried out by some impersonal (strictly determined) method and is uninfluenced by human choice. That is to say, the interviewers are not allowed any freedom in deciding which members of the population shall be included in the sample. This independence of selection from human judgment is a prerequisite for ensuring that every member of the population shall have a known chance of being included in the sample-which is what we mean by randomness.

Quota sampling differs from random sampling in several minor ways, but the fundalnentdl difference is that, once the general breakdown of the sample is decided (e.g., how many men and women, how many people in each age-group it is to include) and the quota assignments allocated to eacb interviewer, the choice of the actual sample units to fit into this framework is left to the interviewer.

Much discussion has centred around the merits of the two techniques. Some experts believe the quota method to be so unreliable and prone to bias as to be almost worthless; others think that, although not as accurate as random sampling, quota sampling can be used safely on some types of inquiry; while some believe that, if careful instructions are given and if sufficient constraints are imposed on the freedom of the interviewer, quota sampling can be made highly reliable, and that the heavy extra cost of random sampling does not result in a sufficient increase in accuracy to be worth while. In general, academic statisticians have criticized the method for its theoretical weakness, while market research workers have defended it for its cheapness and ease of practical application.

This issue is not by any means the most important problem in survey methodology, nor even perhaps in the more narrow field of sampling. Yet the fact is that the controversy has continued -most fiercely in the United States-for many years, largely unaided by any experimental evidence, and that the argument is still conducted mainly on the basis of prejudice and untested assumptions. There is no clear-cut answer to the problem, in the sense that either method could be shown to be preferable in all circumstances. There are some types of survey on which nobody would suggest using a quota sample, while there are others on which random sampling may be impracticable. Still, there remains a large field on which either method could bs used, and research is required to throw factual light on the merits of both techniques. It therefore seemed worth while to embark on a programme of reasearch into quota sampling, in the hope that the results will be of value and interest both to statisticians generally and to market research practitioners.

The present report represents the first, preliminary, stage in the research. It consists of three parts. The remainder of the present part is devoted to an examination of the usual cases for and against quota sampling, followed by a statement of the general aims of this research programme.

The second and major part is a description of current quota sampling practice in this country.

41 2

MOSER-QUO~U Sampling

[Part 111,

The final part then attempts to suggest which are the aspects of quota sampling on which the research should concentrate.

The Usual Case Against Quota Sampling The selection of the final sample units by the quota method is usually criticized on the following grounds : (a) The use of random selection-implying that the probability of inclusion of population units in the sample is known-makes it possible to attach standard errors to the sample estimates.

Quota sampling does not meet the basic requirement of randomness, and it is thus not legitimate, from a theoretical viewpoint, to calculate standard errors for quota sample results.

(6) Quota samplers invariably attempt, as one of their controls, an economic or social breakdown of the sample. Two objections can be made against this type of control. In the first place, it is inevitably based on a hazardous statistical foundation. There are no official or other really reliable figures of social or economic classes. In the second place, these controls are inevitably defined in vague terms. The interviewer uses his or her judgment in deciding to which "class" the respondent belongs, so that there is room for bias. (c) Bias may be introduced because, within the quotas, interviewers may not secure a representative sample of respondents. For instance, although the required number of working-class persons is included in the sample, interviewers may have chosen people towards the upper levels of the working class. Or, again, the top age-group of 65 and over may be filled by persons of 65 and 66, so that the very old people are under-xepresented. Similarly, the sample might underrepresent housewives with outside jobs or with very large families. In all such cases a biased sample would result.

These are hypothetical examples of what is the crucial problem of quota sampling. Given that all the quotas are correctly and honestly filled-i.e., that every sample unit is in the cell to which it properly belongs-is the spread within the cells such that an unbiased sample emerges?

In other words, can human choice do what is expected of random selection?

The extreme form of the unrepresentativeness argument is that, even if the quota sample is correct on the controls and most of the uncontrolled variables as well, it may still be unrepresentative with regard to the variable under study. ( d ) Bias inay arise through the peculiarity of the interview situation. In so far as people are interviewed in the streets or in their offices or factories, biased information may be obtained. (e) Objection is sometimes made to quota sampling, not merely because the interviewer has too much freedom in choosing respondents, but because the method allows altogether too little office control of the fieldstaff. It is more difficult to check on the honesty of interviewers in the case of quota than in that of random sampling; and the former method probably offers more temptations to the unscrupulous interviewer. For instance, investigators may place respondents in the cells where cases are needed or difficult to find, rather than in those to which they really belong. This may apply particularly with vague controls, such as "social class".

The Usual Case for Quota Sampling We now turn to the arguments usually put forward in favour of this method. Some of the points here are answers to the above criticisms, rather than positive advantages: (a) Two points are often made in reply to the first criticism above. Firstly, it is justifiably argued that random samples as se!ected and as achieved are two different things. In practice, some of the original sample is not obtained on account of refusals and non-contacts. In fact, random sampling is not free from bias, and the standard errors are attached to what may, in fact, be a biased final sample. Secondly, it is argued that sampling errors are 0: comparatively small importance as against the very considerable and intractable non-sampling errors which arise in the collection of the data. Consequently, to criticize quota sampling on the grounds that sampling errors cannot be calculated is to concentrate on a relatively minor issue. (The main point of that criticism still .stands, namely that in quota sampling, owing to the absence of the condition of randomness, one does not know the accuracy of any particular sample.)

(b) The main argunient for quota sanlpiing is that it is very cheap. This cheapness is due largely to the much lower travelling costs and to the lack of call-backs. It is difficult to obtain accurate figures, bul it is probable that the averagc random interview in this country costs about twice as much as a qrota one. (c) Quota sampling is much easier from an administrative viewpoint. There is no need to go through the tedious stage of drawing the sample; no problems of non-contacts, call-backs, or substitute lists; no apparent problem of refusals; no need, unless this is specially required, to do much evening interviewing or to send field-workers to out-of-the-way areas. Responsibility for the sample is largely transferred to the individual interviewer, and the office burden is consequently lightened. Interviewers with experience of random and quota sampling usually prefer the more elastic, less controlled and less tiring quota method. (d) If the field-work on a survey has to be completed within a very short time-say, one dayquota sampling may be the only feasible method. (e) In partial answer to criticism (b) above, it is said that only a very few and extremely broad social groups are used, and that disagreement about the class of a respondent would be rare. (f) Quota samplers generally believe that instructions to and constraints on interviewers are sufficient to guard against the main dangers of bias outlined in (c) above, but they mostly agree that this is a matter of belief rather than fact. ( g ) Finally, it is worth pointing out that quota sampling is independent of the existence of lists. As long as good lists-suitable for sampling individuals and households-are available, la the case in this country, this is not a point of substance. But if, for instance, the National Register were to be withdrawn o r to lose its present accuracy, there would then remain only the alternatives of either using one of the other, and less satisfactory, lists; of using area sampling; or of using quota sampling-if this technique has been shown to be of any reliability. (It is worth noting that the National Register is not in fact available to market research organizations for sampling purposes. For national list samples market research bodies would have to depend largely on the Electoral Roll or the Rating List.) (h) Quota sampling is also defended on the grounds that, although the sample may be biased with regard to certain characteristics, it may be quite satisfactory for others.

The main arguments against and for quota sampling have been listed. All of them are to some extent controversial. We are entering on this research with open minds, hoping that at the end at least some of the pros and cons may cease to be matters of conjecture.

The General Aims of the Research Our main aim is to find out what types of bias, if any, are inherent in the method of quota sampling, and to what extent these biases can be eliminated altogether or brought under control by careful training or by the introduction of further constraints on the freedom of the interviewers in selecting the sample. In other words, we should like to know not merely how accurate quota sampling is, but also how accurate it can be made by the use of various refinements.

In the long run we should like to accumulate data to indicate when particular biases, if they are found to be uncontrollable, are important, and when they can safely be ignored. It is often said that quota sampling is quite good enough for some purposes. We want, if possible, to give this statement greater precision.

In short, this research is aimed more at the future than at the present. We felt very strongly, however, that any actual experimentation should be preceded by a study of present-day quota sampling practice. Only in this way can one decide which are the critical aspects of the method needing experimental investigation. Four leading market research organizations in this country --B.B.C. Audience Research, British Institute of Public Opinion, British Market Research Bureau and Research Services, Ltd.-were asked a number of questions regarding their sampling procedures and have given us every possible co-operation. In addition to discussion with their sampling experts, it has been possible to talk to the field supervisors about actual field practice.

We are most grateful for the help given us by the organizations. The information received is summarized and discussed in Part I1 below. It is desirable to retain anonymity, and particulars quoted will not be identified with any one organization by name. (The letters A, B, C and D are used-not, incidentally, in the above order of organizations.)

MOSER-Quota Sampling

[Part 111,

1. The Quota Controls* 1.l. The Standard Controls The standard controls used differ little from one organization to another. Three are used in every case: sex, age, and social status (this last is alternatively called income grade, economic group o r socio-economic group). The age grouping varies in broadness, anything from three to six groups being distinguished. (There is some variation of practice regarding the age question; in some organizations interviewers ask for the actual age, while in others they are told to estimate it.)

The "social status" grouping is also broad. One organization distinguishes five, another four, and the others three groups. This control is discussed in detail below.

Organization B afways uses the additional control: in employment/not in employment; while Organization C always imposes a further constraint by dividing women into housewives and

"occupied" women.

1.2. The Special Controls

Many other controls are used occasionally, if a particular survey seems to require it. The following may be instanced: head/non-head of household; housewife/non-housewife; married/ single. Attempts at control by occupational group are sometimes made. Two of the organizations rarely, if ever, employ controls other than the standard ones.

The important point here is that every attempt is made to keep the controls down to a minimum.

The more complicated the quota scheme, the more restricted the interviewer's freedom, the more difficult her task becomes and the more costly is the whole survey. The attraction of quota sampling is its simplicity of design and execution, and there is a continued resistance to the use of additional controls, in the hope that the work of controls can be done by careful interviewer

instructions.

1 .3. Statistical Basis for the Controls Neither age nor sex present difficulties as regards their statistical basis. The employrr~ent/not in employment control is based on published Ministry of Labour figures, The housewife/occupied woman control is based either on figures from a P.E.P. Report on the Employment of Women or on figures derived from the Social Survey's British Household. The statistical basis of most of the controls (social status is discussed below) is reliable as long as the figures for a single control are required. Nearly all the organizations attempt to vary their quotas regionally, and the figures, when two or three inter-related controls are used, become less well based.

1.4. "Social Status"

This control-under one of its various names-is used by all the organizations and is an integral part of quota sampling. Without it, quota samplers rightly feel, the method would be most precarious. It is, however, the weakest of the controls both in its statistical basis and its definition. We will now discuss the organizations separately, giving for each the most usual practice.

Organization A generally distinguishes five income grades: A. Over £1,000 p.a.

B. £650-£1,000.

C . £400-£650.

D. £225-£400 E. Below £225.

* National quota samples, like random samples, are generally stratlfied by geographical region, urbanirural area and, occasionally, town size. These are not quota controls, and we are here concerned with the selection procedure within the primary sampling units, such as.towns, etc.

19521

Mos~~--QuotaSampling

41 5

These am generally filled in the overall proportions of 5, 10, 20, 55, 10 per cent. The quotas are varied regionally. The breakdown is based on figures derived from past random surveys conducted by the organization and is revised about once a year. Occasionally a broader breakdown is used by combining A B and D $. E. The income grade refers to the head of the household; interviewers ask for occupation and take careful note of details concerning family habits, amenities, type of dwelling and so on. On the basis of this information they decide which class to place the person in. To aid them in this task, their field manual contains particulars of family habits, amenities, etc., and examples of occupations associated with each of the five grades.

The interviewer asks for the actrla! income grade only in cases where she feels some doubt or difficulty in deciding on the appropriate grade.

Organization B distinguishes three social groups: A. Upper middle class.

B. Lower middle class. c. Working class.

The usual over-all breakdown, which is varied regionally, is 5, 20, 75 per cent. The figures are originally derived from the publication The Home Market, and are modified on the basis of the 19.50 edition of that book and of recent Hulton surveys. The assessment is a subjective judgment of socio-economic class, and is made on the basis of the informant's occupation, manner and conversation, with the stress on occupation. In the case of housewives, however, no details of the husband's occupation are asked for, and the class assessment is to be made entirely on her manner, appearance and conversation. The manual gives examples of the sort of occupations found in each class.

Organization C distinguishes three socio-economic classes:

AB. Upper and upper middle class.

c. Middle and lower middle class.

D. Working class.

Interviews are allocated in the overall proportion of 10,20, 70 per cent., and the breakdown is varied regionally. The figures are derived from those used in, and published at the beginning of, the Hulton Survey. Cultural rather than economic factors control the placing. Interviewers are told to classify their respondents according to their first impression of the social type and by observing their "surrounding conditions", including the type of home and the occupation. A list of occupations typical of those found in each class is given in the manual. But interviewers are told that, if the occupation and environment of a respondent do not seem to be in line with the class judged to be appropriate on the basis of his personal characteristics, etc., then the final assessment is to be made on the latter rather than on occupation and environment.

Organizatiorz D distinguishes four economic groups: : Upper class group.

Average Average : Upper middle class.

Average - : Lower and middle working class.

Very poor.

The usual breakdown is 5, 21, 59, 15 per cent. and is derived from figures in the pre-war Horile Market. In assessing economic group, interviewers are told to pay attention more to occupation and general social standing than to earnings. Notes in the manual give guidance on the sorts of occupations, home amenities, and so on, to be found in each group This summary shows that the differences are in emphasis rather than general approach. The statistical basis is, and is often admitted to be, somewhat arbitrary. Survey statisticians would be happy if they had official figures on which to base this control. However, it seems doubtful whether any significant bias is introduced by the use of a breakdown which may be slightly inaccurate. The order of magnitude of the different classes is likely to be right, and this class stratification would have to be very inaccurate to affect overall results materially. In most surveys results are not given for the whole sample but are quoted separately for the different social groups, in which case the wrong size of the groups is even less likely to matter.

Much more serious as a possible source of bias is the vagueness of definition of these controls.

The main objective index available, income, is usually avoided-for obvious reasons. It is admitted

+

+

416

[Part 111,

MOSER-Quota Sumpiing

that the class allocation is largely a matter of the interviewer's subjective judgment and that there is room for bias and differences of opinion, especially with borderline cases. This is a very weak point in quota sampling, and needs to be considered in any experimentation.

2. The Quota Scheme The way in which the quota scheme is constructed constitutes an important difference between organizations. There are two alternatives; the controls may be inter-related or they may be set independently. In the case of Organizatio~zA, they are inter-related on nearly all surveys so that the interviewer is given a scheme of the following type:

Organization B, on the other hand, issues a totally different type of scheme: Social Groups ---

B

1~1-i

I

Sexes

Men

I women

1-1

Age Groups

1

1

Employment In employment

20-29

1

30-49

I

Not m employment

16-19

501 --

1-

1-1

, I

I I

No attempt is made to relate the controls to each other, so that it is possible that all the "In employment" tend to be men, and "Not in employment" women, and that other controls are "paired" in a similar way. Instructions are issued to interviewers to avoid this kind of biasing relationship between controls, and the results are checked with this in view.

Organization C uses inter-related controls in the case of men and occupied women (the controls being age and socio-economic group). In the case of housewives the breakdown is by socioeconomic group only.

Organization D does not inter-relate any of the controls. Interviewers are instructed to aim at a wide spread. Thus, they are told to "get the men and women spread evenly between age and economic groups".

It is difficult to assess how much is gained in accuracy and representativeness by inter-relating the controls, but there can be little doubt that it is to be preferred to the setting of independent controls. The latter may be simpler for the interviewer and less costly; but it is unlikely that, even with explicit instructions, the "pairing" effect, with its resultant bias, can be avoided. The main point in favour of independent controls is that it is easier to find an adequate statistical basis for setting marginal totals (which is all that is necessary for independent controls) than for all the cells of the quota scheme.

3. The Filling of the Quotas 3.1. The Location of the Interviews Random sample interviews are home interviews (except in special surveys) ; in quota sampling, interviews may be made-unless instructions to the contrary are given-at home, in offices, good deal of variation between factories, parks, public places or in the street. There is

19-52]

417

Sampling

E\.~ssER--@uo~~

olganizations in this respect, as weil as between types of surveys within one organization and between interviewers on any one survey. The details below can only give a general average picture : Organization A allows interviews to be made in all types of location. If, in a particular survey, street interviewing is to be excluded (as would be the case if a very complex questionnaire is involved), instructions to that effect are given in the job manual. Interviewers are instructed that they may interview in factories as long as permission is granted, and workers are willingand not ordered-to be interviewed. If a particular factory is very helpful, there is nothing to stop interviewers going back in another survey. On the whole, though, investigators are told to do as much home interviewing as possible and, as a rough approximation, it may be said that two-thirds of all interviews are done at homes.

Organization B interviewers may work in the street or by calling at houses, or places of work, but most interviews are in fact of the first type. The following rough figures show this: Proportion of Interviews Done at Homes All or most

More than half Half Less than half Few or none

Percenta,ye of Interviewers Urban 11 9 14

14

52

Rural 14 17 23

21 25

AN 12

11 17 16

44

Organization C allows n o street interviewing. Housewives are always to be interviewed at home (not in shops or queues), while others may be interviewed in offices, factories, etc. If the sample is controlled by occupational group (as is sometimes the case), all occupational interviews are to be made at the place of employment.

Organization D: On two recent, fairly typical, surveys, home interviews accounted for 36 per cent. of the total, office and factory for 9 per cent. and street and other interviews for 55 per cent.

Factory and office interviewing is allowed for the obvious reason that it catches the sort of people who tend to be missed in ordinary day-time street interviewing. Even so, it is doubtful whether a fully representative sample emerges. Quota samplers then~selveshave suggested that they are liable to under-represent heavy manual workers, miners, dock labourers, and that they may get too many engaged in distribution and transport, and possibly too few unoccupied women.

Supervisors indicate that-within the limits set by the quotas and the instructions-the relative amount of street/home/factory interviewing is entirely a matter of personal preference, coupled with the weather and the investigator's knowledge of the locality. Street interviewing is clearly economical of time, and the common view seems to be that, with a reasonably short questionnaire, it does not affect the quality of the interview detrimentally.

3 .2. The Spread of Interviews It is a common criticism of quota sampling that it is unlikely to achieve an adequate geographical spread within the areas-districts, towns, etc.-selected for sampling. This point, furthermore, is a source of worry to some market research practitioners, and various attempts are made to secure an adequate spread.

Organization A lays down instructions to the effect that a space of at least 20 houses inust be leh between houses in urban areas, add 15 in rural areas. Furthermore, it is suggested to interviewers that they should divide their area into as many segments as they have interviewing days and that they should tackle one segment a day. It is felt that this should be a matter for advice rather than of binding instructions. If the spread in a particular survey is of overriding importance, this organization would probably use a random sample.

Organization B has n o instructions on spacing between houses, but tells its field-workers "not to go farther from your house or business than is necessary to obtain your required daily quota". It should be added that all the surveys conducted by this organization consist of one-day quotas, so that it is not feasible to attempt a wide coverage.

41 8

MOSER-Quota Sanlp1in.q

[Part 111,

Organization C instructs interviewers to spread interviews over the whole town, if possible.

They are told-in the manual-that "having arrived at a street and made your first call, which should not always be the first house in it, proceed along it, lea\:ng a space of 5 houses between each interview until you come to a left turn; then ieave your first road and go along the left turn one until you come to a right turn, etc. . . . When you have started making calls in a road, do not cross to the other side except to take a left or right turn '.

Organization D tells investigators to carry out the interviewing "as near to your home as is compatible with fulfilling your quotas and getting a good mixture of the local population. . . .

Nothing is gained by trying to cover all parts of your area. We are anxious for your sake and our own that travelling should be reduced to an absolute minimum".

The geographical spread of quota samples is clearly a point for investigation. One wants to know whether interviewers tend always to go to a particular area, whether certain parts of a town are systematically under-represented. In making such an examination, one must naturally take a large number of quota samples. There is n o reason to expect one single quota sample, any more than one random sample, to be spread over the whole area. In practice, the quota of, say, 30 interviews in a town is covered by one interviewer, working in only a small part of the town.

In random sampling it is possible to select two or three wards randomly from all the wards in the town, and to confine the sampling to selected wards. If, in the quota case, there were a large number of interviewers spread over the whole town and if one or two interviewers were selected at random for one particular survey, the two ways of tackling the town would be formally equivalent.

It is, in fact, argued by at least one organization that its large turnover of interviewers serves to achieve a reasonable geographical spread of interviews in the long run. The argument is more difficult if the field staff consists of a relatively small number of persons who do the work over a long period of time, always interviewing in the same town or district.

3.3. T iming of Interviews It is not possible to give general figures. Organization D does something like a third of its interviewing in the evening, while others do the great majority of their interviews during the day.

No definite rules are imposed for the timing of interviews.

3.4. Other Restrictions on the Filling of Quotas There are certain other restrictions on the filling of quotas which are worth mentioning: No organization a l l ~ w smore than one person to be interviewed in any one household. In addition, in Organization A the same person must not be included in surveys more than once a year. N o definite restrictions are imposed on the numbers to be taken in any one factory, although interviewers are told to avoid taking "too many" and a check is kept on this at the ofice. Organization B does not allow any person to be interviewed more often than once in three months. (In this organization interviewers are permitted occasionally to take more than one person in a household in rural areas). Organization C tells interviewers that, when interviewing on industrial premises, they may take one person for every 20; and that, if there are less than 20 altogether, they may make one interview. Organization D does not allow more than three interviews in any one block of flats, row of houses or factory; nor may anyone be interviewed in a second survey.

4. Field Organizatiorl Certain aspects of field organization are relevant to the present subject and will be discussed briefly.

4.1 The Interviewers Organization A is the only one among the four which is staffed by full-time field-workers.

The others employ part-time staffs with varying frequency of employment. In the case of Organization B a field-worker may expect an assignment perhaps once every 2 or 3 months and would then be given about 10-20 interviews a day for 2-3 weeks. These are always short interviews-lasting only about 10 minutes. Interviewers in Organization C work ?. days out of 5 on

19521

MOSER-QUQ~~ Sampling

419

the average, and are thus virtually full-time. In Organization D about 250-300 out of the 700-800 interviewers in the pool are in the field at any given time and each of them would do about 15 interviews on any one survey.

It is of interest to note that, in all the organizations, the interviewers are largely local people, i.e., that they are working in the towns or, a t least, districts in which they live. (In one case interviewers are instructed to avoid areas in which they are well known.)

4.2. Training and Supervision All the organizations have field manuals, which include sections on the selection of the sample and quota filling, and in some cases a job manual is issued for each specific survey.

As regards selection of interviewers, one organization accepts all who apply-within reason.

Many are then dismissed if their early interviewing is not up to standard. The other bodies enforce selection procedures of various types. One selects entirely by personal interview; another demands the names of two referees, whose opinions are invariably sought. All place more emphasis on success in early interviewing.

Training-some form of which is used by three of the organizations-mainly consists of supervised interviewing. In one case a fair amount of more formal training is given at the office -on sampling instructions and so on-and this is followed by two days' practice interviewing.

In another, two days' trial interviewing is done and a detailed report on the performance is given to the interviewer. In the third a fortnight's supervised interviewing has to be done.

It is generally felt that one of the main values of supervision is the psychological support it gives to the field-staff. Interviewers are greatly helped in their work by the knowledge that supervisors are available to advise them, to solve any difficult problems and so on. It is difficult to assess just how valuable supervision is in this respect. Survey directors have to decide, on more or less arbitrary grounds, how much money to spend on supervision and, as a result, there is wide variation between organizations in the emphasis placed on the supervisory side of field-work.

The other side of supervision is the attempt, by checks, call-backs and occasional supervised interviewing, to prevent the lowering of interviewing standards and to deter and detect the cheat.

More will be said about this in the next section.

It is widely agreed, and of considerable importance for the present research, that experience plays a very considerable part in quota completion and interviewing; that it is more important than formal training, and that it plays a greater part in quota than in random sampling.

5. Checks A point of great importance is whether the honesty and accuracy of field-workers can be checked. Every organization makes attempts at checks as much in the hope of deterring the cheat as of detecting him. Some checking is possible if the respondents can be traced, and all the organizations now ask for the names and addresses of interviewees (except, in the case of one organization, on political surveys).

Organization A: Supervisors actually call back on 10 per cent. of the respondents, and this is thought to bring about "a fairly healthy respect for the sanctity of quota controls". A check is then made on whether the interview actually took place and on the accuracy of the classification data. N o tabulated results of these checks are available; but, as regards the former, actual dishonesty is said to be quite rare. In the case of classification data, it. is a matter of disagreement between interviewer and supervisor rather than of inaccuracy, and action is taken only if such disagreements are very frequent for one interviewer.

Organization B makes a "postcard audit" on whether the interview took place or not. For each interviewer, one day of his assignment is selected and a postcard is sent to every contact made on that day. About 65 per cent. of the cards come back, nearly all (64 per cent. of the total) saying that the interview did take place. The remainder are either not returned, or come back via the G.P.O. marked "not known" or "insufficient address". If the results seems unsatisfactory, a second day from that particular assignment is checked.

Organization C until recently recorded only the location of the interview, not the address of the respondent. Various checks are now being tried experimentally, partly attempting to

420

MOSER-Quota Sampling

[Part 111;

overcome the weakness of postal checks, i s . , the usually large percentage of cards or letters which are not returned. No tabulated results are yet available.

Organization D makes a postal check on 10 per cent. of the intcrviews. In addition, a 100 per cent. check is made on two or three interviewers selected in turn and upon anyone whose work seems unsatisfactory. On the former a 60 per cent. return rate is usual. Occasionally someone complains that no interview was made. In such a case an explanation is asked for from the interviewer. A small proportion-perhaps 3 per cent.--is returned via the G.P.O., marked "not known" or "insufficient address". Again an explanation is demanded and is quite often satisfactory-there may have been an error in recording the address, or the contact may have moved to a new one without notifying it. In this organization the loss of interviewers through suspected "cheating", in the sense of not making an interview, is between 12 and 15 a year, which compares with the total turn-over rate of some 300.

Nobody claims a great deal for these checks. There is more faith in their psychological value in discouraging the cheat than in their power of detection. Something could be done to improve their value in this direction. Furthermore, none of these checks help in assessing the quality of the quota completion or the interviewing performance.

6 . Refusals It is sometimes said that quota sampling avoids the problem of refusals. This is, of course, not true. Quota sampling hides the influence of refusalq, and we must try to ascertain to what extent refusals bias the samples. Unfortunately, very little information indeed is available on this in any organization.

In Organization A field-workers are supposed to complete forms for all unproductive interviews (this includes-and investigators are to distinguish between-refusals and contacts who were "wrong for the quota"). Few refusals (no accurate figures can be given) are found. Interviewers are asked to record the age, sex, income grade and occupation of the refusals as well as their reasons for refusing.

Organization B asks interviewers to record numbers of refusals, and estimates that about 2 per cent. of persons approached d o actually refuse.

Organizations C and D ask for the numbers of refusals to be recorded on some surveys, but no tabulated findings are available.

In short, we are very much in the dark about the numbers and the characteristics of people who refuse to be interviewed.

7. Actual Field Practice It would have been useful to have given an account of the way in which interviewers actually go about their work, how they plan their quota in the first place, how they plan each day, where they look for their various quotas and so on. T o do this at all properly would have meant watching investigators at work or at least talking to and questioning a large number of them.

This was not feasible. It has been possible, however, to talk to the field supervisors and to form a general picture of field practice.

The chief impression is of how much freedom for individual judgment and variation there remains for the interviewer, after allowing for the various controls and instructions which have been discussed above. By and large, each interviewer can choose whether to cover a small or a large area; whether they interview in the street, in factories or on the doorstep; in the morning, afternoon or evening. The quota cells are generally fairly wide, so that interviewers experience little difficulty early on, although the cases at the end may be quite hard to find.

Here we meet two different approaches to the task. Some interviewers like to spend the first day or two of their assignment looking for the types of persons whom they know to be difficult to find, and hope to spend the rest of their time filling in the easier cells iri a more leisurely way.

The other and, it would seem, much more common approach is to try to get well ahead of schedule in the first days by starting on the easy quota cells, and then to turn to the difficult part ot the assignment with plent] of time to spare. On this approach, investigators tend to start with some street interviewing, turning to the factories and the homes later for their special classes.

During the early, easy stage the investigator is, of course, also onthe look-out for persons to put

19521

Mom-Quota

Sampling

421

into the dficult cells. It has not been easy to find out which the generally difficult cells are, and no figures afe available to show how long it takes investigators to complete various parts of an assignment.

Towards the end of an assignment, whether the scheme consists of interrelated or independently set controls, the interviewer may experience considerable difficulty. One wonders how much dishonesty there is at this stage. It must be very tempting for an interviewer, needing a Female, Class A, aged 65, to "consciously misjudge" a respondent's class to be A, when it is obviously B .or C. No findings on this kind of cheating are available, but the danger is widely recognized. One organization instructs its investigators not to spend a long time looking for the "last person", but to try to get as near to the requirement as possible and then to record the characteristics as found, and on no account to force the person into the wrong quota cell. This is a realistic recognition of the problem.

It may be noted that classification questions are generally asked at the end of the interview.

This is quite safe early in the assignment, when cases are required for all the cells. Towards the end, however, it would lead to the wastage of a number of interviews. One organization allows the classification data to be asked at the beginning if only a small.and difficult part of the quota remains. Another always has the classificatory questions asked at the beginning, recognizing that this may affect the quality of the interview but wishing to avoid wasting any interviews.

PART111.-The Problems for Research On the basis of this survey of current practice, we must now attempt to decide which are the critical aspects of quota sampling on which our research should be focused. We shall not discuss the methods or forms of the experiments, but will merely list the problems requiring attention.

The danger of any research is that too much will be attempted in one experiment. This particular project will certainly need several lines of attack, and a series of experiments, rather than a single one. We shall try to distinguish as far as possible between the major problems, around which the research should be planned, and the subsidiary ones, which should be included if and when the design of the experirhent permits.

One factor increases the difficulty of this research enormously, namely, the absence of data against which the validity of quota samples can be checked. The most easily available data-such as age and sex-are, by definition, correct on quota samples. On other data there are few reliable checks. The results of the 1951 Census will substantially alter this situation, but they will not be ready for some time. The results of random samples-although these may themselves not be properly representative-constitute the best standard available, and will have to be used in judging the accuracy of quote samples.

The Major Questions 1. Representativeness within Quotas Three main groups of possible bias in quota sampling can be distinguished: that due to the basis or definition of the controls; that due to unrepresentativeness within the quota cells; and that due to the 'peculiarity of the interview situation. Without doubt, it is the second of these which constitutes the main danger of bias and which is the top research priority. (a) Are certain types of person systematically under- or over-represented in quota samples?

The collection of detailed classification data on quota samples would show whether an adequate spread is secured within quota cells. If there is any bias here, what are its origins and causes? If this detailed classificatory data were obtained by a large number of interviewers, one would find out the degree of variability with respect to the various factors. This could be compared with similar random material.

(6) Related to the above point ia the question whether certain types of areas are systematically under- or over-represented in quota samples. Furthermore, is the possibly inadequate spread obtained by street and home interviews corrected by factory and office interviews ?

These questions of geographical spread are now, in fact, being investigated. The addresses of respondents interviewed in one town have been taken from the samples of the four organizations, and are being plotted for comparison with random samples for the same area, and with a population density map.

MOSER-Quota Sampling

[Part 111,

(c) T o what extent can any tendency of quota samples to be unrepresentative-with regard to people or areas-be corrected by stricter instructions to interviewers, more extensive training or, most important of all, by the use of further controls? Supplementary to this, what are the effects on interview quality of further restricting the interviewer's freedom and what is the effect on costs?

2. Checks Can currently used checks be improved and others developed so that there is a far stricter control over interviewer honesty and accuracy? Good checks are especially vital on two questions : whether the interview actually took place, and whether any distortion of classification data takes place in order to fill the quotas. Checks on the quality of the interview are, of course, needed in all types of sampling, and are not a special concern of this research.

3. Training and Experience It would be useful to evaluate the importance of training and of experience of interviewers as regards the speed and accuracy of quota completion and the representativeness of the final sample.

4. Social Status What is the extent of disagreement and reliability in placing respondents in social class groups?

Is there a biasing tendency in the grading? (This is a problem in which sociologists would probably have considerable interest.) Can anything be done to make this sort of control more reliable and less vague?

5. Refusals An effort must certainly be made to gain some knowledge of the unsuccessful interviews.

How many people refuse, what sort of people are they, and what are their reasons for refusing?

The Subsidiary Questions 6. The Quota Scheme Is bias introduced by the setting of independent rather than inter-related controls due to the possible "pairing" effect? How does interviewer efficiency differ on the two methods?

7 . Cost

What are the costs of the various components of random and quota sampling respectively?

Data on this can be obtained quite easily as a by-product of the experiments.

8. ClassificationData Is it better to ask the classification data at the end of the interview, thus risking interview wastage, or to ask them at the beginning, with the risk that the quality of the interview may be affected?

9. Location o f Interviews T o what extent is the quality of the interview affected by its location, i.e., whether it takes place in the street, in an office or factory, or on the doorstep?

Two problems of major importance, but of a long-tern~nature, remain: 10. Can quota sampling be organized in such a way that, while retaining its practical advantages, it also becomes more satisfactory from a theoretical viewpoint? If the interviewer's movements were sufficiently controlled, .if it were possible to sample interviewers at random, could quota sampling satisfy the basic conditions of randomness, so that standard errors could legitimately be attached to the results? This point is related to 1 (c) above and is of critical importance.

19521

MOSER-Quota Sampling

423

11. If certain biases are found to be inherent in quota sampling, when do they matter? This is not a question that can be answered by an experiment, but only by the accumulation of survey data over a long p e r i d . The question really is, to what extent are various data of environment, behaviour and opinions-all the subjects which surveys set out to study--correlated with the variables which are controlled in quota sampling and, more important, with those which are not?

The Stages of the Research.

We are proweding in two stages: (a) We are comparing the results of quota and random sampling. These comparisons are being made on the basis of data collected by the professional organizations in the ordinary course of their work, not by an ad hoc inquiry. The comparisons cover data of a classificatory nature, as well as ordinary survey questions.

(6) The major stage will consist of special experiments, designed to throw light on the problems listed above.