Area Sampling or Quota Con trol?-Three Sampling hxperiments BY JOSEPH R. HOCHSTIM AND DILMAN M. K. SMITH Since area sampling is considerably more expensive than quota sampling, it is pertinent to inquire whether increased costs are compensated by suficiently increased accuracy. Three experiments, conducted by the Opinion Research Corporation i n 1945 and 1946, shed considerable light on this question by comparing the results of quota control sampling, area sampling, and "domal sampling" with known data. Area sampling is found to yield the most accurate results, although the less expensive "domal method" is adequate for most cases. In the last analysis, however, the most efficient sampling design (from the point of view of expense) must be determined according to the nature of the problem under investigation. Dr. Hochstim, Chief Statistician for Opinion Research Corporation, holds a Ph.D. in political science from the University of Vienna, and was formerly on the stag of the Psychological Corporation. MY. Smith, Vice-President of Opinion Research Corporation since I9j9, entered the market research field in 1931. Currently, he serves as research consultant to several large industrial firms and trade associations. Recent years have seen heated discus- merits and that each has a sphere in sion about the efficiency and accuracy which it may be properly used.2 of different methods of survey samThe authors are indebted to Mr. Kendrick pling. Area sampling is one of the S. Few for his many helpful suggestions. methods proposed as the answer to a In all discussions of sampling techniques, great many problems. Its proponents one general caution should be kept in mind. have been on the offensive. Quota sam- It would be unreasonable to aim for perfecpling, on the other hand, even though tion in sampling when errors due to wording put on the defensive, still is very widely and sequence of questions and to reliability and validity of the answers might affect the used in market and public opinion results of a survey to a much greater extent research? than errors from sampling fluctuations. We Although discussion has been ex- should always strive For the most efficient tensive, no definitive data on the relative sample, but it would not be wise to double efficiency of the two sampling methods costs in order to reduce a sampling error by I per cent or 2 per cent, especially if little is have been published. Actually, there is done about question wording and other Factors evidence that each method has certain which might possibly affect the results.
74 PUBLIC OPINION QUALRTERLY, SPRING 1948 As for the relative merits of the two (described by R. A. Fisher as the resampling techniques, the defenders of ciprocal of the average of [error]') each method have powerful arguments. over cost. Quota samplers rightfully cite: The nature of the survey, in turn, usually determines efficiency from the a. Considerably lower sampling and point of view of the consumer. In surinterviewing costs. veys dealing with issues and reasons for b. Greater speed in execution of people's opinions (qualitative surveys), sample design and of field work. where the condusions of the user of On the other hand, area samplers the data will be the same whether 70 justly mention: per cent, 75 per cent, or 80 per cent of a. Freedom from the necessity of the respondents hold a certain opinion, estimating the composition of quota samples will usually be betterthe universe and of establishing that is, more efficient. If, however, the quotas. (The longer the time purpose of the survey is to measure since the last census, the greater is exact quantities, as in a sample census the possibility that the composi- or in a survey measuring sales potention of the universe may have tials (quantitative surveys), the area changed. Thus, the sampler is method probably yields more accurate frequently at a loss to make an results. In other words, where we paint intelligent estimate of how large with a broad brush quota sampling is these changes are and even in more efficient, but where we etch with a fine needle area sampling is refera able.^ what direction they go.) Ballot length is another factor inb. Drastic reduction in bias due to fluencing the choice of the sampling interviewer selection of respondents. (When final selection of re- technique. Area sampling will probably spondents is left to the inter- be more efficient with long ballots, viewer, bias may be introduced quota sampling with short ballots. Balbecause of his personal prefer- lot costs are determined by interviewences, convenience, or special ef- ing time, which consists of time spent forts to locate respondents likely in (a) locating the respondent, and (b) actual interviewing. Actual interto give "informed" comments.) viewing time is the same regardless of Criteria for Selection of the type of sampling used, but time Sampling Technique The principal object of each survey sample is to provide the user of the data with the most information at the lowest cost. Consequently, the choice between area sampling and quota control will ultimately be determined by the dollar efficiency of each technique in each individual case.3 Efficiency is defined as the quotient of information
a J. Stevens Stock will show in a forthcoming article how the dollar efficiency of a sample can be computed algebraically. 4There are times, however, when a qualitative survey might become a quantitative one, as is the case when an opinion survey is repeated for the purpose of measuring trends. Here the sample must be kept strictly comparable from one survey to the next in the same way that question wording and sequence must remain unchanged for accurate time-series measurement.
AREA SAMPLING OR QUOTA CONTROL? spent in locating respondents is considerably less when interviews are assigned by quota. Since the relative importance of this time increases inversely to the length of the ballot, the ratio of the cost of locating respondents to total ballot expense is more favorable in the case of a long ballot. In practice, a compromise solution such as the dwelling unit or doma15 sample described below may often be the most eficient one in market and public opinion surveys. Its advantages are: a. Elimination of the worst biases of quota sampling. b. Relatively low cost. Area sampling is considerably more expensive than quota sampling. Since area sampling is being used increasingly, it is important to determine if the additional expense is justified. Accordingly, Opinion Research Corporation has conducted a number of experiments, three of which are reported here. The problems investigated in these experiments were: I. What
are the differences in composition between a quota-control sample and a stratified block sample? (Experiment I ) 2. What happens if the freedom of the interviewers in selecting households within the block is restricted? (Experiment 2) 3. T o go one step further, what is the result when we control the selection of respondents within households and require callbacks if respondents are not at home? (Experiment 3)
Experiment I : Block Sample versus Quota Sample
Eleven cities, each with a population of more than 50,000, were selected. Two interviewers in each city were assigned the same number of ballots. While one interviewer was given a simple quota assignment by sex, age, and socio-economic status, the other interviewer received a block assignment with quotas for sex and age superimposed. The blocks were stratified by census tracts and by average monthly rent. Blocks within each stratum were selected systematically and interviews were assigned in proportion to block size. Instructions to the interviewers called for respondents in poor, average, and prosperous homes in the same proportions as they occur in the blocks. In order to keep the variable of individual interviewer ability constant, the experiment was repeated in a second survey. The same eleven cities and the same twenty-two interviewers were used, but the quota and block assignments were reversed. A number of questions designed to measure certain known population characteristics were included, and the samples obtained were compared with Census data. Results: In both surveys, the block sample showed less bias on education than did the quota sample. (See Table I ) Even in the block sample, however, "Domal" from the Latin domus is defined by Webster as pertaining to the house. The term "domal sampling" as introduced here refers to a sample in which not only blocks but houses and dwelling units within the blocks are predetermined in a systematic way with quotas for one or more characteristics such as sex, age, etc. being assigned.
PUBLIC OPINION QUARTERLY, SPRING 1948 TABLE
I
Actual Figures ( I 940 Census)
Respondents Monthly rent-fourth decile median sixth decile Median number of school years completed Per cent having grammar-school education or less Per cent having high-school incomplete Per cent having high-school complete Per cent having college incomplete Per cent having college complete Per cent owning home Per cent having telephone Per cent having mechanical refrigeration Per cent having automobile
$26.15 29.69 34-04
Survey Figures 1945 and 1946 Block Sample Quota Sample
293 $33.24 36.69 40.34
298 $34.51 39-71 47.69
9 45 I8
23 8 6 44" 63t 69' -$
*Opinion Research Corporation estimate based upon U.S. Census releases, 1945 and 1946.
Research Corporation estimate based upon AT&Treleases, 1945.
S No data available.
t Opinion
there were still too many respondents with college and high-school education and too few with grammar-school education or less. The block sample also showed some improvement over the quota sample with respect to average monthly rent. Census data on rent shown in Table I refer to 1940. Since the surveys were made in 1945 and 1946, the monthly rent reported by respondents was expected to be higher than that shown by the 1940 Census. On three other variables-home ownership, telephone ownership, and mechanical refrigeration-the two samples compare reasonably well with each other and with the estimated actual. No au-
thoritative figures on automobile ownership in the eleven cities were available for comparison. The results from the two surveys, however, are identical. One more point remains for consideration. Even though discrepancies between the two samples might not appear unreasonable, it is still possible that bias might be present in either or both of them. In fact, when respondents were asked if they had ever been interviewed before, 13 per cent in the block sample and 19 per cent in the quota sample said yes-a possible indication that quota interviewers are inclined to approach the same persons over and over again.
AREA SAMPLING OR QUOTA CONTROL? Experiment a: Block Sample versus Domal Sample The purpose of this experiment was to compare a block sample of the type described in Experiment I with a sample wherein both the blocks and dwelling units within the blocks were predetermined systematically so that bias due to interviewer selection would be minimized. The blocks of a Midwestern city of about ~oo,ooopopulation were stratified as described in Experiment I and the sample blocks were numbered consecutively. Assignments in each odd-numbered block called for selection of poor, average, and well-to-do homes proportionate to their occurrence in the block. In even-numbered blocks, however, interviewers were instructed to interview in every nth dwelling unit, proceeding clockwise from a predetermined starting point. A different starting point was selected for each block (for example, house at the NW corner, second house from the NE corner). Corner houses and different sides of the block were given proper representation. When interviewers found nobody at home, or were refused, the interview was taken in the next dwelling unit. The count from one sample dwelling unit to the next was carried on as if the interview were taken in the home originally designated. Because of the nature of the survey, interviewing was restricted to employed persons. Consequently, most of the interviewing was done during late afternoons, evenings, and weekends. Therefore, the number of substitutions was relatively small. In both samples sex and age quotas were superimposed, and to eliminate variations due to interviewer ability
77
both types of assignments were given to all interviewers. Results: The domal sample came closer than the block sample to Census data in the variables tested: rent, education, home ownership, refrigeration. (See Table 2) Thus, the less freedom the interviewer has in selecting dwelling units, the more representative is the sample. Experiment 3: Domal Sample without callbacks versus Area Sample with callbacks6 The area to be sampled in this experiment covered two adjacent counties in Ohio comprising: a. A city of over 200,000 population. b. A number of smaller towns between 2,500 and 25,000 population. c. Villages, open country, and farms in rural areas. Interviews to be taken in each of these areas were assigned in proportion to their populstion. The blocks in the central city were stratified by Census tracts, rent, and number of dwelling units per block, in accordance with 1940 b'lock statistics. Since 12 per cent of all dwelling units in this city were constructed since the 1940 Census, data on their number and location were obtained from the city planner. A sample of blocks was taken in such a way that old and new dwelling units were represented in their proper proportions. Block statistics were not available for the other urban places in the same The authors are indebted to Mr. Frank C. Weber, Vice-President, D'Arcy Advertising Agency, for his suggestions in this sampling experiment and for his permission to publish the results.
PUBLIC OPINION QUARTERLY, SPRING 1948 TABLE
2
Actual Figures (1940 Census)
Respondents Monthly rent-fourth decile median sixth decile Median number of school years completed Per cent having grammar-school education or less Per cent having high-school incomplete Per cent having high-school complete Per cent having college incomplete Per cent having college complete Per cent owning home Per cent having mechanical refrigeration Per cent union members Per cent Negroes
* Opinion
$24.62 27.55 30.71
8 5O
Survey Figures 1946 Domal Sample Block Sample (Automatic (Interviewer selection of selection of households) households)
452 $35.49 39.15 44.35
495 $37.83 41.94 47.23
10
II
33
33
Research Corporation estimate based upon Census releases, 1945,
1- No data available.
ple. Blocks in these towns were numbered consecutively on maps in serpentine fashion, and every nth block was selected for the sample. For rural areas, minor civil divisions were used as primary sampling units and were stratified into groups by size, local option for alcoholic beverage^,^ and distance from the nearest town. Within each group the minor civil divisions were selected at random for the sample. Altogether, seventeen minor civil divisions were chosen, and interviews were assigned in proportion the size of each group.
Within the minor civil divisions, separate samples were taken in villages, in unincorporated rural nonfarm territory, and on farms. In villages, interviewers were assigned every nth dwelling unit on one side of the street. For the open country and farm assignment, square-mile sections were numbered on state highway maps, and every nth section was selected. Interviews were taken at every nth dwelling unit within each section in the sample. Interviewing Procedure. In the city, 7 This stratification was necessary because of the nature of the survey.
AREA SAMPLING OR QUOTA CONTROL? towns, and rural areas, the sample blocks and square-mile sections were numbered consecutively. Split-sample technique was used. All odd-numbered blocks and square-mile sections were put in Group A (the domal sample) and all even-numbered in Group B (the area sample). Interviewers were given starting points and instructions on how to proceed within each block. I n the domal sample, interviewers were told to start at the nth house from the N W (or N E or S W or SE) corner, to proceed clockwise, and to interview in every nth dwelling unit. Only one interview was to be taken in each dwelling unit. In cases of noninterview or refusal, interviewers were instructed to substitute the next dwelling unit, but the count was carried on from the dwelling unit originally designated. Sex and age quotas were assigned. In the area sample, interviewers were given starting points and directions on how to proceed within the block, as in the domal sample. There were, however, no sex and age quotas. Instructions were to interview all persons 18 years old and over in each dwelling unit assigned. Two callbacks were made wherever necessary. In no instance was it left to the interviewer to decide where and whom to interview. Only 2 per cent of the persons in this sample were not reached after two callbacks. All interviewers participated in both parts of the survey. Interviewing was equally divided throughout the week. Results. A comparison of the two obtained samples, as set forth in Table 3, shows very close agreement. This is the more remarkable since the ~ o p u l a tion had increased sharply in the survey area. Consequently, changes in the char-
79
acteristics of the population were di&cu~ltto estimate. An experiment on callbacks reported by Ernest R. Hilgard and Stanley L. Paynes has yielded results which showed differences between first call and later calls. It should be kept in mind, however, that even if there are differences between respondents reached at the first call and those reached at later calls, these differences may not affect materia~lly the total results of a survey unless a relatively large proportion of respondents is not reached at first. Conclusions
In summary, the conclusions from our three experiments are: I. Block samples tend to result in a more representative cross section than quota samples. 2. Mechanical and automatic selection of dwelling units makes the cross section more representative and less subject to bias. 3. The extra expense of callbacks is not always necessary. Experiment 3 indicates that under certain conditions a carefully selected domal sample will produce about as good a cross section as will an area sample. The results of our three experiments are reported here for the benefit of those who are interested in this problem. Each technique reported has its merits. But there will be cases in which one technique has definite advantages over the others; and there also will be cases where a compromise solution, 8
"Those Not at Home: Riddle for Poll-
sters," Public Opinion Quarterly, 8, No. 2
(1944), 254-261.
80
PUBLIC OPINION QUARTERLY, SPRING 1948
such as domal sampling, is most efficient. In any instance the problems presented by the individual survey will
determine which technique is to be used-domal sampling, area sampling, or quota control.
TABLE 3 Survev Fi~ures1946 Area Sample Domal Sample (Automatic (Automatic selection of zelection of households and households; no respondents; callbacks) callbacks)
Respondents Monthly rent-fourth decile median sixth decile Median number of persons per household Median number of children under 18 years Median number of school years completed Per cent with grammar-school education or bless Per cent with high-school incomplete Per cent with high-school complete Per cent with college incomplete Per cent with college complete Per cent manual workers Per cent employed in manufacturing industries Per cent foreign born Per cent with foreign-born father Per cent with foreign-born mother Per cent Negroes Per cent married Per cent owning home Per cent having telephone Per cent having mechanical refrigeration Per cent having automobile Median monthly grocery bill