THOSE NOT AT HOME: RIDDLE FOR POLLSTERS By ERNEST R. HILGARD and STANLEY L. PAYNE THE exacting demands of wartime public opinion and market research have made pollsters look even more carefully at the methods through which "natiofial cross sections" of the population are obtained. Here Mr. Hilgard and Mr. Payne look at still another stumbling block-alas, not the last. This article is an analysis of people whom public-opinion interviewers have difficulty
IN
finding at home. What happens to poll results when they are ignored? Mr. Hilgard, on leave as Chairman of the Department of Psychology at Stanford University, is now working on market surveys for the Otfice of .Civilian Requirements of WPB. Mr. Payne, a veteran Government researcher, is with the Special Services Division, Bureau of the Census.
BOTH PUBLIC OPINION and market surveys, interviews are commonly conducted in homes. But some people are hard to find at home. Thus special precautions are necessary to prevent a bias favoring too large a proportion of stay-at-homes. If those seldom home differ in important ways from those usually at home, this bias may seriously distort survey findings which purport to be based upon a representative sample. Two chief methods are in use for assigning the persons to be interviewed in any selected sample area. The first of these is the quota-control system; the interviewer is told to find a specified number of persons fitting given age, sex, standard of living, or other characteristics. The second method is that of specific assignment; the iliterviewer is told precisely where to go and whom to interview. In the method of specific assignment it is possible to insist upon repeated call-backs to assure inclusion in the sample of those hard to reach. Under the quota-control system, on the other hand, if the interviewer finds no one at home at the first house approached, he is likely to obtain an interview from a neighbor who is at home. Since the quota assignment cannot specify all characteristics, the sample thus tends to include too high a proportion of people who possess the non-specified characteristics of stay-at-homes. An empirical check of some of the characteristics of those hard to find at home is available from the data of a survey of consumer requirements made in November 1943 by the Special Surveys Division of the Bureau of the Census for the Office of Civilian Requirements of the
THOSE NOT AT HOME
255
War Production Board.l/The method used was that of specific assignment, and records were kept of the number of calls which were made before each of the interviews was obtained. It is therefore possible to make a statistical characterization of the interview sample obtained on the first call, on the second call, and on later calls. Analysis of the interviews obtained on the later calls gives a picture of the kinds of people less often home, and provides basis f i r estimating the distortion which would be produced if they were not reached in the survey. A few further details regarding the conduct of the survey are needed before the results can be properly appraised. The sample design is that being followed currently in the Monthly Survey of the Labor Force. Crews of interviewers work in each of sixty-eight areas carefully chosen to give a representative national sample. Within these areas, which consist of single counties or groupsof adjacent counties, smaller geographical segments are chosen for residential listing. The final sample consists of a series of designated dwelling units described specifically, e.g., "apartment 106 at goo Cold Street," "on Highway 6, the first house south of the Ramsdell School." Told precisely where to go, the interviewer is not permitted to exercise judgment in the choice of the house at which to call. In the case of a national sample of persons, the specific individual to be interviewed is also assigned on the basis of the characteristics of the listed households in the r ample.^ For the survey under discussion, however, the unit was the household rather than the person; the only specification was that the respondent be either the housewife or another responsible person at the stated address-a person familiar with the household purchase^.^ Every attempted interview is reported, including calls at houses found vacant. This method of sampling insures
a
Permission to use data from the Survey of Consumer Requirements of November 1943 has been granted by the Civilian Relations Division, Office of Civilian Requirements, the agency sponsoring the survey, and by the Bureau of the Census, responsible for the field work in connection with the survey. ZStudies of the attitude and opinion type using this method of sampling have been done extensively by the Division of Program Surveys, Bureau of Agricultural Economics, U.S. Department of Agriculture, headed by Dr. Rensis Likert. Studies of changes in the interview sample with repeated calls, similar to the analyses presented here, were undertaken previously within the Division of Program Surveys by Mr. J. Stevens Stock. These have unfortunately not been published, but the authors take this method of acknowledging the priority of the investigations under Mr. Stock's direction. Special instructions are necessary to define single-person households, to determine when a lodger belongs in the family, when to be counted as a separate household. These details are not considered essential for the present discussion.
2j6
PUBLIC OPINION QUARTERLY, SUMMER 1944
that the sample properly reflects in-migration and out-migration, a matter of considerable importance during wartime. A national sample of 4935 completed interviews was obtained, including urban and rural interviews. Since a higher proportion of rural interviews was obtained on the first call, an analysis of interview results by calls for the total sample would result in an undue weight to farm and rural interviews. For the present analysis, only those interviews are included which were obtained in places with a population of 2,500 or more in 1940. The total sample considered consists of 3265 interviews from as many urban households. Of these, 2072 were obtained on the first call, 726 on the second call, and 467 on the third or later call. Interviewers were instructed to call back on different days and at various hours of the day. The timing of call-backs varied somewhat, according to the convenience of the interviewer, and according to local conditions such as weather and transportation. CHANGES I N THE INTERVIEW SAMPLE FROM CALL TO CALL
The amount of distortion which may result through relying upon accessible respondents can be determined from the differences in the samples resulting on the first call, on the second call, and on the third or later call, as shown in Table I. Since the survey dealt with purchases of kitchen goods, clothing, and other household articles, the "responsible person" interviewed was in most cases a woman, though there are, of course, an appreciable number of households consisting solely of men. As might be expected, housewives employed outside the home, and other employed respondents, were much harder to reach than respondents not employed outside the home. Although only 21.8 per cent of respondents reached on the first call had employment outside the home, 53.6 per cent of respondents reached on the third and later calls were employed. Households with young children are easier to reach than households whose children are older. These in turn are easier to reach than those with no children at all. Most dimcult of all to interview is the person who lives alone; as the number of persons increases, the easier it is to find a responsible person at home. If the survey had ended with the first interview, there would have been too many housewives not otherwise employed, too many families with young children, too few smaller households. To the extent that
THOSE N O T A T HOME
Table
I
Households Households Households interviewed interuiewed interuiewed on fist on second on third or later call call call
Number of urban interviews Per cent
2072
All householdr znteruietued
726
467
100.0
100.0
100.0
3265 100.0
78.2
57.8
46.4
69.1
21.8
42.2
53.6
30.9
17.2
9.5
6.2
13.9
37.6 45.2
343 55.7
33.3 61.5
36.3 49.8
6.3 24.6 25.6 19.9 23.6
13.1 29.5 23.2 16.5 I7.7
15.2 34.6 16.5 12.7
9.1 27.1 24.4 I 8.7 20.7
3.56 persons
3.11 persons
2.84 persons
3.35 persons
Respondent reporting on household purchases I. Responsible
person, not employed outside home 2. Responsible person, employed outside home
Household composition Having children under two years of age 2. Having older children only 3. Having no children I.
Size of household One person Two persons 3. Three persons 4. Four persons 5 . Five or more persons I.
2.
Average size
21.0
survey data are correlated with these family characteristics, real distortion would have occurred. It should be noted that in the method of specific assignment followed here, the first call differs somewhat from the single calls under the quota-control system, since here the first call is restricted to assigned addresses chosen to randomize the sample. The results reported may be interpreted strictly only in relation to the method of specific assignment, although they have obvious implications for the quota-control method.
PUBLIC OPINION QUARTERLY, SUMMER 1944
258
HOW RESULTS OF INTERVIEWS OBTAINED O N LA'TER CALLS
AFFECT MARKET DATA
Because the market for household appliances and equipment is closely related to such factors as size of family, presence of young children, home ownership, and other characteristics differentiating those easier to find at home from those harder to find at home, surveys depending upon single calls will tend, in general, to overestimate the inventory of items 'such as washing machines which are most common among stay-at-homes. The amount of such over-estimation varies greatly from item to item, as shown in Table 2. Table
Final survey estimates
U~ban households having:
(3265
interviews)
Owned home Washing machine Sewing machine Electric iron Radio Mechanical refrigerator
" Difference
9,;oo,ooo I 1,200,ooo 12,800,ooo 20,500,ooo 21,200,000
14,700,ooo
2
Estimates based on first call
Overestimate if Survey had ended with interviews obtained on first call
(2072
interviews)
Number
IO,OOO,OOO ;oo,ooo I 1,800,ooo 600,ooo 13,400,ooo 600,ooo 20,600,ooo IOO,OOO IOO,OOO 21,300,ooo 14,6oo,,ooo -IOO,OOO
Per cent
5.3 5.4 4.7 0.5" 0.5" -0.7"
not statistically significant
For the purposes of Table 2, it has been assumed that there were 23,000,000 urban households at the time of the survey. No official estimate is available, but the precise number does not matter since the figures are presented for expository purposes only. The estimates based on the first call are those which would have been made had the survey ended with the first call, and only the first-call interviews had constituted the sample. The final survey estimates are based on all interviews obtained. Accepting the final survey estimates as the figures more nearly correct, it is evident that there would have been a considerable overestimation of home ownership, washing machines, and sewing machines, if the survey had stopped with the first call. The size of the over-
THOSE NOT A T HOME
259
estimation, around 5 per cent, is beyond the change to be expected through adding 1193similar interviews to a sample of 2072 interviews; the additional interviews in the final survey do not change the estimates simply by increase in size of sample, but by a real change in the character of the population interviewed. For electric irons, mechanical refrigerators, and radios, there are no appreciable differences in the estimated numbers between the households easier and harder to reach. These items are small-family appliances as well as large-family appliances, and are about as likely to be found in apartments as in separate houses. That radios are found about equally among those interviewed on the first call and on later calls does not mean that a radio survey can dispense with call-backs. While no data are available within the survey, it is probable that the radio-listening habits of those who do not stay at home differ from the radic-listening habits of those more often at home, even though radiepossession does not differentiate between them. The findings also suggest that radio-ownership would be a poor factor to use as a check in determining the adequacy of an obtained sample, since the proportion of radios owned by an inadequate sample (such as the first-call sample here) may not differ significantly from the proportion owned by a more adequate one. For sampling households, washing-machines or sewing machines would provide a much more sensitive check, if one accepts the results of Table 2. This follows because the first-call sample-known to differ from the complete sample-shows greater distortion for these items. EFFECTS OF LATER CALLS ON THE RESULTS OF OPINION SURVEYS
The opinion data from the Survey of Consumer Requirements do not lend themselves well to an analysis in terms of repeated calls, because opinion-type questions were asked only of sub-groups self-selected by their buying experiences. For example, people were asked to rate the degree of inconvenience or hardship of being deprived of certain goods which they were unsuccessful in obtaining. Because only the ones who had been unsuccessful in attempts to buy were included among those questioned about inconvenience or hardship, the differences in rating between those reporting unsuccessful buying experiences on the first call and those reporting unsuccessful buying experiences on later calls were too slight to be significant.
2 60
PUBLIC OPINION QUARTERLY,, SUMMER 1944
Since opinion data are often correlated significantly with economic factors, it may be, inferred that what will be true of market data will also be true to some extent of opinion data. Thus the question, "Do you and your family have more money coming in now than before the war, or not as much?" asked without elaboration, is at once an economic and an attitude question, since replies may be colored by a sense of economic well-being as well as by an actual change in money income. Replies to a question of this type conform to those previously reported, considerable difference being found between interviews on the first and on later calls, as shown in Table 3. Table 3
Hoz~seholds interviewed on first call
Number of urban interviews Per cent More money coming in now than before the war About the same Not as much Not reported
2072 100.0
Households intevviewed on later calIs
I93
I
100.0
All households interviewed
3265 100.0
While among those interviewed on the first call 36.8 per cent reported more money now than before the war, the proportion answering in this way of those interviewed on later calls rose to 39.3 per cent. In surveys in which individuals are sought out instead of households, added difficulties arise because it is so hard to reach young people, both married and single, and because it is so easy to reach old people. A quota assignment (e.g., so many under forty years of age, so many over forty) may result in too few in their early twenties and too many sixty and over. The method of specific assignments results in proportionately too many older people on the first visit, but this disproportion is corrected through call-backs which bring in more younger p e ~ p l e . ~ The possible effect on opinion surveys is evident, since young and old often disagree. 41n the studies referred to in footnote 2, Mr. Stock has shown that first calls have an undue proportion of aged people, and an undersalnpling of young adults, especially of those 20-29.
THOSE NOT A T HOME THE CONCLUSION IS
26 I
. ..
People easily found at home on the first call differ significantly from those found at home only after repeated calls. The latter occur in large enough proportions to make it important for repeated calls to be made in order to represent them in sample surveys. Unless such a course is followed, samples will be distorted in the direction of too large a proportion of responses from households with the characteristics of the stay-at-homes. The data here presented from a survey of consumer requirements have shown the extent of the errors to be expected if interviews made on first calls are depended upon.