Interviewer Performance in Area Sampling

Interviewer Performance in Area Sampling

BY DEAN MANHEIMER AND HERBERT HYMAN Area sampling is subject to appreciable error due to interviewer perTOrmance, even in a ca~efullysupervised survey. This conclusion is suggested by the following study, which finds that interviewers can easily make conscious or unconscious errors in listing dwelling units and in the selection of the sample of dwelling units.

Theye is also a possibility of interviewer error in the selection of individuals within dwelling units.

Mr. Manheimer, until recently a Research Associate of the American lewish Committee, is presently afiliated with the Bureau of Applied Social Research of Columbia University. Dr. Hyman is Research Associate in the National Opinion Research Center.

Accounts of the principles of area sampling are often confined to statements of the theoretical nature of the method. Little reference is made to the fallibility of interviewers in following the requirements of the sample design in practice. Thus Hansen and Hauserl state that area sampling: "does not permit the interviewer discretion in the choice of the individuals to be included in the sample. . . the probabilities of inclusion of the various elements of the -population are known, and consequently the reliabilitv of results from the sample can be measured and controlled."

While this is true in theory, it is clear that, in practice, area sampling is dependent on the field staff. The expert designs the sample on paper, but the field worker is his agent. Only insofar as the field worker is willing and able to follow the sampling instructions will the best designed sample fulfill the expert's requirements.

.

-

This article is an evaluation of the performance of one field crew working on one probability sample. It was possible in this survey to measure the degree to which inaccuracy in interviewer performance resulted in errors in the final sample. However, since such methodological problems were secondary to the substantive content of the survey, some of the data relating to accuracy of interviewer performance are incomplete. Nevertheless, the findings are indicative of the ways in which probability samples can go wrong if not rigidly controlled in the field, and the data clearly demonstrate the need for further methodological research to reduce the bias due to human factors inherent in carrying out probability sampling.

Sample Design

The sample was designed for a cornmunity survey which was conducted in the Fall of 1947 by the National Opin"Area Sampling-Some Principles of Sample Design," Public Opinion Quaflerly, Vol. 9, No. 2 (1945), PP. 183-193.

PUBLIC OPINION QUARTERLY, SPRING 1949 ion Research Center in cooperation with the Department of Scientific Research of the American Jewish Committee. A representative sample of the adult population of a large Eastern city was surveyed. Approximately 1,000 individuals were drawn according to the following design:

Large or consistent errors at any of these stages would jeopardize the quality of the results.

These sources of possible error will now be taken up in turn. However, it is important to summarize first the training system for field workers, as well as certain administrative practices.

Poor performance by an untrained, unskilled crew would not be a fair evaluation of the method in general, but would merely reflect incompetence in managenlent of the pa~ticular survey.

All blocks in the city were stratified according to number of dwelling units per block, and section of the city. Within each stratum, blocks were selected at random. The field man then listed all dwelling units on selected blocks in Interviewer Training and a standard order.2 The household on Administration every fifth line of the listing sheet was Listing was done by a crew of thirdesignated to come into the sample. teen persons, seven of whom had previThereafter, from this sample of house- ous experience in listing procedures unholds, one individual within each house- der the Bureau of the Census. All hold was designated for an interview. thirteen, nevertheless, were given two The sample of individuals was chosen days of individual and group training. in a systematic fashion from an enu- In addition, detailed mimeographed inmeration of all adults within the se- structions were provided.

Sampling was under the continual lected households, which was also obsupervision of three trained workers tained by the field man. As many as who remained in residence in the city three calls, if necessary, were then made during the entire survey. The superat each of the designated households to visors advised the field staff and sought secure the interview, and, to reduce any to maintain highest standards among bias due to non-availability, additional the crew. The crew was told in adcalls were made on a sub-sampling basis. vance that all work would be checked, The final survey results were weighted which acted presumably as an addito correct for the sub-sampling of call- tional spur to care and honesty. backs and for the restriction of sampling ' Following usual practice, the listing was to only one individual per household. done by inspection. The field man was inT~ iranslate this sakple design into structed to decide wherever possible by expractice required accuracy On the part amination of the external a ~ ~ e a r a n cof e a of the field staff at three stages: structure whether it contained one or more A.

I.

in the listing

ing units.

2. Accuracy in the selection of the sample of dwelling units.

3' Accuracy in the individuals within dwelling units.

dwelling units. Where it obviously contained many, the structure was entered. When in doubt, the field man was instructed to enter the structure rather than trust to inspection.

Since he had to enter every fifth dwelling unit in order to enumerate the adults in the household, the listing went beyond mere inspection in at least 20 per cent of the instances.

INTERVIEWER PERFORMANCE IN AREA SAMPLING To further insure high standards, the crew was paid by the hour, rather than on a piecework basis, so that there was no incentive to hurry and do shoddy work. T o make the task easier the listing of dwelling units was always done in a separate stage from the final interviewing.

It would seem, given these conditions of careful recruitment, training, and supervision, that errors in performance would be minimal as compared with the more usual survey situation where one of the important differences, among many others, is the lack of direct supervision. The following results should be interpreted in this light.

I. Errors in Listing Dwelling Units The accuracy of the listing process was first checked by comparing the total number of dwelling units listed by the field man for each block with the number reported by the Bureau of the Census in the 1940 Supplement on Block Statistics. While 1940 data may have been somewhat out-of-date, they were still useful as a first source of control figures that could be checked against other local information from the City Directory for 1945 or from local authorities. Moreover, while there had

85

been a considerable population increase since 1940, estimates by Chamber of Commerce officials were that most population changes had occurred in the suburban areas and that the central city population (on which this evaluation is based) was relatively stable.

The over-all number of dwelling units listed for all sample blocks was exceedingly close to the 1940 Census report. The total listed by the crew was 5,583 as compared with 5,576 in the 1940 Census. However, examination of the block-by-block discrepancies between Census and Survey revealed a systematic pattern relating to the economic character of the blocks. The results are presented in Table I in summary form.

Conceivably, these discrepancies could be due to changes in building construction or demolition, or population changes since 1940 which were in turn reflected in the listing process. There was a small amount of new construction within the city which in all probability occurred in higher rental blocks, so that instances of "overlisting" might be ascribed to new construction. The "underlisting" in poor neighborhoods might reflect destruction of slum areas since 1940. While this may have been true in small measure, the evidence given suggests that the underlisting in

TABLE

I

DISCREPANCIES BETWEENNUMBEROF DWELLING UNITS LISTEDIN 1947 AND NUMBER REPORTED I N I940 CENSUS AS RELATED TO THE RENTAL LEVELOF BLOCKS Number of Dwelling Units Reported by: Median Rental Value in 1940

$35.00 or more $25.00 to $34.00 Under $25.00 Totals

Numbel o f Blocks

42 42 56 140

Listers in I947

Census in 1940

Discrepancy

I537 1815 223 I 5583

I423 1661 2492 5576

+I14 f 154 -261 f7

PUBLIC OPINION QUARTERLY, SPRING 1949 poor neighborhoods was chiefly due to a systematic bias on the part (sf the crew. This in turn resulted in underrepresenting the lower income groups in the final sample.

First, local authorities indicated that slum clearance was of small magnitude and could have been responsible for only a small part of the underlisting.

Second, definite evidence that the underlisting was actual error was derived from the administrative practice of check listing. In cases where the supervisor was suspicious of the number of dwelling units listed, or where the discrepancy with Census was very marked or peculiar in character, a second field man was sent back to check the listing.

Only the most experienced members of the crew were used for checking. While it would have been valuable experimentally to make systematic checks on many blocks, it was only possible to do 13 because of exigencies of time and cost. Table 2 presents the results of the checking.

Checking of the nine low rental blocks indicated definite omissions by

the original listers which were of considerable magnitude. Assuming the checker to be accurate, the omission was 13 per cent (57 out of 425 dwellings).

The instances of checking in higher rental blocks are so few that any conclusion is tentative, but there seems to be no error. Consequently, the data would suggest that the bias in the listing process in area sampling is in the direction of under-representing poorer households in the final sample.

Since the survey was not designed for experimental purposes, we have no quantitative data on the causes of such error, although its locus must necessarily be in the interviewer's performance. Complete incompetence or sheer dishonesty do not seem to be involved.

The past experience of the crew, the careful training, the direct supervision, the written instructions-all these suggest that the field staff was operating at a relatively high level of performance. The presence of supervisors, and the statement orally and in writing that the work would be checked should certainly have been a deterrent to flagrant

TABLE z

Median Rental Value in 1940

$35.00 or more $25.00 to $34.00 Under $25.00 Totals

Number of Blocks

1940 Census Report

2

I 02*

2

9

43 467

I3

612

Total Re~ortedbv: Original The Checker Lister

80

Discrepancy

368

79 46 425

-57

494

5.50

-56

4

+I 0

* T h e two high rental blocks checked were ones in which underlisting had apparently occurred (if the 1940 Census were correct). These were checked because it was most unlikely that there had been any destruction or loss of population since 1940 in better income neighborhoods. Therefore the performance of these listers was suspect.

INTERVIEWER PERFORMANCE I N AREA SAMPLING cheating. But there is still the essential factor of the skill on the part of the field worker required in area sampling.

The listing process requires judgment and care, and is probably more difficult in poor neighborhoods because of crowding of dwellings, doubling up of families, conversion of cellars, stores, and obscure places into homes. Illustrations of these kinds of listing difficulties are found in the reports of interviewers working on another area sample. Thus:

87

11. Errors in the Selection of a

of

AS mentioned earlier, the field staff was instructed to enter every fifth dwelling unit in order to enumerate the adult members. Accuracy of the following kinds was required here: I ) Since the sampling ratio of one in five was specified in order to obtain a particular size of sample, mere carelessness in counting would result in the wrong number of individuals in the final sample, although it would probably not "Nowadays casual observation is bias the nature of the sample. 2) Hownot always adequate. I misjudged ever, if a field man arrived at the fifth dwelling unit and then arbitrarily seseveral dwellings myself." '<The listing of dwelling units lected another unit (e.g., the sixth) beseemed to take too much time and cause of some systematic preference for there was no way-of being sure the special types of households, there would information was accurate in my sec- be a bias. This type of error would tion of tenements and light-house- not necessarily be detectable on the surface of the data, since the field man keeping rooms." would presumably mask his behavior practically had to climb back by false entries on the listing sheet. fences and prowl around through Tentative data on both types of ershrubbery and overgrown bushes to rors are available. Carelessness in countfind houses I knew were there." ing was measured by mere examination "Listing in 'blighted areas'-<haof the listing sheets, and was found to otic dwelling unit organization withbe negligible in magnitude, although in buildings made listing nearly impossible." even intelligent, educated interviewers '$The hardest part of the job for occasionally made errors in counting.

Ideal data on biasing errors in the me was listing in a district with unselection of dwelling units could be numbered flats and apartments which derived only from direct observation. necessitated that I investigate in perHowever, inferential data on bias are son location of all dwelling units in available from a comparison of characflats. It is not possible that it could teristics of the sample of dwelling units be as scientific as desired, primarily with control figures.

In our survey, conbecause of listing problems in heavitrol figures for such a comparison are ly populated sections." ( M biggest ~ difficulty) listing derived from the characteristics of a dwelling units in which family apart- supplementary sample of dwelling units merits had been converted to tiny drawn by the office statistician from apartments and sleeping rooms and the same listings as the field staff used. no numbers had been indicated on (In the course of the survey, it was the doors or bells." found desirable to increase the size of

88

PUBLIC OPINION QUARTERLY, SPRING 1949 TABLE 3

Samples Selected by: Field Staff O f i c e Staff

Measures Related t o Economic Status of Household OCCUPATIONAL DISTRIBUTION OF HEAD OF HOUSEHOLD:

Per cent in professional or proprietor classes Per cent in domestic service, other service, or laborer classes

19%

16%

I5

22

$68.80 53.00 40.20

$69.50 50.60 36.50

50% 2.83

46% 2.71

INCOME DISTRIBUTION O F HEAD OF HOUSEHOLD:

75th percentile Median income 25th percentile MEASURES RELATED TO AVAILABILITY:

Per cent of home ownership Median number of adults in household NUMBER O F CASES the final sample of individuals. As the listings of blocks had already been made, the office statistician merely drew an additional sample of households from the same blocks, by starting at another random point and drawing every Nth dwelling unit. The crew of interviewers was then assigned the supplementary sample along with the original sample of dwelling units they had selected for interviewing.) The characteristics of this supplementary sample may be regarded as a norm since the office statistician would have no motivation to bias his selection. Any departure of the field staff from the instructions for random selection could be revealed by the differences in household characteristics for the two samples. It would have been desirable experimentally to have a large control sample and many measures of household characteristics, but necessarily the size of the sample was restricted by cost considerations, and unfortunately measures of only

643

332

four household characteristics were available.

We might expect that the interviewer would have two major motives to bias the selection of households-he might try to avoid poor homes because of the unpleasantness and squalor, and he might try to avoid homes where no one would be available, because he would then have to make a return visit.3 Two of the characteristics available, occupation and income, relate directly to the first motivation since they measure the economic level of the household. The other two characteristics, size of family and home ownership, relate only indirectly to the second motivation since they are poorly correlated with availability. The data are presented in Table 3.

3 Presumably, there could be factors observable to the interviewer before he enters the household which would be an indication that no one was at home; e.g,, windows shut and shades drawn, lights not on, deliveries standing on doorsteps, etc.

INTERVIEWER PERFORMANCE I N AREA SAMPLING

89 It would seem from these data that dwelling units was chosen from these there is a bias in the sample of house- listing sheets it was chosen only from holds selected by the field staff. In some those that had not fallen on the fifth, manner they departed from the in- tenth, etc., lines. In short, it was chosen structions for random selection and from the list of households that was in obtained a sample of somewhat higher no danger of being biased by a tendency economic level than would be expected. of the field worker to omit unattractive This is demonstrated both in the oc- dwelling units.

Second, there are many instances cupational and income distribution^.^ The data on income distribution sug- where there is some equivocality as to gest the specific manner by which such the order of listing dwelling units. bias operates. The bias does not seem Although detailed instructions were to be in the selection of higher income given the field staff regarding the order households since the upper quarter of of listing for unnumbered dwelling the income distributions are about the units in a multiple-dwelling structure, same in both samples. The difference there were undoubtedly instances where becomes evident as one goes down the the complex physical layout of a buildincome scale and is largest in the lowest ing led to ambiguity about listing. quarter of the distribution, suggesting Despite all instruction, the field man, that the field staff tends to avoid the therefore, must exercise some discrevery low level households. The data on tion. In such ambiguous situations, the occupation show this same process. The listing can be so juggled as to permit difference is greatest in the lowest oc- the more attractive unit to fall on the designated lines. cupational category.

There is no significant bias demonWhile we have no evidence on how the field staff manages to introduce the strated with respect to home ownership bias, we can conjecture as to the mech- and size of family. However, since anism. There are two subtle ways by these two characteristics are only slightly correlated with availability, the which the bias could operate: First, a lister might on occasion omit evidence is not sufficient either to deny listing the dwelling unit which would or confirm the hypothesis that bias, normally fall on the fifth line if that arising from a desire to avoid homes in particular dwelling unit seems to him which no one is available, is present. to be too unattractive in appearance H I . Errors in Selection of Individual for him to want to enter it and make an Respondent within Households enumeration. H e might list the very Measures of the third type of error, next dwelling unit (the sixth) in its accuracy in enumerating adults and stead, and omit listing the unattractive selecting the sample of individuals dwelling unit. If unattractive dwelling units happen to fall on lines other than 4 Appropriate tests of significance have been the fifth, tenth, etc., the field worker applied for conditions of block cluster samwould have no motivation not to list pling. The lower representation of the low groups is significant at the 5 per them since he is not required to go into occupational cent level. Tests on the other economic charthese units to do an enumeration. Now, acteristics do not quite reach this level of when the office-selected sample of significance, but are appreciably above chance.

PUBLIC OPINION QUARTERLY, SPRING 1949 within the dwelling units, can be derived from the following comparison.

In the sample of dwelling units selected by the field staff, the field man enumerated all adults within the dwelling units by age as he came to each selected dwelling. He then brought the enumeration of adults to the main office, and the office personnel selected the sample of individuals from the enumerations by reference to a table of random numbers. This final sample of individuals was then assigned for a future interview. In contrast, in the supplementary sample of dwelling units that had been drawn in the office, the field sta8 was required to select the sample of individuals at the time they enumerated the adults. In order to select the individual at random, they were also given tables of random numbers and a system where, by ranking adults in the household by age, they could make an unbiased selection of a specific adult. This selected person was to be interviewed immediately, if at home. Otherwise, the field man was required to call back.

T o summarize, in the office-selected sample of respondents, accuracy in the selection of respondents is not dependent on field worker performance in applying schemes of random numbers,

whereas in the field-selected sample of respondents accuracy is dependent on field worker performance. This difference in the role of the field man is accompanied by a variation in the motivation to bias the selection. In the fieldselected sample of respondents there might be the motivation (and the opportunity) to misapply the scheme of random numbers in order to conduct an interview immediately. In the officeselected sample there would be no motivation to falsify the enumeration, since the interview was not to be conducted at that time, and nothing would be gained. A demonstration that the field-selected sample shows a higher proportion of available or cooperative respondents would be evidence of this motivation on the part of the field workers to bias the selection of respondents.

The results are presented in Table 4.

It can be seen that there is no significant difference between the two samples. It was thought that if there was a bias operating in terms of the field staff juggling ages in order to interview the most available respondent in the household it would show up on the three characteristics compared for the two samples shown in the above table-per cent of interviews obtained on first call,

TABLE 4

Samples Selected b y : O f i c e Staff Field Staff

Per cent of interviews obtained on first call Per cent of first call interviews rated by field man as completely cooperative Per cent of housewives in total sample NUMBER OF CASES

59%

84 35 643

60%

80 34 332

INTERVIEWER PERFORMANCE I N AREA SAMPLING per cent of first call interviews rated as completely cooperative, and per cent of housewives in total sample.

However, because of the two following uncontrolled factors which might have obscured the differences in characteristics between the two samples, the writers feel that the above results are inconclusive and that further studies should be made of possible interviewer bias in the selection of respondents in probability sampling.

First, the field staff was instructed to find out during the course of the enumeration the hours when each adult was likely to be at home. Consequently, in the case of the sample of individuals selected by the ofice clerk, the first call to obtain an interview with the respondent could be made at a propitious time, which might well have resulted in a higher proportion of successful first calls. In contrast, the hour of the first visit in the sample selected by the field staff was a shot-in-the-dark since there was no prior information by which to guide the time of the visit.

Second, the field staff tended to report that the prior call for an enumeration in the office-selected sample had the effect of warming up and preparing the household favorably for a later visit.

Both these factors should have resulted in a higher proportion of success on first call in the office-selected sample.

Therefore there is a possibility that the effect of any biasing behavior would be cancelled out by these two factors so that the net effect would be to show no difference in the data.

The possibility of such a bias is at least indicated in the reactions of interviewers to area sampling. In another sample conducted by N.O.R.C., the comments from the field show that

91 there is no strong feeling about the need to pick a particular person at random.

Despite indoctrination, these interviewers are not completely convinced of the necessity for following the procedure to the letter.

"It would be a great deal less expensive, saving time and money, to be able to interview any one member of the household who was 21 years of age or over who was at home at the time the interviewer made his or her initial visit."

"I have done other interviewing and have always chosen my respondents at random after instructions as to the number in certain age and economic brackets. I try to get a representative group. This time I am not satisfied with interviews obtained."

"I think it would be all right, if the person you were to interview was not at home to take some other member of the family, because usually the opinions in a household are pretty much the same."

"My reaction was it was a considerable waste of time in getting the required number of interviews which have as good an economic spread as had I approached people on my own."

"I think that more interviews could be made if there weren't a certain one to interview. H e or she should be able to select a respondent from the ones at home at time of visit. But be sure not to select the same type of person in each dwelling unit."

In retrospect, it is clear to the writers how such motivation on the part of the field workers could operate to produce a bias despite the apparent foolproof method of random selection of respondents. The design for "insuring"

92

PUBLIC OPINION QUARTERLY, SPRING 1949

unbiased selection of respondents within households was to have the field worker assign a number to each adult in the household according to his rank order of age. H e then applied the set of random numbers so that an adult with a particular rank was chosen. This, presumably, prevents arbitrariness in using the random numbers.

In spite of this, a field man may often encounter situations that are equivocal with respect to the relative ages of adults within households. Since the ages in an enumeration are always reported by one informant within the household, the reported age may well be vague. For example, it is well known that decimal tendencies in the report of age in the Census (e.g., "around 20," "in the 30's," "around 45") illustrate the vagueness of informants' reports. An informant may not know the age of his lodger, or of his parents, or grandparents. A housewife might be reluctant to admit that she is a little older than her husband and might merely say "he's about my age." Such vagueness would offer a fertile situation for the field man, without many qualms, to alter the order of listing.

Merely on these a priori logical grounds, one would argue that it would be important to enumerate and then rank the adults not only by age, but also by sex, which is not subject to any equivocality in r e p ~ r t . ~ Conclusion The possibility that area sampling is subject to bias arising from the performance of the field staff was examined in one survey. Data indicative of such bias in the listing of dwelling units and in the sampling of households at random have been presented. It is hoped that this discussion will encourage improved application of area sampling to future survey research by emphasizing two commonly neglected realms, the practical and the psychological. Only as we realize that area sampling is dependent on the performance of interviewers-and only as the theoretical requirements of area sampling are translated into a body of effective practice-can we completely achieve the goals described by the sampling experts.

5This improved method for insuring unbiased selection of respondents has been put into operation in recent surveys.