CONTROLLING O R D E R - E F F E C T BIAS

CONTROLLING O R D E R - E F F E C T BIAS

BY WILLIAM D. PERREAULT, JR.* Researchers have long been aware of the importance of questionnaire and interview schedule design on the quality and quantity of response.

Detailed thought is given t o the type of questions that should be asked and to how they should be worded and sequenced. Selecting the appropriate sequence for the questions is a complex issue in questionnaire design. The researcher must be careful to place questions so that he evokes and maintains the respondent's interest, stimulates his attention, and in some cases even overcomes his resistance to answering questions. For example, questions which are of a personal nature ("what is your income?") or those which might prove sensitive to the respondent ("do you approve of interracial marriage?") are frequently placed at the end of the questionnaire. It is felt that by the time the respondent has reached the end of the questionnaire he will be more likely to have adopted a positive response set, and even if he is offended by a question it will not have influenced his response to the other questions. Thus, in most situations, the sequence of questions is a positive factor that opinion researchers use as a tool to improve the quality of the research instrument. Unfortunately, the position of a question may also exert a negative bias on response.

THE PROBLEM

The relative position of an item in an inventory of questions or stimuli may uniquely influence the way in which a respondent reacts to the item.

This phenomenon, referred to as "order effect," may be attributable to any of a number of factors. Landon suggests that early items in an inventory may tend t o act as an "anchor" upon which subsequent responses are made.' From a similar perspective, Kornhauser and Sheatsley note that earlier items of an inventory may create a response set or expectation that influences response to later items.2 Some bias may result from dissonance; in the vein of Anderson, Taylor, and Holloway, the re-

* The author is Assistant Professor of Marketing in the College of Business Administration, University of Georgia, Athens. ' E. Laird Landon, Jr., "Order Bias, The Ideal Rating, and the Semantic Differential," paper presented at the Fall Conference of the American Marketing Association, Boston, Massachusetts, August 1970.

Arthur Kornhauser and Paul B. Sheatsley, "Questionnaire Construction and Interview Procedure," in C. Sellitz, M. Jahoda, M. Deutsch, and S. Cook, Research Methods in Social Relations, New York, Holt, Rinehart and Winston, 1959, pp. 546-574.

ORDER-EFFECT BIAS

545

spondent may begin to alter his true response pattern-as he progresses through an inventory-in a n attempt to be consistent with earlier judgm e n t ~ Sometimes, .~ it is simply the (relative) position of an item which tends to elicit a particular response, regardless of item content. For example, Silk discusses an inventory of items which call for the respondent to check one of two possible alternatives; he noted a respondent bias to checking the first alternative for each item, although this led to an inconsistent response pattern when the whole inventory was Although such order effect is commonly recognized as a source of bias in survey research, it is frequently ignored. This lack of rigor is explained by two considerations. First, models for statistical estimations of order effect are complicated and, more important, appropriate only in restrictive cases.5 Second, if order bias is not treated statistically, it should be controlled by randomizing its effect across respondents. This type of control requires that the researcher produce different questionnaires, comprised of random orderings of relevant items (while maintaining proper overall sequence) for each respondent. The initial expense and clerical difficulties of producing many different questionnaires are obvious and are further compounded by the problems of coding the data from the completed questionnaires into a consistent (machineinterpretable) form. ~ t f i r s thought, t it appears that the ordering and printing of items for a questionnaire could be easily accomplished with the computer. Hughes and Guerrero have, in fact, used on-line terminals and interactive designs to control order bias: each subject was presented stimuli in a random this is a creative solution to the order-bias problem, ~ e q u e n c eAlthough .~ it is also a highly restrictive one. It is certainly not feasible in most public opinion research situations. Their application, however, demonstrates that the problem is not in developing algorithms to (randomly) order survey items; that can be done with ease in any computer language. The problem is in printing questionnaires which have an acceptable appearance-an appearance that will stimulate response. When the order of the questions is random, programing a set of decision rules to control the format and appearance of the finished questionnaires is a timeLee K. Anderson, James R. Taylor, and Robert J . Holloway, "The Consumer and His Alternatives: An Experimental Approach," Journal of Marketing Research, Vol. 3, February 1966, pp. 62-67.

Alvin J. Silk, "Response Set and the Measurement of Self-designated Opinion Leadership," Public Opinion Quarferly, Vol. 35, 1971, pp. 383-397.

See, for example, R. Darrell Bock and Lyle V. Jones, "The Measurement and Prediction of Judgment and Choice," San Francisco, Holden Day, 1968, pp. 75-84; Robert B. Kane, "Minimizing Order Effects in the Semantic Differential," Educafionaland Psychological Measurement, Val., 31, 1971, pp. 137-144.

G . D. Hughes and J. L. Guerrero, "Simultaneous Concept Testing with Computer Controlled Experiments," Journal of Markefing, Vol. 35, 1971, pp. 28-33.

546

WILLIAM D. PERREAULT, J R .

consuming task, and one which in most cases demands considerable programing skill. As a simple example, it might be desirable to print two or more text columns on one page (particularly if standard-size unlined computer paper is used). With the order of the items varying, this would be a considerable problem. Similarly, "keeps" (several lines that should always appear intact in the same text column), upper- and lower-case letters, page numbers, and many other details add difficulty. In short, these format problems have discouraged researchers from employing the computer to control order bias by randomizing its effects.

A SOLUTION

There is, however, a simple way to overcome many of the problems of different questionnaire formats while still1 maintaining the advantage of using the computer to order the items to be printed. The researcher (even o n e with only novice programing skills) can easily produce questionnaires by combining the power and flexibility of "canned" text processing programs with a simple computer program to order the input to the text processing program. Before proceeding with a specific explanation of this type of application, a brief mention of the general capabilities of text processors may be useful to unfamiliar readers.

Computer Text Processing A number of text processing programs are readily a ~ a i l a b l eTypically, .~ input to the programs is a free format "text-string," which consists of both normal text words and editing-specific control characters. The computer reads the text-string, and prints the words according to the control characters. For example, different control characters are used to indicate what material should be kept together in one text column, when new lines, paragraphs, or pages should start, when text is to be capitalized, etc. These control characters, then, determine specific aspects of the format of the output.

The user also controls more general aspects of the format by specifying several control cards. These control cards determine such editing parameters as print column width, number of print columns per page, spacing Although by no means an exhaustive listing, several of the more popular programs are discussed in International Business Machine Corporation, IBM TEXT 360: Reference Manual and Operating Guide, White Plains, N . Y . , IBM Corporation, 1969; H. S. Weiner and P. D. Reilly, TYPIST: A Text Editor Format Program, Ithaca, N . Y . , Office of Computer Services, Cornell University, 1973; Gerald M. Berns, The F O R M A T Manual, Wheaton, Maryland, IBM Washington Scientific Center, 1971; R. Rich, J. Olmer, G . Trotter, D. Brocklebank, and G . Prophet, Info 360: The J H U I A P L Information Package, Silver Spring, Maryland, The Applied Physics Laboratory, Johns Hopkins University, 1972.

ORDER-EFFECT BIAS

547

between lines, whether or not the print column is to be "right justified," etc. By simply changing one of these controls, the user may easily alter the format of the output. Most of the canned text processing programs are simple to use and require little or no prior computer experience.

These programs are most frequently used for producing documents where it is useful to be able to update and modify the content or general format of the text without retyping the whole document.

The advantage of such programs for producing questionnaires rests in the fact that the specific format-control characters are part of the textstring input in the program. The total text-string (in this case, the questionnaire) is broken down into segments (i.e., questions, scales directions, etc.). Each segment contains both the appropriate text and appropriate format controls. These text-string segments are then read into a computer program and reordered. As with any computer problem, the logic of the reordering is completely specified by the programer. Therefore, the researcher may specify broad sequencing specifications that must be met while randomizing the sequence of other material where order effect may be a problem. The result of the reordering is a new, aggregated questionnaire text-string. The controls in that text-string continue to determine the specific aspects of the format of the material. An example may further clarify the application.

AN APPLICATION

A recent (small sample) study which focused on industrial purchasing managers' evaluation of alternative suppliers helps to illustrate the technique. Members of the sample were mailed questionnaires consisting of two types of questions. The first set was a series of paired comparisons in which the purchasing manager indicated the preferred supplier of each pair. In the second set, the respondent indicated the importance of different supplier characteristics by completing a constant sum scale. As in most survey research, there were sequencing requirements for this questionnaire. It was inappropriate for respondents to answer the constant sum scale questions until after the paired comparisons were completed. Within this overall sequence, however, there was an order-bias problem. Pretests (with the series of paired comparisons in a fixed order) indicated that order effect was a potential source of bias.

T o produce questionnaires that controlled this bias with a random ordering of the paired comparisons, while maintaining the broader sequence requirement, a FORTRAN preprocessing algorithm was used in combination with the FORMAT 4 text processing p r ~ g r a m . ~ Berns, op. cit. This particular algorithm was chosen because it is one of the more economical text processing programs and is simple to use.

548

WILLIAM D. P E R R E A U L T , JR.

Procedure The text of the questionnaire was broken into those smaller groupings of phases which would ultimately be printed together in the finished questionnaires. For example, each set of directions and each paired comparison was treated as a distinct segment. Each text segment included the control characters to determine the format of that particular portion of the finished questionnaires. These segments were given indentification numbers and incorporated in the FORTRAN algorithm.

The logic of this algorithm reordered the segments into a total textstring. Each questionnaire started with a unique code number and a set of directions. Next, the algorithm generated random numbers and ranked them. This ranking determined the order in which the paired comparison segments were to be combined. Next, another set of directions was added to the text-string, and finally the set of constant sum scales. A simple index number was used t o number the questions sequentially as they appeared. This procedure was repeated in an iterative fashion until a questionnaire text-string for each person in the sample had been specified. Each questionnaire, therefore, met a set of general sequencing requirements: each began with a code number and a set of directions, followed by a series of paired comparisons, followed by a second set of directions, followed by a series of constant sum scales, and concluding with a "thank-you" message and information concerning return of the forms. Within this overall sequence, however, the order of the paired comparisons was randomized.

Thus, the FORTRAN computer program took care of the ordering of the questionnaire material. Next, the FORMAT 4 program controlled the format of the questionnaires and printed them, taking care of such appearance details as the maintenance of consistent margins, page numbers, and spacing. The questionnaires, which were printed on 8 1/2-inch by 1 1-inch paper with upper- and lower-case letters, appeared to have been individually typed.

Coding of Data One problem of such individual questionnaires is the coding of the data to a usable form. This problem was also overcome. The application discussed above also helps to illustrate this point. As was noted, each time the program produced a new questionnaire a unique code number was assigned and printed. For each questionnaire, the program also produced a corresponding READ statement in computer card form. (In the FORTRAN computer language, the R E A D statement is used to control the input of data to the program.) This READ card was appropriately punched with a list of variables in the same order they had appeared on

ORDER-EFFECT BIAS

549

the questionnaire. When the completed questionnaires were returned, they were coded; first, the questionnaire identification number was coded, and then on subsequent card images the rest of the data were coded in sequential columns. The data were punched, and then prior to the statistical analysis a program was written that performed the "unshuffling" of the data. The unshuffling procedure is simple. First, the questionnaire identification code is read in and used by the computer to identify which questionnaire is being read; then that information is used in the program logic to direct control to the appropriate READ card (generated in the earlier step). The data are read into the machine in this fashion and stored in the computer memory bank, and then may be written out again, in the order specified by the programer, to tape or cards for further processing. This approach will prove effective even with elaborate data format schemes.

Another approach is viable with respect to the coding of the data. It is conceptually simpler than the preceeding approach, but more difficult from the standpoint of clerical effort. Even with standard questionnaires, researchers frequently provide data coders with coding information. This most frequently takes the form of small numbers placed in parentheses unobtrusively in the margin. The numbers represent the columns of the computer coding sheet (computer card) in which the response is to be recorded. In the traditional questionnaire, these numbers would typically be sequential. This approach may also be used with questionnaires with randomized order. The text-string segment for any given question would simply include the controls to print the appropriate column numbers in the margin beside the question. With this approach, the human coder would bear the responsibility for coding the data in the specified columns. Unlike the case of the traditional questionnaire, the coding columns would typically not be sequential.

Because the first approach is really a rather simple computer procedure, it will normally prove more efficient than the second, coderoriented approach. Furthermore, the different ordering of the data on each questionnaire would undoubtedly make the rather dreary task of data coding even more burdensome and error-prone for the human coder.

Response Rate The use of this procedure may have an additional benefit in terms of response rate. In the purchasing manager study discussed above, it was pointed out to recipients of the questionnaire that no one else would receive a questionnaire identical to theirs; of course, the individually typed appearance of the questionnaire reinforced this statement. About fourfifths (21 of 26) of the sample completed and returned the long (nine-

550

WILLIAM D. PERREAULT, JR.

page) questionnaire. It cannot be said with total confidence that the response rate was influenced by the personalized questionnaires; no control group received questionnaires printed in the traditional fashion.

It does seem, however, that if the respondent feels that his response is uniquely important, he may be more inclined to respond. The uniqueness of the individual questionnaire may be emphasized by printing the respondent's name at the beginning of his particular questionnaire. This is an especially straightforward procedure when respondent's names and addresses are already compiled for computer mailing labels.

Cost

The costs of the procedure discussed here should be taken into account in determining the appropriateness of the procedure to a given project.

The costs vary primarily according to (1) how complicated the reordering stage of the process is (i.e., the price of computer time), (2) which text processing program is used, and (3) how much actual text there is on a page of the questionnaire. For the project discussed in this paper, the total computer cost, including supplies, was about 8 cents a page.

Depending on the questionnaire, cost could vary from about 2 cents per page to 12 cents per page (or higher, if prices rise).

For surveys involving a long questionnaire, a large sample, or both, this cost might be prohibitive. In such a case, the randomization procedure might prove useful during pretest stages to test for order effect.

The pretest sample would be randomly split. Half would receive questions in a s t a n d a r d order; t h e remainder would receive questionnaires processed as discussed here. Both response rate and actual response distributions could be tested statistically to determine if there were differences between the two halves. With this additional information, the researcher could evaluate whether order effect was present and whether it needed to be controlled.

Alternatively, if order effect was significant, a series of different questionnaires could be generated, each with its unique ordering. These questionnaires would then be printed in the standard fashion. Each individual respondent would not, therefore, receive a unique questionnaire, but the procedure would be a relatively efficient approach to reducing order-effect bias.

OTHER APPLICATIONS

Although the focus of this article has been on the applicability of combining text editing with a preprocessing algorithm to control other bias in survey instruments, the combination has a number of other useful applications. Frequently, it is advantageous in survey research to person-

ORDER-EFFECT BIAS

55 1

alize advance notice or cover letter^.^ Names and addresses may be read from a mailing list as variables in a preprocessing algorithm and inserted in the text-string of a letter to be processed. The text is then formatted by the program to print the personalized letters. This mechanism also provides a direct procedure to sample randomly from an existing list and simultaneously generate appropriate letters.

The text processing program may also be used to print mailing labels, or print names and addresses directly o n special envelopes developed for use on line printers. Envelopes addressed by the text processing program (with both capital and lower-case letters) are not readily identifiable as computer output.

CONCLUSION

Question sequencing is an important aspect of the art and science of questionnaire construction; question sequence may have both positive and negative effects. This article has discussed a procedure by which public opinion researchers may preserve the positive aspects of sequencing while controlling for problems of other bias. The procedure has additional benefits in that it may also be used as a speedy and relatively economical way to personalize survey research materials such as cover letters, advance notice letters, and even the questionnaires themselves.

An excellent review of the impact that personalized survey materials may have on response has been compiled by Paul L. Erdos, Professional Mail Surveys, New York, McGraw Hill, 1970.