COMMUNICATIONS The purpose of this new section is

COMMUNICATIONS The purpose of this new section is to introduce the lighter side of marketing, where readers can discuss the less technical aspects of marketing research, and comment on material appearing in our journal or elsewhere relating to marketing research. Readers are invited to submit material for this section.

Are Open-ended Questions Worth the Effort?

STANLEY L. PAYNE* >Free-answer

questions are so troublesome that banning them from full-scale surveys has been suggested.

A recent experiment finds them inefficient and no more productive of depth or o f valid answers than checkbox questions ore. Their indicated use may b e confined to the development and pretesting phases o f surveys.

I

ceived, carefully executed, evidently unbiased, controlled experiment. Because Human Organization, official journal of the Society for Applied Anthropology, is not in the normal purview of most market researchers, that report by Barbara Dohrenwend merits special mention here [I]. Among the conclusions from that experiment are the following:

N private discussion, Ben Gaffin sometimes contended-that open questions had no place in large-scale surveys.l He argued that the open, or free-answer, question was inordinately inefficient for whatever contribution it might make to understanding consumer views and attitudes. Problems of interviewer variability, coding inconsistency, lack of comparability in trend surveys, and tabulating complexity combined to make the open question troublesome, time consuming, and expensive beyond its value.

Even if the transfer from open questions in the developmental phase to closed questions in the final form meant increasing the total number of questions, he still championed the closed questions. For example, from answers to the single, exploratory open question, "What do you like about it?" a number of categorical questions might evolve for the final interviews: "Do you like the taste or not?", "Do you like the color or not?", "Do you like the thickness or not?", "Which is the most important to you-its taste, its color, or its thickness?", etc.

Having written a chapter on the free-answer question and its demerits [2], I had some sympathy for the point of view he expressed. At the same time, it seemed almost heresy to think of eliminating that basic type of question from final questionnaires. In writing of its merits, I said that "the free-answer (question) is uninfluenced, it elicits a wide variety of responses, it makes a good introduction to a subject, it provides background for interpreting answers to other questions. It can be used to solicit suggestions, to obtain elaborations, to elicit reasons, to evaluate arguments, to explore knowledge and memory, and to classify respondents." The discussions with Ben did little more than increase my tendency to look at open questions critically; I continued to use many of them in full-scale surveys.

Now, from an unexpected source comes an incisive indictment of open questions based upon a well-con-

* Stanley Payne

The experimental interviews lend support to the criticism that open questions are less efficient than closed questions.

At the sime time, there is no evidence that open questions possess the advantage of being more productive of depth. . . .

Neither is there any direct evidence that open questions produce more valid answers. . . . The general conclusion to which we are led by the evidence in the experimental interviews is that closed questions offer more definite advantages than open questions in research interviews.

In' comparison with most marketing studies, this experimental study was small and limited in coverageonly 32 respondents chosen at random from women undergraduates at Cornell University. Its lack of size and scope was balanced by careful design and rigorous procedure. The design employed a 4 X 2 x 2 x 2 pattern.

Each of four interviewers, all having master's degrees and an average of two years of full-time work as professional interviewers, made eight two-part interviews; one part was factual and one part attitudinal, one part by open and one by closed questions, with the order of presentation of the parts and question types rotated over all possible combinations. Each respondent had earlier been exposed to a contrived standard situation which was the subject for both fact and attitude questions.

Nine "response properties" were measured for the two parts of each interview. The findings on six of those properties may be briefly summarized as follows: ( 1 ) Length

of response averaged about nine lines for open

questions and three lines for closed questions. relevant, unambiguous, nonrepetitive proportions of the answers-did not differ significantly for the two question types.

(3) Pertinence of answers was at about the same level for both types of questions, somewhat relieving open ques-

( 2 ) Usability of response-meaningful,

is a survey director and consultant, Payne

Survey Research, Chicago.

IBenjamin H. Gaffin, who died in 1959, was the founder of Ben Gaffin & Associates, and United States Interviewing Corporation.

417

Journal of Marketing Research,

Vol. I1 (November 1965), 417-19

JOURNAL OF MARKETING RESEARCH, NOVEMBER 1965

41 8

tions from the criticism that they allow respondents to wander from the topic.

(4) Self-revelation in answers to open questions was higher than in answers to closed questions on facts (Who was there? Who said what?, etc.), but the reverse was found in answers to questions on attitudes (Feelings about participating? Feelings of others?, etc.). The latter result goes directly counter to most expectations.

(5) Correctness of factual reporting, or known validity, was higher with the closed questions.

(6) Reasonableness of subjective responses, a judgmental measure, appears to be more affected by subject matter and position in the interview than by question type. Dr. Dohrenwend reaches the conclusion here that "in the face of resistance" closed questions yield more valid responses on subjective topics than do direct open questions.

The full report on the experiment analyzes interactions of form, subject matter, part of interview, and interviewer in considerable detail. It warns of limitations and is cautious in generalizing to normal survey operations. The above summary of six properties contains the most pertinent findings, however, and is presented as fairly as I find possible in a condensed version.

One response property not referred to in that report is the relative number of different ideas mentioned by respondents when answering open and closed questions.

My experience is that the longer responses for open questions tend to consist largely of elaboration and explanation of the few ideas originally expressed rather

than a proliferation of additional ideas. In an unpublished experiment where I was able to compare depth probing with lone reason-why questions, much more verbiage but an increase of only about a sixth in reasons per respondent was elicited through probing. The addition of unusual ideas was negligible for the overall survey.

As a consequence of studying Dr. Dohrenwend's report, I am much more inclined toward the view that open questions should be eliminated from full-scale surveys wherever possible. To quote from her last sentence "our results suggest that it is generally to the investigator's advantage to use well-tested closed questions rather than open questions."

The open question does have important application in the preliminary phases of interview researches. It should be employed there for the development of categorical, checkbox questions which can anticipate the range of relevant views or attitudes likely to be expressed, or to eliminate a need for asking reason-why questions of every respondent, or to provide quotes which may add interest to the report.

REFERENCES 1. Barbara Snell Dohrenwend, "Some Effects of Open and Closed Questions on Respondents' Answers, Human Organization, 24 (Summer 1965), 175-84.

2. Stanley L. Payne, The Art of Asking Questions, Chapter 3, Princeton, New Jersey: Princeton University Press, 1951.

Should the AMA Encourage Arbitration

of Trade Disputes ?

SIDNEY HOLLANDER, Jr .* >The

advantages of AMA arbitration of trade disputes are brought out b y means of an actual case history.

M

EMBERS of the American Marketing Association may be interested in how the ~ssociatiokwas instrumental in effecting the friendly settlement of a trade dispute. Arbitration under the auspices of a business or professional association is fairly common in certain fields (i.e., American Institute of Architects, the Air Transport Association, and some textile and insurance groups). But as far as I, and some knowledgeable elder statesmen whom I asked, know, this is the first instance

* Sidney Hollander, Jr., is a senior partner of Sidney Hollander Associates, Baltimore.

where this has happened within our group. It appears such a logical extension of the Association function, however, that it is presented here with the idea that it may serve as a useful precedent.

The content of the dispute is unimportant for the present report; briefly, the issue was whether a research agency can refuse to pay for admittedly faulty interviewing purchased in another city without giving the field interviewing service a chance to make good. After heated correspondence had deteriorated to mere reiteration of each party's position, it was agreed to submit the dispute to arbitration. Since both parties were AMA