Thoughts About Meaningless Questions

Thoughts About Meaningless Questions

BY STANLEY L. PAYNE THE penchant of many respondents for answering questions which have no meaning for them poses a major problem for public opinion researchers. Normal distribution of responses or consistent responses are in themselves no guarantee that the question is meaningful to the respondents. Although it is diffi-

cult to detect meaningless questions, the use of short questions, understandable terms, filter questions, scaling techniques and careful pretesting can help to keep meaningless questions within reasonable limits.

The author is Co-Director of Special Surveys, Cleveland.

W H E N Tide magazine reported some time ago that Sam Gill had gotten 70 per cent of the public to voice opinions about a fictitious issue, the experiment furnished some amusement but was not taken very seriously in the public opinion field.' Some thought his approach was nayve, some dismissed it as ridiculous, and many seemed to feel that it should best be ignored as needlessly reflecting ill on opinion research in general. Until now, no contributor has seen fit to mention it in T h e Public Opinion Quarterly.

The implications of Gill's findings still deserve serious consideration, however. His experiment should prompt a continuing search for evidence of lack of meaning in other supposedly meaningful questions.

Those of us who are doing research through question and answer techniques need to cultivate a much keener appreciation of the ever-present danger of divining significance from baseless or wrongly based "results." At the risk of exposing my own na'ivett and with the use of some ridiculous examples, I should like here to indulge in a few reflections on meaningless questions and answers in the belief that their implications are not meaningless.

1 "How Do You Stand on Sin?", Tide magazine, March 14, 1947: "Which of the following statements most closely coincides with your opinion of the Metallic Metals Act? It would be a good move on the part of the United States.

It would be a good thing but should be left to individual states.

It is all right for foreign countries but should not be required here.

It is of no value at all."

688

PUBLIC OPINION QUARTERLY, WINTER 1950-51 THE OMNIPRESENCE OF MEANINGLESSNESS

I . Let zts first accept the fact that respondents do answer questions that may be meaningless to them. Solemn answers to Gill's question on the fictitious Metallic Metals Act furnish one example. For another, anyone who has heard a radio program demonstration of the fast double-talker interviewing volunteers from the audience cannot avoid being impressed with the fact that the subjects, confused as they are with his utterly senseless questions, nevertheless are earnest in the answers they give. Riesman and Glazer have pointed out that "people do respond, in overwhelming numbers, to the social demand that they have an opinion."'

Particularly when a question without meaning is asked in the normal course of interviews in which the preceding questions range from clear to abstruse, many respondents are not going to quibble over meanings but will give answers that they think will satisfy the occasion and not reflect on their ignorance. For a ridiculous example may I suggest something like this: "Which of the new fashions now being talked about for women's hats do you think will be most popularGraddits, Freebles, or Pronlies?" While this talk of women's hats should be obvious nonsense to everyone, you can wager that when asked in an apparently reasonable way in what appears to be logical context, it will be answered by some high proportion of respondents.

The point is, however, that a sincere and well-intended question on a serious plane can be just as meaningless to some respondents as "Graddits, Freebles, or Pronlies" is to all respondents. The people for whom these serious questions lack meaning may give answers to them in even higher proportions than in the case of the patently ridiculous example.

2. T h e distribultion of answers to a "meaningless" question is not always random but may be patterned after some known or unknown predisposition. In the above example, if the three choices truly did not have any overtones of meaning or semblances to other words, it might be expected that the expressed opinions would divide into equal thirds.

It would be fine if this pointed a way for us to detect meaningless questions. On that theory we could confine our suspicions of meaningless questions to those where the results showed a possible random character as 50-50, 33-33-33, 25-25-25-25, etc.

ZRiesman, David and Nathan Glazer, "The Meaning of Opinion," The Public Opinion Quarterly, 1948,Vol. 12, N6. 4, p. 633.

THOUGHTS ABOUT MEANINGLESS QUESTIONS

689

The problem of detecting a meaningless question is not so simple, however. Many things may intervene to destroy the random pattern.

For one thing, we know that with oral questions there is a tendency to choose the last-mentioned alternative. Very likely in our women's hat case some people would choose, "the last one." In that event, a 30-30-40 split might be even more suspicious than a 33-33-33 distribution. Of course, this predisposition to take the last-named could be circumvented by rotating the three alternatives in successive interviews so that the telltale equal division might again appear.

Even then, for some unknown reason, respondents might be predisposed toward one or another of a group of meaningless alternatives, again destroying the random pattern. For example, if "Graddits" should bring to mind the mortar board of the graduate, more respondents might make that choice because it would take on some meaning for them.

Another general tendency people display is to seek the middle ground in their answers to a degree-type question. "Do you think the Mayor's program for arboreal reverberation goes too far, is about right, or doesn't go far enough?" A model for expressed opinions on such a senseless issue would be balanced, like 20-60-20 or 15-70-15. Answers coming out in such fashion to a middle-ground question might lead us again to suspect that particular issue as meaningless. It is at least interesting to observe how often such evenly balanced distributions occur.

The fact that people gravitate toward familiar things in making their choices is another form of predisposition that might sometimes nullify the random theory. An earlier paper3 showed this non-random distribution of answers to the question: "In which company (of four listed) do the men running the plant show the most interest in the welfare of their workers?"

83% Company K Company L I5 Company M I Company N I These answers seem far from meaningless until we check them against the answers to a more fundamental question: "Which one of these companies do you feel you know most about ?"

3 Barlow, Walter G., and Stanley L. Payne, "A Tool for Evaluating Company Community Relations." The Public Opinzon Quarterly, 1949, Vol. 13, No. 3, p. 405.

690

PUBLIC OPINION QUARTERLY, WINTER 1950-51

Company K 84% Company L I3 Company M 2 Company N I The resemblance of the two distributions is so marked that we are justified in asking whether the first one was actually answered in terms of interest in worker welfare or only as an aura effect of general familiarity. Had the first question asked about which company has the greatest "propensity for reincarnation," we would have immediately recognized that it was meaningless. Since it speaks of something apparently sensible, however-"most interest in welfare of workers9'-it at first strikes us as being meaningful. But, is it? At least, there is a suspicion that it has little meaning in its own right.

You may be very well acquainted with various makes of calculating machines. Yet, a question as to which company has the best pension plan -Friden, Marchant, or Monroe-would probably be meaningless to you no matter how important it is to the industrial relations departments of these companies.

Four types of predisposition have been mentioned-the tendency to take the last alternative, the inclination to read a meaning into a word on the basis of its similarity to known words, the middle-ground proclivity, and the appeal of the familiar. Probably many other predisposing factors are at work to give senseless answers an appearance of being meaningful, or non-random.

3. It is virtually impossible from the answers to the question itself to ascertain the proportion of respondents for whom it is meaningless.

As has just been indicated, so many different and unknown predispositions may affect the answers that th'e element of randomness may be obscured. Also, practically all questions we are likely to ask are serioilsly intended and do have real meaning to large or small parts of the public.

This means that random choices and predisposed selections can be intermixed with meaningful replies until there is no hope of unscrambling them.

Attempts have been made to calculate the amount of "guesswork" included in answers to isolated questions, but such attempts are doomed to failure. One example of such an attempt from my own experience involves the introduction of an outlandish alternative in a question.4 Payne, Stanley L., T h e Art of Asting Questions, Princeton University Press, forthcoming.

THOUGHTS ABOUT MEANINGLESS QUESTIONS

691

"When you speak of profits, are you thinking of profit on the amount of sales, on the amount of money invested in the business, on year-end inventory, or what?"

On sales 22% On investment 18 On inventory I4 On other bases 10 Don't know 37 If one chooses to take the 14 per cent who selected "year-end inventory" as an index of guessing, he can deduct that number of percentage points from the 22 per cent who selected "sales" and the 18 per cent who selected "investment." This leaves a net of only 8 per cent and 4 per cent who "know" what they are talking about when they speak of profit on "sales" or "investment." This would be one way of guessing at the amount of guesswork in these answers.

However, if we stop to consider more carefully the problem that seemed to be solved so quickly by this expedient, we realize that we are actually up against a more involved question of semantics. A mistake is made in assigning absolute and distinct values to "knowing" and "guessing" instead of recognizing that these two opposing ideas are connected along a continuum. Here is a suggestion of only four grades of the ideas that merge along this continuum: some people "really know," some "think they know," some make "informed guesses," and some make "outright guesses."

Although it complicates this problem even further, we should realize that some people, incorrect though they may be, nevertheless "think they know" that profit is figured on inventory. Others, knowing better, simply misspeak themselves much as they make ridiculous and unintentional mistakes in answering questions in an intelligence test.

With all these possibilities in mind, we begin to see how futile it is to attempt calculating the amount of "guesswork" in any single question.

Another ridiculous example may throw more light on this matter of measuring guesswork: Suppose we get answers like these to the question : "Is the earth flat or round ?"

Round 95% Flat 3 Don't know 2

692

PUBLIC OPINION QUARTERLY, WINTER 1950-51

Can we say that since 3 per cent "guess" that the world is flat, the same percentage must be guessing that the world is round, thus bringing the level of knowledge down to 92 per cent? No, because some of the 3 per cent are not guessing and many of the 95 per cent do not know. It may seem impertinent to ask whether you yourself really know that the earth is round, but do you? Some few people in certain religious sects today may "know" through faith in Biblical references to the "four corners of the earth" (Isaiah XI, 12; Revelation VII, I) that the earth is flat even more definitely than you know that it is round.

Or, if you are inclined to think that the example is pointless, consider for a moment what the answers might have been in the year 1490 and then what they might have been about fifty years after Columbus. Such changes in "knowing" the very shape of the earth could not be adjusted by mathematical means.

4. Even the fact that answers to cz question are borne out by subsequent behavior does not necessarily prove that a question is meaningful.

Unfortunately, the real-life situation itself may have an Alice-in-Wonderland quality about it for many respondents. Here are the seven names which confronted voters in the 1950 Ohio Republican primary for the office of State Treasurer: St. Clair K. Archer, Jr. Harold B. Collier Herbert Hoover Herbert L. Hoover Lawrence R. Lyons Kenneth C. Ray Roger W. Troey Now, it seems reasonable to suppose that many voters marked a choice on their ballots with no advance knowledge about any of these candidates. They were as much in the dark in the voting booth as they would have been in an interview situation. Their behavior when up against an actual voting choice, then, was probably little different from what it might have been in a pre-election poll. Agreement between the pre-election poll and the election itself could occur without proving that either situation had much realism for these voters.

Another Alice-in-Wonderland experience in real life may happen when a person looks into the classified telephone directory for a service

THOUGHTS ABOUT MEANINGLESS QUESTIONS

693

with which he has had no previous experience. Let us say that he needs to have his roof patched. Under the "Roofers" classification he is surprised to find literally scores of roofers, from which he has to make his choice without any real knowledge to back it up. Again, the actual number of phone calls to each roofer from the directory might not differ from the results of a telephone directory survey, but this would not prove that the survey question had much meaning in terms of respondents' knowing which roofer was which.

5. Consistency of replies is no proof that a question is full of meaning.

In fact, the more universally meaningless a question is, the more alike the distributions of replies should be over the course of time. In our ridiculous example of women's hats we should expect to find the same result five years from now as today.

Along somewhat the same line, it should be recognized that the outmoded stability run, showing cumulative distributions of replies after the first roo interviews, the second roo, the third 100, etc., has nothing to do with the intrinsic value of a question. This is to say, a meaningless question stabilizes every bit as soon as a meaningful one does.

Likewise, a low proportion of "no-opinion" or "don't know" responses does not necessarily indicate that a question is full of meaning to respondents. One might, in fact, argue that a high proportion of "no-opinion" answers indicates that uninformed respondents readily recognize their inability to answer the question and that the few who reply do see the meaning in it.

TOWARDS MORE MEANINGFUL QUESTIONS

What moral is to be drawn from all this discussion? All that has been stated or implied so far is that respondents do give answers to questions they don't understand, that meaninqless questions can only sometimes be detected, that there is a possibility of some element of meaninglessness in every question, and that there is no way of determining the proportion of guesses from the structure of answers to a question. This would seem to be equivalent to dismissing the problem as insolvable. Still, a number of precautions can be taken to reduce the meaningless factor.

I. In the first place, we should condition ourselves t o recognize issues or terms that aye likely to be meaningless t o large sections of the public under study. In filter questions asked by the Gallup Poll, 45 per

694

PUBLIC OPINION QUARTERLY, WINTER 1950-51

cent of the public said they did not know what a "lobbyist" was, 41 per cent did not understand the term "socialized medicine," 46 per cent could not describe a "filibuster," and 88 per cent either said they didn't know or gave incorrect descriptions of "jurisdictional trike."^ To be understood by high proportions of people an issue must be close to their interests, timely, specific, and concrete. Stuart Chase has conjectured that "more than half of the questions asked by accredited pollsters since 1936 should probably never have been attempted."' His statement might seem more reasonable had he said that these questions should never have been attempted unless recast into more understandable form. If, for example, the term "watered stock" is explained to people, they may have very definite ideas about it, whereas without explanation they may think the term has something to do with cattle.

2. For another thing, we should avoid using abstractions or concept words for which philosophers require definitions before they can begin to argue. Answers of a dozen different people to a question like this next one can have a dozen interpretations: "Should our country be more active in world affairs ?" There are three concept ideas requiring definition in this one short question-"our country," "more active," and "world affairs." What is meant by "our countryv-you and I, the government as a whole, the State Department, our industrial leaders, or what? What is meant by "more active"? By "world affairs"? Stuart Chase has suggested that we might as well substitute the word "blab" for such abstract ideas.7 That is, the question might almost as well read "Should blab be blab in blab 2'' so far as conveying any generally understood meaning- is concerned.

3. T w o points that cannot be mentioned too often are that qzlestions shoald be short and that the words used shozlld be simple. Self-evident though these two principles may appear, they nevertheless are continually being ignored. Forty-word questions containing many multi-syllable words are being asked of the public every day. An earlier paper was given over to a demonstration of the effect of length and complexity, and its findings need not be repeated here.' Sufice it to say that by any AIPO news service releases, Feb. 12, 1947; March 11, 1949; March 23, 1949; May I, 1949.

Chase, Stuart, T h e Proper Study o f Mankind, New York: Harper & Bros., 1948.

7 Chase, Stuart, T h e Tyranny o f Words, New York: Harcourt, Brace & Co., 1938.

Payne, Stanley L., "Case Study in Question Complexity," T h e Public Opinion Quarterly, 1949, Val. 13, NO.4, P. 653.

THOUGHTS ABOUT MEANINGLESS QUESTIONS

695

readability score survey questions would seldom appeal to that part of the public having less than grade-school attainment.'

So much for what we need to do with a question taken by itself.

4. In addition, filter questions or other approaches can and should be used to eliminate those respondents who lack understanding or show unawareness of an issue. Asking them to define terms or to give examples, as recommended by George Gallup in his quintamensional plan, is a summary device for sorting out those to whom the key question would be meaningless." Filter questions are a good idea generally.

We might just as well consider hermits to be representative of the general public as to take answers to a single question as indicating opinion on a complex issue. However, by relying too heavily on devices such as filter questions we may tend to forget about improving our key questions. A better approach is first to do all we can to improve the understandability of the key question and then to consider the filter question as only an extra precaution.

5. Scaling techniques are valuable also in that questions which do not "scale" with others on the same topic are then recognized as being outside the intended universe of content.'' Many of the response "errors" that cause removal of a question from a scale no doubt result from some meaningless feature. Rewording the same idea can sometimes increase the degree of understanding and restore the revised question to the scale. That is to say, rejection of a particular question from a scale does not necessarily mean that the intended issue does not belong there but may signify only that it was ineptly stated.

6. Pretesting is, of course, the most generally used method of assuring that questions are comprehended as intended. Too often, however, test interviewers seek merely to duplicate as nearly as possible the expected interview situation with attention to timing, boredom, irritating questions, and such. They may report back that a question "works," when all they have actually determined is that it elicits answers. Proper incorporates frequent insertion of queries such Terris, Fay, "Are Poll Questions Too Difficult?", T h e Public Opinion Quarterly, Vol. 13, 2, p. 314. lo Gallup, George, "The Quintamensional Plan of Question Design," T h e Public Opinion Qnarterly, 1947, Vol. I I, No. 3, p. 385. l1 Stouffer, Samuel A., Louis Guttman, Edward A. Suchman, Paul Lazarsfeld, et al, Measurement and P~erlirtion,Studies in Social Psychology in World War I I , Vol. IV, Princeton: Princeton University Press, 1950.

No.

696

PUBLIC OPINION QUARTERLY, WINTER 1950-51

as: "What did you think of when I mentioned the Mayor's program for arboreal reverberation? Was that question clear to you?" This detailed pretesting is still the most direct means of exposing meaningless questions in advance.

Constant awareness of the problem, use of concrete words, attention to brevity and simplicity, inclusion of more than one question, and thorough pretesting are the chief protections against meaninglessness.

Many other things may be done to improve a question once it is generally meaningful, but these few points seem to be the basic ones to our major purpose in asking questions.