Case Study in Question

Case Study in Question

Complexity

BY STANLEY L. PAYNE Many poll questions o#er several alternatives, among which respondents are asked to choose. W h e n questions are "tight" the order in which these alternatives are presented makes little diflerence in the choices of respondents, but when the questions are "loose" more respondents tend to choose the alternative which is stated last. A n experimental survey indicates that questions which are long and which contain dificult words tend to be "loose" and should be avoided whenever possible.

T h e author is Co-Director of Special Surveys, Cleveland. This article is based on a survey made while he was Research Director, Opinion Research Corporation.

Questions ideally should be as short as possible and contain only simple words.

Although this admonishment has probably been stated to question worders more often than any other, little tangible evidence has ever been presented to show that brevity and simplicity actually are important. Consequently, this basic rule may be overlooked too frequently.

The purpose of this article is to emphasize by means of concrete examples the need for short questions made up of simple words. These examples should be a convincing demonstration of the possible difficulties involved in lengthy queries and sesquipedalian expressions.

American Petroleum Institute Survey As part of a national study for the American Petroleum Institute, two carefully matched cross-sections of the national public, each consisting of about 3,200 persons, were interviewed on two

forms of a questionnaire.' These two forms were different from each other in only two respects. One of them carried a brief description of the oil industry, while the other had no description.

The second difference, and the one of interest here, is that the statement of alternatives was reversed in 16 of the questions, thus: Form ADO you think that this tax is too high, or about right?

Form BDo you thin4 that this tax is about right, or is it too high?

The Theory It was thought that if the order of stating the alternatives did have any effect, the combination of results from the two wordings would cancel out this effect. But more important from the Approval for publication of these results has been granted by the Oil Industry Information Committee.

654

PUBLIC OPINION QUARTERLY, WINTER 194g-50

standpoint of the present discussion, it was hypothesized that the extent to which respondents were confused by the question might show up in the degree to which the answers varied as the alternatives were transposed.

Under this theory, a "tight" question would be one in which the order of stating the alternatives makes no significant difference. For example, the answers to the two variations of the above question come out like this: Form A

Form B

Too high 31% 31% About right 43 41 I I Too low No opinion 25 27 The differences here are almost negligible. The "too high" choice is taken by the same proportion whether stated first or last and the "about right" choice increases by 2 percentage points when stated last, making the average difference I per cent in favor of the last alternative. We can say that the order of presenting the alternatives in this question does not affect the answers and that the question is a tight one-not confusing by this test.

On the other hand, the theory was that if reversing the alternatives gave widely differing results, then the question must have some element of confusion, or looseness. None of the 16 questions proved to be a serious offender on this score, but answers to this next one show the greatest differences: Form ADO you think of filling station operators as employees of the oil companies, or as independent businessmen like hardware merchants and lumber dealers who own their own stores?

Form BDo you think of filling station operators as independent businessmen like hardware merchants and lumber dealers who own their own stores, or as employees of the oil companies?

Form A

As employees 29% As independent businessmen 49 Qualified answers 15 N o opinion 7

Form B

37% 42 I5

6

Under the theory, this question is not very tight because reversing the alternatives does affect the replies. Whichever alternative is presented last is chosen on the average by 7% per cent more of the total respondents than when it is presented first.

Nine Tight Questions

In the experiment, g of the 16 questions proved to be tight, in that differences in results between the two forms did not exceed 2 per cent. These nine tight questions are shown below in only one of the two versions used. In the two-way questions the second version was obtained by merely switching the two alternatives about, while in questions stating more than two alternatives the two extreme alternatives were reversed.

Do you think this tax is too high, or about right?

As has already been shown, whichever alternative was stated last had an average I per cent advantage.

H o w would you rate the way oil companies treat their dealers-do they give them good treatment, just fair, or poor treatment?

N o difference was observed in the two versions.

CASE STUDY I N QUESTION COMPLEXITY

O n the whole, how would you say oil companies treat their workers-do they give them good treatment, just fair, or poor treatment?

The last alternative had a I per cent advantage.

From what you 4now or have heard, which would you say gives the customer the most courtesy and attentiongrocery stores, drug stores, or f l i n g stations?

The last alternative had a '/2 per cent advantage.

Would you say that the present price of gasoline is high, about right, or lou) in comparison with the prices of other things you buy?

The first alternative had a I per cent advantage.

Would you say the price of gasoline, including taxes, has gone up, come down, or stayed the same as it was 15 or 20 years ago?

The first alternative had a I per cent advantage.

O n the whole, do you thin4 the oil industry makes too much profit, a fair profit, or too little profit?

The first alternative had a I % per cent advantage.

Do you think there should be more government regulation of the oil industry than there is now, less regulation, or about the same amount?

The first alternative had a I '/2 per cent advantage.

Would you say this gasoline tax per gallon is higher, lower, or about the same as it was 15 or 20 years ago?

The first alternative had a 2 per cent advantage.

No claim of perfection is made for these nine tight questions. The only point to be understood about them is that they have the merit of producing

655

essentially the same results when the alternatives are asked in reverse order.

This conceivably might happen even with weak and misunderstood questions, as well as with excellent ones.

Seven Loose Ones

We know, however, that something is wrong with the other 7 of the 16 questions because the switching of alternatives does produce statistically significant differences in replies. These differences range from 4% per cent to the 7% per cent in the example already cited. The seven loose questions are: Do you think of filling station operators as employees of the oil companies, or as independent businessmen like hardware merchants and lumber dealers who own their own stores?

As already indicated, whichever alternative was stated last had an average 7% per cent advantage.

Do you think of the oil industry as being owned by a few large investors, or by thousands of small stockholders?

The last alternative had a 7 per cent advantage.

Well, which of these statements comes closest to your own idea of how gasoline and oil prices are decided: T h e oil companies get together and set prices for their products, or each company sets its own prices to meet competition?

The last alternative had a 6 per cent advantage.

Some people say that at the rate we are using our oil, it will all be used u p in about 15 years. Others say we will still have plenty of oil loo years from now. Which of these ideas would you guess is most nearly right?

The last alternative had a 4% per cent advantage.

Well, do you think the price has

PUBLIC OPINION QUARTERLY, WINTER 1949-50 656 gone u p because the price of the gaso- the dashes might cause some respondline itself has increased, or because taxes ents to jump the gun.

Do you think the oil companies hold have increased, or both?

The last alternative had a 4% per back new developments-such as ways for increasing gasoline mileage-or that cent advantage.

Is it your impression that there is a they are quick to adopt new developgreat deal of competition between the ments?

Another possibility is that because the companies who manufacture and sell gasoline and oil, a medium amount, or example, "ways of increasing gasoline mileage," goes with whichever alternaonly a little competition?

The last alternative had a 4% per tive comes first, respondents tend to cent advantage.

Do you think that oil companies hold back new developments-such as ways for increasing gasoline mileage-or that they are quick to adopt new developments?

The first alternative had a 5% per cent advantage.

grab it as a handle in making their decisions. Without direct testing of these possible explanations, it is necessary to withhold judgment as to whether this question is an exception that proves the rule or a special case that does not come under the general hypothesis that the last alternative has the greater weight.

The Rule and the Exception

Concept Words

In the first six of these seven loose questions, the alternative that seemed to be favored was the one which was stated last in the question. In other words, an alternative appears to be selected more often when it is mentioned last than when it is mentioned first. Perhaps this indicates that respondents tend to remember best the last words they hear.

In any case, it has been observed before that in a verbal question, the alternative stated last usually has the greater drawing power.

Answers to the seventh loose question indicated, however, that in this one instance the alternative stated first had the greater drawing power. But this may be a very special case. It might be conjectured that respondents tend to answer it early, even before they hear the second alternative. The wording does appear especially conducive to this possibility since the pauses indicated by

Cursory inspection of the two sets of questions gives the impression that the seven loose questions are more complex than are the nine tight questions. This impression is true, as will be shown shortly. Nevertheless it is only in certain respects that the one group is more complex than the other. For example, there are no more vague concept words in the loose group than in the tight group. The tight questions have as many concept words (treatment, attention, profit, regulation, etc.) as the loose questions do (independent, investors, competition, developments, etc.).

"No Opinion" Answers

Another factor which does not appear to account for the relative tightness or looseness of these questions is the degree to which opinion is set on the various issues. It is generally conceded that phrasing can sway opinions most easily when those opinions are not strongly

CASE STUDY I N QUESTION COMPLEXITY held. In this experiment, however, examination of the no opinion percentages, which also are associated with the strength of opinion, does not indicate that opinions on the loose questions are less well formed than the other opinions are.

T o the contrary, g per cent is the average proportion of no opinion answers to the loose questions as compared with 18 per cent for the tight questions. When the middle-ground answers (qualified, in between, about the same, etc.) are included with the noopinion answers, the total averages 21 per cent for the loose questions as against 27 per cent for the tight ones.

These figures seem to lead to the somewhat paradoxical idea that opinions happened to be held less strongly on the tight questions than on the loose ones.

where in the neighborhood of or less.

20

words

Difficult Words Another difference between the two groups of questions is found in the ratio of difficult words to total words. For example, one word in every eight in the loose questions, but only one in every twelve in the tight questions, has more than two syllables.

Count of the number of affixes (prefixes and suffixes) per IOO words shows much the same situation. The loose questions average more than 40 affixes to IOO words, while the tight questions average fewer than 30.

Thus, we find not only that the loose questions tend to be longer than the others, but also that they are stated in Inore difficult terms. This illustrates the need for simplicity as well as brevity.

Number of Words Flesch Scores A simple count of the words in the In his book, The Art of Plain Talk., two groups of questions brings out a Rudolf Flesch outlined a method of more positive conclusion. scoring the readability of written maThe shortest of the tight questions terial. H e computes his scores from has only I I words while the longest con- average sentence length, number of aftains 26 words. Against this, the length fixes in IOO words, and number of perof the loose questions ranges from 21 sonal references in IOO words. His scorto 46 words. On the average, the loose ing method is intended for use on questions are half again as long as the straight reading matter and not for tight ones-31 words to 22 words. isolated questions given verbally. HowIf we count only the words used in ever, we may apply his method to expressing the alternatives, that is, ex- survey questions if we keep in mind cluding the introductory or stage-set- that using his score gives such questions ting phrases, we find that this same every benefit of the doubt from the ratio holds. The statement of the alter- standpoint of their being stated in terms natives in the loose questions is half which would hold respondent interest.'

For one thing, Flesch says that written again as long as in the tight ones.

Therefore, it seems clear that one communication can be pitched higher element of tightness in a question is 2 Cf. Terris, Fay, "Are Poll Questions Too brevity. For a not too difficult goal, we Difficult?", Puhlic Opinton Quarterly, Vol. 13, might try to keep our questions some- NO. 2 ( 1 9 4 9 ) , P. 314.

658

PUBLIC OPINION QUARTERLY, WINTER 1949-50

than oral communication, because the written word is easier to follow. It seems evident also that a sentence is easier to understand in context within a written article than is a question asked in a more or less isolated situation.

According to the Flesch method, the nine tight questions qualify as "standard," or suitable as reading material for people who have completed the seventh or eighth grade. The seven loose questions are rated "difficult" or suited for the reading of people who have completed high school or some college.

Flesch says in contrast that "very easy reading" would consist of sentences

averaging eight words or less with fewer than two affixes a sentence. This style of writing appeals to a potential audience of fourth grade attainments.

It should be noted that Flesch does not necessarily imply that more difficult reading matter cannot be comprehended by people of low educational attainments. His scoring device takes into account only what people at different levels will be likely to read with interest.

Possibly the saving grace in long, complex survey questions is that serious concentration of respondents produces a higher degree of interest in the questions than their text alone would arouse.