QUESTION

QUESTION

WORDING I N P U B L I C POLLS

OPINION

Comments on points raised by Mr. Stagner

George Gallup

Dr. Stagner's criticisms indicate a clearly constructive approach on his part to the problems of public opinion measurement, a s encountered by individuals and organizations actually engaged in this work. It is to be hoped that his article will stimulate further discussion of current polls and polling techniques. The science of public opinion measurement is still adolescent--so relatively new that it will take the combined thinking of many research minds to chart and develop its future. I have suggested from time to time that a national committee of research be set up in this field a s a permanent body to advise the various survey organizations and projects.

The need for such a body is indicated, indeed, by Stagner's present paper. Stagner's conclusions concerning the wording of questions very largely rest, he indicates, upon the findings of Blankenship. But there a r e certain distinct limitations to Blankenship's experiment--limitations which Blankenship himself was careful to point out, but which, it may seem to other observers, have not been sufficiently regarded in the Stagner paper.

Blankenship's study of opinion in Irvington, New Jersey, on the question of the popular referendum on legalizing parimutuel betting in June, 1939, dealt with a question which apparently was of no great interest to a large section of the population. The official vote in Irvington brought out slightly less than 7,000 voters, which represented something smaller than one-third of the Irvington voting population and less than 20 percent of the adult population. In comparing the results on the different question forms on separate samples (which averaged about 350 each), with the election returns, however, Blankenship used all people interviewed who were registered voters and who indicated an intention to vote in the parimutuel referendum--an average of about 58 percent of the samples. It seems clear that Blankenship's samples contained too great a proportion of people who were not sufficiently interested in the election to register their vote at the polls, for the samples to

260

SOCIOMETRY

be directly comparable with the election results. In Blankenship's total sample of 3,000, the "no opinionn vote on what he calls the "variable questions" ranged between an average of 24.2 percent and 29.3 percent.

It is not the purpose of this article to examine the Blankenship experiment in detail, but I do think it is worthwhile pointing out here that the results of the Blankenship experiment should, at the very least, be considered a s one contribution to the problem of question wording, and not a s the final word on this issue.

Blankenship himself stated his own reservations a s follows in his conclusions: "The q u e s t i o n form (as f a r as suggestion i s concerned) would m a t t e r l e a s t when pub1 i c opinion was v e r y For example, d e f i n i t e l y c r y s t a l l i z e d on a s u b j e c t when pub1 i c o p i n i o n i s so d e f i n i t e l y c r y s t a l l i z e d a s on t h e s u b j e c t of who i s i n t h e r i g h t i n t h e European war, t h e question form would probably m a t t e r l i t t l e , i f a t all.....

The one t h i n g t h a t t h i s study d e f i n i t e l y i n d i c a t e s I s t h e need f o r f u r t h e r r e s e a r c h * " I

.....

To ignore this very sensible note d caution, to expand Blankenship's single experiment (carried out with small samples on issues where sentiment was not crystallized) into a general rule applicable to all issues at all times, a s Stagner has done, is probably to go far beyond, if not contrary to, Blankenship's own findings.

Stagner himself states that "biased question formsn have an effect on poll results only "when opinions a r e vague and weakly held," (as they clearly were in the case of the Irvington experiment).

Blankenship's conclusions concerning the negligible influence of question wordings when opinion is definitely formed, tally closely with the Institute's own experience in conducting more than 1,000 surveys in the past seven years. Differences in wording have had little o r no effect on issues where there was a low no-opinion vote, that is, issues on which the people have made up their minds. The split-ballot technique, in which the Institute has done considerable research, furnishes abundant evidence to support this point.2 In the split-ballot technique, two forms of a question a r e printed for purposes of experimentation on the effects of i ~ s y c h o l o g i c a l Record, Vol. I 1 1 , No. 23, pp. 401-402.

2 ~ e ebelow, p. 2 6 4 .

PUBLIC OPINION POLLS

261

different wording--let us say, an "A" form and a "B" form.

The "A" form is put to one half the voters in the cross-section and the "B" form to the other half--both halves containing exactly comparable types of voters.

Rare is the instance in which the difference between the two wordings produces a variation in results greater than the expected variation due to size of sample, when opinion on the issue is definitely formed. It can, therefore, be stated that the "suggestive" effect of a positive o r negative type of question o r of a difference in wording is largely a function of the degree of crystallization of opinion on an issue, and tends to decrease to zero a s the no-opinion vote tends to zero.

Most war issues belong to this latter category.

Relying on Blankenship's finding that, in the Irvington, New Jersey experiment, the objective wording proved "more reliable" than the subjective form, Stagner then goes on to condemn the @subjective" form of wording, which he claims the Institute uses more frequently than the objective type. The subjective type, it will be recalled, is defined a s a question containing the word "you"; for example, "Do you think the C.C.C. camps should o r should not be continued?" The so-called "objective" question, on the other hand, is stated in terms of the third person, thus: "Should o r should not the C.C.C. camps be continued? " It is necessary to point out, however, that among research men, there is considerable disagreement a s to the relative merits of the two types.

Floyd Allport has advanced exactly the opposite conclusion to that of Stagner. He believes there should be not fewer "subjective" questions but more: "ens ider, f o r example, quest ions deal ing w i t h such issues as a measure t o reorganize t h e Supreme Court, I n these questions t o alter neutrality restrictions. the average c i t i z e n can t h i n k of the proposal on1 y as something a p p l y i n g t o an a b s t r a c t i o n such as t h e S t a t e o f t h e Nation. It i s vague i n i t s i m p l i c a t i o n s , and a p a r t from himself.....

By asking questions of t h i s type we take t h e issue away from t h e i n d i v i d u a l and " t h i r d - p e r sonizen it. I n order t o discover how t h e i n d i v i d u a l stands i n t h e most dynamic and pragmatic sense, we must b r i n g t h e problem back f o r him i n t o the f i r s t person.....

Some of t h e p o l l i n g questions used a r e indeed couched i n these f i r s t - p e r s o n i z e d terms; but t h e y seem t o be f a r o u t numbered by questions o f t h e third-personized v a r i e t y . A p r e l i m i n a r y count o f 198 o f t h e questions asked over the p e r i o d from 1936 t o 1940 i n t h e American I n s t i t u t e p o l l

....

262

SOCIOMETRY shows t h a t t h e l a t t e r stand i n a r a t i o t o t h e former o f three t o oneend

It would seem, then, that the Institute stands under fire from both sides. Stagner claiming that there a r e too many first-personized questions, and Allport asserting that there a r e not enough. Obviously, there is need for further research here.

In i t s own work, the Institute has tried to apply the rule of common sense to the matter of question wording.

In the past seven years of polling, it has endeavored to formulate the questions a s closely a s possible to the issue a s it presents itself for debate in Congress o r before the people in public meetings o r through the newspapers and the radio. The choice of word forms has thus been governed chiefly by the nature of the issue, rather than by elaborate and rigid- rules conforming to one theory o r another.

Since Stagner relies heavily upon Blankenship, it is interesting to note that Blankenship himself, in attempting to correct the rigidity of his own conclusions, admitted the need for ballot questions that closely conformed to the wording s f the actual parimutuel amendment on which he was attempting to measure opinion. He write^:^ " I t would have been i n t e r e s t i n g t o v a r y t h e wording o f t h e horse-racing issue from a d i s t i n c t l y d i f f e r e n t t o a v e r y s i m i l a r wording as used I n t h e proposed amendment, except t h a t t h e wording of t h e amendment was so comp l i c a t e d and long t h a t t h i s would have been impossible. if i t could have been done, the most s tmt lar form of the ques-

tton would presumably have been found to be most valtd tn terms of the method used." ( i t a l i c s mine)

That Stagner has apparently overlooked a number of Blankenship's cautions and warnings is but the least of the criticisms of this particular analysis. F a r more serious is Stagner's distortion of the t e r m s "positive" and "negative" question wording to mean something entirely different from what Blankenship intended them to mean. Let us consider this point with care.

The positive type of question, according to Blankenship's original thesis, is the "affirmative interrogative," as follows: "Is it desirable?" The negative type is represented by 3 ~ u b l i cOpinion Q u a r t e r l y , Vol. @, No. 2 , p. 255ff. @ ~ s ~ c h o l o g i c Record, al Vol. i l l , No. 2 3 , p. 401.

PUBLIC OPINION POLLS

263

the question: YIs it undesirable?" These distinctions between positive and negative a r e clearly distinctions in grammatical word constructions.

But Stagner applies the positive and negative test to Fortune and Institute results in an entirely different, and in the light of Blankenship's meaning, unjustifiable manner.

After saying that questions utilized by the two survey organizations were classified a s objective, subjective, positive, negative, etc., "using these terms a s defined by Blankenship," Stagner continues (p. 247): " P o s i t i v e r e f e r s t o an item s t a t i n g an i n t e r v e n t i o n i s t proposal, e . g . Should we d e c l a r e war on Germany and send our army and navy abroad t o f i g h t ? N e g a t i v e r e f e r s t o items phrased t o l e a d t o r e j e c t i o n o f i n t e r v e n t i o n , o r c a l l i n g f o r a peace conference among t h e Great Powers."

Stagner is obviously not using Blankenship's standard of grammatical word construction, but a special standard of his own in which "positive" means a question that puts forward an interventionist proposal in the affirmative interrogative; whereas, "negative" means a question that puts forward an isolationist proposal also in the affirmative interrogative. What it amounts to, therefore, is that Stagner is objecting to the fact that the A.I.P.O. has, in the period under consideration, polled more often on issues putting forward interventionist proposals than on issues putting forward isolationist proposals. This, then, is an essentially different matter from objecting to the wording of a question; it is objecting to the content of the questions. Stagner in effect blames the Institute for having polled on specific issues of foreign policy a s step-by-step these have reached the level of public discussion. Such a criticism can only be rejected in the most categorical way, for it is obvious that the task of a polling agency is to poll on issues a s they present themselves, not a s one wishes them to arise.

In connection with Stagner's contention that Fortune surveys show less interventionist sentiment than do A.I.P.O. surveys, I would like to point out that in the most recent Fortune Survey, (August, 1941 issue), Fortune showed 53.7 percent willing to risk some kind of shooting war to help Britain. The Institute's most recent survey on the convoys issue found 54 percent favoring the use of our Navy for convoy duty. The percentage which the Institute has found favoring convoys has not changed materially for some time.

We now turn to an examination of the influence of "emotionally-toned stereotypes" and "prestige-bearing names" upon opinion. Stagner's criticism of certain A.I.P.O. questions on that score is well-taken. No one is more conscious of certain e r r o r s in that connection than the A.I.P.O. staff. Never theless, the Institute has discovered in several hundred splitballot experiments that the supposed influence of stereotypes and prestige-bearing names is rather less than has commonly been supposed, when opinion is well-formed. The use of the name "President Roosevelt" has been found to have either no effect at all, or only a very small effect one way o r another upon the results in such cases. Nor has the phrase "German government" in a question produced any appreciable difference in results from the name "Hitler." As evidence of the firmness with which the public holds its opinions on certain subjects, the Institute has even found in its split-ballot experimentation that in some cases where opinion is pretty definitely crystallized the use of arguments prefixed to the question, using an unfavorable argument on one form and a favorable argument on the second, has brought no variation in results.

A few of these cases may be cited (see next page).

Owing to the expected variation between split-ballot results due purely to size of sample, a deviation of 3 percent is not statistically significant.

Concerning Stagner's suggestion that the A.I.P.O. sample may have a small interventionist bias a s a result of failure to include sufficient numbers of women and people in the lowest economic group, I welcome the opportunity to say that our sampling on war issues gives equal weight to men and to women and gives proper weight to each ecanomic group.

It has been our policy in the Institute to refine our sampling techniques a s rapidly a s possible. Although our survey in the 1940 election, where we had an average state-by state e r r o r of only 2.4 percent, was by far the most ccurate survey that has ever been conducted in the 48 states,'we have made some further changes in our sampling techniques, incorporating lessons learned from our experience in this period.

At the present time, s o far a s I know, our samples do not contain any bias in either direction, either interventionist or noninterventionist..

5 ~ a n i e l Katz, "he Pub1 ic Opinion Polls and the 1940 Election," Public Opinion Quarterly, Vol. 5, No. I.

PUBLIC OPINION POLLS

265

PRESTIGE-BEARING NAME Do you agree w i t h President Roosevelt t h a t t h e United States should f i g h t t o prot e c t Canada against a t t a c k by any country?

Do you t h i n k t h a t the United States should f i g h t t o p r o t e c t Canada a g a i n s t a t t a c k by any country?

YES

DON'T KNOW

DATE -

Sept. 1939

74%

26%

(12%)

75%

25%

( 9%)

n

~t

HAVEN'T HEARD ABOUT LEASE-LEND

Have you heard about

the President 1s

Lease-Lend B i 11 now

before Congress? Do

you t h i n k Congress

should pass t h i s

bill?

YES

QUAL. ANSWERS

55%-

11%

20% 14%

( 9%)

Feb. 1941

Have you heard about t h e Lease-Lend Bi 11 now before Congress?

Do you t h i n k Congress should pass this b i l l ?

55%

11%

20% I%$

( 9%)

Feb. 194 I

Do you approve or disapprove o f t h e Lease-Lend B i 1 l ?

55%-

6%

29% 10%

( 4%)

March 1941

Do you approve or d isapprove of the Pres ident I s LeaseLend B i l l ?

57%

7%

25% 1 1 %

(5%)

March 1941

NO

u.

EMOTl ONALLY TONED-STEREOTYPES H i t l e r says t h a t t h e P o l i s h question i s s e t t l e d and England and France have no reason t o continue t h e war w i t h Germany. Do you agree?

YES

14% 86%

DON'T KNOW

(12%)

DATE

Oct. 1939

DATE

266

SOCIOMETRY

The German Government says t h a t t h e P o l i s h question i s s e t t l e d and England and France have no reason t o c o n t i n u e t h e war w i t h Germany. Do you agree?

Y E

DATE

DON'T KNOW

14% 86%

(14%)

Oct.1939

PRESENTATI OH OF ISOLATIONIST vs. INTERVENTIONIST ARGUMENTS AGREE

DISAGREE

20%

72%

Some people say t h a t s i n c e Germany w i l l p r o b a b l y defeat Russia w i t h -

i n a few weeks and t h e n t u r n her

f u l l strength against Britain, it

i s more i m p o r t a n t t h a n ever t h a t

we h e l p B r i t a i n . Do you agree,

or disagree w i t h t h i s ?

71%

19%

SomepeoplesaythatsinceGermany i s now f i g h t i n g Russia, as w e l l as B r i t a i n , i t i s n o t as necessary f o r t h i s c o u n t r y t o h e l p B r i t a i n . Do you agree o r disagree, w i t h t h i s ?

HO OPIl410~~d

(8%)

DATE

J u l y 1941

n

"

(10%)

OBJECTIVE POSITIVE vs.03JECTIVE WORDING CONSTRUCTION Should t h e U n i t e d S t a t e s s t o p giving aid t o Britain?

YES

9

10%

8%

Should t h e U n i t e d S t a t e s s t o p COKg i v i n g a i d t o E r i t a i n , o r s h o u l d STOP TlNUE we c o n t i n u e t o h e l p t h e B r i t 7% 89% ish?

ti 0 OPINION 14.beF

DATE

-0NO OPINION (4%)

DATE Feb.1941

CHANGING OF WORDING FRO# NEGATIVE TO POSITIVE CONNOTATION Do you t h i n k i t i s indecent f o r men t o wear t o p l e s s b a t h i n g suits (trunks without s h i r t s ) f o r swimming?

Do you t h i n k i t i s a l l r i g h t f o r men t o wear t o p l e s s b a t h i n g s u i t s (trunks without s h i r t s ) f o r swimming?

NO

YES

fi

OPlHlON

DATE

32%

68%

(5%)

June 1939

68%

32%

(3%)

n

n

PUBLIC OPINION POLLS

267

Stagner and others have commented on the relative merits of the multiple-choice question, which Fortune uses almost exclusively and the Institute on occasion, and the question with a simple statement of the issue calling for "yes," "no," o r "no opinion," which the Institute uses.

The multiple-choice question, which has come to be known a s the "cafeteria" question, because it s e t s before the respondent a whole s e r i e s of possible answers ranging from one extreme to the other, is best suited for probing public opinion in i t s early development stage. While no final judgment can be passed a t this time, there a r e a t least three drawbacks to the multiple-choice type of question.

F i r s t , the procedure frequently enables respondents to avoid expressing an opinion on the main issue.

Second, because multiple statements always tend to overlap, it is virtually impossible to add up the responses to form a clear-cut conclusion a s to what public opinion really is on a particular issue. In order to draw conclusions, one must group together the various responses in a more o r l e s s arbit r a r y fashion.

A third objection is illustrated in the results of a preliminary Institute experiment, which shows that when respondents a r e given a s e r i e s of choices running from one extreme to another, there is an all-too-human tendency to avoid the ext r e m e s and take the middle positions. A full report of the Tnstitute's experiment in that connection will be made during the coming year.

The Institute does not contend that the poll question calling for a straight "yes," "no," or, "no opinion" response is always better than any other. But i t does contend that the purpose of the straight question and the multiple-choice question is not identical. The multiple-choice o r &cafeterianquestion is particularly fit for polling on attitudes and expectations. When there is no clear-cut issue before the public in a particular field, the "cafeteria" question can frequently give a valuable description of the trends and cross-currents of sentiment in that field. But when a clear-cut issue a r i s e s , the straight question with " yes," "no," or, "no opinion" alternatives brings the attitude to mould itself into the framework of the issue and is, therefore, the only one that can accurately measure opinion on that particular issue. Before an election the people may have general attitudes and expectations a s to what kind of people the candidates should be, and regarding their program and optimum qualifications; but when the day of the election comes, they have to make a choice between a strictly limited number of alternatives. Nor is this condition limited only to elections.

268

SOCIOMETRY

It applies a s well to a particular piece of legislation before Congress which involves either enactment o r rejection.

In such cases, multiple-choice questions may give a false picture of the situation and tend to help the respondent dodge the real issues on which he is called to make a choice.

Certainly the Institute must take exception to Stagner's statement that in the " Gallup organization. there has undoubtedly been some unconscious bias I n favor of intervention (upper income Easterners poll high percentages for interventionist policies) which has repressed the observation that a number of items showed defects in construction....."

This statement hardly takes into account the actual The conditions under which question wordings a r e arrived at. wording is established only after thorough preliminary testing in the field and after conferences in which both interventionist and non-interventionist points of view a r e represented by members of the staff, with a view to eliminating conceivable sources of bias.

One further point in conclusion. Stagner raises the question whether opinion data have "validly reflected public thinking regarding an aggressive anti-Axis policy and aid to Britain." I t is likely that the answer will be found not s o much in theoretical studies which a r e themselves open to further theoretical refutations and criticisms, a s in practical studies of those other criteria of public opinion which existed even before polls. The key to the question raised by Stagner is to be found in the votes of Congress, in popular reactions to the measures advocated by the Chief Executive and enacted by the Legislature. Have polls been a good guide to the subsequent behavior of the public? This is a practical test which cannot be escaped. The Institute figures a r e available for any o r all to examine in this respect.

...