Journal of the American Statistical Association, June 1984 newer users can evaluate the usefulness of the technlque for the problem at hand. Chapters 3 and 4 are used to introduce BMDP program usage conventions. In Chapter 3 new users are given a sample program, annotated output, and guidelines on how data should be prepared for the analysis. Chapter 4 contains an overview of BMDP syntax rules and usage notes. Job-control language is introduced in just enough detail to enable the uninitiated to run ajob on an IBM or similar system. Features common to all BMDP programs are introduced in Chapters 5, 6, and 7. Topics presented include paragraph structure, formats, editing and transforming data, and BMDP Save Files. The section on Save Files is especially clear and makes this facility more readily accessible to persons who are not fluent in IBM's Job Control Language (JCL) and file terminology. The remaining chapters describe statistical methods implemented in the 40 programs that comprise the BMDP library. Chapters 8, 9, 10, and 12 discuss those programs treating data description. Capabilities described in these chapters include simple data description, description and comparison of treatment levels, plots and histograms, and treatment of missing data. Chapter 11 contains an extensive description of frequency table analysis together with a variety of examples. Numerous regression modeling techniques are available. Five linear regression techniques are presented in Chapter 13. Chapter 14 covers nonlinear regression, including a derivative-free algorithm and stepwise logistic regression. Five different techniques for performing ANOVA and MANOVA are presented in Chapter 15. Chapters 16 and 17 cover nonparametric analyses and cluster analysis. Chapter 18, "Multivariate Analysis," discusses techniques such as factor analysis, canonical correlation, and stepwise discriminant analysis. Chapter 19 treats life-table and survival analyses. Chapter 20 contains descriptions and examples of programs that perform spectral and Box-Jenkins time series analyses. BMDP has long had a reputation for being computationally rigorous. Annotated computing formulas for all multivariate methods, tests, and measures used in frequency table analysis and random number generation are given in Appendix A. Appendixes B, C, D, and E treat special topics, such as methods to increase the size of programs, the JCL needed to run BMDP on an IBM system, and a list of selected articles from BMDP Communications. The 1981 edition of the BMDP user's guide represents a substantial improvement over previous editions. The type and layout are easier to read and are similar to that found in JASA. The editors have reached a compromise between a manual that tries to be a complete course in data analysis and one that assumes that users are concerned only with statement syntax. BMDP's reference manual is one of the few that provides examples that consider both simple and difficult data analyses. Each chapter's presentation illustrates virtually all of the available program options. The presentation is appropriate for technically sophisticated scientific users who need concise information about how to use this powerful research tool.
REFERENCE
McCLAVE, JAMES T.. and DIETRICH. FRANK H . cisco: Dellen.
(1982).Srotisri<s (2nd ed.).San Fran-
The Art of Asking Questions. Stanley L. Payne. Princeton, NJ: Princeton University Press, 1980. xvi + 249 pp. $5.95 (paperback).
Stanley Payne's classic treatise on survey question wording has been reissued in paperback. In a new foreword George Gallup, Sr., observes, "I have found none of Stanley Payne's trenchant observations to be any less appropriate today than they were almost thirty years ago, when the book was written" (p. vii). The reviewers agree with Gallup's assessment. This lively, sensible, and readable book will continue to provide useful guidance to professional survey planners for at least another 30 years, and probably for as long as human beings are involved in asking and answering survey questions. Although written in a lighthearted and humorous tone, The Art of Asking Questions is a serious attempt to catalog general rules for formulating individual survey questions. In doing so, Payne incorporates insights into the use of words and language, the nature of respondents, the importance of familiarity with the survey topic, and the mechanics of questionnaire construction itself. He emphasizes the need to match the frames of reference of the respondent and the questionnaire designer, to use words that are unambiguous and convey the intended meaning of a question, and to formulate the content and structure of response categories to capture accurately the respondent's reply. Payne's central thesis, stated at the outset (p. 16), is that problems in question construction can generally be traced to hidden and unwarranted assumptions; the purpose of the book is to provide the researcher with effective strategies for critically evaluating survey questions to reveal those assumptions and to eliminate their potential biasing effects. Yet as the title says, this is an "art" book and not a "science" book. For example, Payne gives a very complete exposition of the advantages and disadvantages of the use of open-ended questions without precisely prescribing when they are preferable to multiple choice questions and when they should be avoided. In the final analysis, judgments about the effectiveness and necessity of their use are left to the researcher. To take another example, Payne presents an enumeration of various ways of "loading" a question, which he defines as wording a question to lead some respondents to give different answers than they would give to another wording of what was intended to be the same issue (p. 77). His examples cover a wide range of ways in which unintended distortions in response might be introduced, and his prescriptions in this area ought to be considered by any conscientious questionnaire designer. One large question remains, though: If two wordings produce different results, which one is less loaded? The answer may not always be clear. A. BUBOLZ Although Payne's main focus is on the stimulus presented by single THOMAS Iowa State University questions, he occasionally addresses broader topics. His insights into the use of questionnaires to elicit public opinion (e.g., the crystallization of attitudes), the distinction between having an opinion and intensity of opinion, and the possible need to ask more than one question to obtain a meaningful answer (e.g., the discussion of Gallup's "quintamensional Geigy Scientific Tables, Vol. 2. design") are valid today and are now receiving some attention in the social science community (Turner and Martin 1984). C. Lentner (ed.). Basle: CIBA-Geigy, 1982. 240 pp. $19.95. While some areas of current questionnaire design practice have been Just over half of this volume is devoted to a collection of unusually anticipated by Payne's work, other areas of interest have also developed extensive tables with which to carry out statistical tests and confidence in the last 30 years. For example, in addition to the semantic and moprocedures. The use of these tables is explained in a clear but tightly tivational issues that Payne considered, cognitive issues affecting rewritten section that is, in effect, a 50-page manual on statistical methods. spondents' replies to survey questions have become important considThe remainder of the volume is a digression on mathematical tables, erations. Modern question designers are aware that they must consider symbols, definitions, and formulas. Welcome inclusions are Duncan's imperfect human memory, judgment heuristics that seem to be irramultiple comparisons test, distribution-free tolerance limits for the bi- tional, effects of repeated measurements in panel surveys, and effects nomial parameter, and confidence intervals for the Poisson parameter. resulting from the selection of particular examples as part of a survey question. Contemporary designers must weigh advantages and costs of Serious omissions are tables of the binomial and Poisson distributions. Collected primarily for medical doctors, the material would serve redundancy, extra cues, reference period lengths, and respondent rules. equally well for any user of statistical methods. To avoid being misled, Payne offers little guidance in these areas. The intervening decades have also seen considerable progress in questhe medical doctor should augment his shelf with at least one other book, say, McClave and Dietrich (1982). The Geigy Tables would be a strong tionnaire evaluation methods, which have moved well beyond the splitcandidate for the one book to include in a traveling statistical consult- ballot technique Payne advocates. While the book focuses on the wordant's briefcase; a very large amount of data and information is contained ing of individual survey questions, contemporary approaches to questionnaire design involve strategies for using multiple measures of varin very little space. iables of interest to survey designers. The wide variety of contemporary W. A. THOMPSON, JR. approaches share some basic features: (a) They assume responses to a single question have a "structure," (b) they use multiple items and University of Missouri-Columbia
Book Reviews
479
various procedures for combining responses to estimate the structural possibility of doing all surveys under interagency agreements dictate parameters (e.g.,factor analysis, multidimensional scaling), (c) they use the use of contractors to meet a substantial amount of the federal govtechniques for assessing the quality of the parameter estimates (vali- ernment's needs for survey work. dation studies), and (d) they use structural equation or other models This working paper does not examine the merits of the alternatives that link the response error estimates to the survey estimates. Attention but presupposes that the decision to contract out has been made. It focuses on response error variance in addition to response bias and on examines the technical skills required of personnel within the agency, item nonresponse in addition to response errors. the factors to be considered in preparing the request for proposal (RFP), These model-based approaches have largely developed out of the psy- the selection of the contractor, and the post-award activities needed to chometric and econometric domains. Other evaluation techniques in- make the project a success. These facets of the contracting process are clude frame of reference probing (e.g., Belson 1981), reinterviews, and examined from two perspectives: (a) existing practices and (b) what (less appropriate for attitudinal than for factual questions) record should be done within existing regulations. There are no recommenchecks. Another promising development, arising from increased use of dations for change in existing laws and regulations. A limited number centralized telephone interviewing facilities, is the detection of ques- of interviews were conducted with personnel of contracting agencies tionnaire problems by systematically monitoring interviewerlrespond- and with representatives of contractors to provide the basis for some ent interactions. (admittedly imperfect) generalizations concerning current practice. Methodological research over the 30 years since Payne wrote his book The word "should" appears in many of the substantive paragraphs. has modified some of his principles and recommendations. For example, In that sense, it is a prescriptive document-a "how-to" manual. In answer position biases are not as predictable as he implies, stating the some areas it is too prescriptive. With the liberal use of "should," it other half of a binary choice can be harmful, long questions are some- presents an ideal that can seldom be realized. The reader is not adetimes better than short ones, and open-ended questions are not a pan- quately warned concerning the idealized nature of the prescriptions. As acea for identifying the important dimensions for forced choice items. a result, the neophyte project officer who reads the document may be On the other hand, some of Payne's specific empirical findings do rep- profoundly shaken by his or her inability to measure up to the specilicate well (see Schuman and Presser 1981), although broader gener- fications. alizations remain elusive. We tried out some of Payne's advice by exIt is simply not possible for every project officer on a survey contract amining the questionnaires of the large demographic surveys conducted to be experienced and knowledgeable in forms clearance, sample design, by the Census Bureau. We looked in the questions for Payne's problem questionnaire design, interviewer training, editing and coding, data manwords and found very few. With only two exceptions, a question con- agement, and analysis. Nor is it always possible to have someone on taining a problem word also made a provision for the problem that Payne the project team who is qualified in all of these areas. The use of a said might arise. Of the items free of Payne's problem words, several project team, incidentally, is a good idea and the authors are to be had already been identified as potential problems on other grounds by commended for recommending it. independent panels of experts. So Payne's word list may be useful in It is clear that shortcomings of the agency's project team may be identifying some potential problems, although in this instance the expert compensated for by selecting an ethical and capable contractor with panels identified many more potentially troublesome items than could high performance standards. But the chapter on contractor selection be spotted by relying solely on the list. does not provide a set of rules that will guarantee the selection of the Certainly, the issues involved in questionnaire design research have best contractor-nor should it be expected to. The authors wisely chose expanded since The Art of Asking Questions was published in 1951, and not to catalog the alleged abuses surrounding the matter of contractor significant strides have been made in moving the field from an art to a selection but focus, instead, on some guidelines that should help in the science. Yet in many important ways the insights and conclusions of selection process. These guidelines appear both in the writing of the this book remain remarkably fresh and useful. Stripped to its essentials, RFP and in the evaluation of proposa1s:With few exceptions, one cannot survey research still requires asking questions of respondents. Stanley disagree with the authors' statements of what should be done. But there Payne, speaking across three decades, still offers wise counsel on how are some exceptions. For one thing, it is usually unrealistic to specify to do just that. the response rate that must be achieved. Instead, it may be a better The Center for Survey Methods Research practice to require the proposer to specify in detail the plan for achieving U.S. Bureau of the Census a high response rate, indicate what response rates have been achieved in the past with similar practices, and then figure his costs accordingly. Otherwise, an inexperienced contractor (or an unscrupulous one) may REFERENCES agree to the specified response rate and underestimate the costs of achieving it. When the survey produces a smaller response rate and the BELSON, W. A. (19811, The Design and Understanding of Survey Qlrestions, Aldershot, Engcost plus fixed fee budget is exhausted, the agency has no alternative land: Gofver. but to increase the budget or accept a lower response rate. The specSCHUMAN, H., and PRESSER, S. (19811, Questron nnd Answers in Artrtlrde Slrrveys. Exification of a minimum response rate thus works to the disadvantage of periments on Qlrestion F o r m , Wording, and Conrexr, New York: Academic Press. an experienced and ethical contractor. TURNER, C.F., and MARTIN, E. (eds.1(1984), Surveying Subject~rePhenomena, New York: Basic Books. We also disagree with the emphasis on subject matter experience. There are instances in which such experience is extremely important as, for example, in translating the goals of the study into measurement Contracting for Surveys. methods, questionnaire formats, and analysis plans. But if the study is Federal Committee on Statistical Methodology (Subcommittee on fundamentally statistical in nature, the requirements should focus on Contracting for Statistical Surveys). Washington, DC: Office of statistical competence, not subject matter competence. However, we strongly support the recommendation that a proposed contractor's prior Management and Budget, 1983. v + 58 pp. $10.00 (paperback). work, both in the area of subject matter and the survey methodology, Contracting for surveys is an important part of the statistical activities be reviewed and the appropriate references contacted. The authors of many government agencies. It provides an alternative to in-house properly note that agencies do not have lasting memories, so contractors conduct of surveys or the use of interagency agreements (generally with with poor performance records continue to receive awards, even in the the Bureau of the Census) to provide the needed services. same agency. Rather than being a manifestation of universal optimism, The in-house conduct of surveys is not feasible when surveys are it is probably partially a result of the transitory tenure of project offisporadic and do not justify the maintenance of a statistical staff for this cers-a problem that is also addressed by the authors. purpose. An OMB directive, OMB Circular A-76, provides rules for This working paper is probably the best available compendium of deciding whether to do work in house or to contract it out, based on actions that, if taken, would increase the validity and usefulness of surcomparative costs and other considerations. Such cost comparisons veys conducted for the federal government under contract. One would tend to favor contracting out when the statistical needs of the agency expect it to be required reading for anyone assigned the job of writing are not large and continuing. a survey RFP, serving on a selection panel, or monitoring a survey Interagency agreements with the Bureau of the Census are not always contract. Its tendency to be idealistic is easily offset by the information feasible. The resources of the Bureau are not limitless, the subject mat- it contains on the contracting process. Many contractors could also ter may be considered inappropriate, or the needs of the agency may benefit from reading it, particularly the chapter on evaluating proposals. not be consistent with the confidentiality policies of the Bureau of the EDWARD C. BRYANT Census. Westat, Inc. The infeasibility of maintaining adequate in-house staffs and the im-