Citizen Surveys: Improving

by Mark Ellickson

Citizen Surveys:

Improving Local Government Performance Using Community Feedback

Citizen surveys are a powerful tool for local government managers and elected officials seeking to improve government performance and community livability. Unlike other forms of community engagement, professionally designed and implemented citizen surveys provide accurate and unbiased feedback that is representative of the whole community, not just the vocal few. Citizen surveys are also extremely versatile. Survey results can assist local governments with strategic planning and budgeting, measuring citizen satisfaction with municipal services and identifying areas for improvement, assessing community needs and prioritizing those needs, tracking resident perceptions of community quality of life, and evaluating government performance, to name a few. Citizen perceptions of community characteristics commonly measured and monitored using citizen surveys include the following: • Public Safety • Economic Health • Schools/Education • Natural Environment • Image/Reputation • Overall Appearance • Sense of Community • Recreation and Wellness Opportunities • Mobility/Traffic • Built Environment/Housing

Designing and implementing a credible citizen survey involves a series of interrelated steps that require specialized expertise, attention to detail and precision. Specifically, the various steps in the survey process include planning, data collection, sampling, questionnaire design, data analysis and report writing, and communication of results.

Planning

The most important step in the survey process is having a clear understanding of the goals and objectives of the study. This step is critical because it determines the data collection method, sampling procedure, and the types of questions to consider when designing the questionnaire. A professional survey research firm will work closely with local government officials during this stage to ensure that everyone is onboard with the project’s goals and objectives before proceeding. Careful planning will also facilitate the development of a realistic timetable for completing the survey. Data Collection

There are three primary data collection methods associated with citizen surveys — telephone, mail, and online. Historically, telephone surveys have been the data collection method of choice but due to caller ID, call blocking, cell phones, telephone scams, and public fatigue with telephone surveys in general, response rates have plummeted (see Figure 1) raising concerns about the accuracy of telephone survey results. Currently, the most common method of conducting citizen surveys is by mail. Mail surveys have several important advantages over telephone surveys including a significantly higher response rate due in part to its less intrusive nature (mail surveys are self-administered), anonymity of survey responses, and the ability to convey survey legitimacy via a city’s logo on the mailing envelope and signed cover letter by a major city official such as the mayor or city manager. Also, respondents of self-administered surveys (mail and online) are less likely to suffer from “social desirability bias” (i.e., giving “socially acceptable” responses) as is the case with telephone surveys. Finally, mailing lists of community households are regularly updated by the USPS and have nearly 100% coverage in most areas. No other data collection method comes close to that standard.

Online surveys are a relatively new data collection alternative. They can be significantly less expensive than mail or telephone surveys, easier to

Figure 1. Telephone Survey Response Rates: 1997 – 2018. Source: Pew Research Center

complete, and more likely to attract younger respondents. On the downside, response rates are generally low and comprehensive lists of email addresses do not exist, thus eliminating the possibility of drawing a scientifically valid sample from a community. Consequently, combining mail surveys with an online component is currently the best method of conducting a citizen survey. This hybrid approach combines the strengths of a mail survey (accurate mailing list, scientifically valid sample, convenience, legitimacy) with those of an online survey (younger respondents, ease of completion). with a known margin of error (e.g., +/- 5.0%). In contrast, a survey based on a nonprobability sample is open to anyone interested in taking the survey. Since selection bias is not controlled, survey results cannot be generalized back to the population with any degree of confidence regarding their accuracy.

A recent trend involving probabilitybased mail citizen surveys is to post a link of the mail survey on a city’s website and its social media sites shortly after the mail survey cut-off date. The online version of the “official” mail survey is referred to as an “opt-in” survey because it allows all residents in the community not randomly selected to receive a questionnaire to participate in the survey. Since it is conducted after the “official” survey has been closed, survey results from the “opt-in” survey (nonprobability sample) can be compared to those from the “official” probability-based survey. In some instances, it may be possible to merge the two data sets if the demographic and geographic profiles of the two groups of respondents are similar. This “hybrid” data set has the “potential” to be a representative sample of the community, while at the same time maximizing community input.

Questionnaire Design

Questionnaire design is arguably the most difficult step in the survey research process. The correct choice of words and phrasing of a question is critical if all respondents are to interpret questions the same way. A well-constructed questionnaire will elicit survey responses

OFFICES AROUND THE COUNTRY AND IN YOUR BACKYARD.

Sampling

Ideally, everyone in the community would be asked to participate in a citizen survey and sampling would be unnecessary. Unfortunately, time and money are limited, and some form of sampling is required. There are two general types of samples — probability and nonprobability. Probability samples are based on random sampling techniques that give residents an equal opportunity to participate in the survey thus eliminating selection bias. Findings from a probability sample can be generalized back to the population 12655 Olive Blvd. Suite 200 St. Louis, MO 63141

WWW.SIKICH.COM

314.275.7277

that are valid, reliable and accurate. To do so, the questions must be unbiased, clear and concise; while response options must be balanced, mutually exclusive and exhaustive. Survey questions should flow logically from one section to another, and each question must provide actionable information linked to specific survey goals and objectives.

Data Analysis And Report Writing

Once the survey data has been collected and before any statistical analysis takes place, the demographic profile of survey respondents must be compared to that of the population surveyed using U.S. Census data. If significant differences exist, the sample data must be statistically weighted to reflect the population demographic profile.

Statistical analysis of the sample data typically includes frequencies and crosstabulations.

Frequencies report the number and percentage of people that responded to each category of a survey question. This is the most basic output from a citizen survey. Crosstabulations are frequencies broken out by demographic and geographic subgroups to determine if statistically significant differences exist.

Other types of analysis often associated with citizen surveys include the following: • Trendline Analysis: examines changes in citizen opinions and behaviors over time • Benchmarking: compares survey results to those of other “similar” communities • Importance – Satisfaction Matrix: identifies city services of high importance, but low satisfaction • Key Driver Analysis: weighs the relative importance of factors when predicting an outcome • GIS Mapping: mapping survey results by geographic location, e.g., zip code

Most written reports consist of an executive summary, description of the survey methodology, and demographic profile of survey respondents. The main body of the report will integrate written analysis of the survey questions with judicious use of tables, charts and figures. A copy of the questionnaire along with a tabular display of the crosstabulation analysis of every survey question by all demographic and geographic subgroups is usually included in the appendix.

Communication Of Results

To maximize the usefulness of a citizen survey, a city should share the results externally with the media and public by posting a copy of the report on the city’s website and social media platforms. Results should also be shared internally by referring to them in discussions with elected officials for long-term strategic planning and shaping budget priorities, and with department heads and employees for purposes of tracking customer service quality and delivery of those services and adjusting resources accordingly.

Conclusion

Local government managers and elected officials seeking to improve government performance and community livability should consider augmenting traditional performance measures with ongoing citizen surveys that take the pulse of the entire community. In doing so, they broaden the scope of community participation, narrow the gap between community perceptions and city knowledge of those perceptions, and expand avenues of public communication and trust.

Mark Ellickson, Ph.D. (Political Science), is president of Opinion Research Specialists, a full-service research firm located in Springfield, Missouri. Since 1991, Opinion Research Specialists has conducted hundreds of community surveys throughout Missouri and provided local government decision makers with accurate, unbiased resident feedback. Mark can be reached at mellickson@ ellicksonresearch.com or at (417) 889-4506. For more information about Opinion Research Specialists and its staff, visit ellicksonresearch. com.