Uncontrolled Market Data

CHECK STATION 1

same time period. Another type of market study relies on time-series data. Here, the firm chooses a single geographic area and varies its key decision variables over time to gauge market response. The firm might begin by setting a high price and a low advertising expenditure and observing the market response. Some time later, it may increase advertising; later still, it may lower price; and so on. Time-series experiments have the advantage that they test a single (and, one would hope, representative) population, thus avoiding some of the problems of uncontrolled factors encountered in cross-sectional studies. Whatever the type, traditional market tests and studies are expensive— often extremely so. A very rough rule of thumb holds that it costs $1 million and more to conduct a market test in 1 percent of the United States.

The last decade has seen an exponential increase in management’s use of Internet-based controlled market tests. For instance, Google might set out to test the effect of different Internet ads on customer click-through rates. By randomly assigning Internet visitors to different types of ads, it can compare the average response rate of 20,000 visitors seeing one ad to 20,000 visitors seeing the alternative ad. Because the populations of randomly selected visitors will be essentially identical, any difference in response rate can be attributed to the different ad treatment. The company Omniture provides clients immediate feedback on different Web-page designs by randomly testing specified alternatives and immediately comparing their effectiveness. The credit-card company Capital One does much the same thing when it runs controlled tests of different credit-card solicitations, systematically varying features such as the interest rate and cash-back percentage. Running and analyzing these newest kinds of controlled experiments can be accomplished at ever decreasing costs.3

An airline is considering expanding its business-class seating (offering more room and amenities to business travelers at a slightly higher fare than coach). Which method, survey or controlled market study, would you recommend to gather information for this decision?

In its everyday operation, the market itself produces a large amount of data. Many firms operate in multiple markets. Population, income, product features, product quality, prices, and advertising vary across markets and over time. All of this change creates both opportunity and difficulty for the market researcher. Change allows researchers to see how changing factors affect

3For a detailed discussion of controlled randomized market tests, see I. Ayres, Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart, Chapter 2 (New York: Bantam Dell Publishing Group, 2007). Of course, controlled market tests are not new. Fifty years ago, David Ogilvy, the lion of advertising, extolled the virtue of mail-order advertising because its impact was immediately testable. Either readers clipped the coupon, or they didn’t.