If you do a large number of tests, falsely significant results are a problem. Remember that a 95% chance of something being true means there is a 5% chance of it being false. This means that of every 100 tests that show results significant at the 95% level, the odds are that five of them do so falsely. If you took a totally random, meaningless set of data and did 100 significance tests, the odds are that five tests would be falsely reported significant. As you can see, the more tests you do, the more of a problem these false positives are. You cannot tell which the false results are - you just know they are there. Limiting the number of tests to a small group chosen before the data is collected is one way to reduce the problem. If this isn't practical, there are other ways of solving this problem. The best approach from a statistical point of view is to repeat the study and see if you get the same results. If something is statistically significant in two separate studies, it is probably true. In real life it is not usually practical to repeat a survey, but you can use the "split halves" technique of dividing your sample randomly into two halves and do the tests on each. If something is significant in both halves, it is probably true. The main problem with this technique is that when you halve the sample size, a difference has to be larger to be statistically significant. The last common error is also important. Most significance tests assume you have a truly random sample. If your sample is not truly random, a significance test may overstate the accuracy of the results, because it only considers random error. The test cannot consider biases resulting from non-random error (for example a badly selected sample). To summarize: • • • •
In statistical terms, significant does not necessarily mean important. Probability values should be read in reverse (1 - p). Too many significance tests will turn up some falsely significant relationships. Check your sampling procedure to avoid bias.