Significance of Statistical Analysis in A/B Testing A/B testing is one of the most used tool for web testing and conversion optimization. Today, most of the websites are doing A/B testing at some point in time to optimize their conversion rates. It allows the marketing professionals and website owners to make a more accurate decision regarding the changes and modification that should be done on a site to achieve an accelerated rate of conversion. The proposed modifications in the websites are not just judged with intuition, but there is a deep statistical analysis of the factors that affect the browsing behavior of the visitors. Similarly, it also protects you from making major mistakes that could harm the reputation and conversion behavior of your site. Talking about the procedures to get a better statistical result, first of all, we create a variation page of the original page we own on the website; then we split the visitors by diverting them on two different pages for the same URL. Now, each of the variations will be browsed by a fixed proportion of the audience, and they will surely show some implications. Finally, we collect the data regarding the performance of the web pages, also called as metrics. Now comes the work of statistical analysis when we analyze the data and pick the best among them. But, how can to know that which one is the best? Look at the mistakes we make while choosing a winner page Foremost task in choosing the best is that we are able to understand the common mistakes that we can make in the meantime. There are two common mistakes that we make while picking the winner page: a.
b.
We do not consider the null hypothesis. We take a brief look at the data, compare the figures of conversion rates, and claim the one with the highest conversion as winner page. But we tend to overlook the fact that there is no such difference between the conversion rates. We ignore the conditions that originate the differences of conversion rates between the variation and control page. We call it "false positive." The second mistake we make; after looking at the data when we see no major differences, we conclude that our hypothesis was wrong, the control page is the best case we have got on our site. But, is it so? I think, its an Apple to Apple comparison, where we are considering the performance metrics of a page designed by us with another page that we believe is the best. I mean, it may happen that the variation page was not competent enough to beat the control page, but it does not make our control page a winner. This factor is called a "false negative."
How do we avoid these mistakes in A/B testing? There is a very simple answer to this question. For saving us from making these blunder mistakes in our A/B testing, we set a proper sample size and define the parameters for our test. To avoid the error of false positive, we should consider the use of confidence level. It is
also called as the statistical significance of the A/B testing. For example in MockingFish A/B testing, the optimum confidence level is set to 95%, where a result having a confidence level of 95% or more is considered as the winner. To avoid the mistake of false negative, we need to define some additional parameter that will give a more certain result. One parameter that MockingFish uses is the minimal difference in performance we wish to test, and another parameter is the probability of detecting that difference, if it is detected. This probability factor is also called as statistical power and is usually taken as 80%. To avoid the mistake of false negative, we need to define some additional parameter that will give a more certain result. One parameter that Mocking Fish uses is the minimal difference in performance we wish to test, and another parameter is the probability of detecting that difference, if it is detected. This probability factor is also called as statistical power and is usually taken as 80%.