Do’s and Don’ts of split A/B testing that only expert testers can tell
A/B testing or split testing is quite simple in concept. Create a variation, split the traffic, set your goal, and launch the experiment. However, the concept and the real world implementation are two different things. They have to match with each other in order to prove the hypothesis as right. In the case of A/B testing, the concept and implementation match up to a great extent. However, there is still some need to have practical aspects in the mind while implementing it. What you have read in the books, blogs, and forums might not always be true for every situation. Considering this, I have compiled a list of several do’s and don’ts of A/B testing collected from the works of various CRO experts and also the reviews of various users of the MockingFish A/B testing tool. So, let’s just see what they have to say. Let’s first dive into the part where you should avoid certain things in A/B testing. Don’ts of A/B testing 1. Don’t ever think about testing the control and variation separately It doesn’t matter you receive a very small website traffic or you have thousands of visits each day, it’s always foolish to test your original (control) and variation page separately. If you test the control in the first month and then test the variation in the next
month, you are already messing with how A/B testing works. The demographics, density, and quality of the traffic that was there for the first month is not going to remain the same for the second month. So, it doesn't make a sense testing two pages with same objective and goal at a different time frame. In fact, it may be possible that variation might be actually worse, but your results might show better sales with it after testing it. And long after implementing it, you will find that "you were wrong; variation actually sucks". Keynote: Just split the traffic between the two versions and test them simultaneously in the same time frame. 2. Don’t stop the test early; at least before 30 days The concept of statistical significance in A/B testing determines how confident is your A/B testing result. If you stop the test early, the A/B testing tool will have insufficient data to calculate the statistical significance. As a result, the confidence level would either show as very low or would be manipulated by the insufficient data to wrongly show it as a highly confident result. This is called as the false positive result, and trust me you don't want to fall a prey to it. Keynote: Run the test for a significant amount of time, don't stop it prematurely. Moreover, iterate the test for multiple times to get reliable results from your A/B testing tool. 3. Don’t include regular visitors in the variation’s share of traffic While splitting the traffic among the variation and the control, remember not to include the regular visitors in the test. They should keep seeing the original (control) page itself. The reason is, a sudden change on the site would be easily caught by the regular visitors, and it may come to them as a surprise. Furthermore, there is a chance that the test page may fail the test and never be implemented. That is, the visitors who were seeing the original page before the test, and were being shown test page during the test will again start to see the original page after the test. This frequent change might not smell sober to your regular visitors and they would start to see the site with that fishy look. Keynote: Always include the new visitors in the test page’s share of traffic. New visitors would never know if a testing is going on. However, for that your A/B testing tool should have the feature of URL and audience targeting.
4. Don’t ever overrule the test results A/B test results could be sometimes peculiar to believe. Your gut might favor the variation while the result might say the original page is the best. No matter, how much insane the results are, you should not simply overrule them. The pink colored button might not appeal you on your green themed website, but the A/B test results could show it as a winner. I can understand that your gut doesn't allow you to believe the result, but don't just scrap it outright. Keynote: Don’t reject the results outright. If you are not confident about it, just iterate it for a ton of time till you are confident about the result. Remember, your goal is to win conversions, not to appeal your eyes. Do’s of A/B testing The do's are quite simple. Most of them were also explained in the keynotes below every don'ts. So, let see what else should you do in your A/B testing experiment. 1. Know how long you should test before stopping it Already explained, stopping the test prematurely can cause make the results show wrong results. Therefore you should run the test for quite a sufficient time to make sure it has reached the desired confidence level and it is statistically significant. How would you do that? You should do that by looking at your traffic size, and the number of variations (recommended to test one change at a time). If your website receives a well to do traffic, you might reach the significance level real quick, while with low website traffic, you should carry on the test for quite a time. You can use an online A/B Split Test Duration Calculator, to help you decide a suitable time for your test. 2. Always show the same page to a particular visitor If a visitor is seeing the control page while the test is live, then he should always see the control page only. Same goes with the variation page. Showing different pages on different visits by the same user would make him confused and may result is blunders in some situations as well. For example, you are testing your promotional offers. On the first visit, the user might see the offer and decide to make a purchase, say after an hour. Next time, the offer just vanishes because he is now seeing the control page. That's not
good, and you know it. 3. Make the testing changes uniform across the website There are certain elements on a page which appear on multiple pages of your website. For example, the buy now or Add to cart button will appear on every product page of the website. So if you are testing the orange color for your Add cart button or Buy Now button, you should change it across the site, not just on a single page. Showing Orange color on one page and green color on another page to the same visitor will deviate your test result. 4. Do several iterations to get a more confident result Truth is, A/B test results are not always at their best in the first iteration. Sometimes, even you might feel that there is something wrong with the result. There is no need to get discouraged, A/B testing could be a bit complex but it is always helpful. Just iterate your tests for as many times. Note down all the results. Finally, put out all the positive results together and calculate the average, do the same with the negative results. To be more confident, you should also calculate the variance from all the iterations done. In this way, you will have a more confident and statistically significant result.