7 do’s and don’ts of ab testing updated

7 Do’s and Don’ts of A/B Testing

A/B testing can gain fortunes if it is done right. You can find tons of articles and posts explaining the best A/B testing ideas and hacks. They all pull up pretty well and provide some valuable information. However, I don't want to be complex and high end with my post. I will keep it simple and fundamentals oriented because somewhere in the temptation of bigger benefits, we always tend to ignore the simple things which actually matter.

So, in the same path, I have complied a list of 7 simple Do's and Don'ts of A/B testing. Following these do and Don’t will definitely allow you to run an error-free A/B testing experiment, and gain preciser results. So, let’s hop-in.

1. Do test multiple variables- There isn’t any scarcity of variables that can influence the sales and conversion of your website. There are tons of them and always available for testing on different grounds. As an example, you can effectively test these variables: - Color - Size - Templates - Offers - Subject line

- Layout - Position etc.

Variables are undoubtedly endless. The more variables you test, the better insights you gain in your marketing strategy, and much better results you get from your business initiatives. So, identify and test as many variables you can.

2. Don’t test all variables at once- Well, something being available in massive abundance also create a confusion on ‘which one to use’,’how many to use’. Overwhelmed by tons of variables, we start to think it would be so time-taking to test each of them one by one. So, why don’t we test them all at once? It would save a lot of time.

This is a mistake and a totally wrong approach to test even the multiple variables at a time; testing them all at once is even far away to think.

When you test multiple variables at a time, it becomes difficult to isolate the interpretations from a test result. For example, if you have tested your Landing page CTA button by modifying two variables- color and position, and the variation has won, you will get confused in deciding which variable made the variation better- changing the color or position?

So, always test one variable at a time to gain confident results, and implications that are easier to interpret.

3. Do test simple variables- A/B testing is a huge domain. From small startups to biggest enterprises, all do testing at some point in time. In fact, they do it regularly. When you are just starting up with the A/B testing, do not get overwhelmed by what others are doing. Start with being simple and run the tests for small and simple variables initially- say, button color, size, position etc.

While advance testing is beneficial, there is nothing bad in starting with the simple tweaks at the beginning and then advancing the approach gradually.

4. Don’t hesitate to head up with Advanced Testing- The last point advocated the simple testing approaches at the beginning. However, once you gained enough experience with the simple and basic tests, it's time you go for the advanced ones. Don't hesitate to deploy complex tests such as A/B/n testing, Page level testing, Mobile testing and so on. Advanced testing structures will provide two almost completely different page designs to test.

PS: Don’t get confused with my statement that you should not test multiple

variables at a time. In advance testing, if you consider the whole page as a single variable, it would be easier to grasp.

For example, You can test your multi-page checkout structure by replacing it with a one-page checkout. In this case, the whole checkout page (s) would be considered as a single variable because you are considering to rewrite the whole page into something completely different from the original page.

5. Do regular testing- A/B testing isn't something that you do once and sit relaxing hands on hands waiting for the predicted conversion improvements. I don't see any reason for why we should not always be testing. Moreover, why shouldn't we run multiple experiments at a time if still have enough variables and website traffic? With this point, in my do's list, I strongly support continuous and multiple testing.

For example, If you are already running a test for CTA on a landing page, I don't see any harm in deploying another test for layout change on the category page.

PS: If you are a new entrant and do not have enough website traffic, avoid multiple testing for a while till you have a strong bandwidth and experience.

6. Don’t Stop at first statistical significance- Keep an open mind. Don't trust everything you see, especially when it comes to the statistical significance of the A/B testing results. It is possible that you could have made any of the numerous mistakes in A/B testing, affecting the statistical significance of the result.

For example, just because one headline performed better than the original, it doesn't mean the test was significant.

The issue that prevails here is the false positives. False positives incorrectly show a difference between the test pages and present a less confident data as statistically significant one. The more you check your results, the more you would confront a result affected by false positives. If you stop your test early at the first sign of significance, there is a strong chance to end up with the deceptive false positive results.

7. Do check multiple times if the results are significantOnce you reach the first statistical significance, analyze the result. If the difference between the pages is too small, it may be possible that variable you tested was not influencing the users. If there is a drastic difference between the tests which you didn’t expect, determine whether the test results are

statistically significant. You can use any of the Statistical Significance tools available, for example, Optimizely provides a good tool here. If Statistical Significance is not confident enough, don’t hesitate to re-test the same.

Otherwise, you might just fall prey to false positives or just blame your A/B testing tool.

To conclude Do’s and don’t for A/B testing could be numerous. In fact you might have read various others that are not listed here. I have tried to not repeat the generic points and included some really important ones which are surrounded by various Myths and misconceptions. That’s why, I might have missed many others, what do you think? Do you have any addition to the list from your side?