Multivariate and A/B Testing Written by Sandro Alberti; sandro@fen-om.com Sandro is a Web designer/developer who enjoys educating others about design, both in theory and practice.
INTRODUCTION
"Testing variations of your Web site automatically",... THAT is what this is about. Both multivariate and A/B tests take care of automated performance measurements, based on predetermined variations. The difference between the two? Multivariate: Creates/measures combinations (variations within multiple sections) in one single page. A/B: Creates/measures entirely different variations of a page.
In both cases, what is measured is how effectively traffic is driven to a particular goal, which can be defined in 5 different ways: One-to-one: Both 'multivariate' and 'A/B' can be set up for one-to-one scenarios (the starting page, although it can appear in many different ways, is one page, and traffic is measured to one single result page). This is useful if you are only interested in measuring the traffic that ends up at a single, particular goal page, from one defined start page. Many-to-one: And they can be set up in site-wide headers, for example, which allows many-to-one testing (many starting pages vary in the same ways, and lead to one result page). Similar to one-to-one, but it doesn't matter where traffic is arriving from, in the site, as long as it arrives at the single, particular goal page. One-to-many: They can also be set up for one-to-many scenarios (traffic measured from the variable page to 'any' page). This set-up is a bit 'looser' than the rest, but still measures whether visitors choose to remain in your site after visiting your variation starting page. Many-to-many: This works like one-to-many, but allows for different variation starting pages and different result pages.
Page 1
Time on page: Sometimes it is valuable to measure visitors who remain on a variation test page for a while, even if they don't continue on to another page in the site. In those cases, variation programs can count a conversion after the page has been loaded in the browser for a certain amount of time. This method is useful for online demos, videos, or other pages that don't receive a lot of traditional conversions. Now, as software for variation testing becomes ever more accessible, you can easily test multiple changes simultaneously, saving you time, speeding up your optimization schedule, and increasing your profitability.
SOME DETAILS
Beyond what each of the 2 processes achieves ('multivariate' and 'A/B'), it's worth delving a little bit more into the differences, before selecting one for testing: -----Variability Essentially, multivariate testing can be thought of as a series of numerous A/B tests performed on one page at the same time. At any given moment the variation software will select and display one single 'combination' of variables. So, one might think that a multivariate test with 16 combinations is pretty much the same as an A/B test with 16 page designs. However, there is one critical difference: the number of variables ('variability' of the options). As defined by Google in its variation-testing software (Website Optimizer): "The number of combinations is calculated by multiplying the number of variations for each section." In our example, consider that, under multivariate testing, the 16 combinations could be produced from... 4 page elements/ sections, each of which alternates between 2 variations (a header with 2 variations, a main image with 2 variations, a call-to-action with 2 variations, and a 'go' button that is designed in one of two different ways): 2x2x2x2=16. Or, it could be page with only 3 varying elements/sections, where the header has not only 2 but 4 variations, the main image has 2 variations, and a call-to-action has 2 variations (but the 'go' button never changes): 4x2x2=16. AND... multivariate testing is not only for adding variation to static, 'discreet' page elements/sections. With multivariate testing, you can also modify things like... CSS styles in the head of the document!!! So, a single variation could be... change in color and font size for the ENTIRE page. Or even a complete new style sheet for the entire page (the variable could be the <link> call to an external style sheet). On the other hand, if you have created 16 separate page designs for A/B testing, you really only want to change ONE single element or section in each (the same in all 16 pages), so that you know WHAT it is that your audience preferred. Once you add one more element/section to modify, you are undertaking 'multivariate' design. So the 16 pages in multivariate testing, and the 16 made for A/B testing,... they can't be exactly the same. At most, only one of the multivariate pages will look like one of the A/B pages. -----Look and Feel The level of aesthetic control tends to be greater under A/B testing that under multivariate testing. Under multivariate testing, the visual/experience changes are very relatively focused (although numbering more than those in A/B testing). Typically, if what you want to test is the look and feel of a
Page 2
page (and maybe only one section/element changes in a minor way), A/B testing has been invented for that purpose. And when the look and feel is pretty much established, but you need to gauge the effects of changes in elements such as headers, product images, calls to action, or buttons, multivariate testing is there. However, we have seen that, when we make complete style sheets into variables, we can get the same level of visual control as in A/B testing. So, with a little bit of work, look-and-feel can also be brought into the multivariate equation. -----Duration of Experiment/ Number of Views Multivariate: Google Website Optimizer mentions that multivariate testing is a good choice if "your page receives more than ~1,000 views a week". This makes sense if you consider what we have reviewed: there is great variability in multivariable testing. Even if you keep you total combinations to a minimum, it takes as many days to run the experiment as the number of combinations. So with 16 combinations, the experiment should run 16 days. And... this is IF your current conversion rate is 30% and you are looking for a variation that will offer 20% improvement (you need to have some analytics in place previously, to know what your conversion rate is now). Duration becomes half if your current conversion is 45% and you expect a 20% improvement. Duration also becomes half if your current conversion is 30% and you expect a 30% improvement. Google Website Optimizer offers an online calculator for you to estimate the duration of your multivariate experiment:
www.google.com/support/websiteoptimizer/bin/answer.py?answer=61688
A/B: Runs until you have a clear winner. This could vary, of course, but results could be satisfied in one week (although the more time the better). From Google Website Optimizer: "Generally, we recommend around 100 conversions per page variation over the course of your experiment. For example, an experiment with 3 page variations will typically need around 300 conversions before drawing any conclusions."
HOW TO DESIGN VARIATIONS
Wikipedia describes 3 main methods for the design of variations: (mainly applicable to multivariate testing) Discrete Choice: Models how people make tradeoffs in the context of a purchase decision. Optimal Design: In various iterations, tests the maximum number of creative permutations in the shortest period of time (considering relationships, interactions, and constraints). Taguchi Method: Reduces variations but still give statistically valid results on individual content elements.
SOFTWARE FOR TESTING VARIABLES Avenseo Conversion Works Adobe Omniture Amadesa DIVOLUTION Maxymiser Google Website Optimizer Vertster SiteSpect
Page 3