Monday, October 18, 2010

Taguchi Testing - Is It Good For Landing Page Optimization?

Taguchi Testing 

Is It Good For Landing

Page Optimization?



The short answer is: Taguchi Testing is Fractional Factorial Multivariate Testing and is therefore not a good tool for landing page optimization. Let's elaborate on that.

What Is Taguchi Testing?

Taguchi Testing is a variation of fractional factorial multivariate testing that was developed in the 1950's and 1960's by a Japanese mathematician named Genichi Taguchi. His testing methods, which were originally developed to improve manufacturing quality control, have gained popularity in the field of landing page optimization.

Multivariate tests are simply tests that have more than one input variable. Full factorial tests are tests that analyze every possible combination of inputs variables. Fractional factorial tests attempt to isolate and test only the subset of inputs that are deemed in advance to be important to the output. The Taguchi method is a set of partial factorial techniques that attempt to determine the best combination of attributes in the presence of a lot of variance or noise.

As mentioned, the Taguchi method was developed for use in manufacturing quality control. The Taguchi method has now been adopted as a tool for landing page optimization. The differences between a manufacturing floor and a landing page make Taguchi testing an incorrect tool for landing page optimization. This will be examined later in this article.  


Full Factorial Testing

The most well-known landing page optimization tool that uses full factorial testing is the Google Web Site Optimizer. This is probably the best free tool available for multivariate testing (testing lots of factors at once) on landing pages. The Google Web Site Optimizer will statistically determine which combination of landing page variables are most likely to produce the highest number of conversions. The Google Web Site Optimizer in its current state does not analyze interactions between the landing page variables being tested but only tells how each tested combination of variables performed in comparison with all others.

The Google Web Site Optimizer is also a great tool for A/B split-testing. This involves testing only one variable. Typically only two variations of one variable are being tested against each other. One variation is declared the winner when the Web Site Optimizer calculates that it has achieved an established percent certainty that it converts better than the opponent. For example, we might declare a variation to be the winner as soon as we are 80% certain that this variation outperforms its opponent.

For A/B split-testing, I prefer to use an Excel model that performs the same statistical test as the Google Web Site Optimizer (a one-tailed, two-sample, unpaired hypothesis test of proportion) but doesn't require any of the set-up steps that the Google Web Site Optimizer does. Here is a link to a blog article describing this Excel model with a video showing its use.


Taguchi Method Drawbacks

Taguchi's method of fractional factorial testing for landing page optimization has major drawbacks compared to full factorial landing page testing. They are as follows:

1) Fractional factorial methods assume that interactions between variables do not exist. This assumption is totally invalid for landing pages. Very strong variable interactions normally do exist on landing pages. For example, any landing page designer knows that a mismatch between a landing page headline and the body text will wipe out conversions.

Typically you will find lower order interactions occurring on landing pages. Lower order interactions are interactions that occur between a small number of variables, usually 2 or 3. Higher order interactions (interactions between more than 3 variables) are less common and usually much less important. In order for an interaction to be material and important, one of the factors usually has to be significant on its own   If you ignore interactions during landing page optimization, you will most likely not get the best results.

2) Fractional Factorial methods can only be used to test a small number of landing page combinations simultaneously. Typically the upper limit of the number of separate landing page combinations that can be tested simulataneously using fractional factorial methods is several hundred. Brainstorming marketers will quickly hit this limit after coming up with just a few factors and a couple of variations of each factor. 

Some landing page optimization terminology should be presented here. A variable or factor is an element on the landing page that you are varying during the test. A value is one of the states that a variable or factor (these two terms both mean the same) can take during your test. The branching factor is the number of values that an single variable or factor can take. Each variable has its own specific branching factor. A recipe is a unique combination of variable values available for a test. Another way of expressing this point (#2) would be to say "Fractional Factorial methods can only be used to test small number of recipes simultaneously."

3) Fractional Factorial methods are highly restrictive to test design. Fractional factorial methods do not allow the test designer much freedom when choosing the number of variables or the branching factor for each variable. The Taguchi method uses a matrix structure that works with less than two dozen very specific combinations of number of factors and branching level for the factors. The test designer must construct the test using one of those combinations of factor levels and branching factors. Full factorial methods have none of these restrictions.

4) Fractional Factorial methods require guessing at which factors to include in test. The restrictive nature of Fractional Factorial test design requires that the test designer pick the factors that he or she believes to be most important. The individual biases of the test designer will affect the selection of factors to include in the test.

5) Allocating more bandwidth to the baseline is not possible with Taguchi. The baseline is the current recipe that we are trying to beat with new recipes. It is very important that measurements of the baseline be valid because these measurements are the basis for comparison against results obtained for each recipe tested. To ensure validity of the baseline's measurements, it is a good idea to allocate at least 15% data collection (bandwidth) to sampling the baseline recipe. This type of data throttling is not possible with Fractional Factorial methods such as Taguchi. It is easily done with Full Factorial test methods.


Reasons for the Taguchi Mismatch

Genichi Taguchi developed his testing methods in the 1950's and 1960's to improve quality control on the manufacturing environment. His methods have become popular today in the field of landing page testing. The differences between manufacturing environment, for which the Taguchi method was intended, and today's landing page environment  create the mismatch that makes the Taguchi not the best choice for landing page optimization. Here are the main reasons for the mismatch:

1) Expensive manufacturing prototypes vs. free landing page prototypes. Retooling a production line for a new recipe is expensive. One of the major goals of Taguchi was to keep testing cost down by reducing the number of recipes to a minimum. In landing page testing, there is no additional cost to create more recipes (new variations of a landing page that will be shown to site visitors).

2) Manufacturing costs require a small test sizes vs. unlimited landing page test sizes. The high costs of manufacturing prototypes made small test sizes necessary. The Taguchi method keeps test size small by guessing at and testing only the most important factors. On the other hand, Full Factorial landing page testing methods and the low cost of creating new landing page recipes enables simultaneous testing of millions of recipes.

3) Small manufacturing test sizes could not test, and therefore did not assume, interaction between variables. Landing pages are known to have very strong interactions between variables. The Taguchi method was designed to assume no interaction between variables. That assumption can easily lead to incorrect results during landing page testing.

4) Manufacturing environment tests are smaller because statistical significance is normally reached quicker. Landing pages normally have low conversion rates and therefore require much larger test sizes to reach statistical significance. Manufacturing environment tests normally are designed to have a high probability of success. Landing page success rates (conversion rates) are typically below 1%.

5) Manufacturing test data is often continuous vs. Landing page data which is discrete and unrelated. Continuous data allows the test researcher to take a smaller number of samples and interpolate results for intermediate data points that were not collected. The possibility of interpolating continuous variable test results allows for smaller test sizes. Landing page variables are typically discrete, unrelated choices and therefore and do not allow interpolation for intermediate data ranges that were not collected.


Summary

Taguchi testing's origins in the manufacturing environment make it not the best tool for landing page optimization. Full Factorial methods should be used whenever possible to account for variable interaction and to allow for the widest possible number of recipes being tested.

Excel Master Series Blog Directory

Statistical Topics and Articles In Each Topic

Posted by at
Labels: , , , , , , , ,

1 comment:

Subscribe to: Post Comments (Atom)