# Taguchi Testing

# Is It Good For Landing

# Page Optimization?

The short answer is: Taguchi Testing is Fractional Factorial Multivariate Testing and is therefore not a good tool for landing page optimization. Let's elaborate on that.

**What Is Taguchi Testing?**

Taguchi Testing is a variation of fractional factorial multivariate testing that was developed in the 1950's and 1960's by a Japanese mathematician named Genichi Taguchi. His testing methods, which were originally developed to improve manufacturing quality control, have gained popularity in the field of landing page optimization.

Multivariate tests are simply tests that have more than one input variable. Full factorial tests are tests that analyze every possible combination of inputs variables. Fractional factorial tests attempt to isolate and test only the subset of inputs that are deemed in advance to be important to the output. The Taguchi method is a set of partial factorial techniques that attempt to determine the best combination of attributes in the presence of a lot of variance or noise.

As mentioned, the Taguchi method was developed for use in manufacturing quality control. The Taguchi method has now been adopted as a tool for landing page optimization. The differences between a manufacturing floor and a landing page make Taguchi testing an incorrect tool for landing page optimization. This will be examined later in this article.

**Full Factorial Testing**

The most well-known landing page optimization tool that uses full factorial testing is the Google Web Site Optimizer. This is probably the best free tool available for multivariate testing (testing lots of factors at once) on landing pages. The Google Web Site Optimizer will statistically determine which combination of landing page variables are most likely to produce the highest number of conversions. The Google Web Site Optimizer in its current state does not analyze interactions between the landing page variables being tested but only tells how each tested combination of variables performed in comparison with all others.

The Google Web Site Optimizer is also a great tool for A/B split-testing. This involves testing only one variable. Typically only two variations of one variable are being tested against each other. One variation is declared the winner when the Web Site Optimizer calculates that it has achieved an established percent certainty that it converts better than the opponent. For example, we might declare a variation to be the winner as soon as we are 80% certain that this variation outperforms its opponent.

For A/B split-testing, I prefer to use an Excel model that performs the same statistical test as the Google Web Site Optimizer (a one-tailed, two-sample, unpaired hypothesis test of proportion) but doesn't require any of the set-up steps that the Google Web Site Optimizer does. Here is a link to a blog article describing this Excel model with a video showing its use.

**Taguchi Method Drawbacks**

Taguchi's method of fractional factorial testing for landing page optimization has major drawbacks compared to full factorial landing page testing. They are as follows:

**1) Fractional factorial methods assume that interactions between variables do not exist**. This assumption is totally invalid for landing pages. Very strong variable interactions normally do exist on landing pages. For example, any landing page designer knows that a mismatch between a landing page headline and the body text will wipe out conversions.

Typically you will find lower order interactions occurring on landing pages. Lower order interactions are interactions that occur between a small number of variables, usually 2 or 3. Higher order interactions (interactions between more than 3 variables) are less common and usually much less important. In order for an interaction to be material and important, one of the factors usually has to be significant on its own If you ignore interactions during landing page optimization, you will most likely not get the best results.

**2) Fractional Factorial methods can only be used to test a small number of landing page combinations simultaneously.**Typically the upper limit of the number of separate landing page combinations that can be tested simulataneously using fractional factorial methods is several hundred. Brainstorming marketers will quickly hit this limit after coming up with just a few factors and a couple of variations of each factor.

Some landing page optimization terminology should be presented here. A

**variable**or

**factor**is an element on the landing page that you are varying during the test. A

**value**is one of the states that a variable or factor (these two terms both mean the same) can take during your test. The

**branching factor**is the number of values that an single variable or factor can take. Each variable has its own specific branching factor. A

**recipe**is a unique combination of variable values available for a test. Another way of expressing this point (#2) would be to say "Fractional Factorial methods can only be used to test small number of recipes simultaneously."

**3) Fractional Factorial methods are highly restrictive to test design.**Fractional factorial methods do not allow the test designer much freedom when choosing the number of variables or the branching factor for each variable. The Taguchi method uses a matrix structure that works with less than two dozen very specific combinations of number of factors and branching level for the factors. The test designer must construct the test using one of those combinations of factor levels and branching factors. Full factorial methods have none of these restrictions.

**4) Fractional Factorial methods require guessing at which factors to include in test.**The restrictive nature of Fractional Factorial test design requires that the test designer pick the factors that he or she believes to be most important. The individual biases of the test designer will affect the selection of factors to include in the test.

**5) Allocating more bandwidth to the baseline is not possible with Taguchi.**The baseline is the current recipe that we are trying to beat with new recipes. It is very important that measurements of the baseline be valid because these measurements are the basis for comparison against results obtained for each recipe tested. To ensure validity of the baseline's measurements, it is a good idea to allocate at least 15% data collection (bandwidth) to sampling the baseline recipe. This type of data throttling is not possible with Fractional Factorial methods such as Taguchi. It is easily done with Full Factorial test methods.

**Reasons for the Taguchi Mismatch**

Genichi Taguchi developed his testing methods in the 1950's and 1960's to improve quality control on the manufacturing environment. His methods have become popular today in the field of landing page testing. The differences between manufacturing environment, for which the Taguchi method was intended, and today's landing page environment create the mismatch that makes the Taguchi not the best choice for landing page optimization. Here are the main reasons for the mismatch:

**1) Expensive manufacturing prototypes vs. free landing page prototypes.**Retooling a production line for a new recipe is expensive. One of the major goals of Taguchi was to keep testing cost down by reducing the number of recipes to a minimum. In landing page testing, there is no additional cost to create more recipes (new variations of a landing page that will be shown to site visitors).

**2) Manufacturing costs require a small test sizes vs. unlimited landing page test sizes.**The high costs of manufacturing prototypes made small test sizes necessary. The Taguchi method keeps test size small by guessing at and testing only the most important factors. On the other hand, Full Factorial landing page testing methods and the low cost of creating new landing page recipes enables simultaneous testing of millions of recipes.

**3) Small manufacturing test sizes could not test, and therefore did not assume, interaction between variables. Landing pages are known to have very strong interactions between variables.**The Taguchi method was designed to assume no interaction between variables. That assumption can easily lead to incorrect results during landing page testing.

**4) Manufacturing environment tests are smaller because statistical significance is normally reached quicker. Landing pages normally have low conversion rates and therefore require much larger test sizes to reach statistical significance.**Manufacturing environment tests normally are designed to have a high probability of success. Landing page success rates (conversion rates) are typically below 1%.

**5) Manufacturing test data is often continuous vs. Landing page data which is discrete and unrelated.**Continuous data allows the test researcher to take a smaller number of samples and interpolate results for intermediate data points that were not collected. The possibility of interpolating continuous variable test results allows for smaller test sizes. Landing page variables are typically discrete, unrelated choices and therefore and do not allow interpolation for intermediate data ranges that were not collected.

**Summary**

Taguchi testing's origins in the manufacturing environment make it not the best tool for landing page optimization. Full Factorial methods should be used whenever possible to account for variable interaction and to allow for the widest possible number of recipes being tested.

**Excel Master Series Blog Directory**

Statistical Topics and Articles In Each Topic

- Histograms in Excel
- Bar Chart in Excel
- Combinations & Permutations in Excel
- Normal Distribution in Excel
- Overview of the Normal Distribution
- Normal Distribution’s PDF (Probability Density Function) in Excel 2010 and Excel 2013
- Normal Distribution’s CDF (Cumulative Distribution Function) in Excel 2010 and Excel 2013
- Solving Normal Distribution Problems in Excel 2010 and Excel 2013
- Overview of the Standard Normal Distribution in Excel 2010 and Excel 2013
- An Important Difference Between the t and Normal Distribution Graphs
- The Empirical Rule and Chebyshev’s Theorem in Excel – Calculating How Much Data Is a Certain Distance From the Mean
- Demonstrating the Central Limit Theorem In Excel 2010 and Excel 2013 In An Easy-To-Understand Way

- t-Distribution in Excel
- Binomial Distribution in Excel
- z-Tests in Excel
- t-Tests in Excel
- Overview of t-Tests: Hypothesis Tests that Use the t-Distribution
- 1-Sample t-Tests in Excel
- Overview of the 1-Sample t-Test in Excel 2010 and Excel 2013
- Excel Normality Testing For the 1-Sample t-Test in Excel 2010 and Excel 2013
- 1-Sample t-Test – Effect Size in Excel 2010 and Excel 2013
- 1-Sample t-Test Power With G*Power Utility
- Wilcoxon Signed-Rank Test As a 1-Sample t-Test Alternative in Excel 2010 and Excel 2013
- Sign Test As a 1-Sample t-Test Alternative in Excel 2010 and Excel 2013

- 2-Independent-Sample Pooled t-Tests in Excel
- Overview of 2-Independent-Sample Pooled t-Test in Excel 2010 and Excel 2013
- Excel Variance Tests: Levene’s, Brown-Forsythe, and F Test For 2-Sample Pooled t-Test in Excel 2010 and Excel 2013
- Excel Normality Tests Kolmogorov-Smirnov, Anderson-Darling, and Shapiro Wilk Tests For Two-Sample Pooled t-Test
- Two-Independent-Sample Pooled t-Test - All Excel Calculations
- 2-Sample Pooled t-Test – Effect Size in Excel 2010 and Excel 2013
- 2-Sample Pooled t-Test Power With G*Power Utility
- Mann-Whitney U Test in Excel as 2-Sample Pooled t-Test Nonparametric Alternative in Excel 2010 and Excel 2013
- 2-Sample Pooled t-Test = Single-Factor ANOVA With 2 Sample Groups

- 2-Independent-Sample Unpooled t-Tests in Excel
- 2-Independent-Sample Unpooled t-Test in Excel 2010 and Excel 2013
- Variance Tests: Levene’s Test, Brown-Forsythe Test, and F-Test in Excel For 2-Sample Unpooled t-Test
- Excel Normality Tests Kolmogorov-Smirnov, Anderson-Darling, and Shapiro-Wilk For 2-Sample Unpooled t-Test
- 2-Sample Unpooled t-Test Excel Calculations, Formulas, and Tools
- Effect Size for a 2-Independent-Sample Unpooled t-Test in Excel 2010 and Excel 2013
- Test Power of a 2-Independent Sample Unpooled t-Test With G-Power Utility

- Paired (2-Sample Dependent) t-Tests in Excel
- Paired t-Test in Excel 2010 and Excel 2013
- Excel Normality Testing of Paired t-Test Data
- Paired t-Test Excel Calculations, Formulas, and Tools
- Paired t-Test – Effect Size in Excel 2010, and Excel 2013
- Paired t-Test – Test Power With G-Power Utility
- Wilcoxon Signed-Rank Test As a Paired t-Test Alternative
- Sign Test in Excel As A Paired t-Test Alternative

- Hypothesis Tests of Proportion in Excel
- Hypothesis Tests of Proportion Overview (Hypothesis Testing On Binomial Data)
- 1-Sample Hypothesis Test of Proportion in Excel 2010 and Excel 2013
- 2-Sample Pooled Hypothesis Test of Proportion in Excel 2010 and Excel 2013
- How To Build a Much More Useful Split-Tester in Excel Than Google's Website Optimizer

- Chi-Square Independence Tests in Excel
- Chi-Square Goodness-Of-Fit Tests in Excel
- F Tests in Excel
- Correlation in Excel
- Pearson Correlation in Excel
- Spearman Correlation in Excel
- Confidence Intervals in Excel
- Overview of z-Based Confidence Intervals of a Population Mean in Excel 2010 and Excel 2013
- t-Based Confidence Intervals of a Population Mean in Excel 2010 and Excel 2013
- Minimum Sample Size to Limit the Size of a Confidence interval of a Population Mean
- Confidence Interval of Population Proportion in Excel 2010 and Excel 2013
- Min Sample Size of Confidence Interval of Proportion in Excel 2010 and Excel 2013

- Simple Linear Regression in Excel
- Overview of Simple Linear Regression in Excel 2010 and Excel 2013
- Simple Linear Regression Example in Excel 2010 and Excel 2013
- Residual Evaluation For Simple Regression in Excel 2010 and Excel 2013
- Residual Normality Tests in Excel – Kolmogorov-Smirnov Test, Anderson-Darling Test, and Shapiro-Wilk Test For Simple Linear Regression
- Evaluation of Simple Regression Output For Excel 2010 and Excel 2013
- All Calculations Performed By the Simple Regression Data Analysis Tool in Excel 2010 and Excel 2013
- Prediction Interval of Simple Regression in Excel 2010 and Excel 2013

- Multiple Linear Regression in Excel
- Basics of Multiple Regression in Excel 2010 and Excel 2013
- Multiple Linear Regression Example in Excel 2010 and Excel 2013
- Multiple Linear Regression’s Required Residual Assumptions
- Normality Testing of Residuals in Excel 2010 and Excel 2013
- Evaluating the Excel Output of Multiple Regression
- Estimating the Prediction Interval of Multiple Regression in Excel
- Regression - How To Do Conjoint Analysis Using Dummy Variable Regression in Excel

- Logistic Regression in Excel
- Logistic Regression Overview
- Logistic Regression Performed in Excel 2010 and Excel 2013
- R Square For Logistic Regression Overview
- Excel R Square Tests: Nagelkerke, Cox and Snell, and Log-Linear Ratio in Excel 2010 and Excel 2013
- Likelihood Ratio Is Better Than Wald Statistic To Determine if the Variable Coefficients Are Significant For Excel 2010 and Excel 2013
- Excel Classification Table: Logistic Regression’s Percentage Correct of Predicted Results in Excel 2010 and Excel 2013
- Hosmer-Lemeshow Test in Excel – Logistic Regression Goodness-of-Fit Test in Excel 2010 and Excel 2013

- Single-Factor ANOVA in Excel
- Overview of Single-Factor ANOVA
- Single-Factor ANOVA Example in Excel 2010 and Excel 2013
- Shapiro-Wilk Normality Test in Excel For Each Single-Factor ANOVA Sample Group
- Kruskal-Wallis Test Alternative For Single Factor ANOVA in Excel 2010 and Excel 2013
- Levene’s and Brown-Forsythe Tests in Excel For Single-Factor ANOVA Sample Group Variance Comparison
- Single-Factor ANOVA - All Excel Calculations
- Overview of Post-Hoc Testing For Single-Factor ANOVA
- Tukey-Kramer Post-Hoc Test in Excel For Single-Factor ANOVA
- Games-Howell Post-Hoc Test in Excel For Single-Factor ANOVA
- Overview of Effect Size For Single-Factor ANOVA
- ANOVA Effect Size Calculation Eta Squared (?2) in Excel 2010 and Excel 2013
- ANOVA Effect Size Calculation Psi (?) – RMSSE – in Excel 2010 and Excel 2013
- ANOVA Effect Size Calculation Omega Squared (?2) in Excel 2010 and Excel 2013
- Power of Single-Factor ANOVA Test Using Free Utility G*Power
- Welch’s ANOVA Test in Excel Substitute For Single-Factor ANOVA When Sample Variances Are Not Similar
- Brown-Forsythe F-Test in Excel Substitute For Single-Factor ANOVA When Sample Variances Are Not Similar

- Two-Factor ANOVA With Replication in Excel
- Two-Factor ANOVA With Replication in Excel 2010 and Excel 2013
- Variance Tests: Levene’s and Brown-Forsythe For 2-Factor ANOVA in Excel 2010 and Excel 2013
- Shapiro-Wilk Normality Test in Excel For 2-Factor ANOVA With Replication
- 2-Factor ANOVA With Replication Effect Size in Excel 2010 and Excel 2013
- Excel Post Hoc Tukey’s HSD Test For 2-Factor ANOVA With Replication
- 2-Factor ANOVA With Replication – Test Power With G-Power Utility
- Scheirer-Ray-Hare Test Alternative For 2-Factor ANOVA With Replication

- Two-Factor ANOVA Without Replication in Excel
- Creating Interactive Graphs of Statistical Distributions in Excel
- Interactive Statistical Distribution Graph in Excel 2010 and Excel 2013
- Interactive Graph of the Normal Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Chi-Square Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the t-Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Binomial Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Exponential Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Beta Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Gamma Distribution in Excel 2010 and Excel 2013
- Interactive Graph of the Poisson Distribution in Excel 2010 and Excel 2013

- Solving Problems With Other Distributions in Excel
- Solving Uniform Distribution Problems in Excel 2010 and Excel 2013
- Solving Multinomial Distribution Problems in Excel 2010 and Excel 2013
- Solving Exponential Distribution Problems in Excel 2010 and Excel 2013
- Solving Beta Distribution Problems in Excel 2010 and Excel 2013
- Solving Gamma Distribution Problems in Excel 2010 and Excel 2013
- Solving Poisson Distribution Problems in Excel 2010 and Excel 2013

- Optimization With Excel Solver
- Maximizing Lead Generation With Excel Solver
- Minimizing Cutting Stock Waste With Excel Solver
- Optimal Investment Selection With Excel Solver
- Minimizing the Total Cost of Shipping From Multiple Points To Multiple Points With Excel Solver
- Knapsack Loading Problem in Excel Solver – Optimizing the Loading of a Limited Compartment
- Optimizing a Bond Portfolio With Excel Solver
- Travelling Salesman Problem in Excel Solver – Finding the Shortest Path To Reach All Customers

- Chi-Square Population Variance Test in Excel
- Analyzing Data With Pivot Tables
- SEO Functions in Excel
- Time Series Analysis in Excel

## No comments:

## Post a Comment