Design of Experiment (DOE) has been around for hundreds of years. I have used it extensively in my career, but I’m always surprised to see marketing departments that don’t fully use DOE or even basic A/B testing (or create A/B tests without understanding the importance of sample size).
One of the earliest known examples of DOE was from the 1700s, when a doctor tried to limit variation and control variables in patients who had scurvy. A couple hundred years later, DOE was brought to the forefront by an English statistician named Ronald Fisher. Along with developing the methodology of ANOVA (analysis of variance), he pushed the use of statistical methods in studying genetics.
Currently DOE is used extensively in medical experiments, in genetic studies, and in engineering (testing machinery parts, for example), but again, a field which I believe is severely lacking in the use of proper statistical experiments is marketing.
Some of the benefits of DOE include the ability to:
- Isolate variables such as competition, economic effects, industry trends, and seasonality
- Determine the effect of a variable, combined effect of variables, or level of a variable
- Have statistical significance behind granular campaign results
Although there are cases where it’s not possible to create a controlled experiment, i.e. if there is mass marketing such as TV and/or radio, when possible it should be thought of as a necessary investment. What I mean by “necessary investment” is that, in my experience, I have seen people on the business side who do not want to hold back a control group because they will not reach as many people or prospects and will therefore ultimately lose out on potential acquisitions and/or incremental sales. What they are missing is that, even if there is a lesser opportunity in the short-term, the learnings from the experiment are a return on an investment that can greatly benefit future campaigns.
Below is my concept for experimental design flow:
Hypotheses
Although you can occasionally start with a simple hypothesis or hypotheses, it is best to reach these with the utilization of data. This could be running high-level numbers that indicate a trend, comparing different segments to one another, or looking at pre- vs. post-, or this-year vs. last-year numbers.
Test Design
Once you have done this analysis, it is essential to put a great deal of time and effort into the creation of a clear and concise hypothesis. Everything else that comes after this (operational resources, campaign cost, and, ultimately, the success of the experiment) hinges on whether this step was thoroughly considered. I have seen cases where an experiment was created and the marketing was implemented, months went by while we waited for the data, and when it was time to measure the campaign, we realized that the marketing team wanted to see the effect of multiple variables and not just the single variable that was controlled for. At that point it is necessary to explain that the hypotheses they want cannot be answered with statistical confidence.
In order to create a proper test design, it is essential to consider aspects such as: the number of test groups necessary, how long the test should run, and most importantly, how big of a sample size you need. Make sure that much due diligence is spent on this, because if you want a statistical read on your results, this step is crucial.
Implementation
The Implementation phase is not so much science as much as it is project management. If you are implementing the test yourself, make absolutely sure that it is being done correctly from an operational standpoint. If another team or individual is implementing the test, diligently follow up and ensure that implementation is being done correctly. At the end of the day, even if it is someone else’s fault that the test doesn’t get implemented properly, nobody is going to be happy that the test was a waste of time and resources.
Analysis
Finally, the circle completes once we get back to the Analysis phase. This is where you would use a T-Test, ANCOVA, etc., to statistically measure the results of the test. The learnings from this can then be used to create further hypotheses. For example, if the change in price on a particular item seemed to work, then maybe this price change can be tested on a similar item.
Experimental design is something that is not used as much as it should be in marketing, but from my experience, it does seem like there is a greater awareness now than there was 10 to 15 years ago. As people in your organization see you implementing this as a standard practice and see the benefits it brings, it’s not tough to win people over to your side and evangelize them on the benefits of Design of Experiment.