At heart, Google’s website optimizer just tells you which version of a landing page converts better
Sometimes, a tool is simple and elegant in purpose, but that purpose is obscured by the technical brush you have to clear to get to it. Google’s website optimizer is a case in point.
Google Website Optimizer is a tool for directing people to different versions of a web page and counting how many undertake the desired action on each.
That’s it for the core value proposition. End of story. The sentence I just wrote should be emblazened in large bold letters on the front page of the web optimizer tool.
Now, some might complain that this core value proposition leaves out some key elements of what the tool does and what distinguishes it in the market place. I’ll list those in the order of descending guilt I felt in leaving them out:
- Website optimizer uses statistical methods for determining the extent to which your results are not just due to random chance. Specifically, it tells you the proportion of times you can expect the winner in your test to really win. The fact that it also gives you raw count data means you are free to apply your own methods also.
- Google website optimizer is just a special case of Google Analytics, with the implication that a lot of your understanding of that tool can just transfer over.
Then, there are things, heavily emphasized in the website optimizer introductory material, that I don’t regret leaving out at all. I’m listing these in decreasing order of irritation I experienced wading through them:
- Google website optimizer has a wizard that guides you setting up your experiments. Truth is I had a love/hate relationship with this feature. It does give you a good understanding of the basic things you need to designate. However, the approach is extremely cookie cutter and inflexible. In particular, it requires you to pass tests in setting up any experiment that may run counter to what your real end goal is, necessitating you back out some of the things the tools requires before you actually run the tests. Google even recognizes that some of the wizard’s requirements can be contradictory for the tests you actually want to run when they tell you to ignore some of the wizard’s warnings in their section on Advanced A/B testng.
- The javascript code in web site optimizer is often redundant and leads to diminished maintainability, particularly on advanced web pages using ajax. Some of the website optimizer help files actually point you to code, based on a deprecated API, that will not work with the current version of website optimizer (here’s the main culprit from the FAQ, the help version of this FAQ inside the wizard contains the deprecated code).
- The tool overly focuses on experimental design by forcing you into it right at the outset. The problem is that you often cannot achieve the gold standard of full factorial experimental design on which the tool was originally based. There are two related issues: (1) Often enough your concepts are developed to the point where you have different candidate solutions but not the rigorous theoretical basis for a full factorial design; (2) Full factorial designs can require an impractical amount of data collection to yield statistically significant results (actually implying that you are not stating your theory at the right level of generality, suggesting a special case of reason 1 just listed).
- Initially, it is not clear that website optimizer just counts one conversion event per visitor. Website optimizer is built on the premise that web pages are designed so that people only have one real option for converting. That premise works fine in a number of ecommerce scenarios where you want the person to make an order or not. It’s more questionable where the person may have a number of graded options, many of which fall short of fully placing an order. In particular, website optimizer may be less suitable for affiliate marketers.
