Everyone is familiar with the television game show Who Wants To Be A Millionaire. It has been a rousing international success: 81 different versions of the game with 39 versions currently airing, spanning almost just as many languages. In India alone there are five
versions for five different languages. The central theme of the movie Slumdog Millionaire is around the show.
At Webtrends, we have been running our own semblance of a quiz show for the last few years at site optimisation events across Europe. The format is simple – seven multiple choice questions of which-version-won, the person with the highest number of correct answers scooping a prize. 300 or so ‘contestants’ later and the failure rate stands at a mind-shattering (or perhaps not so mind-shattering) 85%. That means only 15% of people have managed to answer more than three of the seven questions correctly, or about 45 of the 300 contestants, all of whom might consider themselves to be a digital marketing expert.
If we go one step further: no one has ever got all quiz questions correct. And one more step: no one has ever even got all but one correct.
This is a testament to the unpredictability of online shoppers in an ever-mutable digital space. Despite the clear need to better understand online and device-specific visitor behaviour and the capability of generating millions of pounds in uplift, site optimisation hasn’t yet become the fixture it should be for brands today. It’s not surprising to find that even in larger businesses, A/B and multivariate testing has not been embedded firmly into the culture and processes of the company. In comparison, TV quiz shows first appeared in the 1930s and quickly became a fixture by the 1950s when everyone owned a TV. Site optimisation has been around for about 10 years now – a shorter period of time in absolute terms yes, but in 21st century terms an exponentially lengthier period.
Site optimisation vs. the game show
“I’d like to ask the audience.”
*Insert high tension music here*
Asking the audience is exactly what testing is about – let your visitors decide the ‘right answer’ for themselves. With a large enough sample population, the statistics behind site optimisation tell us what imagery, messaging, layout, funnel, etc. visitors respond more favourably to (or not). Not all visitors are the same, so you could also break your data down by segments to start serving different versions of the same page based on, for example, operating system, source, or previous browsing behaviour.
There are many parallels to be drawn between game shows and site optimisation. Fortunately, in the latter there is no wrong answer, risk of ridicule, or (heaven forbid!) cheesy host or hostess. Let’s share a few lessons, shall we?
Lesson 1: A good game show should have a degree of simplicity and be easy enough to follow
Who Wants To Be A Millionaire has been a great success partly due to its simplicity. It doesn’t have an absurd premise that is too difficult to follow. Site optimisation should be the same, especially for those just getting into the discipline. To get into it, start simple before you jump into more complex testing. Simplicity will also ensure you don’t get held up in the development or Q&A phases of your testing project, and quite often you will find that simple changes can have quite a profound effect.
Lesson 2: Ask the audience, phone a friend, or ask the expert
To achieve success in game shows like Who Wants To Be A Millionaire, at some point you’ll need to get some help – by asking the audience, phoning a friend, or asking the expert. In testing, the same applies – you will achieve success if you get others involved and if you put the proper experts in place. The former will allow you to get great test ideas you hadn’t thought of before (think about asking for ideas in your email signature!). The latter may be even more important – in Forrester’s State of Online Testing 2011 paper, it is stressed that the “online testing process requires too much perpetual planning and considerations to be left to non-specialists or a committee.” Without question, the execution of an online testing programme is a people business.
Lesson 3: A bit of calculated risk never hurt anyone
Spin the wheel, take a chance and you could win more. It’s the same in testing – test radical and you can vastly increase the amount of lift you might see from a winner. Inciting a strong visitor response is the best way to reach statistical significance more quickly (make sure you note that down, people managing low-traffic sites). Conversely, you might see a strong negative impact, but either way, you learn and you learn more quickly. And you can always minimise risk by simply throttling down the exposure of your test to a smaller percentage of visitors.
Lesson 4: Good game shows are fun for people because they can interact and play along
Testing is and should be fun. Put your reputation on the line and guess which version of a test will win with high stakes (e.g. a drink at your local watering hole after work) and have some fun with it. By spreading this challenge around the office regularly, you can initiate a culture of testing and get some great ideas from your colleagues in the process.
Lesson 5: Final Answer? There is no final answer
Testing should be continuous, so while one test may have a few variants – A, B, C and D – that doesn’t mean you stop right there. One of those will be the winner, and then it’s time to think about how to augment that winner further. Behaviour is constantly changing in the digital space and to keep up with changing patterns and trends as well as the mobile and tablet explosion, one must continuously optimise.
Final thought
John Carpenter, the first million-dollar winner in the USA, had a final question of: “Who was the only US President to appear on ‘Rowan and Martin’s Laugh-in’?” When Regis Philbin asked him the question, John decided to call his father for the ‘phone-a-friend’ lifeline. Instead of asking his father for help, he told him “Um… I don’t really need your help. I just wanted to let you know that I’m going to win the million dollars.”
With testing embedded in your current processes, it should be just that simple.
