At BookerZzz we very well know that, in the fast going online market, it is important to continuously work on improving our product. A lot of companies base their improvements on trends or on what others do. We prefer to take that extra moment and analyze if the improvement we have in mind really is the right one. How do we do that? By using the method of A/B-testing.
In this article we give you an insight on how to do A/B-testing based on our internal experiences at BookerZzz.
What is A/B-testing?
In order to use the method, you need to know what A/B-testing is exactly. The short version: A/B-testing is a quantitively research where you compare two variations in order to find out which variant works the best.
With A/B-testing you divide the target group you want to test in two separate groups to be able to show the different variations. You’ll have the A-group, in other words the control group, and there is a B-group. The A-group will get to see the original version of the website, the version that is normally shown when visiting the website. And the B-group will get to see the variation, the version you expect to do better.
Below you see an example of an A/B-test for our label BungalowSpecials:
Why A/B-testing?
The answer to this question is simple: you learn a lot from A/B-testing. You get to know your target group better, you learn in which way visitors use your website and what the consequences are of the changes you make to your website. And even when the results of an A/B-test are negative, you still learn and you can adapt the original plan.
Next to the learning part, A/B-testing is also a way of proving how users behave based on data. It often appears, and it is no different at BookerZzz, that there are a lot of ideas from different people. Randomly implementing those different ideas to please people or because someone screams the loudest, will eventually make you regretting doing so. Unless you are lucky, but then again… do you want to play it on luck when you can test it and be sure about it? A/B-testing will ground your ideas and prove you if it will have a positive or negative effect when implementing.
How to do A/B-testing in your company?
With various ideas it can be hard to prioritize the different tests or even deciding on where to start. Therefor you can use the so called PIE-framework. PIE stands for Potential, Importance and Ease. Every idea for an A/B-test should go through this framework before the actual testing.
How to use the PIE-framework for A/B-testing?
Determine the potential of an A/B-test.
Take a look at the website-page you want to release the test on. What can be optimized? What can be gained with it? Think in ways of sessions, conversions, but also the opinion of your website visitors. When an important visitor tells you something valuable, you must not ignore it.
Determine the importance of the website-page
When you want to test a website-page it is good to look at how valuable that page is for your website. If the page doesn’t have enough traffic for example, it makes no sense to give the test a high priority. The chances of gaining a lot out of it are slim and the test will take a long time before it shows a significant result. Recommendable is to not only look at traffic, but also costs and revenue.
Determine how much work the implementation is
At BookerZzz we like to be ambitious, but we simply don’t always have to man power to accomplish everything we want. Therefor we have to look at how much time an implementation of an A/b-test takes. If an implementation would take a lot of our time, but won’t give us big results, it might be good to lower it on our priority list.
Combining all the aspects of the PIE-framework, and giving a test an average score (by giving each element a score from 0 to 10), it gets easier to prioritize the A/B-tests.
Below another example of an A/B-test for our label HotelSpecials
When is an A/B-test positive?
Before you launch an A/B-test you write a hypothesis of what you expect to be the result. Based on this you determine what you have to test and the metrics you need to look at when deciding if the test is positive or negative. There are a few things you need to keep in mind when doing this:
Don’t only measure one metric
At BookerZzz we like to look at the total picture instead of death staring ourselves on one metric. It can for example happen that our conversion goes up, but the amount of session and revenue lower. That is of course not the outcome you want it to be. That is why you need to take the side effects in consideration when deciding if the results of an A/B-test are positive or negative.
Take a deep dive into your data
It is a mistake to only look at the overall results of a test, because you won’t learn where the results are coming from. This is why it is always smart to take a deep dive into the data. Check for example the results per device. Is there a difference in behavior on mobile, tablet and desktop? You can also look at your sources; where is the traffic coming from and what effect does it have? Last but not least: compare new visitors versus returning visitors. Do they behave differently from each other and what percentage are they from the total amount of visitors?
Make sure your test is significant
Next to positive data you want a test to be significant. Implementing a new idea based on coincidentally positive results is almost the same as implementing an idea without testing. In order to get significant test results you have to generate traffic to the website-page you are running the A/B-test on. When the results are insignificant after all, it is advisable to redo the test or run an optimized version of the test.
In case you are a small company we wouldn’t recommend using A/B-testsing but rather use survey’s or usability test. What we often do at BookerZzz is that we launch an A/B-test on the bigger labels and implement it later on the smaller labels as well when there is a significant positive result.
The 3 most important things we learned at BookerZzz doing A/B-tests
- The danger of wanting to change too much
A/B-testing is fun and it can make a company and its employees enthusiastic (maybe even motivate them). The only risk is that you want to change too many things on your website at once. It is important to only change the element you want to test and that you leave the rest the same. It is very tempting to change everything around the element you want to improve. However, this will result in a neutral test, because you end up not knowing what change made the difference.
- Change negative results in something positive
For our label BungalowSpecials we wanted to improve the filtering on the website in order to help our visitors navigate easier through the offers. We upgraded our filters by making them stand out more. Our hypothesis was that people who filter more have more interaction with our website and are tending to book faster.
With high hopes we launched the A/B-test, getting to deal with a major disappointment when we almost immediately found out the result where negative. We had to stop the test before the actual end-date in order to not harm the numbers even more. But instead of going off crying at the desk of our CEO Remco, we decided to look into the data to find out why our hypothesis was wrong.
Ends up we weren’t that wrong after all… The change of the look and feel of the filters worked perfectly fine. What went wrong was that when using the filters, website users wouldn’t get the right results; our filters in general weren’t working well. So, through this A/B-test we found out a feature didn’t work properly. And by fixing that problem, our original hypothesis ended up being right after all.
- Always go for high quality
Running a lot of tests, implementing new ideas fast and seeing positive results is what every company wants. However, it is important to guarantee the quality of an A/B-test at all times. You rather want to spend a bit more time on an implementation and get it right, than to go fast and implement half developed ideas. Only this way you will achieve the best results in the long run.
