A simple (yet powerful) guide to AB Testing Development
Over the past decade, companies had no choice but to take the online user experience more seriously than ever before, to increase their online sales, customer loyalty and achieve their business goals.
As the online businesses race towards becoming the best amongst their competitors, CRO and AB Testing have started to play a significant role to improve digital channel performances. From large companies such as Google, Facebook, Amazon, Netflix, Microsoft to Start-ups, started experimenting with different concepts to gain additional customers or users to their website.
In this article, we are providing a simple guide to develop A/B tests. However, before doing that, it is important to understand the basics, which will help to provide a solid foundation to understand the success of CRO.
What is CRO (Conversion Rate Optimization)?
There are many definitions out there of Conversion Rate Optimization. For example, Econsultancy defines CRO as a “process of optimizing site to increase the likelihood that visitors will complete that specific action”  . HubSpot provides an action-focused definition of CRO by adding enabling people to take action when they visit a website. “By designing and modifying certain elements of a webpage, a business can increase the chances that site visitors will “convert” into a lead or customer before they leave”.
In simple terms, the definition of CRO is
“Making changes to improve metrics.”
The changes can vary from small – like a simple headline change, to large – where you could be adding a new feature to your product. The changes can be on your website, emails, mobile Apps, Search keywords, banners, or even physical entities. The ultimate thing is by making these changes, you are improving your business metrics, performance indicators or KPIs. Moreover, this process of continuous improvement of KPIs by making changes is CRO.
 What is conversion rate optimisation (CRO) and why do you need it? https://econsultancy.com/what-is-conversion-rate-optimisation-cro-and-why-do-you-need-it/
 The Beginner’s Guide to Conversion Rate Optimization (CRO) https://blog.hubspot.com/marketing/conversion-rate-optimization-guide
What is A/B Testing?
As part of the CRO program, you are coming up with ideas to make changes. These change ideas are commonly known as Hypothesis.
As with any research, a hypothesis is always validated by different forms of testing. The most common form of testing a hypothesis within the CRO program is A/B Testing (or ABn Testing) where version A signifies the control or original – what is currently live, and version B is the new variation created based on the hypothesis.
For example, Neil – an Optimisation Consultant has completed some research on the e-commerce site he is involved with. Based on the analytics data, he identified that on the product details pages, the price of the product is presented just below the title of the product, which some users were ignoring. His idea is to change the position of this price and make it closer to the main Call to action (CTA) button so that users can see the price before adding the item to the basket and more people will complete their purchase through this website.
This is a simple example of an AB test to find out if Neil’s hypothesis is true – by moving the price closer to the CTA, more people will complete their purchase.
A hypothesis can create more than one variation. In such cases, your AB test becomes ABn test where A still signifies the control and after that, you have variation B, variation C and so forth to signify the other variation.
In case if you are interested in changing multiple elements and test a combination of different elements, you can setup a Multivariate Test (MVT).
Let’s go back to Neil’s research, and this time, he noticed that when visitors landed on the homepage of the best retailer website, they completely ignored the Sale link and also did not notice the key USP (see below).
Figure 2: Example of a Multi-variate test where two different elements are going to be tested with multiple variations
He wants to test out two different versions of the Sale message against the control version:
- Control Sale message: Up to 50% off! Shop Sale
- Variation 1 Sale message: Sale now on! Get Upto 50% off
- Variation 2 Sale message: Up to 50% off! Hurry while stock lasts!
At the same time, two versions of the main heading to promote free delivery and return messaging against the control:
- Control Heading: Welcome to the BestRetailer site – number 1 in sustainable products
- Variation 1 Heading: Number 1 Sustainable Shopping site with FREE Delivery and Returns
- Variation 2 Heading: Free Delivery and Returns! BestRetailer for Sustainable products
Neil is interested in finding out not only which individual sale message or heading works best to increase the browsability and purchase, but also to see which combination of these messages work best. This is where he sets this test up as a multi-variate test or MVT to test out all of the combinations against the control.
All MVT or multi-variate tests are AB tests – with a lot of variations. For example, the total number of combinations in this example will be 9 (including the control). Each of these combinations can be considered as a variation of an AB test.
Testing tools that provide the option of setting up MVT usually gives the user some additional options – for example, excluding specific combinations and reporting on the performance of individual elements or variates.
What is Server-Side testing?
Client-side testing allows users to test almost everything. There can be cases where a hypothesis might be testing additional features – such as the pricing structure, new site search algorithm, anything that will require backend changes to the site and you cannot test them with the client-side testing. This is where server-side testing becomes handy. The backend developers can create the variation, and with the help of the testing tool, they can launch the test by splitting the traffic.
For example, let’s say Neil identified that the site search on the Best Retailer website is not returning the right set of results. The developed built a new search algorithm. But Neil wants to make sure that the new search algorithm is actually performing better in comparison to the existing search.
In this case, he requests the AB Testing development team to create a server-side test and using the server-side testing tool, the development team can launch the new version of the search alongside the existing version by splitting the traffic.
There are 5 core components of an A/B test:
- Test Plan or Test Brief
- Design of the A/B test variations
- Development of the variations
- Tool Setup
1. Test plan or Test Brief
You can download a test plan template from here.
2. Design of the A/B test variations
Depending on the changes being made with the test, the design of the variations needs to be specified. Ideally, it is great if the A/B testing developers get access to the raw design files (e.g. Photoshop, Figma, Zeplin etc). That way, when the A/B Test developers are building the variations, they can rely on the exact specification from the raw file (e.g. colour code, pixel amount etc.).
If the A/B test needs to be developed across all devices (e.g. Desktop, Tablet, Mobile), then the design should specify how the variation will be displayed across all of those devices. Additionally, if there are specific scenarios that would impact how the variation will be displayed (e.g. clicking on an accordion/tab), the design should also provide information such as active state, highlighted state.
3. Development of the A/B test variations
4. Tool Setup
Depending on the testing tool, the setup of an A/B testing tool may vary. Additionally, the type of test can also dictate the setup of the test. For example, if it is an MVT, the developer needs to define the area where the changes are being made. If it is a redirect test, then the right parameters need to be used.
The A/B testing developer also needs to think about the targeting conditions based on the test plan. Who should be targeted, where is the test going to be live, if the site is developed using SPA, what needs to be done to ensure that the experiment is running at the right location.
Metrics setup is another important part of this stage of A/B testing development. The developers need to go through the list of metrics and implement them. Note that in some testing tools, you can save the metrics and re-use the saved metrics. Whereas in some other tools, every time you create a new test, you need to create the metrics.
When implementing the metrics, A/B testing developers would need to make sure that they are tracked in variations, as well as in Control. Once this is done, the test developer will need to prepare this for QA of the variations.
5. A/B Test QA
Once the development is complete, the test must go for QA. Test QA Engineers at this stage would pick up the test plan, check the variations against the design and the test plan to make sure that the variation is rendering as it should. They will report all bugs, which will then need to go back to the developer to fix. Furthermore, once the developer fixes the bugs, they will need to go through a re-QA process.
This is a significant part of any A/B test. Without QAing the test, the test should never go live to the end-user and you can read why A/B Testing QA is really important. The last thing you want is your test becomes invalidated by a bug in the variation.
QA Engineers also need to make sure that the metrics are getting the right data in control as well as in variation. Depending on the testing tools, the way of checking metrics can vary. However, in most cases, the metrics that are being passed to the testing tool server can be found on the network tab of the browsers.
A/B testing comes in different forms. However, all tests have some fundamental components – and they will define the quality of the tests that you are building for your CRO / Experimentation program.