A simple (yet powerful) guide to AB Testing Development

Over the past decade, companies had no choice but to take the online user experience more seriously than ever before, to increase their online sales, customer loyalty and achieve their business goals.

As the online businesses race towards becoming the best amongst their competitors, CRO and AB Testing have started to play a significant role to improve digital channel performances. From large companies such as Google, Facebook, Amazon, Netflix, Microsoft to Start-ups, started experimenting with different concepts to gain additional customers or users to their website.

In this article, we are providing a simple guide to develop A/B tests. However, before doing that, it is important to understand the basics, which will help to provide a solid foundation to understand the success of CRO.

What is CRO (Conversion Rate Optimization)?

There are many definitions out there of Conversion Rate Optimization. For example, Econsultancy defines CRO as a “process of optimizing site to increase the likelihood that visitors will complete that specific action” [1] . HubSpot provides an action-focused definition of CRO by adding enabling people to take action when they visit a website. “By designing and modifying certain elements of a webpage, a business can increase the chances that site visitors will "convert" into a lead or customer before they leave”[2].

In simple terms, the definition of CRO is

“Making changes to improve metrics.”

The changes can vary from small – like a simple headline change, to large - where you could be adding a new feature to your product. The changes can be on your website, emails, mobile Apps, Search keywords, banners, or even physical entities. The ultimate thing is by making these changes, you are improving your business metrics, performance indicators or KPIs. Moreover, this process of continuous improvement of KPIs by making changes is CRO.

[1] What is conversion rate optimisation (CRO) and why do you need it? https://econsultancy.com/what-is-conversion-rate-optimisation-cro-and-why-do-you-need-it/

[2] The Beginner's Guide to Conversion Rate Optimization (CRO) https://blog.hubspot.com/marketing/conversion-rate-optimization-guide

What is A/B Testing?

As part of the CRO program, you are coming up with ideas to make changes. These change ideas are commonly known as Hypothesis.

As with any research, a hypothesis is always validated by different forms of testing. The most common form of testing a hypothesis within the CRO program is A/B Testing (or ABn Testing) where version A signifies the control or original – what is currently live, and version B is the new variation created based on the hypothesis.

A/B Testing simply helps you to check if the changes you are making are truly improving the target KPIs. It takes the metric performance of the current version and compares it with the new version where the changes have been applied.

For example, Neil – an Optimisation Consultant has completed some research on the e-commerce site he is involved with. Based on the analytics data, he identified that on the product details pages, the price of the product is presented just below the title of the product, which some users were ignoring. His idea is to change the position of this price and make it closer to the main Call to action (CTA) button so that users can see the price before adding the item to the basket and more people will complete their purchase through this website.

This is a simple example of an AB test to find out if Neil’s hypothesis is true – by moving the price closer to the CTA, more people will complete their purchase.

Figure 1: Example of a simple AB Test where Version A or Control where the price is just below the title of the product. Version B, where the price is just above the Buy now CTA button.

A hypothesis can create more than one variation. In such cases, your AB test becomes ABn test where A still signifies the control and after that, you have variation B, variation C and so forth to signify the other variation.

In case if you are interested in changing multiple elements and test a combination of different elements, you can setup a Multivariate Test (MVT).

Multivariate Test or MVT is a form of a test where you want to find the impact of the combinations of more than one change across multiple places within a page or site section. Let us break this definition down with an example.

Let’s go back to Neil’s research, and this time, he noticed that when visitors landed on the homepage of the best retailer website, they completely ignored the Sale link and also did not notice the key USP (see below).

Figure 2: Example of a Multi-variate test where two different elements are going to be tested with multiple variations

He wants to test out two different versions of the Sale message against the control version:

• Control Sale message: Up to 50% off! Shop Sale
• Variation 1 Sale message: Sale now on! Get Up to 50% off
• Variation 2 Sale message: Up to 50% off! Hurry while stock lasts!

At the same time, two versions of the main heading to promote free delivery and return messaging against the control:

• Control Heading: Welcome to the Best Retailer site – number 1 in sustainable products
• Variation 1 Heading: Number 1 Sustainable Shopping site with FREE Delivery and Returns
• Variation 2 Heading: Free Delivery and Returns! Best Retailer for Sustainable products

Neil is interested in finding out not only which individual sale message or heading works best to increase the browsability and purchase, but also to see which combination of these messages work best. This is where he sets this test up as a multi-variate test or MVT to test out all of the combinations against the control.

All MVT or multi-variate tests are AB tests – with a lot of variations. For example, the total number of combinations in this example will be 9 (including the control). Each of these combinations can be considered as a variation of an AB test.

Testing tools that provide the option of setting up MVT usually gives the user some additional options – for example, excluding specific combinations and reporting on the performance of individual elements or variates.

This is another form of A/B test where you are simply redirecting the variation traffic to your newly developed page(s). One important thing to consider when setting up a redirect A/B test is to use a rel=canonical tag. This will tell Google and other search engines that the content is similar to the original URL and that your SEO ranking is not penalised when running the test.

What is Server-Side testing?

Most commonly used A/B testing methods are client-side – i.e. the changes are applied on the browser using JavaScript. This does not require any code push from the server and marketers and A/B testing developers can rely on the testing tool to set up the A/B test.

Client-side testing allows users to test almost everything. There can be cases where a hypothesis might be testing additional features – such as the pricing structure, new site search algorithm, anything that will require backend changes to the site and you cannot test them with the client-side testing. This is where server-side testing becomes handy. The backend developers can create the variation, and with the help of the testing tool, they can launch the test by splitting the traffic.

For example, let’s say Neil identified that the site search on the Best Retailer website is not returning the right set of results. The developed built a new search algorithm. But Neil wants to make sure that the new search algorithm is actually performing better in comparison to the existing search.

In this case, he requests the AB Testing development team to create a server-side test and using the server-side testing tool, the development team can launch the new version of the search alongside the existing version by splitting the traffic.

There are 5 core components of an A/B test:

1. Test Plan or Test Brief
2. Design of the A/B test variations
3. Development of the variations
4. Tool Setup
5. QA

1. Test plan or Test Brief

Also sometimes known as ‘Blueprint’, a test plan or test brief should provide all the necessary information about the A/B test. It should include, but not limited to, the hypothesis, audiences, targeting conditions, URL where the test will be launched, variation information, metrics that need to be tracked, QA scenarios – i.e. anything that would define any aspect of this A/B test will need to be documented on the Test Plan. This should be treated as the reference points for everyone involved in this CRO programme to learn more about an individual test.

You can download a test plan template from here[1].

2. Design of the A/B test variations

Depending on the changes being made with the test, the design of the variations needs to be specified. Ideally, it is great if the A/B testing developers get access to the raw design files (e.g. Photoshop, Figma, Zeplin etc). That way, when the A/B Test developers are building the variations, they can rely on the exact specification from the raw file (e.g. colour code, pixel amount etc.).

If the A/B test needs to be developed across all devices (e.g. Desktop, Tablet, Mobile), then the design should specify how the variation will be displayed across all of those devices. Additionally, if there are specific scenarios that would impact how the variation will be displayed (e.g. clicking on an accordion/tab), the design should also provide information such as active state, highlighted state.

3. Development of the A/B test variations

Once the test plan and design variations are ready, A/B test developers or Solutions Engineers start the development of the variations. Depending on the changes that are being made with the variations, the test developers would write the code using JavaScript / CSS / HTML. They can use the Browser Console to run the code and check if the desired changes are happening or not. Once they are happy, they can move into the tool setup.

4. Tool Setup

Depending on the testing tool, the setup of an A/B testing tool may vary. Additionally, the type of test can also dictate the setup of the test. For example, if it is an MVT, the developer needs to define the area where the changes are being made. If it is a redirect test, then the right parameters need to be used.

The A/B testing developer also needs to think about the targeting conditions based on the test plan. Who should be targeted, where is the test going to be live, if the site is developed using SPA, what needs to be done to ensure that the experiment is running at the right location.

Metrics setup is another important part of this stage of A/B testing development. The developers need to go through the list of metrics and implement them. Note that in some testing tools, you can save the metrics and re-use the saved metrics. Whereas in some other tools, every time you create a new test, you need to create the metrics.

When implementing the metrics, A/B testing developers would need to make sure that they are tracked in variations, as well as in Control. Once this is done, the test developer will need to prepare this for QA of the variations.

5. A/B Test QA

Once the development is complete, the test must go for QA. Test QA Engineers at this stage would pick up the test plan, check the variations against the design and the test plan to make sure that the variation is rendering as it should. They will report all bugs, which will then need to go back to the developer to fix. Furthermore, once the developer fixes the bugs, they will need to go through a re-QA process.

This is a significant part of any A/B test. Without QAing the test, the test should never go live to the end-user and you can read why A/B Testing QA is really important. The last thing you want is your test becomes invalidated by a bug in the variation.

QA Engineers also need to make sure that the metrics are getting the right data in control as well as in variation. Depending on the testing tools, the way of checking metrics can vary. However, in most cases, the metrics that are being passed to the testing tool server can be found on the network tab of the browsers.

Conclusion

A/B testing comes in different forms. However, all tests have some fundamental components – and they will define the quality of the tests that you are building for your CRO / Experimentation program.