When creating A/B testing variations, it is essential to design them in a way that not only produces the desired results but also abides by the following key principles. Here are the top five principles to be considered when designing successful variations for A/B testing.   

Branding

One of the key aspects in A/B testing is keeping the brand consistent, it is essential to ensure that any changes comply with the visual and messaging identity of your brand. This uniformity of all elements and channels strengthens the brand identity and creates trust in your audience.  

Customer Journey  

It is crucial to also focus on the customer Journey which involves optimizing the path users take from the first contact with your brand to the final action, such as making a purchase. The goal is to enhance this experience. Once you create hypothesis on which touchpoints could be the most important in this process, you can create experiments based on that and measure how they impact the user behavior and conversion rates.
 
For example, we introduced a countdown timer for the next day delivery. That same countdown timer was used on the PDPs, Basket page and the Checkout page. Having the same design for the countdown timer throughout the journey made a significant difference on the overall conversion rate.

Link - https://guessthetest.com/test/with-or-without-the-next-day-delivery-timer/ 

User Interaction  

Another key area of high importance when designing variations is interaction. This covers the engagement with the user’s interface from the point of the first interaction to the point of conversion. When optimizing user interaction, the goal is to enhance it and make it more intuitive. It is important to strike the balance between being subtle enough not to distract the customer, but also engaging enough to achieve the desired results such as conversion rates.  

We know that many people in this industry believe that CTA tests are the simplest form of tests and that people should be thinking bigger and bolder. No doubt about that! However, there have been cases where making small changes can have a large impact. 

Examples of such variations would be button responsiveness - testing different feedback mechanisms such as animation on click or when hovering over the mouse - interactive elements such as sliders or expandable sections (read more), or utilizing subtle animations or motion effects on some of the critical elements like CTA’s in order to grab the user’s attention without being too distracting.  

Sliders are also crucial in this context; creating sliding animations such as image carousels or product showcases can make the experience more engaging for users..  

As for a specific example of successfully creating a sliders variation, the testing team producing tests for Burton Publishing has decided to create an experiment for the client testimonial section, making it more interactive by creating a slider to increase the conversion rate. The results confirmed the hypothesis. Not only did choosing the winner improve sales by 16.2% compared to the static version, but also it increased the Revenue Per Visitor by 30.7%.   

Link - https://guessthetest.com/test/with-or-without-the-testimonials-slider/ 

Content Area 

When crafting experiments, focusing on the content area is also essential. This is to examine and find out how the content arrangement affects user engagement and conversion rates. The ultimate goal of such an experiment is to enhance user retention, readability, and user engagement. This can be done by modifying the layout, content type, and overall presentation.   

There are several ways of changing the content area, which may include increasing or decreasing the content length, incorporating multimedia elements (such as videos and graphics), adjusting call to action placement, headline testing, and experimenting with  the ratio of image to text content. Another way of creating a variation would be producing a hero image for one of the tests and another one without such an image. Tinuiti ran this test for American Automobile Association of California with the idea that a large Click to call button would make it easier for users to request a quote.   

The A/B tests have proven that indeed the interface not having the hero image and leading straight to the quote section, increased the click to call conversions by 180% making it a clear winner in this particular case.   

Link - https://guessthetest.com/test/with-or-without-the-hero-image-on-mobile/ 

Window Resizing  

If the requested tests are designed for several devices, each variation also needs to be created specifically to cater for the window size of the device it will be used on. The goal is to ensure we adapt to the layout, design, and functionality of all screen sizes to assess how these changes impact usability and conversion rates.   

There are several aspects of window resizing that need to be considered:   

• Functionality: Seeing that all interactive elements such as dropdowns and sliders work correctly on all devices  
• Layouts: making sure the layout is consistent across devices and remains visually appealing  
• Visibility: ensuring all texts, interactive elements, and images are visible without having to zoom on smaller screens 

You can apply the window resizing tests by using Fluid Grid Layouts, which use the ratio percentage of the grid instead of fixed pixels, or adaptive images that adjust depending on the device. 

In conclusion  

Every test should not only determine which variations work better but also provide a deeper understanding of user behavior and preferences. These design principles will help you and your team to create the variations that not only “works”, but also follows the site itself and help you achieve your CRO objectives.  

Reference: www.GuessTheTest.com 

When creating A/B testing variations, it is essential to design them in a way that not only produces the desired results but also abides by the following key principles. Here are the top five principles to be considered when designing successful variations for A/B testing.   

Branding

One of the key aspects in A/B testing is keeping the brand consistent, it is essential to ensure that any changes comply with the visual and messaging identity of your brand. This uniformity of all elements and channels strengthens the brand identity and creates trust in your audience.  

Customer Journey  

It is crucial to also focus on the customer Journey which involves optimizing the path users take from the first contact with your brand to the final action, such as making a purchase. The goal is to enhance this experience. Once you create hypothesis on which touchpoints could be the most important in this process, you can create experiments based on that and measure how they impact the user behavior and conversion rates.
 
For example, we introduced a countdown timer for the next day delivery. That same countdown timer was used on the PDPs, Basket page and the Checkout page. Having the same design for the countdown timer throughout the journey made a significant difference on the overall conversion rate.

Link - https://guessthetest.com/test/with-or-without-the-next-day-delivery-timer/ 

User Interaction  

Another key area of high importance when designing variations is interaction. This covers the engagement with the user’s interface from the point of the first interaction to the point of conversion. When optimizing user interaction, the goal is to enhance it and make it more intuitive. It is important to strike the balance between being subtle enough not to distract the customer, but also engaging enough to achieve the desired results such as conversion rates.  

We know that many people in this industry believe that CTA tests are the simplest form of tests and that people should be thinking bigger and bolder. No doubt about that! However, there have been cases where making small changes can have a large impact. 

Examples of such variations would be button responsiveness - testing different feedback mechanisms such as animation on click or when hovering over the mouse - interactive elements such as sliders or expandable sections (read more), or utilizing subtle animations or motion effects on some of the critical elements like CTA’s in order to grab the user’s attention without being too distracting.  

Sliders are also crucial in this context; creating sliding animations such as image carousels or product showcases can make the experience more engaging for users..  

As for a specific example of successfully creating a sliders variation, the testing team producing tests for Burton Publishing has decided to create an experiment for the client testimonial section, making it more interactive by creating a slider to increase the conversion rate. The results confirmed the hypothesis. Not only did choosing the winner improve sales by 16.2% compared to the static version, but also it increased the Revenue Per Visitor by 30.7%.   

Link - https://guessthetest.com/test/with-or-without-the-testimonials-slider/ 

Content Area 

When crafting experiments, focusing on the content area is also essential. This is to examine and find out how the content arrangement affects user engagement and conversion rates. The ultimate goal of such an experiment is to enhance user retention, readability, and user engagement. This can be done by modifying the layout, content type, and overall presentation.   

There are several ways of changing the content area, which may include increasing or decreasing the content length, incorporating multimedia elements (such as videos and graphics), adjusting call to action placement, headline testing, and experimenting with  the ratio of image to text content. Another way of creating a variation would be producing a hero image for one of the tests and another one without such an image. Tinuiti ran this test for American Automobile Association of California with the idea that a large Click to call button would make it easier for users to request a quote.   

The A/B tests have proven that indeed the interface not having the hero image and leading straight to the quote section, increased the click to call conversions by 180% making it a clear winner in this particular case.   

Link - https://guessthetest.com/test/with-or-without-the-hero-image-on-mobile/ 

Window Resizing  

If the requested tests are designed for several devices, each variation also needs to be created specifically to cater for the window size of the device it will be used on. The goal is to ensure we adapt to the layout, design, and functionality of all screen sizes to assess how these changes impact usability and conversion rates.   

There are several aspects of window resizing that need to be considered:   

• Functionality: Seeing that all interactive elements such as dropdowns and sliders work correctly on all devices  
• Layouts: making sure the layout is consistent across devices and remains visually appealing  
• Visibility: ensuring all texts, interactive elements, and images are visible without having to zoom on smaller screens 

You can apply the window resizing tests by using Fluid Grid Layouts, which use the ratio percentage of the grid instead of fixed pixels, or adaptive images that adjust depending on the device. 

In conclusion  

Every test should not only determine which variations work better but also provide a deeper understanding of user behavior and preferences. These design principles will help you and your team to create the variations that not only “works”, but also follows the site itself and help you achieve your CRO objectives.  

Reference: www.GuessTheTest.com 

We have previously shared how to create targeting conditions that are dynamic and only include a visitor when they perform certain actions. That method requires updating the Global JS (Global Project JavaScript) of Convert. There is a small challenge from a maintainability perspective.

For example, if you are running 100s of experiments, maintaining all those conditions withing Global JS might cause some unforeseen issues. This has pushed us to come up with a different approach. Utilizing the JS Audience creation, that will do the same thing, without having to touch the Global JS Convert script.

Here's a step-by-step guide on how to create a custom trigger audience using JavaScript conditions.

Steps to Create a Custom Trigger Audience

1. Create a New Audience

Go to your Convert dashboard and create a new audience. Name it something descriptive like "Open Cart Drawer".

2. Add a JavaScript Condition

Drag and drop the JavaScript condition component into the audience rules conditions.

Here is an example for how the Audience setup should look like:

3. Write a Self-Calling Function

Within the JavaScript condition, you'll need to write a function that defines your custom logic.

Here’s a sample code snippet to get you started:

(function () {
  if (window.bucketed === true) {
    return true;
  }
 function waitForElement(selector, callback, count = 1000, interval = 250) {
    if (document.querySelector(selector)) {
      callback();
    } else if (count > 0) {
      setTimeout(function () {
        waitForElement(selector, callback, count - 1, interval);
      }, interval);
    }
  }
  waitForElement('body', function () {
    document.body.addEventListener('click', function (e) {
      if (e.target.closest('.button-selector')) {
        if (window.bucketed !== true) {
          window.bucketed = true;
          window._conv_q = window._conv_q || [];
          window._conv_q.push(["executeExperiment", "experimentID"]);
        }
      }
    });
  });
})();

Explanation

Initial Condition Check
When the page loads, the script checks if ‘window.bucketed’ is ‘true’. Since it is initially ‘undefined’, the condition fails, and the user is not included in the experiment.

Waiting for Element and Adding Event Listener

• The ‘waitForElement’ function waits for the entire document body to load and then adds an event listener to it.
• This listener checks for a click on an element matching the selector ‘.button-selector’.

Triggering the Experiment

• When the specified element ‘.button-selector’ is clicked, ‘window.bucketed’ is set to ‘true’, and the custom experiment trigger snippet is executed.
• This Custom Experiment Trigger snippet will recheck the entire experiment again, including the "Locations" and "Audiences".
• While rechecking the "Audiences", it will find that ‘window.bucketed’ is true and will then bucket the user into the experiment.

Implementation Tips

• Custom Variables: Customize the variable ‘window.bucketed’ and the event listener selector (‘.button-selector’) based on your specific use case.
• Experiment ID: Replace "experimentID" with the actual ID of your experiment.
• Event Listeners: You can add different event listeners or conditions as needed to fit your specific audience criteria.

Conclusion

By using custom JavaScript conditions, you can create highly specific and dynamic audience segments in Convert. This approach ensures that your experiments are triggered only for users who meet your precise conditions, allowing for more accurate and relevant testing.

Special thanks to Tansen and Shakil, two of our Solutions Engineers, for their expertise and insights in shaping the content and ensuring its accuracy.

We have previously shared how to create targeting conditions that are dynamic and only include a visitor when they perform certain actions. That method requires updating the Global JS (Global Project JavaScript) of Convert. There is a small challenge from a maintainability perspective.

For example, if you are running 100s of experiments, maintaining all those conditions withing Global JS might cause some unforeseen issues. This has pushed us to come up with a different approach. Utilizing the JS Audience creation, that will do the same thing, without having to touch the Global JS Convert script.

Here's a step-by-step guide on how to create a custom trigger audience using JavaScript conditions.

Steps to Create a Custom Trigger Audience

1. Create a New Audience

Go to your Convert dashboard and create a new audience. Name it something descriptive like "Open Cart Drawer".

2. Add a JavaScript Condition

Drag and drop the JavaScript condition component into the audience rules conditions.

Here is an example for how the Audience setup should look like:

3. Write a Self-Calling Function

Within the JavaScript condition, you'll need to write a function that defines your custom logic.

Here’s a sample code snippet to get you started:

(function () {
  if (window.bucketed === true) {
    return true;
  }
 function waitForElement(selector, callback, count = 1000, interval = 250) {
    if (document.querySelector(selector)) {
      callback();
    } else if (count > 0) {
      setTimeout(function () {
        waitForElement(selector, callback, count - 1, interval);
      }, interval);
    }
  }
  waitForElement('body', function () {
    document.body.addEventListener('click', function (e) {
      if (e.target.closest('.button-selector')) {
        if (window.bucketed !== true) {
          window.bucketed = true;
          window._conv_q = window._conv_q || [];
          window._conv_q.push(["executeExperiment", "experimentID"]);
        }
      }
    });
  });
})();

Explanation

Initial Condition Check
When the page loads, the script checks if ‘window.bucketed’ is ‘true’. Since it is initially ‘undefined’, the condition fails, and the user is not included in the experiment.

Waiting for Element and Adding Event Listener

• The ‘waitForElement’ function waits for the entire document body to load and then adds an event listener to it.
• This listener checks for a click on an element matching the selector ‘.button-selector’.

Triggering the Experiment

• When the specified element ‘.button-selector’ is clicked, ‘window.bucketed’ is set to ‘true’, and the custom experiment trigger snippet is executed.
• This Custom Experiment Trigger snippet will recheck the entire experiment again, including the "Locations" and "Audiences".
• While rechecking the "Audiences", it will find that ‘window.bucketed’ is true and will then bucket the user into the experiment.

Implementation Tips

• Custom Variables: Customize the variable ‘window.bucketed’ and the event listener selector (‘.button-selector’) based on your specific use case.
• Experiment ID: Replace "experimentID" with the actual ID of your experiment.
• Event Listeners: You can add different event listeners or conditions as needed to fit your specific audience criteria.

Conclusion

By using custom JavaScript conditions, you can create highly specific and dynamic audience segments in Convert. This approach ensures that your experiments are triggered only for users who meet your precise conditions, allowing for more accurate and relevant testing.

Special thanks to Tansen and Shakil, two of our Solutions Engineers, for their expertise and insights in shaping the content and ensuring its accuracy.

We know that targeting the users at the right time and at the right situation to start an experiment is extremely important. Part of an A/B Test setup, you need to target the right area of the site. Whether it is the homepage, all product details pages or your checkout funnel, in case if your targeting is not correct, you will not get the right data against the metrics that you are tracking.

In some cases, the targeting can be more sophisticated. For example, if you are running an experiment on the mini-basket or cart drawer area, you only want to start the experiment when a user opens the mini-basket. Prior to that, the users should not be part of the experiment.

In Convert testing tool, you can use the “JS Targeting Condition”. However, one challenge with this is the time limitation – we have noticed that Convert will wait for 2 minutes for that condition to be fulfilled. Once the period has exceeded, the user will not be bucketed into the experiment even if he fulfills the condition.

Convert JS Targeting Condition Note - stating a 2 minutes cutoff

Going back to the mini-basket example, a user might open the mini-basket after 2 minutes. And as a result, the user will not be bucketed to the experiment even though the condition is fulfilled.

How to resolve this:

Luckily, there is a way and we have found a work around to resolve this issue.

Step 1: Configure Convert Global Project JS

We need to use the Global Project JS and configure the Global Javascript.

Convert - Global Project JS configuration

Step 2: Add a new mutation observer to the Global JS

Add the code below, make the necessary changes as needed:

// listen for targeted element
const targetNode = document.getElementById('element-id); // parent under which the changes occur
const config = { attributes: true, childList: true, subtree: true };
const callback = function(mutationsList, observer) {
       if (!!document.querySelector("target-element")) { // target element or trigger condition
             observer.disconnect();
             window.targetElementTouched = true;
             window._conv_q = _conv_q || [];
             window._conv_q.push(["executeExperiment","experimentId"])
      }
};
const observer = new MutationObserver(callback);
observer.observe(targetNode, config);

Step 3: Add a JS condition in the experiment Audience.

Rename targetElementTouched to the name of the element, e.g. cartDrawerElement.

window.targetElementTouched == true 

Convert JS Targeting Condition Setup

And voila – you have got the right JS targeting condition setup without having to worry about the 2 minutes rule or when the user is performing the necessary action to be bucketed into the experiment!

Summary:

In Convert, JS Condition of an experiment only runs for 2 minutes. Therefore, we cannot trigger the experiment when a user performs the action after 2 minutes of landing on the page. A MutaionObserver is added to the global project javascript which listens for the change and assigns a window variable for it. After that we can execute a particular experiment as shown above with

window._conv_q.push(["executeExperiment","experimentId"]). 

The triggers for the experiment will be checked again in which case it would fulfill the condition and user will be bucketed into the experiment.

Special thanks to Raihan, one of our Software Engineer, for his expertise and insights in shaping the content and ensuring its accuracy.

We know that targeting the users at the right time and at the right situation to start an experiment is extremely important. Part of an A/B Test setup, you need to target the right area of the site. Whether it is the homepage, all product details pages or your checkout funnel, in case if your targeting is not correct, you will not get the right data against the metrics that you are tracking.

In some cases, the targeting can be more sophisticated. For example, if you are running an experiment on the mini-basket or cart drawer area, you only want to start the experiment when a user opens the mini-basket. Prior to that, the users should not be part of the experiment.

In Convert testing tool, you can use the “JS Targeting Condition”. However, one challenge with this is the time limitation – we have noticed that Convert will wait for 2 minutes for that condition to be fulfilled. Once the period has exceeded, the user will not be bucketed into the experiment even if he fulfills the condition.

Convert JS Targeting Condition Note - stating a 2 minutes cutoff

Going back to the mini-basket example, a user might open the mini-basket after 2 minutes. And as a result, the user will not be bucketed to the experiment even though the condition is fulfilled.

How to resolve this:

Luckily, there is a way and we have found a work around to resolve this issue.

Step 1: Configure Convert Global Project JS

We need to use the Global Project JS and configure the Global Javascript.

Convert - Global Project JS configuration

Step 2: Add a new mutation observer to the Global JS

Add the code below, make the necessary changes as needed:

// listen for targeted element
const targetNode = document.getElementById('element-id); // parent under which the changes occur
const config = { attributes: true, childList: true, subtree: true };
const callback = function(mutationsList, observer) {
       if (!!document.querySelector("target-element")) { // target element or trigger condition
             observer.disconnect();
             window.targetElementTouched = true;
             window._conv_q = _conv_q || [];
             window._conv_q.push(["executeExperiment","experimentId"])
      }
};
const observer = new MutationObserver(callback);
observer.observe(targetNode, config);

Step 3: Add a JS condition in the experiment Audience.

Rename targetElementTouched to the name of the element, e.g. cartDrawerElement.

window.targetElementTouched == true 

Convert JS Targeting Condition Setup

And voila – you have got the right JS targeting condition setup without having to worry about the 2 minutes rule or when the user is performing the necessary action to be bucketed into the experiment!

Summary:

In Convert, JS Condition of an experiment only runs for 2 minutes. Therefore, we cannot trigger the experiment when a user performs the action after 2 minutes of landing on the page. A MutaionObserver is added to the global project javascript which listens for the change and assigns a window variable for it. After that we can execute a particular experiment as shown above with

window._conv_q.push(["executeExperiment","experimentId"]). 

The triggers for the experiment will be checked again in which case it would fulfill the condition and user will be bucketed into the experiment.

Special thanks to Raihan, one of our Software Engineer, for his expertise and insights in shaping the content and ensuring its accuracy.

One of the core things that create a lot of back and forth between the developers and product owners is that the information needed about a test that needs to be developed is simply not there up front. Even though you are well versed on what the hypothesis is and what you want to test, because of the missing information, the development can be stalled. We have seen some tests that are supposed to be delivered in three days’ time, took over a week as some of the key information was missing from the original test brief.

Here are the 5 most important things that the A/B testing developers need to develop and deliver the test in the most efficient way:

1. Variation designs

Obviously! And that includes what changes need to be made, any functionalities that need to be added and / or how the changes are reflected across different devices

2. Metrics

Unless you are going to track the metrics by yourself – specify the primary KPI and all other metrics you want to track. There could be custom metrics that the A/B testing developer need to implement as part of the test development across all variations (including control!)

3. Targeting pages / screens

Where is this going to be live? Homepage? Product Listing Pages? Product Details pages? What is the URL structure? If it is a sitewide test – is it truly a sitewide? What about the checkout funnel (which can be structured differently from the rest of the site)?

4. Audiences

Who will be part of this experiment? What are the audience criteria? What about visitors coming from PPC campaigns? Are there any restrictions?

5. Edge cases / Anomalies

If a product is out of stock or on sale, the product details page is displayed differently. How is that going to be reflected for the variation? It is very easy to miss the edge cases – but for the test developer, it is important to ensure that they are covered up front

The rule is simple – the clearer the information is up front, the less time spent on unnecessary back and forth!

One of the core things that create a lot of back and forth between the developers and product owners is that the information needed about a test that needs to be developed is simply not there up front. Even though you are well versed on what the hypothesis is and what you want to test, because of the missing information, the development can be stalled. We have seen some tests that are supposed to be delivered in three days’ time, took over a week as some of the key information was missing from the original test brief.

Here are the 5 most important things that the A/B testing developers need to develop and deliver the test in the most efficient way:

1. Variation designs

Obviously! And that includes what changes need to be made, any functionalities that need to be added and / or how the changes are reflected across different devices

2. Metrics

Unless you are going to track the metrics by yourself – specify the primary KPI and all other metrics you want to track. There could be custom metrics that the A/B testing developer need to implement as part of the test development across all variations (including control!)

3. Targeting pages / screens

Where is this going to be live? Homepage? Product Listing Pages? Product Details pages? What is the URL structure? If it is a sitewide test – is it truly a sitewide? What about the checkout funnel (which can be structured differently from the rest of the site)?

4. Audiences

Who will be part of this experiment? What are the audience criteria? What about visitors coming from PPC campaigns? Are there any restrictions?

5. Edge cases / Anomalies

If a product is out of stock or on sale, the product details page is displayed differently. How is that going to be reflected for the variation? It is very easy to miss the edge cases – but for the test developer, it is important to ensure that they are covered up front

The rule is simple – the clearer the information is up front, the less time spent on unnecessary back and forth!