Image Shadow of structure

6 reasons why independent manual testing is a must for Quality Assurance of A/B Testing

Over the years, I have heard many people saying that their consultants, and in some cases, the developer who is building the test is also doing the QA of the variations. This is potentially hazardous and prone to missing out on finding bugs before the test is live. As a result, the research and data-backed hypothesis test or A/B test could bring back incorrect results. In this article, I have summarized six key reasons as to why you should be doing independent manual QA of all of your variations.

1. Developers and consultants are too close to the test:

Your developer and potentially your consultant are too close to the test that they are building – making it very easy to miss out small but important details if they are in charge of QA

2. Emulators are not the real thing:

“Vege hotdog tastes the same as the real hotdog” Sorry but they are not the same. Your end-users will not use an emulator – they will use the real device and browser. If you are not manually checking them on actual device/browsers, there is a potential that you will miss out on finding issues specific to real browsers

3. Interactions:

If you are not manually checking the variations, you might miss out on issues related to interactions to the page/variations. This could be opening an accordion, clicking on a button or going through the funnel itself.

4. Checking goal firing:

If you are not doing a QA across all browsers manually, you might not be able to test out whether your metrics setup are correct. In a worst-case scenario, you might look at your result after a couple of weeks and notice that your primary metric did not fire properly for some browsers or at all!

5. Breakpoints and changing device display mode:

If you are using emulators, you might miss out any issues related to switching the mode from Portrait to Landscape or vice versa. By QAing the variations on actual mobile/tablet devices, you can easily spot check if the variation is displaying correctly on both modes, but also when the user is switching between the two, the behaviour is as it should.

6. Tests from a Human Perspective:

Manual QA helps to quickly identify when something looks “off.” Automated test scripts don’t pick up these visual issues. When a QA Engineer interacts with a website or software as a user would, they’re able to discover usability issues and user interface glitches. Automated test scripts can’t test for these things.

This is why, here at EchoLogyx, our dedicated QA Engineers always use actual devices and test all variations on the targeted browsers to find issues. They have to be thorough to make sure that no bugs are present in any variations or any development work before we deliver the work. They need to check all possible scenarios and their target is to break the work that our engineers are doing. Essentially our QA team are the gatekeepers to approve whether the test is ready to go live or not. This significantly reduces the risk of getting a bad user experience to the end-users who will be using the site.

Related Posts

Goals Troubleshooting/QA In VWO

A/B Testing, AB Test Development, Quality assurance

Goals Troubleshooting/QA in AB Tasty

A/B Testing, AB Test Development, Quality assurance

Best practices to implement the snippet of AB testing tools

A/B Testing, AB Test Development, Quality assurance

Let's have a chat

If you want to find out a bit more, just get in touch. We love a chinwag, and we'd love to help you out.

Contact us

Don’t just take our word for it…

Our customers are our biggest advocates. That’s why we let them do the talking.

  • Working with EchoLogyx has simply changed the development lifecycle of our website. Their experienced A/B testing developers and QA engineers have significantly increased our efficiency.

    Farhan Hussain

    eCommerce Director, eSpares
    4 3 2 1 0
  • I’d 100% recommend EchoLogyx – they have really sort of raised the bar of my expectation, of how you should work with developers and also their efficiency of completing the project.

    Lucy Collins

    UX Consultant, Web Usability
    4 3 2 1 0
  • They come to me saying, this can be done better. This proactiveness is what I most like about the EchoLogyx team. They have got a strong QA team as well so that nothing goes out live without going through a thorough QA.

    Nabil Al Jannati

    Brand Manager, Ninety Percent
    4 3 2 1 0
  • Our experience of EchoLogyx is resoundingly positive. They’re very thorough and detailed. We’ve got a great working relationship with our optimisation managers.

    James Amborse

    Head of Operations, RedEye Optimisation
    4 3 2 1 0

Leaving so soon?

Get 20% OFF on your first project

When you Sign up to our email newsletter

You will receive:
  • Top tips on A/B testing development and QA
  • Testing tools and their performance
  • QA and Bug Free testing

Contact us

Why don't you get in touch to find out more about our services? Fill in the form below: