A friend of mine has recently asked if I can help him with creating an Excel file to calculate the power of A/B test results. I found one calculator online here which I think is pretty sleek but the only problem is, it is online. So if you want to do your own calculation using your excel file, you will need to know the maths behind it.
Rather than getting into the details of what the statistical power is, in simple terms, the statistical power determines the possibility of Type II error (or false negative) within the result that you are seeing from the test.
I have created this Excel file which follows the same calculation that has been used by the AB Test Guide (https://abtestguide.com/calc/) to calculate the statistical power of any A/B test.
Instruction to use this excel file to calculate statistical power is pretty self explanatory:
- Select whether the test is 1-Tail or 2-Tail (in most cases, if you are running A/B tests using major tools that are in the market - it should be 2-Tail)
- Select the confidence level (90% / 95% / 99%)
- Add the visitor numbers for your control and variation
- Add the conversion numbers (NOT Conversion Rate) for your control and variation
That's it! It will show the power of the test - in other words - the possibility of Type II errors (the lack of it).
