All about Sample-Size Calculations for A/B Testing: Novel Extensions & Practical Guide

While there exists a large amount of literature on the general challenges and best practices for trustworthy onlineA/Btesting, there are limited studies on sample size estimation, which plays a crucial role in trustworthy and efficient A/B testing that ensures the resulting inference has a sufficient power and type I error control. For example, when sample size is under-estimated, the statistical inference, even with the correct analysis methods, will not be able to detect the true significantimprovementleading to misinformedandcostly decisions. Thispaper addresses this fundamental gapbydeveloping newsample size calculation methods for correlated data, aswell as absolute vs. relative treatment effects, both ubiquitous in online experiments. Additionally, we address a practical question of the minimal observed difference that will be statistically significant and howit relates to average treatment effect and sample size calculation. All proposed methods are accompanied by mathematical proofs, illustrative examples, and simulations. We end by sharing some best practices on various practical topics on sample size calculation and experimental design.