During the last couple of years, I have witnessed a lot of mistakes that people make during A/B testing design and post-analysis. I would like to discuss here three of such mistakes and how to overcome them.
Use Mann-Whitney for comparing medians
The first mistake is not an appropriate use of Mann-Whitney test.The method is massively misunderstood and misused, because most people use this test ,assuming that it is approximately non-parametric “t test” for medians. In reality, we can use Mann-whitney test only to check if there is a shift in our distribution.