Why Overlapping Confidence Intervals mean Nothing about Statistical Significance

The statement above is wrong. Overlapping confidence intervals/error bars say nothing about statistical significance. Yet, many make the mistake of inferring a lack of statistical significance. Likely because the inverse — non-overlapping confidence intervals — implies statistical significance. I’ve made this mistake. I think part of the reason it is so pervasive is that it is often not explained why you cannot compare overlapping confidence intervals. I’ll take a stab at explaining this in this post in an intuitive way. HINT: It has to do with how we keep track of errors.

Read More