Estimating Total Experimentation Impact
<p>Data-driven organizations often run hundreds or thousands of experiments at any given time, but what is the net impact of all of these experiments? A naive approach is to sum the difference-in-means across all experiments that resulted in a significant and positive treatment effect and that were rolled out into production. This estimate, however, can be extremely biased, even if we assume there are no correlations between individual experiments. We will run a simulation of 10,000 experiments and show that this naive approach overestimates the actual impact delivered by <em>45%</em>!</p>
<p><a href="https://towardsdatascience.com/estimating-total-experimentation-impact-ab6cd56bffb"><strong>Website</strong></a></p>