Experiments on Returns on Investment
<p>When we run an experiment, we are often not only interested in the effect of a treatment (new product, new feature, new interface, …) on revenue, but in its <strong>cost-effectiveness</strong>. In other words, is the investment worth the cost? Common examples include investments in computing resources, returns on advertisement, but also click-through rates, and other ratio metrics.</p>
<p>When we investigate causal effects, the gold standard is <a href="https://en.wikipedia.org/wiki/Randomized_experiment" rel="noopener ugc nofollow" target="_blank">randomized control trials</a>, a.k.a. <strong>AB tests</strong>. Randomly assigning the treatment to a subset of the population (users, patients, customers, …) we ensure that, on average, the difference in outcomes can be attributed to the treatment. However, when the object of interest is cost-effectiveness, AB tests present some additional problems since we are not just interested in one treatment effect, but in the <strong>ratio of two treatment effects</strong>, the outcome of the investment over its cost.</p>
<p><a href="https://towardsdatascience.com/experiments-on-returns-on-investment-34a1953c5f16"><strong>Website</strong></a></p>