Variance is a measurement of dispersion which measures the amount of “noise” in a metric or experiment results. Higher variance is associated with larger confidence intervals, and leads to experiments requiring more sample size to consistently observe a statistically significant result on the same effect size.Reducing variance can lead to shorter experiment run times due to the lower sample required. Because of this, techniques have been developed to reduce the variance in experiment results in order to reduce run times and increase confidence.At Statsig, we use a form of CUPED based on a 2013 Microsoft paper (Deng, Xu, Kohavi, & Walker). This is automatically applied to experiments at Statsig, and is run for the topline results on key metrics in Pulse. This observably leads to significant variance reduction in the large majority of metrics where CUPED can be applied.Refer to our launch post for CUPED for more details.
CUPED - Controlled-experiment Using Pre-Existing Data
CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied).
Another common technique for reducing noise is Winsorization, which is a way to manage the influence of outliers.Winsorization refers to the practice of measuring the percentile Px of a metric and setting all values over Px to Px.This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors.
The metrics you use can dramatically influence the sensitivity of your analysis. The transformations above, in addition to techniques like creating threshold-based flags, can let you trade-off exact numbers for significantly more power. Please refer to our blog post on the topic for more information.
