Rollout Alerts

Get notified when experiments or feature rollouts cause metric regressions beyond preset thresholds.

Overview

On Statsig Cloud, Rollout Alerts are evaluated every 24 hours. Alerts only trigger if the cumulative metric delta is statistically significant lower / higher your threshold, which helps reduce alert noisiness.

On Statsig Warehouse Native, Rollout Alerts are evaluated every time Metric Results are loaded in a feature gate / experiment. Loading Results on the first day will help provide more real-time visibility during this window. Alerts only trigger if the metric delta is statistically significant lower/higher than your threshold, which helps reduce alert noisiness.

Finally, all stats methodologies you’ve enabled for your experiment/gate rollout (CUPED, Sequential Testing, etc.) will be applied to alert calculations.

Rollout Alerts do not alert at the topline metric value level, but rather at the experiment/feature gate level. This means that even if you have an experiment allocated to 10% of your users, but the metric change within that 10% allocation breaches the set threshold, you will be alerted. All alerts you receive will be in the context of a specific experiment or feature gate and to debug/resolve the alert you will be directed to the offending experiment or gate in question.

Setting up a Rollout Alert

To set up a metric alert, go to the Metrics tab —> Metrics Catalog and search for the desired metric.

Once in the Metric Detail View, go to the Alerts tab, and tap + Create Alert. As part of Metric Alert configuration, you will be asked to configure the following inputs-

• % Change- The metric delta threshold after which you want to trigger an automated alert. As noted above, this delta is in the context of the feature gate rollout or experiment the metric is being measured in and is not a top-line metric value change across your whole user-base.
• Minimum Participating Units- Set a minimum threshold for the number of unique units emitting the metric in each test group before triggering an alert. We surface the 25th / 50th / 75th percentiles of metric-emitting unit counts per group across your gates and experiments to help you choose a reasonable threshold.
• Direction- Positive or negative, depending on whether you want to be alerted based on your metric exceeding vs. dropping below a target threshold.
• Subscribers- By default, all creators of a feature gate/experiment with a metric that has an alert configured will be notified if an alert is fired. You can also add additional, global subscribers to a metric alert, who will be notified if any feature gate or experiment regresses the metric beyond the target threshold. Note that in a large project adding yourself as a global subscriber of a metric risks being noisy.

Once a Rollout Alert has been configured for a given metric, you will see an “alert” alarm bell icon next to the metric inline in the Metrics Catalog. You can also filter for metrics with alerts set on them via the standard metrics filtering affordance next to the search bar.

Determining the Right Threshold

To help you configure the right threshold for your Rollout Alert, there is a preview of how much the metric has moved in the context of any feature gates or experiments containing that metric in the Scorecard.

To see how a given metric has trended over a longer period of time, hover over the metric delta for a given feature gate/experiment, and tap on a data point to view more details. This will open the time-series for the metric, with a configurable date range picker.

Alert UX

If a Rollout Alert is triggered, all subscribers and the relevant gate/experiment creator(s) will receive a notification via email, in the Statsig Console, and via Slack for users who have configured Slack notifications for their Statsig accounts.

Tapping on View Alert will take you into the Diagnostics page of the offending feature gate or experiment that triggered the Rollout Alert.

Once in the Diagnostics section, scroll to the Rollout Alerts section and select the active alert. You will see the metric delta trend with the threshold overlayed. To resolve the alert, tap Resolve inline, which will give you the option to either resolve the alert (+ provide an explanation), or snooze the alert for a specified period of time. The alert will show as resolved both inline in the Rollout Alerts section, as well as on the top of the Pulse page for the given gate/experiment.

Viewing Alert History

To view alert history, go to Metrics tab —> Metrics Catalog and select the metric you want to view alert history for. Then go to Alerts —> … menu in the Experiment and Gate Alerts section, and select View Alert History. You will be able to see all instances of the alert firing, being resolved, or being snoozed, as well as reason provided for resolution, who resolved the alert, and at what time.