Safe Rollouts
Releasing a new feature always carries risk. Even small changes can introduce regressions. Safe Rollouts reduce this risk by releasing to a subset of users while monitoring key metrics for issues.
Safe Rollouts is a GrowthBook Pro and Enterprise feature.
Overview
Safe Rollouts are a type of feature rule that lets you release a feature to a subset of users and automatically monitor guardrail metrics for regressions. Guardrails can be any metrics that matter to your team, such as error rates, latency, or conversions.
The rollout runs as a short-term A/B test:
- The Control receives the existing value.
- The Rollout receives the new value.
GrowthBook analyzes the impact of the rollout on your selected guardrail metrics and provides status updates indicating whether it’s safe to ship, whether issues have been detected, or whether you should consider rolling back.
How to Add a Safe Rollout
1. Select "Safe Rollout" When Adding a Rule
When adding a new rule to your feature, select Safe Rollout from the list of options.
2. Configure the Safe Rollout
- Rollout Value: Add or select the value to serve during the rollout. This is compared against the Control (default value).
- Split Attribute: Choose the attribute for assigning users to the Control and Rollout groups (for example,
idororganization_id).
3. Define Metrics and monitoring
- Data Source and Assignment Table: Confirm your data source and the experiment assignment table for exposure tracking.
- Guardrail Metrics: Select one or more metrics to monitor for regressions during the rollout.
- Monitoring Duration: Set how many days to collect data before making a decision.
4. Monitor the Rollout
Once the rollout is published, GrowthBook begins monitoring the guardrail metrics. The rollout status updates automatically based on results.
Status indicators:
| Status | Meaning |
|---|---|
| X days left | The rollout is in progress. Monitoring continues. |
| Unhealthy | Traffic is imbalanced. Check your implementation. |
| Guardrails Failing | Regression detected. Consider reverting the change. |
| Ready to ship | No regressions detected and safe rollout duration completed. Ready to release fully. |
| No Data | No traffic detected after 24 hours. Check setup. |
Guardrails are analyzed for failure using one-sided sequential testing, allowing you to roll back as soon as statistical significance is reached without fear of false positives. Your safe rollout is automatically monitored for implementation errors with sample ratio mismatch and multiple exposures checks.
5. Take Action
At the end of the monitoring period (or sooner if issues arise), you can:
- Release the feature to 100% of users.
- Revert the feature to the control value.
- Continue to monitor.
FAQs
Why is the rollout capped at 50%?
Safe Rollouts rely on a control group to detect regressions. Limiting the rollout group to 50% ensures there is always a reliable baseline for comparison. Automatic increases in traffic and more customizable rollout percentages are scheduled soon.
What happens if a guardrail metric degrades?
If a guardrail metric crosses a significance threshold indicating a regression, the rollout status will reflect the issue. The feature is not automatically rolled back—you retain control over whether to stop or revert the rollout. (But automatic rollback is planned! Get in touch for more details.)
What should I use as my guardrail metrics?
Choose metrics that represent key system health or business outcomes. This might include error rates, latency, or conversion rates—whatever signals that the change is working as intended and not causing harm.
Choosing too many guardrail metrics could increase the chance of false positives, so find a balance between a good mix of critical metrics without adding too many.
How is this different from an experiment?
Safe Rollouts use the same analysis engine as GrowthBook experiments but are designed for operational decision-making, not learning. The primary goal is to ensure a safe release, not to measure long-term impact.
What happens if there isn’t enough data?
Safe Rollouts are built to help you ship confidently and reduce the chance of negative regressions. If any guardrail fails, we recommend rolling back your test.
If test results are still inconclusive after the set duration for your safe rollout, we recommend shipping, because there’s no clear evidence that the feature is harmful.
Safe Rollouts therefore bias towards action. If you are more uncertain about a feature and want to learn generally about its impact, we recommend running a regular Experiment.